Author: Haroon Khalil

If we had combined all the partitions we derived from the substringsBetween program, we would have ended up with 320 tests. This number is even larger for more complex problems. Combinatorial testing is an entire area of research in software testing; I will not dive into the techniques that have been proposed for such situations, but I…

chYou can simplify your understanding of the different test cases by using the same input seed for all of them, as we noticed in an observational study with professional developers described in my paper with Treude and Zaidman (2021). For each partition, you then make small modifications to the input seed: just enough to meet…

On and off points belong to specific partitions, so they also serve as concrete test cases for the partitions. This means testing all the boundaries of your input domain is enough. Nevertheless, I often try some in and out points in my tests. They are redundant, because the on and off points exercise the same…

When you are exploring inputs and outputs, identifying partitions, and devising test cases, you may end up considering a boundary to be an exclusive partition and not a boundary between two partitions. It does not matter if a specific case emerges when you are identifying partitions or in the boundaries step. Each developer may interpret…

The pragmatic answer to this question is to understand the risks of a failure. What would be the cost of a failure in that part of the program? If the cost is high, it may be wise to invest more in testing, explore more corner cases, and try different techniques to ensure quality. But if…

Describing iterative processes in writing is challenging. My explanation may have given you the impression that this process is fully sequential and that you move to the next step only when you have completed the previous one. However, the entire process is meant to be iterative. In practice, I go back and forth between the…

Now that you have a clear understanding of how to systematically devise test cases based on specifications, here are a few pragmatic tips I have learned over the years.

The developers of the Apache Commons Lang framework (the framework where I extracted the implementation of the substringsBetween method) are just too good. We did not find any bugs there. Let’s look at another example: one implemented by me, an average developer who makes mistakes from time to time. This example will show you the value of…

I propose a seven-step approach to derive systematic tests based on a specification. This approach is a mix of the category-partition method proposed by Ostrand and Balcer in their seminal 1988 work, and Kaner et al.’s Domain Testing Workbook (2013), with my own twist: see figure 2.4. Figure 2.4 The seven steps I propose to derive test cases based on specifications.…

Being systematic is good, but we should never discard our experience. In this step, we look at the partitions we’ve devised and see if we can develop interesting variations. Variation is always a good thing to have in testing. In the example, when revisiting the tests, I noticed that we never tried strings with spaces.…