Category: Specification-Based Testing

When you are exploring inputs and outputs, identifying partitions, and devising test cases, you may end up considering a boundary to be an exclusive partition and not a boundary between two partitions. It does not matter if a specific case emerges when you are identifying partitions or in the boundaries step. Each developer may interpret… Continue reading Partition or boundary? It does not matter!

The pragmatic answer to this question is to understand the risks of a failure. What would be the cost of a failure in that part of the program? If the cost is high, it may be wise to invest more in testing, explore more corner cases, and try different techniques to ensure quality. But if… Continue reading How far should specification testing go?

Describing iterative processes in writing is challenging. My explanation may have given you the impression that this process is fully sequential and that you move to the next step only when you have completed the previous one. However, the entire process is meant to be iterative. In practice, I go back and forth between the… Continue reading The process should be iterative, not sequential

Now that you have a clear understanding of how to systematically devise test cases based on specifications, here are a few pragmatic tips I have learned over the years.

The developers of the Apache Commons Lang framework (the framework where I extracted the implementation of the substringsBetween method) are just too good. We did not find any bugs there. Let’s look at another example: one implemented by me, an average developer who makes mistakes from time to time. This example will show you the value of… Continue reading Finding bugs with specification testing

I propose a seven-step approach to derive systematic tests based on a specification. This approach is a mix of the category-partition method proposed by Ostrand and Balcer in their seminal 1988 work, and Kaner et al.’s Domain Testing Workbook (2013), with my own twist: see figure 2.4. Figure 2.4 The seven steps I propose to derive test cases based on specifications.… Continue reading Specification-based testing in a nutshell

Being systematic is good, but we should never discard our experience. In this step, we look at the partitions we’ve devised and see if we can develop interesting variations. Variation is always a good thing to have in testing. In the example, when revisiting the tests, I noticed that we never tried strings with spaces.… Continue reading Step 7: Augment the test suite with creativity and experience

It is now time to transform the test cases into automated JUnit tests. Writing those tests is mostly a mechanical task. The creative part is coming up with inputs to exercise the specific partition and understanding the correct program output for that partition. The automated test suite is shown in listings 2.3 through 2.7. They… Continue reading Step 6: Automate the test cases

With the inputs, outputs, and boundaries properly dissected, we can generate concrete test cases. Ideally, we would combine all the partitions we’ve devised for each of the inputs. The example has four categories, each with four or five partitions: the str category with four partitions (null string, empty string, string of length 1, and string of length > 1), the open category with four… Continue reading Step 5: Devise test cases

Bugs in the boundaries of the input domain are common in software systems. As developers, we have all made mistakes such as using a “greater than” operator (>) where it should have been a “greater than or equal to” operator (>=). Programs with such bugs tend to work well for most provided inputs, but they… Continue reading Step 4: Analyze the boundaries