Category: Specification-Based Testing

If two testers performed the specification-based testing technique I described earlier in the same program, would they develop the same set of tests? Ideally, but possibly not. In the substringsBetween() example, I would expect most developers to come up with similar test cases. But it is not uncommon for developers to approach a problem from completely different… Continue reading The role of experience and creativity

The two methods we tested have no state, so all we had to do was think of inputs and outputs. In object-oriented systems, classes have state. Imagine a ShoppingCart class and a behavior totalPrice() that requires some CartItems to be inserted before the method can do its job. How do we apply specification-based testing in this case? See the following listing.… Continue reading How does this work with classes and state?

The seven-step approach I propose works for requirements of any granularity. Here, we applied it in a specification that could be implemented by a single method. However, nothing prevents you from using it with larger requirements that involve many classes. Traditionally, specification-based testing techniques focus on black-box testing: that is, testing an entire program or… Continue reading Requirements can be of any granularity

A little duplication is never a problem, but a lot of duplication is. We created 21 different tests for the substringsBetween program. The test code was lean because we grouped some of the test cases into single test methods. Imagine writing 21 almost-identical test cases. If each method took 5 lines of code, we would have a… Continue reading Go for parameterized tests when tests have the same skeleton

Testing nulls and exceptional cases is always important because developers often forget to handle such cases in their code. But remember that you do not want to write tests that never catch a bug. Before writing such tests, you should understand the overall picture of the software system (and its architecture). The architecture may ensure… Continue reading Test for nulls and exceptional cases, but only when it makes sense

Sometimes, your goal is to exercise a specific part of the functionality, and that part does not use one of the input values. You can pass any value to that “useless” input variable. In such scenarios, my recommendation is to pass realistic values for these inputs.

Picking concrete input for test cases is tricky. You want to choose a value that is realistic but, at the same time, simple enough to facilitate debugging if the test fails. I recommend that you avoid choosing complex inputs unless you have a good reason to use them. Do not pick a large integer value… Continue reading When in doubt, go for the simplest input

If we had combined all the partitions we derived from the substringsBetween program, we would have ended up with 320 tests. This number is even larger for more complex problems. Combinatorial testing is an entire area of research in software testing; I will not dive into the techniques that have been proposed for such situations, but I… Continue reading When the number of combinations explodes, be pragmatic

chYou can simplify your understanding of the different test cases by using the same input seed for all of them, as we noticed in an observational study with professional developers described in my paper with Treude and Zaidman (2021). For each partition, you then make small modifications to the input seed: just enough to meet… Continue reading Use variations of the same input to facilitate understanding

On and off points belong to specific partitions, so they also serve as concrete test cases for the partitions. This means testing all the boundaries of your input domain is enough. Nevertheless, I often try some in and out points in my tests. They are redundant, because the on and off points exercise the same… Continue reading On and off points are enough, but feel free to add in and out points