Confirmatory Testing

Every now and then this idea of confirmatory testing pops up. There’s nothing wrong with it as long as it’s complemented with disconfirmatory testing, or I should probably just say with testing.

Although I don’t want to generalize too much in this article, I still think that what I’m about to say on these lines is something that happens too often. If that’s not the case at your company or workplace, you are a lucky one and you might be doing it right :)

Anyway, there’s many times a popular belief that all testers do is they show that something works (it might be worded slightly differently). But that’s really not the whole story.

When I start testing something, I’m obviously interested in seeing if it works at all. That might help me see where the benefit for the user is or it might save me some time. These might be some first experiences I have with the product but it’d be foolish to stop at that.

As I go on experiencing the product, I create theories and hypotheses about how the product works. These need to be looked at more closesly and tried to be disproved. Testing is, in essence, a disconfirmatory process.

Let’s now have a look at some concrete examples of confirmatory testing.

Automation

Using code to check things is a confirmatory process. I’m not saying it’s wrong or it should not happen. Not at all. It’s useful and it should be present. What I’m saying is that is should be complemented with testing.

An example in pseudocode might be:

go to "/search" page;
type "test event" into the search field;
check that event object ({...}) is in the response body;
check that suggestion is visible on the frontend;

This might be something you recognise. But when you look at it more closely, you can see that:

• it checks only what should happen, it does not test what might go wrong
• it’s rather one-dimensional kind of check, it does not consider many other areas that might contain trouble — accessibility or performance to name just two

If your testing includes only such kind of simple checking, it’s really not enough and you might run into serious trouble because of bugs that are not uncovered with such a confirmatory process.

Demos

Demos are kind of popular at many workplaces. At the end of a sprint, there’re demos in which people present how something works. Testers are sometimes asked to conduct such demo sessions (been there, done that).

But again, showing that something works is not testing. It’s perhaps part of testing, if at all. I understand that testers might temporarily play a different role, so if they want to conduct a demo, so be it. But they still need to be aware that this is not testing. In my experience, it might be difficult to hold opposing ideas in my head, so beware.

Checklists

I think this is a similar idea to automation. Testing code is to a computer what checklists are to humans. It comes down to output checking, to expected and actual results.

Checklists might be useful as well. But they need to written in a way that it leaves enough room for autonomous thinking, creativity, and invention. Let’s see two examples:

* fill in "test event" and see a suggestion with the same name

and:

* test how different input values are handled

Which of these two examples leaves more room for autonomous thinking and tester’s own creativity and invention?

I think the latter by a lot. If seen by an experienced tester, they immediately start thinking about various inputs and problems that they would like to explore.

However, the key part here is the word “experienced” because one has to have some training to come up with test ideas and to be able to imagine what might go wrong. It might be that (test) managers are scared of giving responsibility to an individual tester. But in my view, it’s a far better way.

What do you think? Is confirmatory testing the only testing that should happen? Why? What problems might remain in the product if this is the only testing performed? How do you test? How do other testers around you test?

It might be worth being a bit more aware of what exactly I’m doing when testing. And if I find myself doing too much confirmatory testing, stop and think what I can disprove.