Not All Bugs Are Found By Automation

There is a lot of content that says all we should do is automation. I disagree with that. There are more than one reason but instead of listing them, I will share one example of a missed bug.

A couple of days ago, I was involved in investigating one bug that came to us from our customer care. That basically means our customers complained about it. It was a different part of the app than what I was familiar with but I also wanted to learn a bit more.

What I found was basically this: there were two applications involved, a user would log into one and then was able to go to the other one with the same credentials. The problem was that this worked only for a period of time. What was happening was that the first app already started using different authentication scheme, so after some time, about half a minute or so, it called /revoke-token endpoint and revoked a token that it didn’t need. Except that the second app was relying on the very token.

The users were left logged out from the second app after this period of time. That was unfortunate because it happened to be an important part of the whole system. These two apps are designed to work closely together.

Anyway, it was eventually fixed and we carry on like before. However, what are the lessons here? And how would you find such a bug?

I thought about it and concluded a few things:

• Frontend application was involved, it was the frontend app in a browser that called this endpoint, so any testing that has a potential to catch this bug has to rely on a browser.
• It was not caught by any of our automation checking. And I think it couldn’t have been.

Why do I think automation is not the best approach here?

Mostly because when you write automation checks, you also want speed from them. You write them in a way that they run fast. You even find shortcuts that make for a faster run in a pipeline.

But this bug needed time. It needed someone close to a real user who would try to use the app and accomplish a task the app was designed for. Even if you try hard, automation is far from this.

I’m not saying this is not possible to catch using automation. But it is unlikely. What are the chances that you would write something like this in Cypress?

cy.wait(40000);

This would simply not make it past a pull request code review. But it was probably the only thing that had a chance to catch this bug.

There are bugs like these. And real humans are simply better in catching them — be it in a code review, or in experiencing and exploring the app. Automation and exploratory testing are two complementary approaches applied on different problems. It’s not a question of which one, but a question of how we can combine them in a way that it’s beneficial to us.