The Danger Of Written Test Procedures

  • to write a test
  • to write a test case
  • to write a check
  • and likely a bunch of other word expressions
  • What does random mean? Random is really not random in many cases. I think this would simply lead to choosing an item that’s closest to the tester on the UI (for example right on the homepage as opposed to in a category), which is everything but random.
  • How should I choose the item? Can an item be added to a cart from different places? E.g. from a product detail page as well as from a listing page.
  • Is one way of choosing preferable over other ways? Is one way more used with real customers? What are the ways we want customers to be able to use? Are there any ways we want to block from using? E.g. do we want to build a platform that considers accessibility? An example of that might be being able to control the system using a keyboard only.
  • What does an item mean? E-shops sell all sorts of products, they might differ in their properties, in how they are represented in databases, in how the system works with these representations. I’ve seen e-shops successfuly add some items into a cart while fail when adding different items into the same cart. It’s not given that when one item can be added into a cart, it will be automatically true for all items.
  • It seems to me that the test procedure suggests I should choose an item that’s also in stock. How about if I choose one that’s not in stock at the moment? Can that still be added to a cart? Why? Why not? Perhaps it can still be added but the context changes and instead of buing it straight away, it will be reserved and shipped when in stock again at some point in the future.
  • How about items that are discounted? Should they be included in the pool of items I’m “randomly” choosing from? Or perhaps it’s a new test idea that’s worth experimenting with.
  • How about a choice of a device? Many e-shop platforms are built with responsiveness in mind. That means they could be used on various devices, screen sizes, operating systems, etc. It’s a good idea to gather evidence about how the system actually behaves in different environments.
  • What and why should happen in the background? Should the system subtrack the number of items in stock? Or should this happen only after the order is completed?
  • Should I add just one item? Why not more? Why does it seem to suggest that adding one item equals adding more items? The system might very well behave differently depending on the quantity of items being added to a cart. An example could be discounts — a customer gets a discount for higher quantity of the product.
  • What does a shopping cart mean? There might be more instances of it. E.g. when a customer logs in, the shopping cart might be personalised. Or perhaps there are already other items in the shopping cart, which might play a significant role. Simply put it, it says a shopping cart, not the shopping cart, so I don’t really know what it means.
  • Should we log user’s actions? Peerhaps we want to monitor users more closely so we can use the data for marketing purposes. That leads to all sorts of ethical questions that are usually nowadays translated into GDPR (in Europe) and informing users about these practices before we gather any personal data.
  • It’s difficult to capture examply what is meant by such steps. The steps are many times ambiguous, too general, they omit important aspects, they can’t really capture tacit knowledge. That can lead to more confusion than not having any written procedure at all.
  • Written test procedure might limit the thinking a tester puts into the task. Some testers might simply take the two steps, build a mental model based on that, perform the testing, and mark the test procedure as passed. All the questioning might be left out because the tester was given these steps and not given an open-ended task.
  • It’s not easy to follow procedures all the time. If this is all you are asked to do, you might find it boring. And people don’t like boring tasks. So you might start thinking about different things, daydreaming, etc. You might also start thinking about other ways of performing the steps, which is paradoxically exactly what you should do. But you should do it consciously, not as a by-product.
  • Written test procedures tend to get obsolete quickly. Systems change, documentation not so much. It’s expensive to keep documentation in sync with the actual system.
  • If the intention of written test procedures is to guide junior testers, it’s still bad. Junior testers need to learn to think about their testing. That’s the first and foremost task they have. It’s better to give junior testers challenges, do pair testing with them, debrief with them after their test sessions.




Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

What Is Serverless Computing

Azure Functions Java CI/CD pipeline in Azure DevOps

How to set up your PowerShell

Kubernetes Compute Resource Management and Upper-limits

Thread Management and Thread Scheduling

Running a Kubernetes lab for Windows Users

How to mock Salesforce External Objects

System Design Concepts: Dynamo, 16 Sheets of Paper That Changed the World

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Pavel Saman

Pavel Saman


More from Medium

Why Do We Test?

It’s the QA’s fault.

Ananta Wali, QA manager at Codal, discusses her role and the importance of rigorous software…

Why to use Loggers?