What is it? Why is it important for a tester?
Let’s have a look into American Psychological Association’s dictionary:
any influence a researcher may have on the results of his or her research, derived from either interaction with participants or unintentional errors of observation, measurement, analysis, or interpretation. In the former, the experimenter’s personal characteristics (e.g., age, sex, race), attitudes, and expectations directly affect the behavior of participants. In the latter, the experimenter’s procedural errors (often arising from his or her expectations about results) have no effect on participant responses but indirectly distort the research findings.
This has a significance for testers as well because:
- testing is experimentation
- testers observe, analyze, measure, interpret
- testers often set expected outcomes, and they have expectations about the product they test
In reality, it can go something like this. A tester expects the product is working correctly and there are no bugs. She simply wants to confirm the product is working correctly. In line with the experimenter effect, she will likely get such results, that is no bugs, everything works.
Another tester might come to the product with different expectations — there are bugs waiting to be found. She will more likely find results that support this expectation.
Experimenter effect has been described in depth by various scientists, for example by Rosenthal in 1980s. It’s nothing new, yet I don’t see that many testers being concerned with it (but they should be).
The experimenter effect is typically mitigated by blinding strategies. But what about testers? What do you do to mitigate this bias?
As BBST courses mentioned, there’s not so much we can do apart from biasing ourselves towards finding bugs. Yes, that’ll end up being a bias as well, but on the safe side. It’ll provide some false positives but that’s probably better than not noticing important problems.
Last but not least, is this something testers are aware of? Some are, but in general? To be honest, as I read online material on LinkedIn and here on Medium, it seems to me testers speak about checking that the product works way too often. I’d like to see more posts and discussions focused on risk — what can go wrong.