Referees, testers, and bias

There’s one day until the scheduled release day of Make-Or-Break (MOB) software. You’ve been working long hours testing MOB, to the point that it visits you in your dreams (where everything is broken and the developers are all on vacation). You keep pushing yourself though, because you haven’t hit that warm, fuzzy “Let’s ship it!” feeling. (Or as I like to think of it, the point where you’re willing to “Put your name on it!“) You’re testing the print functionality of MOB, a piece that everyone on the product team has been lauding as well-tested and successful.

But something doesn’t seem right here.

For a brief moment you think you might have a bug to investigate, but no… it’s fine. We’ve tested this piece already. We’re shipping tomorrow. You didn’t really see anything. You move on.


I’ve been reading a lovely little book called Scorecasting. (I recommend it to anyone interested sports, statistics, or just the interesting factors that influence people. Or anyone ready for a good argument, because I’ve found a few of the chapters to be flawed in their approaches.) There is a great chapter that examines the influences behind home field advantage, and it basically uses statistics to boil away things like travel and crowd influence on players, and settles mainly on referee bias as the primary factor. (I imagine Mark Cuban has this chapter printed out and framed in his office with the authors’ autographs.) The authors then discuss some of the psychology behind the referee bias; they suggest that they aren’t biased out of any kind of conscious favoritism, but:

In trying to make the right call, they are conforming to a larger group’s opinion, swayed by tens of thousands of people witnessing the exact same play they did.

In addition, there is the possibility that the referees are swayed by a suggestion from the league that when the home team wins, that organization makes more money in sales, and it’s therefore better for the league. That seems like a bit of a stretch, I know, but then they put it in terms that any one of us can understand:

If your boss sent a subtle but unmistakable message that Outcome A was preferable to Outcome B, when you were forced to make a difficult, uncertain, and quick decision, how would you be inclined to act?


In this light, it sounds like you missed that print functionality bug in MOB because of home team bias. The crowd thought the product worked. Your bosses wanted the product shipped. Bias — not a conscious decision — in the heat of the moment led you to make what seems like a bad decision to an impartial (and less pressured) observer.

My point — if I have one other than sharing a snippet of Scorecasting — is that bias is real and it is not limited to software testers. Even folks paid to be completely impartial imposers of rules like referees are subject to bias’s skewed grasps. The question is how do we limit bias? In sports, it will remain mostly fact; there will always be paying fans attending sporting events, and influencing referee calls in various, often indirect, ways.

It seems that once you are exposed to factors that cause bias, there’s no avoiding it completely. If you’ve been on the product team for MOB and have been hearing about the schedule and praise for the print functionality, you can’t unhear those things and shed the bias they spawned. But you can be aware of it and talk about it. Maybe you shouldn’t test that piece; find someone on another project. Maybe you should discuss your concerns with the team instead of testing it alone. And, maybe, if the bugs gets to production, you just have to explain the bias that contributed to the miss.

For now, all this talk of bias is making me hungry for sports. And, oh look, Game 7 of Thunder vs. Grizzlies is on…

Page with Comments

  1. Huh! I always pointed to simple self-doubt when I talk myself out of bugs or enhancements, but I never considered how the doubt could rise from the environment around me! I think when I’m in the current rushing something out, it is hard to resist the push. Interesting points, indeed.

    1. Thanks for the comment! I’d say that self-doubt is a component as well for most of us. Maybe the broader point is that it is helpful to be able to identify and talk about the factors that contribute to us missing or finding bugs.

Leave a Reply

Your email address will not be published. Required fields are marked *