APRIL 3, 2018

Why I Prefer Good Testing Over Excellent Testing – Part 1

I gave a 15 minute talk on this topic at StarCon in January 2018. While I enjoyed the challenge of fitting everything I wanted to say into 15 minutes (I'm not usually great at being concise), I know I could have gone into a lot more detail on some of the information if I had more time. Therefore, I've also submitted this as a full-length topic to PNSQC, which takes place later this year. The neat thing about this conference is that they also require a peer-reviewed paper to accompany each presentation. So I've decided to do a series of blog posts that elaborate on the information in my original 15 minute talk. Having some material prepared in advance would be useful if I do end up needing to write a formal paper, but I also just wanted to spend some more time thinking about the topic anyway.

So here's part 1!

Why is this topic interesting / relevant?

Although I'm currently a Software Development Manager, until recently I've mostly been in test-focused roles throughout the rest of my career. I used to think that if only we had more time for testing, we'd eventually find all of the important issues. But I started noticing that most escaped defects I could remember were things I never would have found, or couldn't have prevented anyway. So, while I believe good testing will always be important, instead of investing in truly excellent testing I'd rather assume things WILL go wrong and be well prepared to handle failure.

What is the difference between Good and Excellent testing?

To me, excellent testing is thinking of as many test cases as you can, trying your best to run them all, and continuing to test until you run out of time. Good testing, on the other hand, still involves thinking of as many test cases as you can, but choosing to run only the ones that will give you the most information about how well your product is working. Then, spend your remaining time planning for how you'll detect and resolve failures that you weren't able to anticipate.

What are the other blog posts in this series going to cover?

To illustrate how I became convinced that good testing is better than excellent testing, I'll share some stories about things that have happened to me lately. These will include defects I've missed that I wouldn't have found with all the time in the world, and cases where I have been saved by solid monitoring and roll-back strategies. I'll also share ideas for how to get started with planning for good testing, and preparing for failure detection and handling. Finally, I'll provide some comments on what I think all of this means for testers, for developers, and for how to approach software development in general.

, TF