Here’s a quote from a book on beachcombing that ties perfectly with software testing: “a beachcomber’s life is a demanding one that calls for discipline and zeal… it’s the endless seeing that causes the psychic strain. It’s the richness of life in the tidal zone.” After working through numerous internal builds there is a psychic strain in keeping my eyes open to something I’ve seen what feels like (and may be) dozens of times.
I am literally getting close to the final round of testing on a project. I can’t write projects specifics but I can write about how it feels at the end of testing and tactics I use late in the game.
The end requires more discipline than creativity – my opinion. The bugs I find in the final cycles tend to be less interesting, less exciting but necessary to find and deliver.
I like the comparison to beach combing because I collect sea glass – ok here goes a digression (I didn’t really know I digress so often until I’ve been working with Mike Kelly who gracefully has pointed it out to me). I grew up in Boston and spent many summers on Cape Cod – being near water is important to me – now years later I live in a ‘burb of Chicago, a mile from Lake Michigan near a beautiful beach. I beach comb and collect sea glass even here in the Midwest. If you don’t know what sea glass is you can learn about here. I happen to have what I would consider a pretty decent collection that’s taken years to accumulate. Yes, years. Collecting sea glass takes patience. So does collecting the final round of bugs.
Software bugs are like sea glass – interesting small broken bits that we scavenge for. Consider this phrase from Wikipedia on beachcombing: “Sophisticated recreational beachcombers use knowledge of how storms, geography, ocean currents, and seasonal events determine the arrival and exposure of rare finds.” Certainly compares to what an experienced tester can have up their sleeve compared to someone new in the field. Knowledge of how bugs breed and cluster, how moving a build to a different environment (testing to production or preproduction) can uncover configuration issues and each release can expose rare finds.
I want to point out some distinct WIFFMs and be less abstract:
Recognize some testing requires raw discipline. Even sophisticated combers deal with the grueling aspect of many hours hunched before finding the final bits and sometimes without a find at all. I use the term raw discipline because in all honesty, the final rounds of testing can require more discipline and focus than earlier rounds (at least for me). I’m tired right? The big splashy bug finds are gone and now all that remains all the small shards. And here we go again with yet another build.
So what’s going on during these final rounds of testing? Like sea glass – when I think I’ve found a broken bit, I turn it over slowly and evaluate. Always be willing to discard a find.
Each build provides opportunity for new finds. I feel it is my obligation to stay focused and disciplined at the end of release cycle. But I am human and I get bored, I fidget, I long for the great finds from the early builds, I miss the gotcha bugs while I’m left combing through an application looking about for a small paltry bucket of low priority defects – these little low bugs are not as joyful to find. But then again, I remind myself that I’m part of a team that fine-tunes a product before it releases. I can part of what makes a product – shippable.
I would consider this combing process another element of what I introduced as sweep testing. For as broad as the strokes of testing are at the beginning, the strokes at the end are subtle, a fine filtering. It takes ingenuity in the beginning to feel like I’m outwitting the coders to think of conditions and permutations that make an application twist. But at the end, it’s about finding comfort for the team that product will stand well in production.
And my tactics are different too. I have to force myself to stay seated – yikes, do I want to admit this? But yes, I do because there is – or at least for me the reality that at the end of product build, it is simply a hard hoe. I have to stay focused and alert – after all late introduced bugs are possible and can destroy a release.
I have small personal tactics that work for me. And I will distinctly explain them. (I would have thought this might be dull but email tells me that people want specifics so here goes …)
I barter with myself– can I finish this cycle within X period of time? When I’m done this cycle, I’ll take a stretch. Late in the day, I’ll step out – buy a coffee and since I have the good fortune to work from home often and live near the beach I’ll head over to the Starbucks by the lake. Take 15 minutes, drink expresso, stare the water, and refresh my mind. And then it’s time to get back and go another round. I remind myself of my obligation to deliver, that sometimes it’s discipline which isn’t the brilliant or exciting part of testing but the commitment.
I use my notes. I’ve kept my notes all along the way and now I use them. I go back and I read. What were my first questions about the product? Sometimes I am amused to see how much I didn’t know in the beginning and how much more I know about an application after months or weeks of testing. But the basic questions from the beginning help me at the end – how? – because they’re centering questions – usually the high level questions are guidance to remember what’s important.
I look through my notes, where did I find issues? What areas was I confused about before, because confusion holds opportunity. I use my old Excel files where I’ve captured details. I use worksheets in Excel files to separate ideas by testing areas or categories and now I have those notes, thoughts, tests, bugs – it’s just a method of being organized and not that I’m advocating one method over another – it’s about returning to my notes, not what my notes look like or how I write them. When I’m highly organized I have notes from every release I touched, I know what tests found what bugs. It’s not a documentation heavy process, I write without effort.
I often re-run a short risk analysis in my head. I ask myself continually: what would be awful if it happened in production? I find this to be the single most centering sobering thought to focus my energies. The thought of a possible slip to production is what pulls me through the end.
