The trace matrix has always disturbed me. For me, there’s something uncomfortable about taking the complexities of testing and reducing all that intellectual activity, thinking, and effort and squeezing it into a small tidy table that allegedly proves testing is complete that doesn’t feel right.
I’ve tried blogging about the trace matrix before but the topic irritates me to the point where I have I haven’t been able to write on the topic. That’s a pretty strong reaction to a simple “tool” and my skepticism and concern about the use of this tool remains to this day. My intent in writing about the matrix now is to share my opinion, give exposure to a testing topic, and hopefully hear about other people’s experiences and thoughts about the matrix.
If you’re unfamiliar with the trace matrix, let me explain its classic layout and alleged benefit. The trace matrix begins with building a table (often done in Word). There are a few variations on the matrix but often the matrix contains four columns. Number each requirement and place the requirement number in the first column, followed by the requirement itself in the second column, followed by the test case number(s) or reference points in the third column. I’ve also seen the third column used for the test case number and/or test step numbers and the fourth column used for the test case name. There are variations on the theme but this is the basic layout.
The alleged benefit is that by showing a one to one requirement to test case (or multiple test cases), you will be assured that testing is complete and that all requirements have been covered. Wikipedia has an entry on the matrix, referring to it as the traceability matrix which is the name used at times. The layout shown on Wikipedia is not a layout I’ve seen in practice despite having seen many matrices’ since I began working (intermittently) with regulated software in 2001. But layout isn’t a concern of mine; my concern is with the belief that a simple mapping of requirements to test gives assurance of coverage.
When I first began working with regulated software, I was sent to FDA auditor training. I was taught that the trace matrix – along with bug reports, complaints, validation plans and final reports would be the first materials requested by an auditor. Before entering the world of regulated software, I had been in software testing since 1994 but had never heard of the trace matrix. And so, to begin with on the basis that I had worked at numerous software companies and seen many a good product ship and had never heard of the matrix, I approached the matrix with skepticism.
For some years now, I’ve observed the people who embrace and promote the trace matrix seem to be people who don’t actually like testing. I know this is a generalization but trace matrix advocates seem to be people who would like testing better if testing were a containable manageable finite task.
For the record, I wrote one of the trace matrices on a very large project just a couple of years ago. Yep, me – not someone else but me personally. I recognize its requirement in a regulated environment where an auditor will look for the trace matrix as one of the first documents. But I also don’t expect most auditors know much about testing nor have much if any, real life experience with testing. I expect most auditors have been taught the rote mechanics of checking requirements to test cases with a checklist mentality. I feel I can say this because I attended auditor training and I know with what detachment I was taught that a glance at the matrix/table to ensure no “empty boxes” appear was taught as an assurance that testing is complete had me cringing. This type of investigation lacks thinking.
The trace matrix advocates I’ve worked with often seem to be date-focused more than product-quality-focused. When a date is the most important aspect of a project, testing becomes a nuisance. I’ve tried to get inside this type of thinking – the thinking that testing is a nuisance instead of an intellectually fun puzzle to solve – I try to understand the mindset because I think its mind expanding to try to fundamentally understand someone whose train of thought differs so radically from mine own.
When I finally got to a mental state where I thought I could see this different perspective, I could see how testing would be awful. It’s so unpredictable, takes wild turns, refuses to be time boxed, forecasted and is hopelessly never complete. Don’t you love that part of testing? It’s never done. It’s what makes testing a wide awake type of job. There always another permutation, combination, O/S patch – some other bizarre combination of data and functionality that twists creating splendid bugs. It’s what requires testers to think hard, because sometimes testing is about outwitting a system –sure we check core functionality works but good testers, great testers look to find more. And all that type of investigative thinking doesn’t fit neatly in a trace matrix or on a line item on an MS Project Plan.
What I love about testing is what PMs and auditors hate. Now, I’m using big basic words like a child speaks with no shades of grey but that is how it feels and how I see the behaviors play out. PMs often want testing contained to a line item on a MS Project Plan but then nasty bugs show up with no regard to proximity of the ship date. Can’t we find all the bad bugs first? This question has amused me for years.
On the flip side of this internal mental argument about the trace matrix – and I’m about to argue back and forth here – I can see some benefits of the matrix as long as skepticism remains – deep skepticism. It may be convenient to have all the requirements housed together somewhere – whether a requirements tool is used or a well-maintained spreadsheet. It is convenient to have a central repository. Although I have yet to see a product, regulated or not that truly has all the requirements written despite sincere efforts. Perhaps in building a matrix, this forced housekeeping task could highlight an item or two that have been overlooked – either a requirement or a test – perhaps.
But what a long tedious road to hoe in the hopes of finding an oversight, couldn’t a better method be used? I would, if I were an auditor, long to hear or read the internal thought process of what was tested, what was deemed a higher risk and handled accordingly, to hear the thinking behind the trace (or an FMEA or MMEA). I cannot find a single tool that I would want to in place of hearing whether orally or through recorded prose what someone thought, or how a team thought and then executed with that knowledge. I would, of course, want the story and not a table.
I’ll try the other side again; I think some small assurance can be gained in identifying one test was conducted against each requirement. But this side of my internal arguement doesn’t work for me as I find a one to one map is such a thin blanket of comfort that I find no warmth in the tool. And an over-reliance on the matrix as proof positive that everything has been covered is naïve. Perhaps, and I’m trying again to find value where I have previously found little to none, perhaps by seeing a long list of tests to a particular requirement an auditor would surmise the essential risk and priority of a particular requirement and that would lead them to conduct a more thorough review – the matrix would give an auditor a blatant call to attention.
Is that possible call to attention worth the massive time suck? The matrix creates volumes of pure busy work. A shame truly that the time and labor spent building the matrix couldn’t be better directed to more testing. And the larger the application being tested, the more impossible the task becomes. Not to mention the maintenance. And who can then read and digest very large matrices?
Recently, I sat with a new client. The topic of the trace matrix came up and I could feel that same mental and physical reaction of mine to the topic. My face must have given some indication of my opinion (and if you know me personally, you can imagine). The client stopped and asked what I thought about the matrix, I nodded and chose my words. I explained my concerns, ranted a bit about how testing isn’t about a one-to-one map game and gave an example of how the trace could fail. The response? The client agreed it seems he shares my skepticism. I was asked if I could, and I quote “apply critical thinking to test this product.” That I can work with. And at the end of this project, if some mapping is done to fulfill an auditor’s request, so be it. But if the auditor doesn’t know more than to look for a one to one mapping, or believes that a completed matrix shows thoroughness then the auditor isn’t using their own critical thinking.
