I am making this post to share my thoughts on testing certifications & standards and the recent activity that took place at CAST 2014.Since the activities that took place at the CAST 2014 conference, I’ve had several people ask questions or make comments via Twitter. As I do not see Twitter as an easy place to clarify deep thinking, I decided to write this blog post to state my thoughts.
This post has multiple parts:
Part 1: My thoughts on the Professional Tester’s Manifesto
Part 2: Questions Asked
Part 3: CAST 2014, The back-story to current activities
Part 4: The Professional Tester’s Manifesto
Part 1: My thoughts on the Manifesto
First, I want to review the statements made within the manifesto, statements from the manifesto are in bold – my thoughts follow.
That standards compliance is no substitute for knowledge and skills, and that possessing a certificate demonstrates neither.
That companies have been convinced that only certified testers should be hired.
That organizations who use certification as a surrogate for rigorous selection processes place the quality of their testing at risk.
For several years when I was working as a test manager, I interviewed and hired numerous people. Once during an interview, a candidate pulled out a certificate and held it up for me, telling me he should be hired instantly due to his certification. I asked that he put the certificate away and talk with me. After discussion, it was my assessment that the candidate had little working understanding of how to test a product. Perhaps he had memorized material or had someone else take the exam for him – whatever had been the case; there was no evidence that the candidate would be able to perform well. As this event took place some years ago, I will refrain from trying to recall more specifics.
Given this same scenario, if I did not have a testing background but needed to hire someone, perhaps I would have been convinced based on the certificate that the candidate was equipped for the job.
If a company is hiring testers and the person or persons interviewing do not understand testing (which is often the case with HR), I do not believe that hiring person is qualified to make a decision – or even qualified to establish a pool of candidates for others to interview. This is where certification is dangerous – at first glance, it sounds good, it seems like it should provide some assurance that a person is qualified. But as I have found, having a certificate does not provide evidence of a person’s knowledge or skill.
In my work, my own personal experience as an employee and consultant, I have not worked at a company requiring certification. I have heard from several colleagues in Europe that companies do require certification and I find this concerning. Further I have heard colleagues say they have paid the money, taken the exams and become certified just so they can find work or keep employment even though they do not believe the information gathered from certification has helped them.
Given that set of circumstances (people believing they must be certified to find or keep their jobs), I believe it is important for people within the testing community who do not believe in certification to say something. Is the Professional Tester’s Manifesto perfectly worded? No. Should a statement be added that says something such as: I am certified but I do not believe in certification. I am certified but have not found the certification to help me with my work. Yes, I wish we had included that statement but with 60 signatures collected, I cannot change the wording without an amendment process and recollect signatures. At this time, I’m not going to do that.
That organizations who make money from creating or promoting standards and certifications are biased in their thinking by the potential financial rewards of convincing organizations that only certified testers are professional testers. Those organizations may include those who sell training, consulting or other related services.
I find it disturbing that certification companies are making money, while lone individuals feel they cannot fight against the increasing number of companies requiring certification. I have concerns that the same scenario could eventually take place in the US.
As the certification companies have significant influence and their own marketing dollars to spend promoting their view, I believe that testers who do not believe in certification lack this same organized voice to state their opinion – this further enables the certification companies to their benefit. And while people in the testing community agree (against certification) but have not pooled together to make a collective voice against certification, the situation continues. It is my hope that this statement signed (hopefully) by a large number of testers, will give us this voice.
That testing benefits from diversity and not homogeneity: that testing is not a profession that can be standardized but instead needs to remain an intellectual professional activity.
The knowledge needed for a tester varies based on what type of software a tester is working with and the type of testing being executed.
a) For example, a tester working with ecommerce software and focused on performance testing may need to test spike conditions as well as simulating other load conditions. In this case, a tester would likely do best to have knowledge of math and statistics.
b) For example, a tester working with business intelligence software and focused on pulling accurate data into a data warehouse may need to test data cleansing routines as well as simulating data loading from a variety of sources. In this case, a tester would likely do best to have knowledge of SQL and stored procedures.
c) For example, a tester working with a medical device and focused on science lab simulation may need to test data transmissions and multi-user scenarios. In this case, a tester would likely do best to have knowledge of FDA regulations, GLP – good lab practices, GDP – good documentation practices.
All three of these examples come from my own personal experiences as a professional tester in the past twenty plus years testing.
While it is true that the context around these examples could not possibly be covered in a more generic testing certification, there is no evidence that the base knowledge obtained by the certifications is sufficient base knowledge. There is also no evidence that the materials within the certification are developed or continue to evolve in a timely enough manner to address the every changing software/hardware/technology world in which software testers work. So while the certifications could not possibly address the vast array of software contexts, there is no evidence that the base knowledge acquired is “the right base” or a “sufficient base” or that the “base is updated in a timely fashion.”
In each of these examples, a tester would do best to have a thorough understanding of the software solution and its intended use. As you can tell from the three examples, each context is entirely different and therefore the knowledge background varies accordingly. With certification, little information to none of this material is addressed and yet these are three real examples. In these cases, certification would not help. I base this last statement on the fact that although I do not hold a certification, I have read and reviewed the body of knowledge for one certificate program and purchased and reviewed the book of another certification – and in each of these three working experiences, did not find anything from the body of knowledge that would have assisted me in this work. In all three examples, I expanded my own knowledge from an assortment of books, regulations and other materials.
I am not saying the information from the certifications is detrimental but I am saying I have not found the information from the certifications to be instrumental in getting the job done.
That choosing not to be certified does not mean I do not take my profession seriously. It is because I take my profession seriously that I choose not to be certified.
I am consciously not certified. I believe obtaining a certification would further enable certification companies a financial benefit while providing me with nothing further than the ability to say, I am certified.
Each year, I continue to invest in my education. I continue to learn technology, skills – at times technical and at times “soft” skills. I openly share the books that have shaped my education on my account on a free website known as: www.librarything.com I write articles, I tweet and I do all online activities by my own name, making it clear what my background, knowledge and experiences are. I take my profession seriously. I do not believe in certification.
Part 2: Questions Asked
Some questions I have been asked:
What about having a standard set of definitions and terms?
Having a standard definition and understanding of terms such as: test case, test plan, and test strategy is one of the most common arguments in favor of certification. In each work environment I have been in, including environments where many people where certified there has never been a singular understanding of terms. In each environment, on each team, people have needed to come together to understand what each other means and come to some type of informal understanding of the terms based on the context. So clearly a standard set of definitions have not been achieved.
Do you speak at conferences hosted by certification organizations?
I do speak at numerous conferences each year. In some cases, I do not know where all the funds come from or where all the funds are spent. The one time I was asked by a certification company to speak at a conference, I said I could only speak if I was given an opportunity to have a panel discussion on certification. I also was clear in stating that I am opposed to certification but would welcome a healthy debate on the topic. The response was they would select someone else to speak. I did not attend.
Why are you speaking out against certifications now?
I am a long-standing member in the software testing community. Although I typically avoid heated public debates and prefer instead to more quietly and peacefully pursue my career and business, I believe the issue over certifications is a growing issue. I recognize that silence can imply compliance and I am not willing to do so any longer. I believe it is time for me to be a vocal opponent as many of my colleagues have been already.
What do you hope to achieve with the “Professional Tester’s Manifesto?”
I hope that we can find agreement in the community that certification is not effective. I hope that while certification companies and certified testers have one another to rely upon that we as a group of people – otherwise not affiliated and connected can discover that we have many like-minded people who will band together. If we can collect a large enough number of signatures, we can make a unified statement and hopefully make a change. After all, it was the signing of the Agile Manifesto that drew and continues to draw a tremendous change in our industry. Change is possible. We may not agree on many topics but may the signing of this statement, align us on this one topic.
Why don’t you believe the current certifications are beneficial?
There is an absence of peer validation of the “certification” standards and procedures by recognized experts outside the “certification” organizations. There is no evidence that these organizations’ “certifications” provide a useful or meaningful baseline of competence or effectiveness for companies hiring testers. The fact that an organization creates a concept of “certification” and then proclaims that those who complete materials and testing that have not been peer validated are “certified” is an exercise in circular logic.
Part 3: CAST 2014, The back-story to current activities –
At CAST 2014, James Christie (@james_christie ) gave a presentation on standards and certifications. (AST recorded his presentation; the video will be available sometime this week – meaning the week starting Monday August 18th) through AST (@AST_News). To summarize, Jim’s presentation highlighted disadvantages of both (certifications and standards). Although I cannot claim to know what people were thinking in the audience, it appeared as though many people in the audience were in agreement with the presentation. At the end of Jim’s presentation, several questions were raised – the most notable question being: what can we do? (I don’t recall who asked this question.)
Following the format of the conference I held up my “pink card” out to speak – the pink card signifies to the facilitator that I as an audience member have something urgent to say. To summarize, I suggested that since it was the start of the conference (at this point it was the end of the first session of the conference) and many of us seemed to be in agreement on a position against certifications and standards) that perhaps we could “do something” that would make an impact. It seemed to me then (and now) that many people have been in agreement on this topic for years. In my view, there was “energy” and support in the room to do something.
Iain McCowatt (@imccowatt ), Fiona Charles ( @FionaCCharles) along with Jim Christie and I talked at the end of Jim’s session. As each of us had other obligations for the remainder of the day, we agreed to think about what took place and meet again at the end of the day.
At the end of the day, I had drafted a statement – which statement is now being referred to as the “Professional Tester’s Manifesto.” The statement was drafted and circulated at CAST2014. Sixty signatures were collected at the conference. Numerous people have asked to sign this statement since the conference. A website is being built to provide that opportunity. At this time, the site is not up. The text of the statement is included at the end of this post for reference.
In addition, Iain wrote a petition that is now referred to as “Stop 29119.” See the website: ipt.io/tiul The petition is a call to action, the call being made to the ISO president.
The two statements are separate. The statements serve a different purpose.
Part 4: The Professional Tester’s Manifesto
The exact words from the statement follow.
A website will be up (hopefully within the week) for people to review and sign.
The Professional Tester’s Manifesto
I, as a professional software tester, believe:
That standards compliance is no substitute for knowledge and skills, and that possessing a certificate demonstrates neither.
That companies have been convinced that only certified testers should be hired.
That organizations who use certification as a surrogate for rigorous selection processes place the quality of their testing at risk.
That organizations who make money from creating or promoting standards and certifications are biased in their thinking by the potential financial rewards of convincing organizations that only certified testers are professional testers. Those organizations may include those who sell training, consulting or other related services.
That testing benefits from diversity and not homogeneity: that testing is not a profession that can be standardized but instead needs to remain an intellectual professional activity.
That choosing not to be certified does not mean I do not take my profession seriously. It is because I take my profession seriously that I choose not to be certified.
Excellent post. Thank you !
Karen: Thank you.
Karen,
Thanks for posting this. I was away last week and didn’t follow any of the tweets from CAST, so this is the first I’m hearing of the Manifesto. I especially appreciate that your post shows both a calm demeanor and your passion on the subject. Sometimes this topic gets dealt with far too shrilly (from both sides) and we end up drowning the reason out.
I look forward to the site being completed and adding my signature to the Professional Tester’s Manifesto.
Karen: Thank you.
As one who believes it is important to be “for” something rather than merely stating what I am against, I am happy to see this post, this manifesto, and I am more than happy to support both.
Karen: Thank you.
Hello Karen, been a while, how you keeping?
I’m no particular fan of tester certification either & I too have chosen not to be certified – apart from for the loose-cannon madness I’m well known for.
However I have to say the The Professional Tester’s Manifesto above reads more like The Anti-Test Certification League Manifesto. I see what is trying to be achieved here and applaud it however to be taken seriously by a community as a whole, IMHO any sub-group would be well advised to avoid polemics at all costs. Respectful suggestion only.
Geoff
Karen: Geoff, it has been a long while. I’m well – other than I miss being in New Zealand. I believe I am known for previously avoiding all “polemics” but this time around, I think it is time to have an opinion. I have not singled out names or organizations. I hope the message is clear. I’m not keen to hear people see it as a rant or an “anti-whatever” but it seems there are times when it is necessary to say: I disagree (with the current certifications).
I am mostly disappointed that the “Professional Tester’s Manifesto.” is a one topic rant that does not do anything to further state of the art.
Are there no more interesting things to be busy with? Or is passion only expressed in voicing disapproval?
Karen: Yes, it is important to me. Yes, I have other activities going on within testing. Guess we don’t share the same view on this one.
Hi Karen,
I thought I’d quickly reply to your thoughtful post, and as a signer of the manifesto, it does not seem unreasonable to express why I did so. I feel that at least at present there are not certifications that can in any meaningful measure provide the quality of a tester nor state which organizations a tester would be a good fit in. While I personally think Dr. Kaner’s proposed approach maybe better than many other approaches that presently exist and that I have heard of, in weighting the two choices: A world where certificates are used as a proxy for human judgment or A world where a certificate’s value is only what the person learned while getting said certificate, I think I would prefer the latter. I may feel the document that was generated at CAST was imperfect, but again, using a dichotomy, I prefer having something for people to see, consider and debate over some nebulous opinion piece on a blog.
Karen: Exactly – that was the point (to provide an opportunity to collect signatures – as a place to register an opposing view.) The manifesto does not offer an alternative to certification. I agree that Dr. Kaner’s proposal offers an alternative – not just dissatisfaction with what currently is.
In regards to having standard definitions, I have spent the last year going over words used both inside and outside of our industry, such as ‘heuristic’ and ‘critique’ and have found that many words we think have standard definitions don’t. The deeper you go into the rabbit-hole, the more diversity of opinions you will find. That being said, in one organization I worked with, nearly half the testers took BBST, and even then we debated and argued what words meant. Even the BBST class seemed to have gotten vocabulary words mixed up in a way that made it hard to know the definition (E.G. Is an oracle the tool that generates the result or the result itself?).
Karen: Indeed. I have been at workshops where more time was spent on definitions than moving onto other topics. Personally I wear out on these dialogues. I have often felt so hungry to learn that I’ve wanted to say – let’s agree on a definition – perhaps imperfect – and move onto other stuff.
In your ‘What do you hope to achieve with the “Professional Tester’s Manifesto?”’ I do find something of concern. I may feel certifications are problematic and are too fiscally concerned, but I don’t want to stifle others, some of whom may feel that a more formal education is a good method to learn. It would be interesting to see a degree in or around testing, but I have not seen colleges jumping to go that route. Furthermore, some forms of education have a ‘certificate’ at the end saying effectively that someone took a class and passed. How do we avoid going from ‘Certification is does not ensure practical knowledge.’ to ‘Certifications are wrong. Those whose learning that is partly developed by certifications are wrong too.’?
Karen: Good questions. I honestly do not have alternative.
Dr. Kaner has been offered the concept of open certification for a long time now and even that has not taken off the way one may hope.
I see myself as someone who was around at the early stages of this field, that my learning and background have been acquired over years of reading and on the job experiences – hard to summarize into an exam.
The manifesto has limitations – we knew that when it was drafted. It is a statement; it is not a resolution.
– JCD
REFERENCING: “That standards compliance is no substitute for knowledge and skills, and that possessing a certificate demonstrates neither.”
This comes off very harsh to those of us who are certified. I agree with the first part of the statement wholeheartedly; however, disagree with the later part. I am a professional tester with 15+ years of experience. I have two technical certifications, one of them for testing. I believe the certification I poses backs up my resume/experience.
—-
REFERENCING: “That companies have been convinced that only certified testers should be hired.”
I have not experienced this in my location, Southeastern United States
—-
REFERENCING: “That organizations who use certification as a surrogate for rigorous selection processes place the quality of their testing at risk.”
I can agree with this statement, but the issue is deeper than just certifications. The same can be said for candidates with extraneous college educations. Just because they have studies and managed to mass memorize enough information to pass a class doesn’t mean they can apply practical knowledge.
—-
REFERENCING: “That testing benefits from diversity and not homogeneity: that testing is not a profession that can be standardized but instead needs to remain an intellectual professional activity.”
Yes and no. You need to have diversity and be able to think outside of the box. You also need to be able to apply practical knowledge when testing. I do not feel standardization of software testing is a bad thing. I would consider it a baseline, or a starting point. I have worked on several projects over the years and so many of my peers lack the intuition to test the boundaries of the system, to create alternate scenarios, to perform negative testing. If each tester within an organization understood some of the same methodologies and then were left to also use their own creativity in testing, that organization would be deadly (in a manner of speaking).
The issue with standardization and terminology is no two people have the same perspective. I have never met two professional testers who actually agree on what the difference between a scenario, a test case, and a test script. However, as professionals, we can come together and agree and document it so that we are on the same page going forward.
—-
REFERENCING: “That choosing not to be certified does not mean I do not take my profession seriously. It is because I take my profession seriously that I choose not to be certified.”
I also take my profession seriously and I am certified. I very much enjoyed the whole experience of taking the class and passing the exam. I am proud of my certification and look forward to obtaining the next level. Is it a money making scheme? – Maybe, but I feel good about it and that is what is important to me. We all make have to make and live with our own decisions.
Thank you Karen for sharing this blog and your insights!! :-)
Hi,
Like it, however there are two things bothering me.
1. companies have been convinced …
Well, addind some would help me to agree. I do a lot of recruiting in different countries. But i do test the ppl. Certifcates are no prerequisite, but not hindering as well.
2nd issue. Choosing to get certified for me was always about obtaining knowledge. I might be overly sensitive, but one could read that sentence like “testers choosing to get certified don’t take their profession seriously” . Sure u did not mean that.
Thx
Karen: Absolutely I believe in obtaining knowledge. I read, study and confer with others every year and after more than two decades still believe I have a lot to learn.
There is a danger in certifications giving the impression that once knowledge is acquired, a person is done being a student. (Ongoing needs to show current work and fluency in the field is quite limited.)
A second concern is that the current “bodies of knowledge” are not open for discussion or review outside of the organizations that have created them – this is a limitation. This is what I was referring to when I wrote:
The fact that an organization creates a concept of “certification” and then proclaims that those who complete materials and testing that have not been peer validated are “certified” is an exercise in circular logic.The fact that an organization creates a concept of “certification” and then proclaims that those who complete materials and testing that have not been peer validated are “certified” is an exercise in circular logic.
I saw your blog post and agree with your comment: The problem is when you do not get a job without one. The problem lies with mistaking a certificate for the real thing. The problem lies with the people and organisations hiring!
In my experience recruiters and HR professionals rarely understand our work and for them certifications are often substituted for vetting people. This is a difficult situation to change and certification companies seem to encourage this same poor cycle by inferring that a certification does that skill checking – that’s an issue.
I have little respect for the _current_ software testing certifications, and _most_ of the software engineering (including testing) standards efforts that I have seen have been deeply disappointing, and seemed to me to have been deeply polluted by self-interest.
Karen: I agree. I have been doing my best to keep my “person” out of the manifesto. I wanted testers in the community to have some mechanism to pool their voices, to be heard but I have no interest in turning this into a personal mission. My interests and commitment to the community are broader than this one topic. I hope people see that.
But I don’t agree with the broader generalizations that (1) certification itself is inherently bad or (2) that the people and organizations who advocate for certification (even the crappy current ones) are all badly motivated or engaged in bad conduct. Your statement “That organizations who make money…” goes too far. It encourages the reader to prejudge the motivations of people and organizations and sometimes, in my experience, that prejudgment would be unfair. There is no need to go this far to make your point. We should be encouraging critical thinking, not prejudice. This is one of the two reasons that I cannot sign this.
Karen: Interesting point. I know myself – and I have watched you as well over the years give so much of your personal time, energy and efforts – and often with no or little pay. People have to make a living. But I do – and I believe others do – hold some frustrations that some people and organizations appear to be making good money and have convinced other companies of the value of certification – based on what I have seen – on little to no evidence. That combination I find concerning.
As an attorney in California, I got to see the operation of (what I considered to be) a healthy disciplinary system for incompetence and dishonesty. We (California attorneys) pay a significant share of our dues to support a court system that polices the profession. The enforcement of professional accountability is one of the components of the most important professional certifications. I don’t believe that we are yet in a position to create such a certification in any branch of software engineering today, but I also don’t think it is disreputable for someone to decide they want to move our profession in that direction. That is not _my_ objective, and it probably never will be. But I don’t condemn people who have that objective.
My second concern is with the negativity of the manifesto. Yes, yes, current certifications are bad, bad, bad. Pheh. Bad certifications. BAD certifications! BAD, BAD, BAD. Where is the positive advocacy here? For example, I think that the context-driven community (of which I am a part) talks too much about politics and not enough about skill. Frankly, I think that is a disease that is as enticing and as cognitively deadly as standards compliance. The manifesto says nothing about professional ethics, professional community, professional education, or the positive meaning of a sense of professionalism. I think a professional tester’s manifesto must focus more on our worth than on the non-worth of others.
— Cem Kaner
Karen: Bad certifications – indeed. I can hear your voice in your comment and imagine you saying the same to me in person. But I also hear your objective critique of the manifesto citing where is the reference to ethics? Darn good point. I did not at the time of writing the manifesto feel that was a necessary topic and was intending for the manifesto to be solely focused on certification as opposed to looping in ethics or professional community issues.
These topics (ethics and professional community) are important. The fact that people within the wider professional community will not speak to me at a conference or in other locations – because we do not share the same views seems ludicrous to me. We should treat one another respectfully even when our views are different. Our debates should be on the substantive matter and not degrade to personal attacks. It is – to me – a shame that such divisions exist within our community. I have typically been known as “quiet” and indeed, I am an introvert who is not known for “picking fights” so I suspect it is with surprise that I would be behind this manifesto. But I also realize that silence can be construed as compliance. I loathe the idea that the current certifications could become more prevalent here in the States (as they are in Europe) and I feel I would be remiss if I did nothing to stop that from happening. The manifesto is not perfect, it does not solve many issues but I had hoped it would provide one simple mechanism for people to make a statement – that statement being I’m a professional tester and I am not in agreement with the current certifications. As independent people without an organization I believe that we have lacked having a mechanism to show that unity. This was my attempt. (And James Christie, Fiona Charles and Iain McCowatt – my co-authors – this is our try.)
Some years ago, I was aware of your attempt create an open certification – a certification costing little to no money. I missed your blog post back in March of this year (http://kaner.com/?p=401 ) where you outline a detailed proposal. Perhaps it is your proposal that offers the next steps – the steps you patiently outlined several years ago. Perhaps now the community is ready ~
Karen – Thank you for posting this. I have been out of software testing for a year now, having pursued a career in software development related to hardware testing. However, I have been keeping up with the community since it is an immense resource in leading edge software development concepts.
I think there is a role for basic software certifications, but not in the form they are being promoted now. When I first started in software testing about a decade ago, my company said they were willing to pay for certifications if I wanted them. I bought the books, read them end to end, and went through and passed all of the certification tests in about a one year period. However, their value was not in the knowledge gained but in providing a framework to discover the community and seek out how the concepts presented applied to the context of the testing I was performing at the time.
These certifications should be a gateway in preparation for a mentorship program of some sort, similar to the RST or BBST sessions offered (or your consulting work). They should be configured to give the new tester a basic introduction to various test concepts and nothing more.