I’ll be in Malmo Sweden this coming week teaching and presenting at the Oredev conference. You can read more here.
If you’ll be attending the conference or if you’re located nearby and want to meet up, send me an email. It’s interesting to meet other testers. I feel most fortunate to have the opportunity to be at the conference. It will also be my first time to Sweden, yahoo!
The week after Oredev, I’ll be presenting at the ExpoQA conference in Madrid Spain. You can read more here.
I’m surprised I haven’t blogged more about both conferences (Oredev and ExpoQA) but have been busy working. And now I need to pack …
I’m teaching a workshop on how to build a test strategy at GLSEC in Michigan. The workshop focuses as much on creative thinking and strategic thinking as drafting a test strategy document.
Over the years, I’ve often focused on deepening and continuing my technology knowledge with less focus on business and strategic thinking. I’m sure this has to do with my background and comfort zone; I’m used to learning new technologies all the time but less used to thinking about business strategy. In recent time, I’m schooling myself in strategy as much as technology. Without a business background, it’s a less familiar learning path. And as a tester (in my experience), I often didn’t have to (or ignored) thinking much about strategy – at least strategy from a team or product point of view. My strategic thinking was more immediately-centered on breaking whatever application I was working with. But from the perspective of leading a team, there is more and other things to think through.
As a test lead, test manager and now as an independent consultant, I’m often more focused on strategy and less frequently working as a test executor. Strategy has become what I’m working on nearly every week if not daily, by either coaching tactical approaches or in writing a test strategy or updating an approach based on new knowledge or a change or fresh obstacle that needs to be addressed.
The writing aspect of crafting a test strategy isn’t a challenge for me. I can write without difficulty (although I have some annoying habits like switching from present to past tense that challenges both my co-writer and different editors). But it is the actual writing that is difficult for some people and I realize this. Write like you’re talking would be my primary advice. Write what you think. And don’t write what you don’t believe you or your team can execute. I think of a test strategy as a statement of work, every aspect of the strategy should be a tangible or should provide insight into the thinking behind the tactical real work to come. Skip the fluff.
One of my teaching objectives is to help people get past the anxiety of writing. Another objective is to show how strategic thinking is reflected in how we execute throughout a project.
So there’s brainstorming in a group and then there is what I think of as brainstorming alone. I often brainstorm alone. The reason for this is that I frequently work alone on projects. The other reason for my solo brainstorming sessions is bundled up in my INTJ personality and my preference to have alone time to think through situations and problems without distractions or other people’s influence (at least sometimes). I like to have my own grounding time before noodling on a problem with other people.
I’ve been thinking about how to describe my solo think sessions. The reason is that I’m actually quite curious about how people analyze things. I’d like to analyze analysis.
I know it sounds heady and perhaps like I’m over-caffeinated and under-slept again but stand back and think about it. Do you know how someone else analyzes and if you could get inside the head and thinking process of someone else to learn how to think better, clearer or just to gain alternate way to think, wouldn’t you want to?
I’ve gone to bookstores and the library to ask if there are books on the analysis of the analysis process but I’ve gained mostly odd reactions from booksellers. Seems like a reasonable question to me but … so I thought I would share some of my own thinking process here – how I brainstorm alone. I’m hoping other people will think about how they think and perhaps share some ideas.
Movement. I’ve found some of my best thoughts come when I’m moving, especially when I’m walking. I’m learning that brain activity and physical movement isn’t surprising – a book I’ve been reading recently is Brain Rules and movement is discussed. I know I’ve long found the drab colored walls of most offices to be exactly uninspiring. When I have a particularly gnarly problem to work through a couple of things happen – here’s one tactic I’ve developed, I have planned walking routes.
Walk. I go for long walks. I’ll take a problem in my head and more or less state the issue to myself. Then I pick a walking route that I know (I have several where I know the length and approximate time so that I can somewhat auto-pilot my walk). I then make timed sessions with myself, challenging myself to think only of X problem until I reach a destination. I can be a pretty disciplined person, sometimes the discipline works for me – it forces me to focus, the physical movement helps and odd solutions and ideas will pop up in my mind. If I find myself without index cards or a notebook in hand to record my ideas, then I use the voice recorder on my blackberry. I’ve been surprised a few times the ideas I’ve recorded – ideas I thought I would not forget, but I’ve been glad I recorded and replayed my notes.
Trying the opposite of movement, sometimes I have planned seated sessions.
Coffee. I mention all the time that I sit in Starbucks and create many test strategies and plans in various Starbuck locations and its true. The combination of coffee and large glass windows seems to work well for me. The only exception is the occasional over-yuppied location or loud days in acoustically poor locations. An alternative is my local library, I have a couple of favorite chairs and locations inside my local library but I’m not allowed to bring coffee in. I’m sure the point isn’t the drink although some recent studies of caffeine and increased brain activity have me justifying caffeine quite nicely. I’ll bet it has more to do with the discipline of the defined think sessions that help me. I think about the advice given to people with sleep issues and how training yourself that when you crawl into bed, you’re there to sleep. I guess Starbucks has become a trained place for me, I walk in and if I’m going to stay, I know I’m there to think through something.
It seems location can matter – sometimes a small influence and sometimes a larger influence but for the price of a coffee and the small effort of moving– it seems worth it. So now what? Once I’m seated or moving what do I do? I’ve been trying to identify specific to be able to articulate this – here’s a few…
Argue. I argue with myself. I’ll work both sides of a conversation in my head. Sometimes I’m thinking about a particular real person or real project obstacle and I’ll argue through an idea or tactical approach I want to use and where I expect opposition. I suppose I am planning my defense of persuasive speech in advance. Other times, the obstacle is less tangible and I just want to get to the root of my idea and by looking for the opposite or an opposition, I can see more specifically and clearly where I’m attached to an idea and why. It’s helpful to understand motives, including my own which I’ve learned are not always obvious.
One important reason arguing alone works for me is that there is no heat or politics. I know when I’m debating with myself that I have no hidden agenda – which isn’t the case when I work with other people where it’s not always obvious what agenda someone may have. With myself, that obstacle is removed and the best of arguing comes to light – for me. The removal of the heated debate may not work for some where rallying for the cause brings out great debates skills but for me, it’s a more gentle probing session and I can see the ideas without the heat of emotions that can sway things.
Prove it. I imagine I have to prove my idea(s) to someone and I step through how I would articulate my idea. I look to see where my reasoning might seem or is thin and I try to find additional reasons.
History. I think through past experiences if I can draw a correlation or comparison and I look for situations that have worked well or situations of past failures to attempt to prevent repeating a pattern. The correlations can be people-related, product-related, time-crunch related. Less than obvious correlations can often be made. For instance how did I deal (well or not) with the last time I was so painfully time-crunched? What worked? And what did I vow to myself not to repeat? And now I go on to ask myself – how? How? What can I actually do about X situation now.
Worst case. I try to envision the worst incident that could take place with a product or a release. This is the complete opposite of thinking about the blubbly happy words used in marketing hype. I try to envision catastrophes and what I realize is that in thinking this way, I’m building a risk analysis in my head. Thinking through the worst of things helps me gain a sense of priority, a sense of where to focus efforts.
Rotate. Another time game I play in my think sessions is limiting time to think about a particular category. The timebox approach has a way of forcing me to think through the most important things first and I personally have to work on letting the smaller things go. It’s a game really, for instance if I pick a time to end working on something, say writing a blog post, will I stop when the time comes and did I do a good enough job in the time I allotted?
An O’Reilly book on web security testing is about to be released. I was a previewer for the book and have been reading chunks of the book these past months. The book is highly readable and packed with ideas. I learned a lot previewing the copy and chatting and emailing with Paco Hope and Ben Walther. See O’Reilly. Cheers to Ben and Paco.
Here’s a look at the table of contents –
Table of Contents
1. Introduction
2. Installing Free Tools
3. Basic Observation
4. Web-Oriented Data Encoding
5. Tampering with Input
6. Automated Bulk Scanning
7. Automating Tasks with cURL
8. Automating Tasks with LibWWWPerl
9. Seeking Design Flaws
10. Attacking AJAX
11. Manipulating Sessions
12. Multifaceted Tests
Here’s where you can find more: http://websecuritytesting.com/
Some of the free tools covered include: WebScarab, cURL, Cygwin and CAL9000. There are several others.
I was pretty tired when I posted last nite and realize I was lean on details. Hopefully this gives a better feel for the book.
I’m at the PNSQC conference in Portland Oregon. Yesterday I taught my SQL for Testers class. Today I’m giving a presentation on building a test strategy and tomorrow I’ll be sharing ideas on storytelling. If you’re at the conference or in the area, email me (karen@karennjohnson.com). And if you’re headed to Powell’s let me know …
The topic of a cascade delete came up on a project the other day and I was surprised to find one of the project members wasn’t familiar with the term or the concept. I thought I would share a short lesson and reflect on testing associated with cascade deletes.
A cascade delete is when an on object is deleted in an application and on the backend in a relational database, the delete cascades through the associated tables in the database removing associated data. (And hopefully both removing the appropriate information and leaving the other data.) The concept isn’t difficult to imagine although there are ripples of what takes place, what can go wrong, what could be tested, and how a failed delete can be found.
Being a fan of examples as a way to define and describe, here’s an example.
Imagine an ecommerce website that sells books – suppose a book is no longer being offered for sale on the site. An administrator of the site might delete the book.
On the delete transaction, the entry for the book would be removed from the book table. Likely the delete has been coded such that when a book is removed from the book table, a trigger deletes the information on the other associated tables (such as a pricing and publisher table). The delete effectively cascades.
If the trigger has not been coded or isn’t working, the result could be orphaned data. Or the delete transaction could fail leaving the book in the database.
I guess this might be categorized as database testing. I suppose. But orphaned data can have an impact on performance when a sufficiently large amount of data has been orphaned and the performance of a site is comprised.
Here’s another way to stumble into a failed cascade delete. Imagine the publisher information associated for the book should have been removed but not all the information for the publisher. After all there could be many other books offered by the publisher. If I look up another book from the publisher and find the publisher information is missing, I might find a page not found or error of another type. A failed cascade delete could be the culprit behind missing data.
Another variation might be the bookseller no longer carrying any books from a particular publisher. Now the delete becomes a delete of a publisher record and all associated books being removed.
I may just be a data geek, but I don’t consider this database testing as much as I consider this as having an awareness of how deletes and for that matter inserts and updates are handled by the application. I like to know how data flows in and through an application.
If I have to wait to back into bugs through the front end of an application then it might take a long while to find them. When data issues from production get reported, a failed cascade delete is a possibility that comes to mind. It might be why a bug isn’t so easily or obviously reproducible – the specific data is always a potential factor in a bug. This is one idea I have in my collection of ideas about how to find and/or reproduce a bug, the ripple effects of a cascade delete.
The trace matrix has always disturbed me. For me, there’s something uncomfortable about taking the complexities of testing and reducing all that intellectual activity, thinking, and effort and squeezing it into a small tidy table that allegedly proves testing is complete that doesn’t feel right.
I’ve tried blogging about the trace matrix before but the topic irritates me to the point where I have I haven’t been able to write on the topic. That’s a pretty strong reaction to a simple “tool” and my skepticism and concern about the use of this tool remains to this day. My intent in writing about the matrix now is to share my opinion, give exposure to a testing topic, and hopefully hear about other people’s experiences and thoughts about the matrix.
If you’re unfamiliar with the trace matrix, let me explain its classic layout and alleged benefit. The trace matrix begins with building a table (often done in Word). There are a few variations on the matrix but often the matrix contains four columns. Number each requirement and place the requirement number in the first column, followed by the requirement itself in the second column, followed by the test case number(s) or reference points in the third column. I’ve also seen the third column used for the test case number and/or test step numbers and the fourth column used for the test case name. There are variations on the theme but this is the basic layout.
The alleged benefit is that by showing a one to one requirement to test case (or multiple test cases), you will be assured that testing is complete and that all requirements have been covered. Wikipedia has an entry on the matrix, referring to it as the traceability matrix which is the name used at times. The layout shown on Wikipedia is not a layout I’ve seen in practice despite having seen many matrices’ since I began working (intermittently) with regulated software in 2001. But layout isn’t a concern of mine; my concern is with the belief that a simple mapping of requirements to test gives assurance of coverage.
When I first began working with regulated software, I was sent to FDA auditor training. I was taught that the trace matrix – along with bug reports, complaints, validation plans and final reports would be the first materials requested by an auditor. Before entering the world of regulated software, I had been in software testing since 1994 but had never heard of the trace matrix. And so, to begin with on the basis that I had worked at numerous software companies and seen many a good product ship and had never heard of the matrix, I approached the matrix with skepticism.
For some years now, I’ve observed the people who embrace and promote the trace matrix seem to be people who don’t actually like testing. I know this is a generalization but trace matrix advocates seem to be people who would like testing better if testing were a containable manageable finite task.
For the record, I wrote one of the trace matrices on a very large project just a couple of years ago. Yep, me – not someone else but me personally. I recognize its requirement in a regulated environment where an auditor will look for the trace matrix as one of the first documents. But I also don’t expect most auditors know much about testing nor have much if any, real life experience with testing. I expect most auditors have been taught the rote mechanics of checking requirements to test cases with a checklist mentality. I feel I can say this because I attended auditor training and I know with what detachment I was taught that a glance at the matrix/table to ensure no “empty boxes” appear was taught as an assurance that testing is complete had me cringing. This type of investigation lacks thinking.
The trace matrix advocates I’ve worked with often seem to be date-focused more than product-quality-focused. When a date is the most important aspect of a project, testing becomes a nuisance. I’ve tried to get inside this type of thinking – the thinking that testing is a nuisance instead of an intellectually fun puzzle to solve – I try to understand the mindset because I think its mind expanding to try to fundamentally understand someone whose train of thought differs so radically from mine own.
When I finally got to a mental state where I thought I could see this different perspective, I could see how testing would be awful. It’s so unpredictable, takes wild turns, refuses to be time boxed, forecasted and is hopelessly never complete. Don’t you love that part of testing? It’s never done. It’s what makes testing a wide awake type of job. There always another permutation, combination, O/S patch – some other bizarre combination of data and functionality that twists creating splendid bugs. It’s what requires testers to think hard, because sometimes testing is about outwitting a system –sure we check core functionality works but good testers, great testers look to find more. And all that type of investigative thinking doesn’t fit neatly in a trace matrix or on a line item on an MS Project Plan.
What I love about testing is what PMs and auditors hate. Now, I’m using big basic words like a child speaks with no shades of grey but that is how it feels and how I see the behaviors play out. PMs often want testing contained to a line item on a MS Project Plan but then nasty bugs show up with no regard to proximity of the ship date. Can’t we find all the bad bugs first? This question has amused me for years.
On the flip side of this internal mental argument about the trace matrix – and I’m about to argue back and forth here – I can see some benefits of the matrix as long as skepticism remains – deep skepticism. It may be convenient to have all the requirements housed together somewhere – whether a requirements tool is used or a well-maintained spreadsheet. It is convenient to have a central repository. Although I have yet to see a product, regulated or not that truly has all the requirements written despite sincere efforts. Perhaps in building a matrix, this forced housekeeping task could highlight an item or two that have been overlooked – either a requirement or a test – perhaps.
But what a long tedious road to hoe in the hopes of finding an oversight, couldn’t a better method be used? I would, if I were an auditor, long to hear or read the internal thought process of what was tested, what was deemed a higher risk and handled accordingly, to hear the thinking behind the trace (or an FMEA or MMEA). I cannot find a single tool that I would want to in place of hearing whether orally or through recorded prose what someone thought, or how a team thought and then executed with that knowledge. I would, of course, want the story and not a table.
I’ll try the other side again; I think some small assurance can be gained in identifying one test was conducted against each requirement. But this side of my internal arguement doesn’t work for me as I find a one to one map is such a thin blanket of comfort that I find no warmth in the tool. And an over-reliance on the matrix as proof positive that everything has been covered is naïve. Perhaps, and I’m trying again to find value where I have previously found little to none, perhaps by seeing a long list of tests to a particular requirement an auditor would surmise the essential risk and priority of a particular requirement and that would lead them to conduct a more thorough review – the matrix would give an auditor a blatant call to attention.
Is that possible call to attention worth the massive time suck? The matrix creates volumes of pure busy work. A shame truly that the time and labor spent building the matrix couldn’t be better directed to more testing. And the larger the application being tested, the more impossible the task becomes. Not to mention the maintenance. And who can then read and digest very large matrices?
Recently, I sat with a new client. The topic of the trace matrix came up and I could feel that same mental and physical reaction of mine to the topic. My face must have given some indication of my opinion (and if you know me personally, you can imagine). The client stopped and asked what I thought about the matrix, I nodded and chose my words. I explained my concerns, ranted a bit about how testing isn’t about a one-to-one map game and gave an example of how the trace could fail. The response? The client agreed it seems he shares my skepticism. I was asked if I could, and I quote “apply critical thinking to test this product.” That I can work with. And at the end of this project, if some mapping is done to fulfill an auditor’s request, so be it. But if the auditor doesn’t know more than to look for a one to one mapping, or believes that a completed matrix shows thoroughness then the auditor isn’t using their own critical thinking.
In recent weeks I’ve had the good fortune to return to do some work on a product I worked on a couple of years ago. I’m working with a product that I have deep history and knowledge of and I’m happy to be working with again.
A tester on the team who’s fairly new to the team needed some help. When I heard what she needed I had to think about it for a bit. This would have been information I would have had to have learned as well. So I did something that I often do and realized it might be worthwhile to blog about and share. I turned to a folder I create for every project and trolled through a stack of my notepad files. My how-to’s files.
When I’m on a project and I learn information that I feel is unique or had been troublesome or obscure in some way, I open up notepad and brain dump. I’ve been doing this for years. I don’t write long flowery sentences. I don’t outline, number or format. I just write. I write what was hard to learn. I write tactical details. I’ve noticed in trolling through some files this week that my files are almost always single topic, often not longer than one page. About the length of a blog post for me. Yep, that’s it. It isn’t high tech; it’s not polished or edited. I just write.
So I was able to share a couple of these how-to files with a tester who was hungry to learn. And as we stepped through the info, I could see the interest in her eyes. You know, you just can’t fake sincere interest. So I stayed with her for a while and we had a learning session. Then I backed off and made sure she was at the keyboard and that she could see where the notes were helpful and encouraged her to update my notes.
Take the notes over I suggested, add to, whatever would help her would mostly certain help the team. I don’t know why these files hadn’t been maintained on the artifacts folder but whatever, that happens sometimes. I added the files now. And now I have the added comfort of knowing someone other than me knows some unexpected tech details of a product.
The session also pointed out to me how much I enjoy teaching. It is a very cool feeling to watch someone’s eyes convey confusion and frustration and to watch that cloud clear away and to be replaced with understanding and know-how.
I was a bit disappointed with myself when she asked a good question and realized that I both knew the answer and could not recall having written a how-to file on the topic pointing out that I miss recording/writing sometimes. I should have written about that. I took mental note that I’d missed writing and committed to writing up the info now and posting it to the team.
So the blunt WIFFM. Find a way to write and share. One place I worked, I used to add files off the intranet. On this project it’s a simple shared drive. Documentation doesn’t have to be a painful overhead. It can be short. It needs to be accessible. And the best time to write it, I’ve come to learn is right when I’m done learning a bit.
I haven’t tested a cookie in a long time, so long I realized I didn’t know where cookies are stored on Vista. Finding cookies on Vista is more of a headache than I can recall in any other Windows versions. There are two directories:
C:\Users\[username]\AppData\Roaming\Microsoft\Windows\Cookies\
C:\Users\[username]\AppData\Roaming\Microsoft\Windows\Cookies\Low\
Even though I had the path, I still couldn’t find the cookies. I finally learned what I needed to do was to remove a protection setting. (Even though I’m an administrator on the Vista laptop I was using and I have the annoying user control protection turned off, I still had to turn this file protection setting off as well.) I found two pieces of information especially helpful:
1. This post on Vista Annoyances: http://www.annoyances.org/exec/forum/winvista/1164397964
2. Learning that in Windows Explorer (on Vista), if you press F10 an old familiar toolbar appears that includes the much-needed Tools menu. Open up Windows Explorer, press F10 and you can see what I’m referring to.
I also need to make changes to my hosts file on Vista and had some trouble finding the file. I found the info I needed here: http://geekswithblogs.net/thibbard/archive/2006/12/13/ChangingYourHostsFileInVista.aspx