Interface. Test the ways we interact with it.
Does the app work well on an array of devices such as phones and tablets?
Does the app work well when used with “one hand?”
Does the app work well when used with “one eye?”
Is the mobile version consist or complementary to the desktop version?
Is the app consistent with UI current constructs in the marketplace?
Does the mobile version address what makes sense when a user is on the go?

On another note, I’ve long thought that “E” for error testing should be added to the SFDPOT mnemonic. Here’s how I would address error testing.

Errors. Test the coded-for known error states and around those error conditions.
Does the app continue gracefully after an error?
Is the message helpful, instructive?
If a user provokes multiple errors in succession, does the app continue?
Can I get to an error state that forces the app to shut down?
What happens when I restart the crashed app – is previous data or state saved?
Is there a method to send an error report?

I think this question about adding “I” was a good question because it raises the awareness of not testing just to the suggestions of a heuristics but continuing on, adding our own thoughts, considering the context and while using a heuristics – thinking beyond a heuristic as needed.

I created tweets for my presentation in advance (and had those tweets posting while I was presenting) so I included those tweets as part of my slides. Discipline & Software Testing slide deck

My presentation kicked off with a video showing the marshmallow test, a well-known test – the video speaks for itself. You can find the video on YouTube here.

Enjoy~

I construct mental images in my mind about the process the data flows through. I consider the originating sources of data and how that data has to traverse to get to the warehouse. On one BI project I worked on, I summarized a chunk of the testing efforts as “the story of a cube” and relayed the process of how the warehouse was built and tested to the business owners by way of the story. My storytelling in this case worked fairly well. I used a bit of humor, I drew some visuals and talked through the process. But that was then and this is now …

A client recently asked me about testing on a BI project where they are using Agile. They are also intrigued about the prospect of using Exploratory Testing but are unsure what Charters could be. So in the background of juggling several tasks and conference work, I’ve been puzzling over this question.

I’ve been building out a mind map to address this question and to help me think it through. Not long ago I referenced I was having a solo brainstorming session on Twitter about this topic. Minus one branch of the map that is client-sensitive, here’s my BI + Agile + ET Mind Map.

In a presentation that I’ve given a few times this past year, I’ve identified ways in which data changes or can be changed through “silent” non-visual processes, the point of the list is to remind myself and other testers of ways things/data can changes that we might not see but still need to consider, detect and investigate. Here’s the list:

1. stored procedures
2. triggers
3. views (in SQL in the sense that a user might not be able to see, query or access data and this “missing” data can “seem” like a defect)
4. indexes
5. caching
(again these last two, items 4 and 5 can make an impact on what a user can access making data seem to be missing, or not available)
6. batch runs (say EOM end of month, EOY end of year processes) that can disrupt the flow of data into a warehouse
7. ETLs (but you expected this one)

I’d be happy to grow this list or mind map. I post items like this to help the testing community – so if you have an ideas to expand this … please share.

An article referencing this mnemonic was published by Better Software in an article called “X Marks the Test Case: Using Mind Maps for Software Design” back in 2006.

Here’s the update card deck .

• Hardware Specific: Try another device, try a device that is close in “class”
• Note the OS, manufacturer, version of device, and the version of software
• Browser
• Device orientation
• Battery status
• Data
• Applications running in background
• Connectivity. Network – wifi – if there was any carrier switching taking place
• Location
• Device Settings
• Drivers, LED notifications, Bluetooth
• SIM Cards
• SD Cards
• Integration with other software
• Language Settings
• Varying Bandwidth
• Input speed and mechanism
• Navigation Path

Thanks to attendees: Kelli Cruz, Nicole Thompson, Cathy Clary, Cherri Shang, Rajiv Bhati, Saraswathy Ramadas, Chris Anders, Sakina Crocker, Jon Hagar, Gary Norris, Anand Ramdeo, and Jean Ann Harrison. And to Eric Proegler who facilitated.

One of the exercises in the class is to give each group a deck of index cards, the cards have about 20 testing mnemonics listed – each card has a mnemonic on the front and back of the card. The mnemonic is meant to inspire testing ideas.

The basic premise is for a tester to flip through the card deck until a testing idea comes to mind. A tester may find one mnemonic in particular helps generate ideas or they may find ideas come from one word of multiple mnemonics. There is no right or wrong way to use the deck. The deck is meant as an idea generator.

For each mnenomic I’ve listed the author as well as a URL to find more information about the mnemonic. If you’re an experienced tester, chances are a single testing-related word generates ideas and if you’ve been around the testing community awhile, you’re probably already familiar with the mnemonics and seeing the mnemonic will be enough of a jumpstart.

As a practical matter, I’ve formatted the mnemonics to an Avery template so that printing card deck is fairly easy to do. Here’s the deck.

I thought I’d take the well-known testing heuristic from James Bach known as SFDPOT (sometimes referred to as San Francisco Depot) and apply that heuristic to mobile testing. If you’re not familiar with this heuristic see:
James Bach: http://www.satisfice.com/articles/sfdpo.shtml

Let’s see how it holds up:

Structure. Test what it’s made of.
Can I download the app?
Can I download an update?
Can I update the app when I have multiple apps to update?
What happens when the operating system of the device is updated?
Does the app write to an SD card? What if the card is filled, removed or changed?

Function. Test what it does.
Does the app or site perform the tasks or features it was designed to?
Does the app or site prevent, block or not include features not intended on a mobile device?
Does the app or mobile website ever direct me to the “full site or desktop version”?
Does the app direct me to turn on necessary related services such as location services if those services are shut off?

Data. Test what it does it to.
Does the app find time related data (if relevant, such as movie show times) based on the device time?
Does the app find locations based on my location (such as movie theaters or hotels)?
Does the app integrate with any hardware such as bar code scanners for payment or other purposes?
If the app tracks rewards or points for frequent purchases, does this data get updated?
If the app integrates with a desktop profile or account information (such as My Profile or My Account), does the app and the desktop version keep information in sync with each other?

Platform. Test what it depends upon.
Does the app use location services?
Does the app depend on any device settings?
Does the app detect the platform it is running on such as a phone or tablet?
Can I find the app in the app store?

Operations. Test how it’s used.
Does the app function when I am moving and traveling around?
What happens when I switch to wifi vs 3G ?
What happens if my wifi connection is down or intermittent?
What happens when a mobile interruption occurs such as an incoming call or text message?
What happens when the device is set to airplane mode?

Time. Test how it’s affected by time.
What happens if the time zone is switched?
What happens when my location is switched?
Does the device auto detect a change in time zones and reset the time (and possibly) the day and date?
Does the app find time-related data based on my device time (such as reservation times available or movies show times)?

Can you think of more specific testing ideas based on your mobile app or mobile website – good.

Can you think of more ideas “generically” ? Good, please email or add a comment to this blog. I’d be happy to build on this for all of us in the software testing community.

Comments are welcome.

Hello -

I’ve shared my thoughts on each of your questions. I have to say – this is all based on my experiences and not necessarily reflective of the broader and global mobile market. Testing like so much else about mobile, will continue to change significantly in the next 1 and 2 years time.

1. What is mix of simulator based testing vs actual device based testing ? What factors decide/influence this mix?

In general, emulators have not been used with clients I have worked with. Occasionally a developer will use an emulator during unit testing but otherwise, emulators do not seem to be used or trusted.

The factor used seems to be a general distrust that an emulator will be accurate and this is based on opinions vs experiences encountered. It is also well known that through Device Anywhere and Perfecto Mobile, you can get your hands on nearly any device and so why tolerate an emulator?

Unreliable to depend upon but a typical experience I’m seeing is that within most teams, there is a natural mix of devices and people generally use their own devices at least part of the time. Bigger companies have funding and I know of a couple of companies that are building out mobile test labs – accumulating what they can.

There are a few “quick and free tools” on the market to execute mobile readiness tests. See the W3C, http://validator.w3.org/mobile/ and Gomez, http://www.gomez.com/mobile-readiness-test/ and dotMobil, http://mtld.mobi/emulator.php See a comment on the next question for my “reaction” to these tools.

2. What are the class of defects or problems simulator based testing cannot discover?

I cannot fairly comment on this as I have more experience with devices than emulators.

I have seen issues reported from the quick tools that I mentioned above but found in most cases, the issues detected were either obvious or more obscure GUI-issues that we (the team) were not likely to address – the net result, not so helpful.

3. With so many tools claiming to do automation for mobile apps – how does one pick? In what ways automating (driving GUI from a tool) mobile app is diff from other software apps/web apps?

HP seems to be pushing their QTP app to handle mobile automation. I do not know anyone using it. Perhaps it works? Also a shocking but true fact – at this point in time, which I do expect will change – I do not know of any company that is successfully using automation in mobile. I know a couple of companies trying different tools, wanting to find a tool but at this time – and again, I think this will change before 1 or 2 years time – most companies are trying to figure out what their app and/or mobile site is going to do, what they need to test, what devices they need to cover, how to gather decent statistics and then will move onto mobile automation. The old expression – you have to walk before you can run seems to be the situation.

And although you didn’t directly ask me, I know that you are keen to continue to learn, as am I – so what do I read and where do I continue to learn?

I learn by other people sharing with me (why I feel it is important to get back to you and share as well.) I look to my colleagues Julian Harty (UK) and Jonathan Kohl (CA) and share with them as all of our schedules permit. Google their names, track down their work and articles.

I learn by experience and am currently (which has been true for the past couple of years) most eager for any mobile work I can get my hands on. When I do not have client work that gets me that experience, I continue with experimenting with a variety of apps and my own devices.

I follow Twitter closely because mobile is so current nearly anything in print is already dated. (You might look to see who/companies I follow as much of my Twitter following is mobile-centric.) That said, I think I own at least the “core” books on mobile offered from Amazon and O’Reilly as of today. I scan every testing newsletter for articles and attend webinars as feasible due to my sometimes crazy schedule. In this last 24 hours I devoured one book from Amazon in e-format only as the book is only days old, titled: Mobile Analytics by Jesus Mena. Straightforward title and rock-solid information.

Like most things I suppose, we find what we are truly interested in – no matter the day of week or time of day. At this time, mobile is a top interest of mine.

Now on an unexpected final note, given that I have taken the time to reply to you – I realize that your questions are not so very different than questions other people have asked me – I have decided to take my replies and post them into my blog. I won’t identify your name as it would be inappropriate of me to do that – but the replies I have shared with you are not different than I would share in my blog – and perhaps this exchange will help another tester? And so the circle of sharing continues.

Happy Testing~

I’m detailing my personal account of a Sony S tablet versus an iPad in hopes it will help someone else who might be on the edge of trying to decide what tablet to go with.

I was curious how the unboxing experience would be because even the initial opening of a device seems to set the tone. When I opened the device and powered up, the opening screen of the Sony S tablet was a system crash message. A restart and the device turned on fine; I completed setup with no further issue but geez that first bad experience left me with doubts.

Like any new device – beyond the setup, I seem to have the same reaction – ok, cool now what? It half makes me smile because I think the issue is data – devices on their own don’t do much – instead it is the applications combined with either our own data or the data we look up that makes a device so interesting. I have no doubt that the commercial market and users whose occupations are outside of the tech arena don’t see it that way – but think about even the iPad ads – it’s the photos, the videos, the books, the contacts, and other bits of data that make a device interesting. It is not the raw merits of the technology as much as the sweet combination of technology and data that make technology compelling.

Ok back to the Sony S tablet experience – the screen was dark, I adjusted the device to be as bright as I could but it just didn’t matter. The Sony S tablet is dark. Also in almost all cases – from using an app to watching video – the entire viewing screen of the device is not put to use – which adds to a disappointing visual experience. Each time I used the tablet, I had this inclination (if not need) to put my reading glasses on because I just couldn’t interact with the device readily and easily visually. I was surprised, I mean this is Sony – shouldn’t the visual experience be great?

After mucking around with the device for two days and having about the same reactions of “not being wowed” I decided I needed to do something. I needed to take my Sony S tablet and go to the Apple store and do my own side by side comparison, app by app if it took that – to compare the Sony tablet versus the iPad – which of course is not just a device comparison but a fundamental OS comparison. It is after all, almost a religious difference to be on a Droid (Google) vs. iPad (Apple) device. It might seem more logical that I would go to Best Buy or somewhere else and compare the Sony S tablet to say the Samsung or Motorola tablets but it wasn’t the comparison I wanted. And as much as I love my Droid phone, I was willing to give up a Droid tablet and move to Apple if I was happier with the device.
Meanwhile I have several issues with Apple’s interfaces and there are some aspects of working with either an iPod or an iMac that I find flawed and not as intuitive as Apple touts. But I do own and use a mix of PC and Mac, Droid and Apple (even before this tablet experiment) – so I’m religiously a bit undefined. In fact, I believe in building and maintaining some knowledge in both technology camps because I test in the mobile space and to completely disavow either part of the market would render me too much of a zealot which I think is inappropriate in my role as an independent consultant and as a software tester in general.

I was a bit concerned how I might look in the Apple store smuggling in an Android tablet but figured I wasn’t stealing anything and besides if a company is so confident of their own product, wouldn’t they welcome the challenge?

Somehow I found a corner of the Apple store and was able to step through what I wanted to in my own side-by-side comparison. Since it was Christmas Eve and the store was busy with last minute shoppers, not too many people cared what I was doing which was perfect for me. In fact the best holiday gift I gave myself this season was the time to run my own comparison tests – this was important to me. If you haven’t noticed by now, I like my gear.

Here goes, first stop: You Tube. The experience of comparing the iPad to the Sony S was almost a closed case by comparing video display and handling on You Tube alone, the iPad has it hands down. The Sony S has a yellow hue to all video, graphics and display – it’s not just the overall darkness but an additional coloring that renders sepia-like-experience. I found myself wanting true coloring. I queued up several videos in a side by side checking visuals as well as sound.

Second stop. Photos, same disappointing graphical experience. What’s with the yellow hue Sony?

At this point, I knew everything else would pale. I figured the data – mail, contacts and calendar would be easy with the Sony S and it was – as long as you have a Google account and use each of those (email, contacts and calendar) then loading up data on the Sony S was easy. But that easy data loading is not compelling enough because Apple handles that too – although Apple’s limited handling of Google calendar is annoying.

Next: device handling and covers. This seems like a minor point but I know that a large amount of ongoing ownership time gets spent in pulling out a device, turning it on/off and that the device cover or protection can influence and impact the overall device experience. So much so I don’t know why most smartphones don’t have a more rugged shell to begin with so we can stop buying cheap plastic cases for devices worth so much more.

After having one phone case that left me disliking a phone, I know the wrong device accessory/case can influence how I interact with the device itself. The Sony S tablet case is one of the worst cases I have experienced. There’s a large Velcro strip loud enough to hear across a room – that’s a deal breaker itself because at this point there are no other cases designed for that particular tablet and the Sony case is awful. One of the most important aspects of owning a tablet is to be able to access and use it quickly – so if it’s a hassle to pull out the tablet and embarrassingly loud – then the case is a detriment to the device.

Meanwhile in addition to being loud, the case itself is not snug enough to protect the device. And finally, the Sony case does nothing to help if the device is left on, the case will store the device and happily drain the battery if you forget to shut the device off before stowing it away. The iPad’s smart cover is cool, works well and it is that simple.

At this point, an Apple staff person came over to help me. Her eyes widened when she realized what I was doing – comparing devices right in the store. Then she asked if she could join me, seems she’d never held a Sony S tablet and she was curious too. My daughter arrived in the store at the same time and I asked both of them to help me with the comparison. It was an interesting hour and at the end, I bought an iPad2. Wow, I’m surprised, still surprised because my Droid phone has been great.

The last step was to remove all my data from the Sony S and return the Sony tablet. I was able to uninstall apps and handle some cleanup easily but not my Gmail. Looking up what other people have written I discovered a horrible “feature” of Gmail – and that is there is no way to sign out or remove your account from a tablet. Search for what other users have discovered and you’ll probably find what I did – people furious because they can’t share their Droid tablet with anyone else because there is no way to switch what Gmail account is logged in. And there is no way to remove email, contacts or calendar data without a factory reset. Not cool, not acceptable.

I have an iPod Touch and iMac so the Apple is familiar but I have been “adjusting” to having an iPad, and have to say, I’m becoming pretty attached to my iPad and have to admit, it is simply amazing.

At the Belgium Testing Days conference, I’ll be giving a keynote called: “Why it matters what I’m called: Quality Analyst or Software Tester.”

I’m conducting a survey on job titles in our field. It would be great if you could help me. Here’s what I need: your name, country, job title and if you’d like to add your company name and any comments – comments relating to how you feel about your job title, that would be great.

I’ll probably out the data together in Excel so if you could send this information like this:

Karen Johnson, USA, Software Test Consultant

And comments like this would be helpful:

I care deeply about my title. I feel my title sends a message to my team mates about what I do and what my role is.

Or –

I don’t care about my title. I just want meaningful work. My team mates know what I bring to the group and the company.

Leave a reply. Email me (if you don’t want to share this information publicly). Or reply in Twitter @karennjohnson

-Thanks Karen