Recently a colleague within the testing community emailed me three questions re: mobile testing. After answering the email, I realized that our exchange of questions and answers might be helpful to share – so following is the Q &A from our exchange (minus personal identification).
Hello –
I’ve shared my thoughts on each of your questions. I have to say – this is all based on my experiences and not necessarily reflective of the broader and global mobile market. Testing like so much else about mobile, will continue to change significantly in the next 1 and 2 years time.
1. What is mix of simulator based testing vs actual device based testing ? What factors decide/influence this mix?
In general, emulators have not been used with clients I have worked with. Occasionally a developer will use an emulator during unit testing but otherwise, emulators do not seem to be used or trusted.
The factor used seems to be a general distrust that an emulator will be accurate and this is based on opinions vs experiences encountered. It is also well known that through Device Anywhere and Perfecto Mobile, you can get your hands on nearly any device and so why tolerate an emulator?
Unreliable to depend upon but a typical experience I’m seeing is that within most teams, there is a natural mix of devices and people generally use their own devices at least part of the time. Bigger companies have funding and I know of a couple of companies that are building out mobile test labs – accumulating what they can.
There are a few “quick and free tools” on the market to execute mobile readiness tests. See the W3C, http://validator.w3.org/mobile/ and Gomez, http://www.gomez.com/mobile-readiness-test/ and dotMobil, http://mtld.mobi/emulator.php See a comment on the next question for my “reaction” to these tools.
2. What are the class of defects or problems simulator based testing cannot discover?
I cannot fairly comment on this as I have more experience with devices than emulators.
I have seen issues reported from the quick tools that I mentioned above but found in most cases, the issues detected were either obvious or more obscure GUI-issues that we (the team) were not likely to address – the net result, not so helpful.
3. With so many tools claiming to do automation for mobile apps – how does one pick? In what ways automating (driving GUI from a tool) mobile app is diff from other software apps/web apps?
HP seems to be pushing their QTP app to handle mobile automation. I do not know anyone using it. Perhaps it works? Also a shocking but true fact – at this point in time, which I do expect will change – I do not know of any company that is successfully using automation in mobile. I know a couple of companies trying different tools, wanting to find a tool but at this time – and again, I think this will change before 1 or 2 years time – most companies are trying to figure out what their app and/or mobile site is going to do, what they need to test, what devices they need to cover, how to gather decent statistics and then will move onto mobile automation. The old expression – you have to walk before you can run seems to be the situation.
And although you didn’t directly ask me, I know that you are keen to continue to learn, as am I – so what do I read and where do I continue to learn?
I learn by other people sharing with me (why I feel it is important to get back to you and share as well.) I look to my colleagues Julian Harty (UK) and Jonathan Kohl (CA) and share with them as all of our schedules permit. Google their names, track down their work and articles.
I learn by experience and am currently (which has been true for the past couple of years) most eager for any mobile work I can get my hands on. When I do not have client work that gets me that experience, I continue with experimenting with a variety of apps and my own devices.
I follow Twitter closely because mobile is so current nearly anything in print is already dated. (You might look to see who/companies I follow as much of my Twitter following is mobile-centric.) That said, I think I own at least the “core” books on mobile offered from Amazon and O’Reilly as of today. I scan every testing newsletter for articles and attend webinars as feasible due to my sometimes crazy schedule. In this last 24 hours I devoured one book from Amazon in e-format only as the book is only days old, titled: Mobile Analytics by Jesus Mena. Straightforward title and rock-solid information.
Like most things I suppose, we find what we are truly interested in – no matter the day of week or time of day. At this time, mobile is a top interest of mine.
Now on an unexpected final note, given that I have taken the time to reply to you – I realize that your questions are not so very different than questions other people have asked me – I have decided to take my replies and post them into my blog. I won’t identify your name as it would be inappropriate of me to do that – but the replies I have shared with you are not different than I would share in my blog – and perhaps this exchange will help another tester? And so the circle of sharing continues.
Happy Testing~