There's no substitute for product testing by real people 'in the wild' says Matt Johnston at uTest
The fact that Apple is now supporting 300,000 apps and Android is comfortably past 100,000 is a source of great pride for the industry.
So much creativity. So much commerce.
But this vast breadth of product also multiplies the potential for failed downloads and buggy user experience.
As a result, the demand for mobile product testing has never been more buoyant. Some content providers use their own teams, others use emulators. But according to Matt Johnston at uTest, the best results are achieved with testers using 'real' handsets in 'real' conditions.
He must be on to something; uTest's mobile business is growing 30x a year. We asked him to elaborate:
What are the advantages of uTest's approach to testing over the alternatives?
We believe that using a lab-based solution like and emulator or a simulator is not real testing at all. There's no guarantee of quality because the conditions are artificial. Labs offer a sterile environment, and they're usually in offices in metropolitan area where coverage is good. Real testing needs to be done 'in the wild' by real people with real handsets that have low battery life and different versions of the same OS. In-house teams can obviously do good work, but the handsets are proliferating so fast, and it's hard to keep up with this using internal resources.
How do you handle this fragmentation?
Well, it's a huge point for us, but then it's also the reason we're in business. We can't claim to test against every single permutation of handsets, OSs, languages, operators and so on – which would run into tens of thousands – but having thousands of testers means we are are using 'crowd sourcing' to reflect the most popular variations at any one time.
How many testers do you have?
There were 10,000 in 2009, 30,000 in 2010 and we're on target for 50,000 in 2011. 60 per cent of them are already full time testers who work for us in their spare time, and around 30 per cent are freelancers. They're in 168 countries, which is a pretty good spread considering the UN recognises 194 countries. We've even got one in the Vatican.
What sort of testing do they carry out?
On mobile it's mostly around loading and the UI than the kind of functional testing we do on the web and desktop. Basically, the problems are less to do with the original coding than the way that products are affected by the fragmentation we've been discussing. To give one example, we had a BlackBerry developer that was adamant it didn't need our help. But then RIM brought out a touchscreen and all hell broke loose. Some of their stuff didn't even load.
How important is mobile to uTest?
I'd like to say we saw the growth coming, but we really didn't. I suppose mobile got going in early 2009, and it's now growing at 30x a year. At this rate, mobile will overtake our web and desktop business in 12 to 18 months.
* Matt Johnston is he VP of marketing and community at uTest
Advertisement
