Testing Android apps
I've been developing an Android app as a spare-time project for 6 months now. I have access to 3 Android devices:
- Samsung Galaxy SII running ICS (my personal mobile)
- Samsung Galaxy Mini running Gingerbread (bought as a temporary dev device when my old HTC Desire gave up the ghost)
- Motorola Xoom II Media Edition (tablet used for day job)
My app runs beautifully on all these devices. I cannot make it crash. It really frustrates me that I get crash reports from other devices and cannot replicate them. For a spare-time project I can't afford to buy lots of devices to test with. Enter AppThwack.
I first heard about AppThwack after I tweeted about one of its competitors - testdroid - trying to find some testimonials from other users. I didn't find any, but I did get a reply from @tdpeterson inviting me to AppThwack's beta.
Some weeks later I finally found a spare few minutes to take a look, and I'm so glad I did!
As they are currently in beta, you need a code to be able to sign up. Mine arrived a few hours after requesting it, and I immediately signed up to give it a try. The service makes it very easy to get started, which is clever, because once you're in you really are hooked - its just too good.
I uploaded the latest release apk I had to hand and launched the UI Exerciser Monkey tests (these are automatic and randomised tests that poke all the buttons and what-not). From sign-up to running tests took less than 2 minutes.
The monkeys found at least two bugs that I was not aware of on that first run. I am torn between horror and delight. The horror and delight turn to embarassment when I realise that I have uploaded an obfuscated apk, and the stack-traces are in gibberese. Lesson#1 right there.
A test run takes a while to complete, but you can watch the results coming in in real time. When you realise what is going on behind the scenes its really impressive: your app is installed, launched, monkeyed, and then uninstalled simultaneously across a whole farm of different devices running different versions of Android. I'd love to see what that looks like :)
You can view the results of your test runs in a number of different ways. I like the "issues by device" mode, which lists all the devices that experienced problems or warnings, and allows you to drill down to see what happened in the monkey log - with a full stack trace of the crash - and then down again into log-cat.
Pause a second and let that sink in. I just ran my app on 43 different devices and Android versions in under 5 mins and now I have instant access to the stack traces from the crashing devices and the full log-cat output. I even get screen-shots from some of the devices - above is from an HTC Evo 4G running 2.3.3. Holy shitcakes!
Oh, and you can filter the log-cat output directly in the web-app. Nice.
OK, now I have some crashes that I want to try to replicate which are going to involve some complex interactions. Its going to take a lot of chaos monkeys a long time to replicate those tests, but the AppThwack guys have thought of that and integrate Robotium testing too.
Robotium was also new to me, but again its a great tool and a doddle to get started:
- download the robotium jar from here
- create an Android test project, set up to test "this" project (not the project you actually want to test)
- set the manifest <instrumentation> tag's targetPackage attribute to be the package of the app you want to test
- copy the basic test code from this tutorial (I confess I find it weird that the tutorial is a pdf, not a web page!)
- tweak the test code to test your app - its a very straight-forward API
- Run the app you want to test in an emulator
- Run your test and watch as it drives your app in the emulator - great fun
Once you have a working test that you want to run over at AppThwack's device farm its really easy to get that going too:
- export a signed apk of the app you want to test (app-under-test)
- export a signed apk of the test project (sign with the same key!)
- upload app-under-test apk
- click the configure icon
- click "add test apk", and upload the test project apk
- click go
- marvel as the results start rolling in
I haven't really tried to fix my crashing bugs yet, but at least there is light at the end of the tunnel now, thanks to AppThwack and Robotium.
I noticed a few funnies along the way, which I plan to mention to the AppThwack guys when I've had a bit more experience:
- Many devices crashed at the same point, but there's no way to collapse common issues together - I found myself opening the log for each device to check the stack-traces to find out the total number of different problems. update: Pawel from AppThwack tells me a new view mode "By Failure" is in the works and will be landing in the beta very soon!)
- Some devices timed out while installing. I'm not sure if this is a problem with my app (seems unlikely?) or some issue with the devices themselves? Its not a catastrophe if its a device problem, but then probably they should be marked as warnings rather than errors. update: Pawel confirmed there were timeouts within the device-farm which I'm sure they'll resolve soon
- I noticed lots of these in the monkey logs:
java.io.FileNotFoundException: /mnt/sdcard/scriptlog.txt (Permission denied)which seem to be just noise (at least as far as I am concerned)
So far so wonderful as far as I'm concerned: my thanks to Trellis Automation for AppThwack, and I really hope the beta goes well and the business is successful!blog comments powered by Disqus