Testing Android apps

I've been developing an Android app as a spare-time project for 6 months now. I have access to 3 Android devices:

• Samsung Galaxy SII running ICS (my personal mobile)
• Samsung Galaxy Mini running Gingerbread (bought as a temporary dev device when my old HTC Desire gave up the ghost)
• Motorola Xoom II Media Edition (tablet used for day job)

My app runs beautifully on all these devices. I cannot make it crash. It really frustrates me that I get crash reports from other devices and cannot replicate them. For a spare-time project I can't afford to buy lots of devices to test with. Enter AppThwack.

AppThwack

I first heard about AppThwack after I tweeted about one of its competitors - testdroid - trying to find some testimonials from other users. I didn't find any, but I did get a reply from @tdpeterson inviting me to AppThwack's beta.

Some weeks later I finally found a spare few minutes to take a look, and I'm so glad I did!

As they are currently in beta, you need a code to be able to sign up. Mine arrived a few hours after requesting it, and I immediately signed up to give it a try. The service makes it very easy to get started, which is clever, because once you're in you really are hooked - its just too good.

Chaos Monkeys

I uploaded the latest release apk I had to hand and launched the UI Exerciser Monkey tests (these are automatic and randomised tests that poke all the buttons and what-not). From sign-up to running tests took less than 2 minutes.

The monkeys found at least two bugs that I was not aware of on that first run. I am torn between horror and delight. The horror and delight turn to embarassment when I realise that I have uploaded an obfuscated apk, and the stack-traces are in gibberese. Lesson#1 right there.

A test run takes a while to complete, but you can watch the results coming in in real time. When you realise what is going on behind the scenes its really impressive: your app is installed, launched, monkeyed, and then uninstalled simultaneously across a whole farm of different devices running different versions of Android. I'd love to see what that looks like :)

You can view the results of your test runs in a number of different ways. I like the "issues by device" mode, which lists all the devices that experienced problems or warnings, and allows you to drill down to see what happened in the monkey log - with a full stack trace of the crash - and then down again into log-cat.

Pause a second and let that sink in. I just ran my app on 43 different devices and Android versions in under 5 mins and now I have instant access to the stack traces from the crashing devices and the full log-cat output. I even get screen-shots from some of the devices - above is from an HTC Evo 4G running 2.3.3. Holy shitcakes!

Oh, and you can filter the log-cat output directly in the web-app. Nice.

Programmed Tests

OK, now I have some crashes that I want to try to replicate which are going to involve some complex interactions. Its going to take a lot of chaos monkeys a long time to replicate those tests, but the AppThwack guys have thought of that and integrate Robotium testing too.

Robotium was also new to me, but again its a great tool and a doddle to get started:

1. download the robotium jar from here
2. create an Android test project, set up to test "this" project (not the project you actually want to test)
3. set the manifest <instrumentation> tag's targetPackage attribute to be the package of the app you want to test
4. copy the basic test code from this tutorial (I confess I find it weird that the tutorial is a pdf, not a web page!)
5. tweak the test code to test your app - its a very straight-forward API
6. Run the app you want to test in an emulator
7. Run your test and watch as it drives your app in the emulator - great fun

Once you have a working test that you want to run over at AppThwack's device farm its really easy to get that going too:

1. export a signed apk of the app you want to test (app-under-test)
2. export a signed apk of the test project (sign with the same key!)
3. upload app-under-test apk
4. click the configure icon
5. click "add test apk", and upload the test project apk
6. click go
7. marvel as the results start rolling in

Early days

I haven't really tried to fix my crashing bugs yet, but at least there is light at the end of the tunnel now, thanks to AppThwack and Robotium.

I noticed a few funnies along the way, which I plan to mention to the AppThwack guys when I've had a bit more experience:

• Many devices crashed at the same point, but there's no way to collapse common issues together - I found myself opening the log for each device to check the stack-traces to find out the total number of different problems. update: Pawel from AppThwack tells me a new view mode "By Failure" is in the works and will be landing in the beta very soon!)
• Some devices timed out while installing. I'm not sure if this is a problem with my app (seems unlikely?) or some issue with the devices themselves? Its not a catastrophe if its a device problem, but then probably they should be marked as warnings rather than errors. update: Pawel confirmed there were timeouts within the device-farm which I'm sure they'll resolve soon
• I noticed lots of these in the monkey logs: java.io.FileNotFoundException: /mnt/sdcard/scriptlog.txt (Permission denied) which seem to be just noise (at least as far as I am concerned)

So far so wonderful as far as I'm concerned: my thanks to Trellis Automation for AppThwack, and I really hope the beta goes well and the business is successful!