Because I'll forget it if I don't write it down...
Follow steveliles on Twitter View my LinkedIn profile
Subscribe to RSS feed

I just checked in a SAX parser based version of dsl4xml to github, and finally got a chance to run the perf tests on an Android device. This is how it looks:

Notice that while it loses about 15% on raw SAX parsing, it still provides approximately an order of magnitude greater throughpout than the next best (raw pull parsing). And of course, its damn easy to write readable unmarshalling code with :)

I also added SimpleXML parsing to my performance tests - it ties for last place (performance-wise) with W3C DOM parsing. Arguably it is more readable and requires less code than the others, though personally I'm not a huge fan.
Comment on this post

I'm so fed up with the Google Play comments and rating system. How can they have got it so wrong? Comments and user-ratings are a bad system to start with, but Google's system is broken beyond belief.

Comment/Rating is a broken idea

Commenting and rating systems are a bad idea to begin with. They pander to the extremes no matter what the subject under comment.

Check any app - any app - on the market: I defy you to find one that is not overwhelmingly rated 1 or 5 stars. Why? Because most people between the two extremes don't care enough or can't be bothered to rate an app.

That leaves the big fans who love the apps unquestionably but, bless 'em, unhelpfully ("★★★★★ Love this app"), and the haters ("★☆☆☆☆ Gross! sucks! don't download!").

The big fans are wonderful, but not helpful, but neither are they being destructive. The haters though, they really get to me. I want my apps to be good. I want them to work for everyone. I want them to be liked. I feel it really personally when my app gets a bad rating. I shouldn't, everyone tells me, but I can't help it - I'm invested in my work.

I've put a lot of time and effort into this thing - poured myself into it. A lot of that time I've given away free. Gratis. No charge. The "pro" version is the price of a newspaper that you'd read once and throw away.

If you download, don't like, ★☆☆☆☆ and walk away, I'm stuck with a bad rating that I can do nothing about, and its cost you nothing (Google Play allows a refund within a couple of hours of the download).

Please don't misunderstand me - I'm not railing against the people who comment and rate negatively - everyone is entitled to an opinion. The problem is the system which Google have created seems designed to make life difficult for app developers, and frustrating for users.

Negative commenters seem to fall into a few categories:

The Unwittingly Unhelpful

★☆☆☆☆ N. O'Tunreasonable on April 20th, 2012 (Motorola shit-hot-superphone-II with version 2.0.1)

"Doesn't work on Motorola shit-hot-superphone-II. FIX IT OR REFUND ME!".

Dude, I'd love to fix it. No really, I would! Ask any one of the dozen or so people who've emailed or tweeted me about a problem and who got a response within hours and a fix within two days at most (best I can do on a personal project - I have a day job!).

Unfortunately I don't have a Motorola shit-hot-superphone-II (its always Motorola, why is that? Oh, occasionally its an HTC, but nearly always Motorola. Curse them).

It works great on my Samsung's, including the Galaxy mini that cost £50 in Tesco. No new crash reports or freezes in my developer console. How in the living hells do you expect me to FIX IT or, for that matter, to refund you? I do not know who you are because Google anonymise you!

I don't blame these guys actually - they rightly expect the app to work, and equally they expect a commenting system to allow some kind of conversation. Unfortunately, the Android eco-system is fragmented to hell and back, bugs are a universal truth of software, and Play's commenting system does not allow threaded responses and does not give the developer access to the commenters identity.

The first two I can handle - fragmentation requires more work, and bugs can be fixed, but I need to be able to communicate with commenters or I can't help them. I'm looking at you Google. With beetled brows.

The Blackmailer

Uses the power of a bad rating to demand whatever features he feels the app should have before benificently conferring his generous 5 stars.

★☆☆☆☆ Dick Dastardly on April 16th, 2012 (HTC Wonderful with version 2.0.1)

"Great app, will rate 5* when you add XYZ feature"

Really, this has happened to me several times. Luckily so far the "requests" have been for features I was already working on, so I've managed to satisfy these without having to bow to any whims. Strangely they do get all gushy afterwards.

The Affronted

Affronted that after skipping the description and reading just the name and maybe glancing at the icon of the app, it turns out not to do what they wanted - the confusion thus engendered renders the app's very existence a personal insult.

★☆☆☆☆ D. Idnot RTFM on April 16th, 2012 (Samsung Universe XVI with version 2.0.1)

"I ecspected this app to XXX but it dusnt it only YYY sooooo ridicolus OMFG I waisted nerly 30 seconds of my life on this thing and then I culdnt make it XXX but it shoud and like whatever this sux, dont waist ur life on this"

OK, nothing much I an do about that, except try to come up with a better name / more descriptive icon / shorter and more pointed description. I guess I just have to hope that other potential downloaders do read the app description and take these kinds of comments with a pinch of salt.

The Cryptic Critic

★☆☆☆☆ on April 16th, 2012 (HTC Wonderful with version 2.0.1)

"Shocking. So baaad. Even the icons suck. WTF!?"

OK, come on guys! Shocking how? What's bad? Why do the icons suck? Give us a frigging clue here! There must have been something about the app that tempted you to download (unless I've mislabelled one of The Affronted), so presumably the issues with the app could have been worked out.

Except Google didn't give me the chance to help you, or to improve the app for those that come after, because I have no way to answer the comment, publicly or privately, or to try to get any further information from the complainant.

Options for Developers on Google Play

Build great apps

OK, this one sounds obvious, but its hard to build a great app on the first shot without any helpful feedback.

As a solo developer its especially hard, actually, to see your own mistakes and evaluate the quality of something when you are so close to the work. If you have a company or a team working on an app there are lots of eyes and minds to spot mistakes, find bugs, and think of improvements.

There are a few things we could do as developers, but I'm not sure of the efficacy of these strategies:

  • Build diagnostics into our own apps - crash reports are very very useful, but miss a lot of vital information (Android version!? Heap size on all memory errors!?). We either have to wait for Google to make improvements or DIY it.
  • Build feedback into our own apps - ask the user to submit feedback from within the app, and record and publish that feedback on the developers website. Doesn't solve the problem of comments on the Google market, but might make it possible to engage in a conversation with a percentage of the users you would otherwise be unable to talk to.
  • Switch allegiance to Amazon's store. Of course, then you have to pay a fee to join, and are subject to an Apple-like review process, and I've no idea if the result is worth it - do Amazon solve any of the problems with Google's market?
  • Build our own market that does a better job. No seriously, I'd love to do this. Of course, its a massive undertaking, and would require an incredible confluence of circumstances (or marketing budget) to really take off.

Self comment

On my free app I added a comment of my own, and I periodically freshen it up so it stays near the top of the stack so new commenters see it. Here's what it says:

★★★★★ steve on March 10, 2012 (Samsung Galaxy S2 with version 1.5.4) I am the developer...

Hi all! Please consider contacting me before leaving a negative review - I am very keen to improve the app, will fix reported bugs quickly, and will add popularly requested features!

If you only leave a comment like "forces close" I can't fix it because it doesn't give me any info to work with - that makes me sad.

Yes, I five-starred my own app. I don't feel bad actually, because how else can I contend with Google's broken ratings/comment system?

Unfortunately I'm not allowed to buy my own paid app, so this technique doesn't work there. I have added more or less the same information in the app description but, as we know, not everyone reads the descriptions.

Please please, if you have any problems at all contact us directly by email - crash reports and comments are great but Google don't give us any way to contact you back!!

To be fair, many (paying) users have emailed me, as has one person who wanted the app but couldn't download either free or paid versions on her device (apparent Market bug! I sent her the paid app for free and contacted the device manufacturer - Samsung - as there's no clear way to contact Google).

Sometimes when I've had negative comments from paying users I've been able to contact them by matching up the purchase record with the information in the comment.

I can't always match them up, but when I can they are never surprised that I was able to contact them. They expect that commenting makes their information available to the developer. Why wouldn't it?

Mark as spam

The only tool Google give developers to deal with comments is a spam/not-spam toggle. I don't think its appropriate to mark genuine comments (and I do think all of the above are genuine comments) as spam.

I understand this tool was added some time ago because there were big spam problems. So far I didn't get any spam comments at all.

Paid vs Free

Strangely, the users of paid apps are typically much more polite and less inclined to negatively review. They are more often inclined to email for help and delighted to get a response.

Could be its the Principle of Commitment and maybe Post-purchase Rationalisation at work? Or maybe its just the demographic of free vs paid users?

I'm strongly considering never publishing a free app again, just for the reduced hassle.

What Google can do to help developers

I don't pretend to have all the answers, but a few small things would make my life as an Android developer much easier:

  • Give developers access to the email address of commenters, or even a way to contact them via Google that keeps them anonymous. I imagine Google think that they are protecting users by anonymising them, but its really not helping anyone.
  • Allow threaded comments. That would be enough - if I could respond, and the commenter was alerted to the response, an awful lot of problems and misunderstandings could be cleared up quickly. This would be massively to Google's benefit, reducing user and developer frustration!
  • Include more device info in crash reports and comments (e.g. Android build versions and heap size for starters)
  • Maintain a list of device specs or emulator configurations that can be used to replicate crashes or bugs reported by users - including things like the damn heap size. I can't afford to buy one of every device on the market - my developer console states "This application is available to over 1245 devices.".

Please Google, please just help us to help our users. Help us to make the Android eco-system better. Help us to generate more profits for you. Everyone wins.

p.s. I found a product-forums thread on this topic where the originating comment dates back to May 2009 and raises many of the same points I've raised here. Clearly Google's priorities lie elsewhere :(
Comment on this post

This post details my experiments parsing the same document with the usual-suspects - DOM, SAX, and Pull parsing - and comparing the results for readability and performance - especially for Android. The parsing mechanisms compared here are:

  1. W3C DOM parsing
  2. W3C DOM and XPath
  3. SAX Parsing
  4. Pull Parsing
  5. dsl4xml (dsl around Pull-parser)
  6. SJXP (thin Pull-parser wrapper using xpath-like expressions)

I hope to add more later - some contenders include: jaxb; xstream; and Simple.

The code for the entire project is in github. You will need to Maven install the dsl4xml library if you want to run the tests yourself, as I'm afraid I don't have a public repo for it yet.

Important Note: This experiment was inspired by some work I did to optimise a slow Android app, where the original authors had used mostly DOM parsing with a sprinkling of XPath.

My ultimate aim was to run these perf tests on one or more real Android devices and show how they compare there.

For this reason if you look at the project in github, you'll see that I've imported the Android 4 jar and used only the parser implementations that are available without additional imports in Android. (OK, the two pull-parser wrappers require very small standalone jars, sorry).

The Android project and Activity for running the tests on a device is in a separate project here.

The XML

The XML file being parsed is a Twitter search result (Atom feed). You can see the actual file here, but this is a snippet of the parts I'm interested in parsing for these tests (the 15 <entry>'s in the document):

<?xml version="1.0" encoding="UTF-8"?>
<feed .. >
  ..
  <entry>
    ..
    <published>2012-04-09T10:10:24Z</published>
    <title>Tweet title</title>
    <content type="html">Full tweet content</content>
    ..
    <twitter:lang>en</twitter:lang>
    <author>
        <name>steveliles (Steve Liles)</name>
        <uri>http://twitter.com/steveliles</uri>
    </author>
  </entry>
  ..
</feed>

The POJO's

The Java objects we're unmarshalling to are very simple and don't need any explanation. You can see them in Github here.

Parsing the Twitter/Atom feed

First, just a few notes on what I'm trying to do. I basically want to compare two things:

  1. Readability/maintainability of typical parsing code.
  2. Parsing performance with said typical parsing code, incl. under concurrent load.

With that in mind, I've tried to keep the parsing code small, tight, and (AFAIK) typical for each mechanism, but without layering any further libraries or helper methods on top.

In working with each parsing mechanism I have tried to choose more performant approaches where the readability trade-off is not high.

Without further ado, lets see what parsing this document and marshalling to Java objects is like using the various libraries.

W3C DOM

DOM (Document Object Model) parsing builds an in-memory object representation of the entire XML document. You can then rummage around in the DOM, going and back and forth between elements and reading data from them in whatever order you like.

Because the entire document is read into memory, there is an upper limit on the size of document you can read (constrained by the size of your Java heap).

Memory is not used particularly efficiently either - a DOM may consist of very many sparsely populated List objects (backed by mostly empty arrays). A side effect of all these objects in memory is that when you're finished with them there's a lot for the Garbage Collector to clean up.

On the plus side, DOM parsing is straight-forward to work with, particularly if you don't care much about speed and use getElementsByTagName() wherever possible.

The actual code I used for the performance test is here, but this is roughly what it ended up looking like:

private DocumentBuilder builder;
private DateFormat dateFormat;

public DOMTweetsReader() 
throws Exception {
    DocumentBuilderFactory factory = 
        DocumentBuilderFactory.newInstance();
    builder = factory.newDocumentBuilder();
    dateFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss");
}

@Override
public String getParserName() {
    return "W3C DOM";
}

public Tweets read(InputStream anInputStream) 
throws Exception {
    Document _d = builder.parse(anInputStream, "utf-8");
    Tweets _result = new Tweets();
    unmarshall(_d, _result);
    return _result;
}

public void unmarshall(Document aDoc, Tweets aTo) 
throws Exception {
    NodeList _nodes = aDoc.getChildNodes().item(0).getChildNodes();
    for (int i=0; i&lt;_nodes.getLength(); i++) {
        Node _n = _nodes.item(i);
        if ((_n.getNodeType() == Node.ELEMENT_NODE) && 
            ("entry".equals(_n.getNodeName())
         ){
            Tweet _tweet = new Tweet();
            aTo.addTweet(_tweet);
            unmarshallEntry((Element)_n, _tweet);
        }
    }
}

private void unmarshallEntry(Element aTweetEl, Tweet aTo)
throws Exception {
    NodeList _nodes = aTweetEl.getChildNodes();
    for (int i=0; i&lt;_nodes.getLength(); i++) {
        Node _n = _nodes.item(i);
        if (_n.getNodeType() == Node.ELEMENT_NODE) {                    
            if ("published".equals(_n.getNodeName())) {                         
                aTo.setPublished(dateFormat.parse(getPCData(_n)));
            } else if ("title".equals(_n.getNodeName())) {
                aTo.setTitle(getPCData(_n));
            } else if ("content".equals(_n.getNodeName())) {
                Content _content = new Content();
                aTo.setContent(_content);
                unmarshallContent((Element)_n, _content);
            } else if ("lang".equals(_n.getNodeName())) {
                aTo.setLanguage(getPCData(_n));
            } else if ("author".equals(_n.getNodeName())) {
                Author _author = new Author();
                aTo.setAuthor(_author);
                unmarshallAuthor((Element)_n, _author);
            }
        }
    }
}

private void unmarshallContent(Element aContentEl, Content aTo) {
    aTo.setType(aContentEl.getAttribute("type"));
    aTo.setValue(aContentEl.getNodeValue());
}

private void unmarshallAuthor(Element anAuthorEl, Author aTo) {
    NodeList _nodes = anAuthorEl.getChildNodes();
    for (int i=0; i&lt;_nodes.getLength(); i++) {
        Node _n = _nodes.item(i);
        if ("name".equals(_n.getNodeName())) {
            aTo.setName(getPCData(_n));
        } else if ("uri".equals(_n.getNodeName())) {
            aTo.setUri(getPCData(_n));
        }
    }
}

private String getPCData(Node aNode) {
    StringBuilder _sb = new StringBuilder();
    if (Node.ELEMENT_NODE == aNode.getNodeType()) {
        NodeList _nodes = aNode.getChildNodes();
        for (int i=0; i&lt;_nodes.getLength(); i++) {
            Node _n = _nodes.item(i);
            if (Node.ELEMENT_NODE == _n.getNodeType()) {
                _sb.append(getPCData(_n));
            } else if (Node.TEXT_NODE == _n.getNodeType()) {
                _sb.append(_n.getNodeValue());
            }
        }
    }
    return _sb.toString();
}

Its worth noting that I would normally extract some useful utility classes/methods - for example getPCData(Node) - but here I'm trying to keep the sample self-contained.

Note that this code is not thread-safe because of the unsynchronized use of SimpleDateFormat. I am using separate instances of the Reader classes in each thread for my threaded tests.

W3C DOM and XPath

XPath is a language for describing locations within an XML document as paths from a starting location (which can be the root of the document (/), the current location (.//) or anywhere (//)).

I've used XPath on and off for years, mostly in XSLT stylesheets, but also occasionally to pluck bits of information out of documents in code. It is very straight-forward to use.

Here's a sample for parsing our Twitter Atom feed. The actual test code is in github.

private DocumentBuilder builder;
private XPathFactory factory;

private XPathExpression entry;
private XPathExpression published;
private XPathExpression title;
private XPathExpression contentType;
private XPathExpression content;
private XPathExpression lang;
private XPathExpression authorName;
private XPathExpression authorUri;

private DateFormat dateFormat;

public DOMXPathTweetsReader() 
throws Exception {
    DocumentBuilderFactory _dbf = 
        DocumentBuilderFactory.newInstance();
    _dbf.setNamespaceAware(true);
    builder = _dbf.newDocumentBuilder();
    factory = XPathFactory.newInstance();

    NamespaceContext _ctx = new NamespaceContext() {
        public String getNamespaceURI(String aPrefix) {
            String _uri;
            if (aPrefix.equals("atom"))
                _uri = "http://www.w3.org/2005/Atom";
            else if (aPrefix.equals("twitter"))
                _uri = "http://api.twitter.com/";
            else
                _uri = null;
            return _uri;
        }

        @Override
        public String getPrefix(String aArg0) {
            return null;
        }

        @Override
        @SuppressWarnings("rawtypes")
        public Iterator getPrefixes(String aArg0) {
            return null;
        }
    };

    entry = newXPath(factory, _ctx, "/atom:feed/atom:entry");
    published = newXPath(factory, _ctx, ".//atom:published");
    title = newXPath(factory, _ctx, ".//atom:title");
    contentType = newXPath(factory, _ctx, ".//atom:content/@type");
    content = newXPath(factory, _ctx, ".//atom:content");
    lang = newXPath(factory, _ctx, ".//twitter:lang");
    authorName = newXPath(factory, _ctx, ".//atom:author/atom:name");
    authorUri = newXPath(factory, _ctx, ".//atom:author/atom:uri");

    dateFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss");
}

private XPathExpression newXPath(
    XPathFactory aFactory, NamespaceContext aCtx, String anXPath
) throws Exception {
    XPath _xp = factory.newXPath();
    _xp.setNamespaceContext(aCtx);
    return _xp.compile(anXPath);
}

@Override
public String getParserName() {
    return "W3C DOM/XPath";
}

@Override
public Tweets read(InputStream anInputStream)
throws Exception {
    Tweets _result = new Tweets();
    Document _document = builder.parse(anInputStream);

    NodeList _entries = (NodeList) 
        entry.evaluate(_document, XPathConstants.NODESET);                  
    for (int i=0; i&lt;_entries.getLength(); i++) {
        Tweet _tweet = new Tweet();
        _result.addTweet(_tweet);

        Node _entryNode = _entries.item(i);

        _tweet.setPublished(getPublishedDate(_entryNode));
        _tweet.setTitle(title.evaluate(_entryNode));
        _tweet.setLanguage(lang.evaluate(_entryNode));

        Content _c = new Content();
        _tweet.setContent(_c);

        _c.setType(contentType.evaluate(_entryNode));
        _c.setValue(content.evaluate(_entryNode));

        Author _a = new Author();
        _tweet.setAuthor(_a);

        _a.setName(authorName.evaluate(_entryNode));
        _a.setUri(authorUri.evaluate(_entryNode));
    }

    return _result;
}

private Date getPublishedDate(Node aNode) 
throws Exception {
    return dateFormat.parse(published.evaluate(aNode));
}

The code ends up being quite easy to read and can be written to nest in a way that mimics the document structure. There is a very big downside - as you'll see later - the performance is atrocious.

SAX Parser

SAX stands for Simple API for XML. It uses a "push" approach: whereas with DOM you can dig around in the document in whatever order you like, SAX parsing is event-driven which means you have to handle the data as it is given to you.

SAX parsers fire events when they encounter the various components that make up an XML file. You register a ContentHandler whose methods are called-back when these events occur (for example when the parser finds a new start element, it invokes the startElement method of your ContentHandler).

The API assumes that the consumer (ContentHandler) is going to maintain some awareness of its state (e.g. where it currently is within the document). I sometimes use a java.util.Stack to push/pop/peek at which element I'm currently working in, but here I can get away with just recording the name of the current element.

I'm extending DefaultHandler because I'm not interested in many of the events (it provides a default empty implementation of those methods for me).

The actual test code is in github, and is actually more complex in order to handle entity-refs via a LexicalHandler, but here's the gist of it:

private XMLReader reader;
private TweetsHandler handler;

public SAXTweetsReader() 
throws Exception {
    SAXParserFactory _f = SAXParserFactory.newInstance();
    SAXParser _p = _f.newSAXParser();
    reader = _p.getXMLReader();
    handler = new TweetsHandler();
    reader.setContentHandler(handler);
}

@Override
public String getParserName() {
    return "SAX";
}

@Override
public Tweets read(InputStream anInputStream) 
throws Exception {
    reader.parse(new InputSource(anInputStream));
    return handler.getResult();
}

private static class TweetsHandler extends DefaultHandler {

    private DateFormat dateFormat = 
        new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss");
    private Tweets tweets;
    private Tweet tweet;
    private Content content;
    private Author author;
    private String currentElement;

    public Tweets getResult() {
        return tweets;
    }

    @Override
    public void startDocument() throws SAXException {
        tweets = new Tweets();
    }

    @Override
    public void startElement(
        String aUri, String aLocalName, 
        String aQName, Attributes aAttributes
    ) throws SAXException {
        currentElement = aQName;
        if ("entry".equals(aQName)) {
            tweets.addTweet(tweet = new Tweet());
        } else if ("content".equals(aQName)) {
            tweet.setContent(content = new Content());
            content.setType(aAttributes.getValue("type"));
        } else if ("author".equals(aQName)) {
            tweet.setAuthor(author = new Author());
        }
    }

    @Override
    public void endElement(
        String aUri, String aLocalName, String aQName
    ) throws SAXException {
        currentElement = null;
    }

    @Override
    public void characters(char[] aCh, int aStart, int aLength)
    throws SAXException {
        if ("published".equals(currentElement)) {
            try {
                tweet.setPublished(dateFormat.parse(
                    new String(aCh, aStart, aLength))
                );
            } catch (ParseException anExc) {
                throw new SAXException(anExc);
            }
        } else if (
            ("title".equals(currentElement)) &&
            (tweet != null)
        ) {
            tweet.setTitle(new String(aCh, aStart, aLength));
        } else if ("content".equals(currentElement)) {
            content.setValue(new String(aCh, aStart, aLength));
        } else if ("lang".equals(currentElement)) {
            tweet.setLanguage(new String(aCh, aStart, aLength));
        } else if ("name".equals(currentElement)) {
            author.setName(new String(aCh, aStart, aLength));
        } else if ("uri".equals(currentElement)) {
            author.setUri(new String(aCh, aStart, aLength));
        }
    }
}

One downside when handling more complicated documents is that the ContentHandler can get littered with intermediate state objects - for example here I have the tweet, content, and author fields.

Another is that SAX is very low level and you have to handle pretty much everything - including that text nodes are passed to you in pieces when there are entity-references present.

Pull Parser

Pull-parsing is the "pull" to SAX parsing's "push". SAX pushes content at you by firing events as it encounters constructs within the xml document. Pull-parsing lets you ask for (pull) the next significant construct you are interested in.

You still have to take the data in the order it appears in the document - you can't go back and forth through the document like you can with DOM - but you can skip over bits you aren't interested in.

Test code is in github, this is roughly what it looks like:

private DateFormat dateFormat;
private XmlPullParserFactory f;
private Tweets tweets;
private Tweet currentTweet;
private Author currentAuthor;

public PullParserTweetsReader() 
throws Exception {
    dateFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss");
    f = XmlPullParserFactory.newInstance();
    f.setNamespaceAware(true);
}

@Override
public String getParserName() {
    return "Pull-Parser";
}

@Override
public Tweets read(InputStream anInputStream) throws Exception {
    XmlPullParser _p = f.newPullParser();
    _p.setInput(anInputStream, "utf-8");
    return parse(_p);
}

private Tweets parse(XmlPullParser aParser) 
throws Exception {
    tweets = new Tweets();

    int _e = aParser.next();
    while (_e != XmlPullParser.END_DOCUMENT) {
        if (_e == XmlPullParser.START_TAG) {
            startTag(aParser.getPrefix(), aParser.getName(), aParser);
        }
        _e = aParser.next();
    }

    return tweets;
}

private void startTag(String aPrefix, String aName, XmlPullParser aParser)
throws Exception {
    if ("entry".equals(aName)) {
        tweets.addTweet(currentTweet = new Tweet());
    } else if ("published".equals(aName)) {
        aParser.next();
        currentTweet.setPublished(dateFormat.parse(aParser.getText()));
    } else if (("title".equals(aName)) && (currentTweet != null)) {
        aParser.next();
        currentTweet.setTitle(aParser.getText());
    } else if ("content".equals(aName)) {
        Content _c = new Content();
        _c.setType(aParser.getAttributeValue(null, "type"));
        aParser.next();
        _c.setValue(aParser.getText());
        currentTweet.setContent(_c);
    } else if ("lang".equals(aName)) {
        aParser.next();
        currentTweet.setLanguage(aParser.getText());
    } else if ("author".equals(aName)) {
        currentTweet.setAuthor(currentAuthor = new Author());
    } else if ("name".equals(aName)) {
        aParser.next();
        currentAuthor.setName(aParser.getText());
    } else if ("uri".equals(aName)) {
        aParser.next();
        currentAuthor.setUri(aParser.getText());
    }
}

SJXP (Pull-Parser wrapper)

The first of the pull-parser wrappers under test, I stumbled upon this one yesterday. I liked the idea behind it so decided to give it a try.

I'm a big fan of callbacks generally, and having spent quite some time working with XPath in the past the idea of using XPath-like syntax to request callbacks from the pull-parser seems tempting.

There was one problem I couldn't work around which seems like either a gap in my knowledge (and the documentation) or an irritating bug - when declaring the paths you have to use the full namespace uri even on elements in the default namespace.

This means that my path declarations even on this shallow document are enormous and I had to split them onto three lines to fit the width of my blog.

Code is in github, this is the gist of it:

private Tweet currentTweet;
private DateFormat dateFormat;
private XMLParser&lt;Tweets> parser; 

private IRule&lt;Tweets> tweet = new DefaultRule&lt;Tweets>(Type.TAG, 
    "/[http://www.w3.org/2005/Atom]feed" +
    "/[http://www.w3.org/2005/Atom]entry"
) {
    public void handleTag(
        XMLParser&lt;Tweets> aParser, boolean aIsStartTag, Tweets aUserObject) {
        if (aIsStartTag)
            aUserObject.addTweet(currentTweet = new Tweet());
    }   
};

private IRule&lt;Tweets> published = new DefaultRule&lt;Tweets>(Type.CHARACTER, 
    "/[http://www.w3.org/2005/Atom]feed" +
    "/[http://www.w3.org/2005/Atom]entry" +
    "/[http://www.w3.org/2005/Atom]published"
) {
    public void handleParsedCharacters(
        XMLParser&lt;Tweets> aParser, String aText, Tweets aUserObject
    ) {
        try {                   
            currentTweet.setPublished(dateFormat.parse(aText));
        } catch (ParseException anExc) {
            throw new XMLParserException("date-parsing problem", anExc);
        }
    }           
}; 

private IRule&lt;Tweets> title = new DefaultRule&lt;Tweets>(Type.CHARACTER, 
    "/[http://www.w3.org/2005/Atom]feed" +
    "/[http://www.w3.org/2005/Atom]entry" +
    "/[http://www.w3.org/2005/Atom]title"
) {
    public void handleParsedCharacters(
        XMLParser&lt;Tweets> aParser, String aText, Tweets aUserObject
    ) {
        currentTweet.setTitle(aText);
    }           
};

IRule&lt;Tweets> content = new DefaultRule&lt;Tweets>(Type.TAG, 
    "/[http://www.w3.org/2005/Atom]feed" +
    "/[http://www.w3.org/2005/Atom]entry" +
    "/[http://www.w3.org/2005/Atom]content" +
) {
    public void handleTag(
        XMLParser&lt;Tweets> aParser, boolean aIsStartTag, Tweets aUserObject
    ) {
        if (aIsStartTag)
            currentTweet.setContent(new Content());
        super.handleTag(aParser, aIsStartTag, aUserObject);
    }
};

private IRule&lt;Tweets> contentType = new DefaultRule&lt;Tweets>(Type.ATTRIBUTE, 
    "/[http://www.w3.org/2005/Atom]feed" +
    "/[http://www.w3.org/2005/Atom]entry" +
    "/[http://www.w3.org/2005/Atom]content", "type"
) {
    public void handleParsedAttribute(
        XMLParser&lt;Tweets> aParser, int aIndex, String aValue, Tweets aUserObject
    ) {                 
        currentTweet.getContent().setType(aValue);
    }
};

private IRule&lt;Tweets> contentText = new DefaultRule&lt;Tweets>(Type.CHARACTER, 
    "/[http://www.w3.org/2005/Atom]feed" +
    "/[http://www.w3.org/2005/Atom]entry" +
    "/[http://www.w3.org/2005/Atom]content"
) {
    public void handleParsedCharacters(
        XMLParser&lt;Tweets> aParser, String aText, Tweets aUserObject
    ) {                 
        currentTweet.getContent().setValue(aText);
    }
};

private IRule&lt;Tweets> lang = new DefaultRule&lt;Tweets>(Type.CHARACTER, 
    "/[http://www.w3.org/2005/Atom]feed" +
    "/[http://www.w3.org/2005/Atom]entry" +
    "/[http://api.twitter.com/]lang"
) {
    public void handleParsedCharacters(
        XMLParser&lt;Tweets> aParser, String aText, Tweets aUserObject
    ) {
        currentTweet.setLanguage(aText);
    }
};

private IRule&lt;Tweets> author = new DefaultRule&lt;Tweets>(Type.TAG, 
    "/[http://www.w3.org/2005/Atom]feed" +
    "/[http://www.w3.org/2005/Atom]entry" +
    "/[http://www.w3.org/2005/Atom]author"
) {
    public void handleTag(
        XMLParser&lt;Tweets> aParser, boolean aIsStartTag, Tweets aUserObject
    ) {
        if (aIsStartTag)
            currentTweet.setAuthor(new Author());
        super.handleTag(aParser, aIsStartTag, aUserObject);
    }
};

private IRule&lt;Tweets> authorName = new DefaultRule&lt;Tweets>(Type.CHARACTER, 
    "/[http://www.w3.org/2005/Atom]feed"
    "/[http://www.w3.org/2005/Atom]entry" +
    "/[http://www.w3.org/2005/Atom]author" +
    "/[http://www.w3.org/2005/Atom]name"
) {
    public void handleParsedCharacters(
        XMLParser&lt;Tweets> aParser, String aText, Tweets aUserObject
    ) {
        currentTweet.getAuthor().setName(aText);
    }
};

private IRule&lt;Tweets> authorUri = new DefaultRule&lt;Tweets>(Type.CHARACTER, 
    "/[http://www.w3.org/2005/Atom]feed" +
    "/[http://www.w3.org/2005/Atom]entry" +
    "/[http://www.w3.org/2005/Atom]author" +
    "/[http://www.w3.org/2005/Atom]uri"
) {
    public void handleParsedCharacters(
        XMLParser&lt;Tweets> aParser, String aText, Tweets aUserObject
    ) {
        currentTweet.getAuthor().setUri(aText);
    }
};

@SuppressWarnings("all")
public SJXPTweetsReader() {
    dateFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss");
    parser = parser = new XMLParser&lt;Tweets>(
        tweet, published, title, content, contentType, 
        contentText, lang, author, authorName, authorUri
    );
}

@Override
public String getParserName() {
    return "SJXP (pull)";
}

@Override
public Tweets read(InputStream anInputStream) 
throws Exception {
    Tweets _result = new Tweets();  
    parser.parse(anInputStream, "utf-8", _result);
    return _result;
}

I like the idea of SXJP and I think that - particularly on more complex documents - it will lead to code that is easier to understand and maintain because you can consider each part entirely separately. It bulks up with boiler-plate though, especially with that namespace issue I mentioned.

Like SAX and "straight" Pull parsing it also suffers the problem of having to manage intermediate state (in my sample its currentTweet). It does allow a state/context object to be pushed into the callback methods, so I could have passed a customised context class to manage my state in instead of passing Tweets.

dsl4xml (Pull-parser wrapper)

This is my own small wrapper around XMLPullParser. The goals and reasons for it are stated at length else-where, but suffice to say that readability without sacrificing speed was my main aim.

Dsl4xml parsing code has a declarative style, is concise, and uses reflection to cut boiler-plate to a minimum.

Actual code is in Github, here's what it looks like:

private DocumentReader&lt;Tweets> reader;

public Dsl4XmlTweetsReader() {
    reader = mappingOf(Tweets.class).to(
        tag("entry", Tweet.class).with(
            tag("published"),
            tag("title"),
            tag("content", Content.class).with(
                attribute("type"),
                pcdataMappedTo("value")
            ),
            tag("twitter", "lang").
                withPCDataMappedTo("language"),
            tag("author", Author.class).with(
                tag("name"),
                tag("uri")
            )
        )
    );

    reader.registerConverters(
        new ThreadUnsafeDateConverter("yyyy-MM-dd'T'HH:mm:ss")
    );
}

@Override
public String getParserName() {
    return "DSL4XML (pull)";
}

@Override
public Tweets read(InputStream anInputStream) throws Exception {
    return reader.read(anInputStream, "utf-8");
}

There are two things I want to point out, which I guess you will have noticed already:

  1. This is by far the shortest and simplest code of all the samples shown.
  2. The code is slightly unusual in its style because it uses an Internal Domain Specific Language. The nice thing (IMHO) is that it is very readable, and even mimics the structure of the XML itself.

Its still early days for dsl4xml, so the DSL may evolve a bit with time. I'm also looking into ways to keep the same tight syntax without resorting to reflection - the aim being to narrow the performance gap between the raw underlying parser (currently a Pull parser) and dsl4xml.

Performance Comparison

I built some performance tests using the mechanisms described above to parse the same document repeatedly.

The tests are run repeatedly with increasing numbers of threads, from 1 to 8, parsing 1000 documents in each thread. The xml document is read into a byte array in memory before the test starts to eliminate disk IO from consideration.

When the statistics for each method have been collected, the test generates a html document that uses Google charts to render the results.

Each parsing method is tested several times and the results averaged to smooth out some of the wilder outliers (still far from perfect, partly due to garbage collection). I ran the tests on my Linux Desktop, Macbook Air, Samsung Galaxy S2 and Morotola Xoom2 Media Edition.

Here is the chart for the desktop (Core i7 (quad) 1.8GHz, 4GB RAM, Ubuntu 11.10, Sun JDK 1.6.0-26). There is a noticeable hump at 4 threads, presumably because its a quad core. Performance keeps rising up to 8 threads, this presumably because the cpu has hyperthreading. After 8 threads the performance slowly drops off as the context-switching overhead builds up (not shown here):

And here's the chart from my MacBook Air (Core i5 (dual) 1.7GHz, 4GB RAM, OSX Lion, Apple JDK 1.6.0-31):

The difference running under Android is, to put it mildly, astonishing. Here's the chart from my Samsung Galaxy S2 running Android 2.3.4, 64Mb heap. I reduced the max concurrency to 4 and the number of documents parsed per thread to 10, otherwise my phone would be obsolete before the results came back :)

Yep, SAX kicking ass right there.

Here's how it looks on a Motorola Xoom 2 Media edition running Android 3.2.2 (with 48Mb heap):

Confirming that SAX is the way to go on Android!

Quick side note about iOS

My friend Matt Preston did a quick port of the DOM and SAX parsing tests to iOS.

He didn't produce a chart (yet!), but the DOM parsing throughput on an iPhone 4S was approximately twice as good as SAX parsing on my Samsung. SAX Parsing on the iPhone churned through on average 150 docs/sec!

Its interesting to note that the iPhone4S runs a 1GHz Cortex A9 CPU clocked down to 800Mhz, while my Samsung is running a 1.2GHz Cortex A9.

Why XPath parsing sucked so bad

The observant will have noticed the charts do not contain figures for the XPath parsing. That's because I dropped it when I realised it was two orders of magnitude slower even than DOM parsing.

This appalling performance seems to be because when executing each xpath expression a context object is created which involves looking up several files on the classpath (and all the inherent synchronisation this entails). I don't intend to waste my time digging into why this can't done once and cached :(.

If you're interested, this is what my threads spent most of their time doing in the XPath test:

"Thread-11" prio=5 tid=7fcf544d2000 nid=0x10d6bb000 
    waiting for monitor entry [10d6b9000]
    java.lang.Thread.State: BLOCKED (on object monitor)
    at java.util.zip.ZipFile.getEntry(ZipFile.java:159)
    - locked &lt;7f4514c88> (a java.util.jar.JarFile)
    at java.util.jar.JarFile.getEntry(JarFile.java:208)
    at java.util.jar.JarFile.getJarEntry(JarFile.java:191)
    at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:757)
    at sun.misc.URLClassPath$JarLoader.findResource(URLClassPath.java:735)
    at sun.misc.URLClassPath.findResource(URLClassPath.java:146)
    at java.net.URLClassLoader$2.run(URLClassLoader.java:385)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findResource(URLClassLoader.java:382)
    at java.lang.ClassLoader.getResource(ClassLoader.java:1002)
    at java.lang.ClassLoader.getResource(ClassLoader.java:997)
    at java.lang.ClassLoader.getSystemResource(ClassLoader.java:1100)
    at java.lang.ClassLoader.getSystemResourceAsStream(ClassLoader.java:1214)
    at com.sun.org.apache.xml.internal.dtm.SecuritySupport12$6.run
        (SecuritySupport12.java:117)
    at java.security.AccessController.doPrivileged(Native Method)
    at
    com.sun.org.apache.xml.internal.dtm.SecuritySupport12.
        getResourceAsStream(SecuritySupport12.java:112)
    at com.sun.org.apache.xml.internal.dtm.ObjectFactory.
        findJarServiceProviderName(ObjectFactory.java:549)
    at com.sun.org.apache.xml.internal.dtm.ObjectFactory.
        lookUpFactoryClassName(ObjectFactory.java:373)
    at com.sun.org.apache.xml.internal.dtm.ObjectFactory.
        lookUpFactoryClass(ObjectFactory.java:206)
    at com.sun.org.apache.xml.internal.dtm.ObjectFactory.
        createObject(ObjectFactory.java:131)
    at com.sun.org.apache.xml.internal.dtm.ObjectFactory.
        createObject(ObjectFactory.java:101)
    at com.sun.org.apache.xml.internal.dtm.DTMManager.
        newInstance(DTMManager.java:135)
    at com.sun.org.apache.xpath.internal.XPathContext.
        &lt;init>(XPathContext.java:100)
    at com.sun.org.apache.xpath.internal.jaxp.XPathExpressionImpl.
        eval(XPathExpressionImpl.java:110)

Conclusions

Readability

Of the mechanisms tested so far, and from the code samples above, I think that dsl4xml produces far the most readable and maintainable parsing code. Of course I am biased.

I think SAX parsing would have worked out to be the most readable of the other mechanisms if it hadn't been for those pesky entity-refs. As it is I have to recommend Pull-parsing as the way to go for readability.

Desktop/laptop xml parsing performance

SAX parsing and the pull-parsing wrappers give comparable performance. Raw Pull-parsing beats the lot by a margin of around 15%. DOM performs relatively badly - around twice as slow as any of the others. Don't go near XPath based parsing unless you like watching paint dry.

Recommendation: Pull Parser for max performance and relative ease of use. Dsl4xml if you want performance and great readability :)

Android xml parsing performance

Avoid XPath at all costs. DOM and pull-parsing appear to have similarly poor performance characteristics. SAX absolutely destroys all the others - roughly an order of magnitude quicker.

Recommendation: SAX, every time. I'll get working on a SAX-based dsl4xml implementation :)

Update (23rd April 2012): Just finished a SAX-based dsl4xml - here's the performance chart for my Samsung Galaxy SII again (also includes figures for SimpleXML):

Final words

The Twitter Atom feed is not particularly complicated - tags are not deeply nested, not too many attributes, no nested tags of the same name, no mixed content (tags and text-nodes as siblings), etc.

I suspect that the performance gap between the different mechanisms widens as the document complexity increases, but as yet have no real evidence to back that up.
Comment on this post

For a readability and performance comparison of different parsing mechanisms available in Android, have a look at my more recent post that compares parsing a Twitter search result using DOM, SAX, and various Pull parsing methods.

The short story: Always use SAX in Android (here's why).

SAX and Pull-parsing are fast, but don't lead to the most readable/maintainable code. Instead, how about a super-simple internal DSL for describing the mapping for unmarshalling from XML to POJO's, with pull-parsing performance? Quick example:

<books>
  <book>
    <title>The Hobbit</title>
    <synopsis>
        A little guy goes on an adventure, 
        finds ring, comes back.
    </synopsis>
  </book>
  <book>
    <title>The Lord of the Rings</title>
    <synopsis>
        A couple of little guys go on an adventure, 
        lose ring, come back.
    </synopsis>
  </book>
</books>

Can be unmarshalled to simple POJO's with this:

import static com.sjl.dsl4xml.DocumentReader.*;

class BooksReader {
    private DocumentReader&lt;Books> reader;

    public BooksReader() {
        reader = mappingOf(Books.class).to(
           tag("book", Book.class).with(
               tag("title"),
               tag("synopsis")
           )
        );
    }

    public Books reader(Reader aReader) {
        return reader.read(aReader);
    }
}

The long story:

I recently had occasion to work on an Android app that was suffering horrible performance problems on startup (approx 7-12 seconds before displaying content).

A look at the code showed up several possible contenders for the source of the problem:

  1. Many concurrent http requests to fetch XML content from web
  2. Parsing the returned XML documents concurrently
  3. Parsing documents using DOM (and some XPath)

A quick run through with the excellent profiler built in to DDMS immediately showed lots of time spent in DOM methods, and masses of heap being consumed by sparsely populated java.util.List objects (used to represent the DOM in memory).

Since the app was subsequently discarding the parsed Document object, the large heap consumption was contributing a huge garbage-collection load as a side-effect.

Parsing many documents at once meant that the app suffered a perfect storm of exacerbating issues: Slow DOM traversal with XPath; constant thread context-switching; massive heap consumption; and huge object churn.

The network requests - even over 3G - were comparatively insignificant in the grand scheme.

Reducing thread context switching

An obvious and inexpensive thing to try at this point was reducing the concurrency to minimise the overhead of context-switching and hopefully enable the CPU caches to be used to best advantage.

I confess I hoped for a significant improvement from this small change, but the difference, while measurable, was too small to be significant (~5-10%).

More efficient parsing

XPath is easy to use, and typically makes it possible to write straight-forward code for marshalling data from an XML document into Java objects. It is, however, horribly slow and a terrible memory hog.

I decided to try an experiment with an alternative parsing method, to see if a worthwhile performance gain could be achieved on one of the smaller documents that could then be applied to others.

I wrote a small test-case confirming the correctness of the existing parsing mechanism and testing the throughput in documents per second, then extracted an interface and created a new implementation that used Pull-Parsing instead of DOM and XPath.

The result was quite pleasing: 5x faster on a simple document. I fully expected the performance gains to be even better on more complex documents, so was quite eager to repeat the process for one of the most complex documents.

However, I had one major concern that put me off: the code for parsing even a simple document was already quite long and had a nasty whiff of conditional-overkill (think: lots of if statements). I wasn't too happy about trading code readability for performance.

I pondered a few alternatives like XStream which I've used a lot for converting from Java to XML but not much the other way around, and SimpleXML which I have used previously and can be nice, but pollutes your model objects with annotations and in some situations can be a real pain to get working.

An Internal DSL for mapping XML to POJO's

In the end I decided to spend just a few hours turning the problem over in code to see if I could come up with something more readable for working with the pull-parser directly.

The result, after an afternoon of attempting to parse the most complex XML file the app consumed, was a small Internal DSL (Domain Specific Language) for declaratively describing the mapping between an XML and the Java model classes, and a 15x performance improvement in startup time for the app (7-12 seconds down to ~0.5s).

The DSL I originally came up with required some boiler-plate code to do the final mapping between text nodes / attributes and the model classes being populated. If Java had a neat syntax for closures this would have been much less irritating :)

As it was the boiler plate irked me - too much stuff getting in the way of reading what was really important. I thought about it a bit in my spare time, and had another shot at it. My aims were:

  1. To make readable, maintainable, declarative code that unmarshalls XML documents to Java objects.
  2. To make unmarshalling XML documents to Java objects very fast (sax/pull-parsing speeds).
  3. To avoid polluting model classes with metadata about xml parsing (no annotations).
  4. To avoid additional build-time steps or "untouchable" code (code generators, etc).
  5. To produce a very small jar with no large dependencies.

The result is starting to take shape in github as dsl4xml. It removes all of the boiler plate in exchange for a small performance penalty due to use of reflection. I don't have comparative performance figures yet, but will post some when I get time.

Another example

XML:

<hobbit>
  <name firstname="Frodo" surname="Baggins"/>
  <dob>11400930</dob>
  <address>
    <house>
      <name>Bag End</name>
      <number></number>
    </house>
    <street>Bagshot Row</street>
    <town>Hobbiton</town>
    <country>The Shire</country>
  </address>
</hobbit>

POJO's: See the source-code of the test-case

Unmarshalling code:

private static DocumentReader&lt;Hobbit> newReader() {
    DocumentReader&lt;Hobbit> _marshaller = mappingOf(Hobbit.class).to(
        tag("name", Name.class).with(
            attributes("firstname", "surname")
        ),
        tag("dob"),
        tag("address", Address.class).with(
            tag("house", Address.House.class).with(
                tag("name"),
                tag("number")
            ),
            tag("street"),
            tag("town"),
            tag("country")
        )
    );

    _reader.registerConverters(new ThreadUnsafeDateConverter("yyyyMMdd"));

    return _reader;
}

A DocumentReader, once constructed, is intended to be re-used repeatedly. The DocumentReader itself is completely thread-safe as unmarshalling does not modify any of its internal state. To ensure thread-safety you must use only thread-safe type converters (see type conversion section below).

A minimum of garbage is generated because we're using a pull parser to skip over parts of the document we don't care about, and the only state maintained along the way (in a single-use context object for thread safety) is the domain objects we're creating.

Type conversion

You can create and register your own type converters. They are used only to map the lowest level xml data to your Java objects - attribute values and CData Strings. The Converter interface looks like this:

package com.sjl.dsl4xml.support;

public interface Converter&lt;T> {
    public boolean canConvertTo(Class<?> aClass);
    public T convert(String aValue);
}

An example Converter for converting String values to primitive int's looks like this:

class PrimitiveIntConverter implements Converter&lt;Integer> {
    @Override
    public boolean canConvertTo(Class&lt;?> aClass) {
        return aClass.isAssignableFrom(Integer.TYPE);
    }

    @Override
    public Integer convert(String aValue) {
        return ((aValue == null) || ("".equals(aValue))) ? 
            0 : new Integer(aValue);
    }
}

Most converters can be thread-safe, but some may require concurrency control for multi-threaded use (example: when converting dates using SimpleDateFormat).

You can use optimised type converters in situations where you know you will not be unmarshalling from multiple threads concurrently. An example is the ThreadUnsafeDateConverter which is used in the example above because it came from a test-case that will only ever run single-threaded. 

public class ThreadUnsafeDateConverter implements Converter&lt;Date> {
    private DateFormat dateFormat;

    public ThreadUnsafeDateConverter(String aDateFormatPattern) {
        // SimpleDateFormat is NOT thread-safe
        dateFormat = new SimpleDateFormat(aDateFormatPattern);
    }

    @Override
    public boolean canConvertTo(Class&lt;?> aClass) {
        return aClass.isAssignableFrom(Date.class);
    }

    @Override
    public Date convert(String aValue) {
        try {
            return ((aValue == null) || ("".equals(aValue))) ? 
                null : dateFormat.parse(aValue);
        } catch (ParseException anExc) {
            throw new XmlMarshallingException(anExc);
        }
    }
}

The alternative ThreadSafeDateConverter looks like this:

class ThreadSafeDateConverter implements Converter&lt;Date> {
    private ThreadLocal&lt;DateFormat> dateFormat;

    public ThreadSafeDateConverter(final String aDateFormatPattern) {
        dateFormat = new ThreadLocal&lt;DateFormat>() {
            protected DateFormat initialValue() {
                return new SimpleDateFormat(aDateFormatPattern);
            }
        };
    }

    @Override
    public boolean canConvertTo(Class&lt;?> aClass) {
        return aClass.isAssignableFrom(Date.class);
    }

    @Override
    public Date convert(String aValue) {
        try {
            return ((aValue == null) || ("".equals(aValue))) ? 
                null : dateFormat.get().parse(aValue);
        } catch (ParseException anExc) {
            throw new XmlMarshallingException(anExc);
        }
    }
}

Missing features

This is still a very new project, and in an experimental stage. There's loads still to do:

  • Experiment with more documents to drive improvements to the DSL
  • More converters for the obvious types (e.g., BigDecimal, BigInteger, File, URI, etc.)
  • Support for namespaced documents
  • Support for CDATA (so far only tested with PCDATA)
  • Performance comparisons with DOM, SAX and non-DSL'd Pull parsing
  • Support for explicit (non-reflective) marshalling of properties
  • Support for SAX parsing instead of Pull-Parsing (see notes below)
  • Performance tests
  • Performance optimisations

Notes

I came across some interesting comments by Diane Hackborn (Android platform developer) in this thread.

Diane points out that SAX parsing is faster than Pull Parsing (at least on Android). I had been under the impression it was the other way around, hence I went with Pull parsing.

Later perf tests show SAX to be much faster on Android, so I will probably refactor to use SAX.

android parser performance
Comment on this post

The Android platform allows you to use all of the normal Java concurrency constructs. You should use them if you need to do any long-running* operations: you must do these off the main UI thread if you want keep your users happy, and the platform even enforces this by displaying an Application Not Responding dialog if an app does not respond to user-input within 5 seconds.

The platform provides a couple of mechanisms to facilitate communication between background threads and the main application thread: Handler's and AsyncTasks. In this article I want to concentrate on AsyncTask.

update, December 2013: I came across a StackOverflow post referencing this article, where the poster was - understandably - confused by my use of "long-running" in the first paragraph. Long-running is a relative term, and here I mean 'more than a few milliseconds and less than ~500ms'.

The main thread has just 16.67ms to render a single frame at 60Hz, so if you do anything on the main thread that gets even close to using up that 16ms you're risking skipped frames.

For operations that are long-running in human terms (500ms+) there are other constructs which may be more appropriate (Loaders/Services) - you might find my book useful, which covers all of the Android concurrency constructs in detail, including much more up-to-date and in-depth coverage of AsyncTask.

There are all kinds of things you will want to do off the UI thread, but AsyncTask (when used from an Activity or Fragment) is only really appropriate for relatively short operations. Ideal uses include CPU-intensive tasks, such as number crunching or searching for words in large text strings, blocking I/O operations such as reading and writing text files, or loading images from local files with BitmapFactory.

The basics of AsyncTask

AsyncTask provides a simple API for doing work in the background and re-integrating the result with the main thread. Here's what it looks like:

new AsyncTask&lt;Param, Progress, Result>() {
    protected void onPreExecute() {
        // perhaps show a dialog 
        // with a progress bar
        // to let your users know
        // something is happening
    }

    protected Result doInBackground(Param... aParams) {
        // do some expensive work 
        // in the background here
    }

    protected void onPostExecute(Result aResult) {
        // background work is finished, 
        // we can update the UI here
        // including removing the dialog
    }
}.execute();

The template methods onPreExecute() and onPostExecute(Result) are invoked such that you can safely update the UI from there.

There is a fourth template method - onProgressUpdate(Progress[]) - which you can implement if you want to update the UI to show progress is being made within the background thread. For this to actually work you will need to invoke publishProgress(Progress[]) regularly from within doInBackground(Param[]).

AsyncTask is generic, and presents three type variables:

class AsyncTask&lt;Params, Progress, Result>

They are used as follows:

  1. Params is the argument type for the varargs array passed in to doInBackground.
  2. Progress is the argument type for the varargs array passed in to onProgressUpdate, and so is also the type (of array) you must use when invoking publishProgress.
  3. Result is the return type of doInBackground, which in turn is the argument type passed in to onPostExecute.

What happens when you execute()?

When execute(Object.. params) is invoked on an AsyncTask the task is executed in a background thread. Depending on the platform AsyncTasks may be executed serially (pre 1.6 and potentially again in 4+), or concurrently (1.6-3.2).

To be sure of running serially or concurrently as you require, from API Level 11 onwards you can use the executeOnExecutor(Executor executor, Object.. params) method instead, and supply an executor. The platform provides two executors for convenience, accessable as AsyncTask.SERIAL_EXECUTOR and AsyncTask.THREAD_POOL_EXECUTOR respectively. (Note: If you are targeting earlier API levels executeOnExecutor is not available, but you have several options - see below).

I have not tested exhaustively, but at least on tablets running HoneyComb the THREAD_POOL_EXECUTOR is set up with a maximum pool size of 128 and an additional queue length of 10.

If you exhaust the pool by submitting too many AsyncTask's concurrently you will receive RejectedExecutionException's - a subclass of RuntimeException which, unless handled, will crash your application.

I suspect that on a resource-constrained device it is probably quite a disaster if you actually have that many AsyncTask's active concurrently - context-switching all those threads will render the cpu-cache ineffective, cost a lot in terms of CPU time, and anyway all those concurrently active threads will likely be using a good chunk of your heap and generating garbage for the GC to contend with.

You might want to consider an alternative Executor configured with a lower max threads and a longer queue, or a more appropriate strategy for managing the background work: for example if you have many files to download you could enqueue a url and a callback to a download-manager instead of executing an AsyncTask for each one.

executeOnExecutor for API levels below 11

AsyncTask gained a nice new method at API level 11 - executeOnExecutor - allowing you some control of the concurrency of your AsyncTask's. If you need to support older API levels you have a choice to make: do you absolutely have to have executeOnExecutor, or do you simply want to use it when it is available, and fall-back to execute otherwise?

The fallback approach

If you want a simple way to take some measure of control where possible, you can subclass AsyncTask, test for the API level at runtime, and invoke the executeOnExecutor method if it is available - something like this:

class MyAsyncTask&lt;Param, Progress, Result> {

    private static final boolean API_LEVEL_11 
        = android.os.Build.VERSION.SDK_INT > 11;

    public void execute(Executor aExecutor, Params... aParams) {     
        if(API_LEVEL_11)
            executeOnExecutor(aExecutor, aParams); 
        else
            super.execute(aParams);
    }

}

I know that at first glance something appears wrong here:

private static final boolean API_LEVEL_11 
    = android.os.Build.VERSION.SDK_INT > 11;

This looks like it will be optimised out by the compiler - a static comparison of a literal integer (11) with what appears to be another static integer (android.os.Build.VERSION.SDKINT), but in fact the upper-case VERSION.SDKINT is slightly misleading - the values in VERSION are extracted at runtime from system properties, so the comparison is not baked in at compile-time.

executeOnExecutor for all API levels

If you insist on having executeOnExecutor available for all API levels you might try this: copy the code for AsyncTask from API level 15, rename it (and make a few small changes as described here), and use that everywhere in place of the SDK version.

AsyncTask and the Activity lifecycle

The Activity lifecycle is well defined and provides template methods which are invoked when critical events occur in the life of an Activity.

AsyncTask's are started by an Activity because it needs some potentially blocking work done off the UI thread, and unless you really really know what you are doing they should live and die with that Activity.

If your AsyncTask retains a reference to the Activity, not cancelling the task when the Activity dies wastes CPU resources on work that cannot update its UI, and creates a memory leak (the Activity and all its View hierarchy will be retained until the task completes).

Don't forget that the Activity is destroyed and re-created even on something as simple as a device orientation change, so if a user rotates their device you will have two copies of your Activity retained until the last AsyncTask completes. In a memory constrained environment this can be a disaster!

For longer operations, e.g. networking, consider whether IntentService or Service are more appropriate to your needs (be aware, though, that Service does not automatically offload work from the main thread!).

For some useful discussion and ideas related to AsyncTask and lifecycle management, see this stackoverflow post.

AsyncTask and good Android citizenship

AsyncTask is in the SDK because it fulfils a common need, but it does not enforce a usage pattern that makes your app a good Android citizen.

In an environment where users switch contexts frequently and quickly (example: receive a phone call while in the midst of writing an email on your phone), it is probably important that your app does not hog resources whilst it is not the current focus.

If, as described above, you've set yourself up to cancel tasks according to the Activity lifecycle methods then you're all set and should not face any issues here.

My book, Asynchronous Android, goes into a lot more detail of how to use AsyncTask, including things like showing progress bars, continuing work across Activity restarts, understanding how AsyncTask works, cancelling running tasks, exception handling, and how to avoid some very common issues that developers face.

AsyncTask is just one of the 7 chapters; the book aims to arm you with a complete toolkit for building smooth, responsive apps by working with the platform to get work done off the main thread.

Comment on this post
Recent Posts
 Older posts...