Invoking Processes from Java

Invoking an external process from Java appears easy enough but there are sooo many gotchas to watch out for. Typical problems that arise include:

1. Hanging Processes - The invoked process "hangs" and never completes (because it is waiting for input that never comes, or for the output buffer(s) to be drained).
2. Failure to execute - Commands that work fine from the cmdline refuse to run when invoked from Java (because the parameters are passed incorrectly).
3. Mysterious issues in production - Peculiar situations where processes cease to work after running happily for some time (the file-handle quota is exhausted because the IO streams are not being correctly closed).

The first two are irritating, but at least they present themselves immediately and are typically fixed before the code leaves the developer.

The last problem is much more insidious and often only rears its head after some time in production (sometimes this is because it takes time and a significant number of executions before it manifests, other times it is because of differences between the development and production environments).

Lets have a look at the general solution to each of these problems. Later I'll list some code that I've been using to invoke processes safely.

Hanging Processes

Symptoms: When invoked, the process starts but does not complete. Sometimes this may appear to be caused by the input that is being fed to the process (e.g. with input A it works but with input B it does not), which adds to the confusion over why the problem occurs.

Cause: The most common reason for this problem is failing to pump input into the program, and drain output buffers from the program, using separate threads.

If a program is consuming sufficient input via standard-input (stdin), or producing sufficient output via stdout or stderr, the limited buffers available to it will fill up. Until those buffers are drained the process will block on IO to those buffers, so the process is effectively hung.

Solution: When you invoke any process from Java, you must use separate threads to pump data to/from stdin, stdout, and stderr:

// invoke the process, keeping a handle to it for later...
final Process _p = Runtime.getRuntime().exec("some-command-or-other");

// Handle stdout...
new Thread() {
    public void run() {
    try {
            Streams.copy(_p.getInputStream(), System.out);
        } catch (Exception anExc) {
            anExc.printStackTrace();
        }
    }
}.start();

// Handle stderr...
new Thread() {
    public void run() {
    try {
            Streams.copy(_p.getInputStream(), System.out);
        } catch (Exception anExc) {
            anExc.printStackTrace();
        }
    }
}.start();

Correctly pumping data into and out of the std io buffers will keep your processes from hanging.

Failure to communicate

Symptoms: You have a command-line that works perfectly when executed at the shell prompt, but invoking it from Java results in strange errors and, perhaps, complaints about invalid parameters.

Cause: Typically this occurs when you try to pass parameters which include spaces - for example file-names - which you escape or quote at the shell prompt.

Example: Running ImageMagick "convert" to add transparent rounded corners to an icon:

convert -size 72x72 xc:none -fill white -draw \
  'roundRectangle 0,0 72,72 15,15' in.png \
  -compose SrcIn -composite out.png

This command-line works fine at a bash prompt, but if you try to invoke it naively from Java it will likely fail in a variety of interesting ways depending on your platform:

public static void main(String... anArgs) {
    // invoke the process, keeping a handle to it for later...
    final Process _p = Runtime.getRuntime().exec(
        "/usr/bin/convert -size 72x72 xc:none -fill white -draw" +
        " 'roundRectangle 0,0 72,72 15,15' /home/steve/Desktop/in.png" +
        " -compose SrcIn -composite /home/steve/Desktop/out.png"
    );

    // Handle stdout...
    new Thread() {
        public void run() {
            try {
                Streams.copy(_p.getInputStream(), System.out);
            } catch (Exception anExc) {
                anExc.printStackTrace();
            }
        }
    }.start();

    // Handle sderr...
    new Thread() {
        public void run() {
            try {
                Streams.copy(_p.getErrorStream(), System.out);
            } catch (Exception anExc) {
                anExc.printStackTrace();
            }
        }
    }.start();

    // wait for the process to complete
    _p.waitFor();
}

Whilst the command-line worked fine at the bash prompt, running the same command from Java results in an error message!:

convert: non-conforming drawing primitive definition 
    `roundRectangle' @ error/draw.c/DrawImage/3143.
convert: unable to open image `0,0':  @ error/blob.c/OpenBlob/2489.
convert: unable to open image `72,72':  @ error/blob.c/OpenBlob/2489.
convert: unable to open image `15,15'':  @ error/blob.c/OpenBlob/2489.
convert: non-conforming drawing primitive definition 
    `roundRectangle' @ error/draw.c/DrawImage/3143.

What's going on!? Basically the command we gave to Runtime.exec has been sliced up at spaces, ignoring the single quotes, and so ImageMagick has seen a very different command-line to the one we presented via the shell.

Solution: The solution this time is very easy: Use the overloaded Runtime.exec(..) methods that accept the command and the parameters as an array of String's. Re-writing our previous example:

public static void main(String... anArgs) 
throws Exception {
    // invoke the process, keeping a handle to it for later...
    // note that we pass the command and its params as String's in
    // the same String[]
    final Process _p = Runtime.getRuntime().exec(
        new String[]{
            "/usr/bin/convert",
            "-size", "72x72", "xc:none", "-fill", "white", "-draw",
            "roundRectangle 0,0 72,72 15,15", 
            "/home/steve/Desktop/in.png", "-compose", "SrcIn",
            "-composite", "/home/steve/Desktop/out.png"
        }
    );

    // Handle stdout...
    new Thread() {
        public void run() {
            try {
                Streams.copy(_p.getInputStream(), System.out);
            } catch (Exception anExc) {
                anExc.printStackTrace();
            }
        }
    }.start();

    // Handle sderr...
    new Thread() {
        public void run() {
            try {
                Streams.copy(_p.getErrorStream(), System.out);
            } catch (Exception anExc) {
                anExc.printStackTrace();
            }
        }
    }.start();

    // wait for the process to complete
    _p.waitFor();
}

Passing your cmdline parameters in a String array instead of as one long String should prevent your parameters from being chewed up and mis-interpreted.

Mysterious issues in production

Symptoms: For a good while things appear to be working fine. Processes are invoked, do their work, and shut-down. After a while a problem occurs - the processes are no longer being invoked, or hang.

Cause: The cause of this is usually exhaustion of the available file-handles, which in turn is caused by failing to correctly close all of the IO streams opened to handle the process IO.

Solution: Careful closure of all standard IO streams opened by the process and streams opened by you to consume the data from the standard streams opened by the process. Note: That's SIX streams in total, not just the three that you open to deal with stdin, stdout and stderr! I also recommend calling destroy on the Process object.

I may be being over-cautious in closing the process's own std streams, but I have seen many cases where closing these streams solved problems of leaked file-handles. (btw., A handy tool if you're running a *nix is lsof, which lists open file handles).

Here's how I recommend cleaning up after your process completes (this assumes that you did provide input via stdin):

public static void main(String... anArgs) {
    Process _process = null;
    InputStream _in = null;
    OutputStream _out = null;
    OutputStream _err = null;
    try {
        _process = Runtime.getRuntime().exec( ... );
        // ... don't forget to initialise in, out, and error,
        // .... and consume the streams in separate threads!
        _process.waitFor();
    } finally {
        if( _process != null ) {
            close(_process.getErrorStream());
            close(_process.getOutputStream());
            close(_process.getInputStream());
            _process.destroy();
        }
        close(_in);
        close(_out);
        close(_err);
    }
}

private static void close(InputStream anInput) {
    try {
        if (anInput != null) {
            anInput.close();
        }
    } catch (IOException anExc) {
        anExc.printStackTrace();
    }
}

private static void close(OutputStream anOutput) {
    try {
        if (anOutput != null) {
            anOutput.close();
        }
    } catch (IOException anExc) {
        anExc.printStackTrace();
    }
}

These days I usually use some utility classes which I've written to wrap all this stuff up and make life a little easier. You can find them in my sjl.io project at github. There's an example of usage in the test source tree - ExternalProcessTest - which invokes ImageMagick.