Thursday, 2 October 2008

Java: how not to make a mess of stream handling

This article was written with Java 6 in mind.
Updated 2008/12/16.

  /**
   * Writes "Hello, World!" to a file.
   */
  public static void main(String[] args) {
    try {
      byte[] data = "Hello, World!".getBytes("UTF-8");
      OutputStream out = new FileOutputStream("output.txt");
      //if write throws an error
      out.write(data);
      //then close will never be called! BUG!
      out.close();
    } catch(IOException e) {
      System.err.println("ERROR");
      e.printStackTrace();
    }
  }

If you aren't careful with streams in Java, you end up with resource leaks. This article addresses some of the pitfalls of stream handling.

The examples that follow use implementations of OutputStream. Which output stream is not important - implementation details are encapsulated by the object and may change between Java versions. It is the developer's responsibility to honour the contract as per the documentation.

OutputStream.close(): Closes this output stream and releases any system resources associated with this stream. The general contract of close is that it closes the output stream. A closed stream cannot perform output operations and cannot be reopened.

New Java developers pick up pretty quickly that they have to close streams. Problems arise when they start thinking about how to do it.

Sections
  • Imperfect, But A Good Default
  • How Not To Close A Stream
  • How To Get The First Exception
  • How To Get The First Exception (Round 2)
  • When One Exception Is Not Enough
  • Getting All The Exceptions
  • What Is The Best Way?
  • Source Code

Imperfect, But A Good Default

  /**
   * Writes data to a stream
   */
  public void sendToStream(byte[] data) throws IOException {
    OutputStream stream = openStream();
    try {
      stream.write(data);
    } finally {
      stream.close();
    }
  }

The above code is a good default: the method always informs the caller that an exception has been raised and always closes the stream. It has one problem: if both write and close throw exceptions, the second exception will be passed up the stack even though it is likely that the first was the root cause. Compile and run DefaultDemo below to see the problem in action.

public class DefaultDemo {

  /**
   * return a stream that throws an exception on 
   * output and close
   */
  private OutputStream openStream() {
    return new OutputStream() {
      @Override
      public void write(int b) throws IOException {
        System.out.println("Writing: " + b);
        System.out.println("Throwing Exception");
        throw new IOException();
      }

      @Override
      public void close() throws IOException {
        System.out.println("Stream closed");
        System.out.println("Throwing Exception");
        throw new IOException();
      }
    };
  }

  /**
   * Writes data to a stream
   */
  public void sendToStream(byte[] data) throws IOException {
    OutputStream stream = openStream();
    try {
      stream.write(data);
    } finally {
      stream.close();
    }
  }

  public static void main(String[] args) {
    byte[] data = { 1 };
    DefaultDemo handler = new DefaultDemo();
    try {
      handler.sendToStream(data);
      System.out.println("OK");
    } catch (IOException e) {
      System.err.println("ERROR");
      e.printStackTrace();
    }
  }

}

Output from DefaultDemo:

Writing: 1
Throwing Exception
Stream closed
Throwing Exception
ERROR
java.io.IOException
 at DefaultDemo$1.close(DefaultDemo.java:23)
 at DefaultDemo.sendToStream(DefaultDemo.java:36)
 at DefaultDemo.main(DefaultDemo.java:44)

The exception received by the caller is the one from close and it is not apparent from the stack trace that write threw an exception. The exception from close is likely to be related to the one from write, so this may not be a big problem.

How Not To Close A Stream

A common "solution" to the problem is seen in this code. (Inexperienced programmers will often try to fit another catch block in there too.)

  /**
   * Writes data to a stream
   */
  public void sendToStream(byte[] data) throws IOException {
    OutputStream stream = null;
    try {
      stream = openStream();
      stream.write(data);
    } finally {
      if(stream != null) {
        try {
          stream.close();
        } catch(IOException ignored) {
          //failsafe
        }
      }
    }
  }

The idea behind this approach is that if write throws an exception, the stream will be closed and the exception will not be hidden. Unfortunately, this method completely ignores exceptions from close as BadDemo demonstrates.

public class BadDemo {

  /**
   * return a stream that throws an exception on close
   */
  private OutputStream openStream() {
    return new OutputStream() {
      @Override
      public void write(int b) throws IOException {
        System.out.println("Writing: " + b);
      }

      @Override
      public void close() throws IOException {
        System.out.println("Stream closed");
        System.out.println("Throwing Exception");
        throw new IOException();
      }
    };
  }

  /**
   * Writes data to a stream
   */
  public void sendToStream(byte[] data) throws IOException {
    OutputStream stream = null;
    try {
      stream = openStream();
      stream.write(data);
    } finally {
      if(stream != null) {
        try {
          stream.close();
        } catch(IOException ignored) {
          //failsafe
        }
      }
    }
  }

  public static void main(String[] args) {
    byte[] data = { 1 };
    BadDemo handler = new BadDemo();
    try {
      handler.sendToStream(data);
      System.out.println("OK");
    } catch (IOException e) {
      System.err.println("ERROR");
      e.printStackTrace();
    }
  }

}

Output from BadDemo:

Writing: 1
Stream closed
Throwing Exception
OK

An exception was thrown by close, but the program doesn't detect an error. If it wasn't for the System.outs, we wouldn't know an exception had been thrown. Some developers cotton on to this an put some error logging in the catch block - this notifies you (if you read the logs), but your program is still not dealing with the error.

You might be tempted to think that the only reason close would throw and exception would be because the resource couldn't be released - unfortunate, but not your problem. This would be a mistake - plenty of streams write data in their close blocks, BufferedOutputStream (inheriting from FilterOutputStream) and CipherOutputStream being two documented examples. If you don't handle close properly, you might commit incomplete data to the stream and not know it.

How To Get The First Exception

It is possible to get the first exception, though the following code is inelegant.

public class FirstExceptionDemo {

  /**
   * return a stream that throws an exception on output and close
   */
  private OutputStream openStream() {
    return new OutputStream() {
      @Override
      public void write(int b) throws IOException {
        System.out.println("Writing: " + b);
        System.out.println("Throwing Exception");
        throw new IOException();
      }

      @Override
      public void close() throws IOException {
        System.out.println("Stream closed");
        System.out.println("Throwing Exception");
        throw new IOException();
      }
    };
  }

  /**
   * Writes data to a stream
   */
  public void sendToStream(byte[] data) throws IOException {
    IOException firstError = null;

    OutputStream stream = openStream();
    try {
      stream.write(data);
    } catch (IOException e) {
      firstError = e;
    } finally {
      try {
        stream.close();
      } catch (IOException e) {
        if (firstError == null) {
          firstError = e;
        }
      }
    }
    
    if (firstError != null) {
      throw firstError;
    }
  }

  public static void main(String[] args) {
    byte[] data = { 1 };
    FirstExceptionDemo handler = new FirstExceptionDemo();
    try {
      handler.sendToStream(data);
      System.out.println("OK");
    } catch (IOException e) {
      System.err.println("ERROR");
      e.printStackTrace();
    }
  }

}

Output from FirstExceptionDemo:

Writing: 1
Throwing Exception
Stream closed
Throwing Exception
ERROR
java.io.IOException
 at FirstExceptionDemo$1.write(FirstExceptionDemo.java:15)
 at java.io.OutputStream.write(Unknown Source)
 at java.io.OutputStream.write(Unknown Source)
 at FirstExceptionDemo.sendToStream(FirstExceptionDemo.java:35)
 at FirstExceptionDemo.main(FirstExceptionDemo.java:57)

This code gives us what is probably the primary cause of the errors, but if we added an input stream the code would get difficult to follow and unpleasant to maintain.

How To Get The First Exception (Round 2)

A less verbose way to get the first exception is shown below.

public class FirstExceptionDemo2 {

  /**
   * return a stream that throws an exception on output and close
   */
  private OutputStream openStream() {
    return new OutputStream() {
      @Override
      public void write(int b) throws IOException {
        System.out.println("Writing: " + b);
        System.out.println("Throwing Exception");
        throw new IOException();
      }

      @Override
      public void close() throws IOException {
        System.out.println("Stream closed");
        System.out.println("Throwing Exception");
        throw new IOException();
      }
    };
  }
  
  /**
   * Writes data to a stream
   */
  public void sendToStream(byte[] data) throws IOException {
    OutputStream stream = openStream();
    try {
      stream.write(data);
      stream.close();
    } finally {
      closeSilently(stream);
    }
  }

  private void closeSilently(Closeable c) {
    try {
      c.close();
    } catch(IOException e) {
      //swallow error
    }
  }
  
  public static void main(String[] args) {
    byte[] data = { 1 };
    FirstExceptionDemo2 handler = new FirstExceptionDemo2();
    try {
      handler.sendToStream(data);
      System.out.println("OK");
    } catch (IOException e) {
      System.err.println("ERROR");
      e.printStackTrace();
    }
  }

}

This code relies on being allowed to call the stream close method twice. If an exception is thrown by a stream operation, the exception is passed up the stack and any exceptions from close are swallowed. If the stream operations succeed and closing the stream throws an exception, it is not swallowed silently because close is called in the try block.

Output from FirstExceptionDemo2:

Writing: 1
Throwing Exception
Stream closed
Throwing Exception
ERROR
java.io.IOException
 at streamdemos.FirstExceptionDemo2$1.write(FirstExceptionDemo2.java:47)
 at java.io.OutputStream.write(OutputStream.java:99)
 at java.io.OutputStream.write(OutputStream.java:58)
 at streamdemos.FirstExceptionDemo2.sendToStream(FirstExceptionDemo2.java:65)
 at streamdemos.FirstExceptionDemo2.main(FirstExceptionDemo2.java:84)

Here, the first exception (from write) is printed to the console.

If the code is modified so that write does not throw an exception, the close exception is reported:

Stream closed
Throwing Exception
Stream closed
Throwing Exception
ERROR
java.io.IOException
 at streamdemos.FirstExceptionDemo2$1.close(FirstExceptionDemo2.java:51)
 at streamdemos.FirstExceptionDemo2.sendToStream(FirstExceptionDemo2.java:63)
 at streamdemos.FirstExceptionDemo2.main(FirstExceptionDemo2.java:81)

Calling close twice should be a safe operation - stream wrappers close their wrapped streams and the code that opens a stream is ultimately responsible for closing it.

When One Exception Is Not Enough

To get all the relevant error messages, you need to aggregate them somehow. This IOException class is an incomplete implementation (this post is long enough already) of a class that does this.

public class CompositeIOException extends IOException {

  private static final long serialVersionUID = 1L;

  private List<Throwable> causes;

  public CompositeIOException(List<Throwable> causes) {
    if (causes == null) {
      throw new NullPointerException();
    }
    if (causes.size() < 1) {
      throw new IllegalArgumentException();
    }
    this.causes = new ArrayList<Throwable>(causes);
  }

  @Override
  public void printStackTrace() {
    super.printStackTrace();
    System.err.println("CAUSED BY:");
    for (Throwable e : causes) {
      e.printStackTrace();
    }
  }

}

Getting All The Exceptions

To minimise the messy code used in "How To Get The First Exception", it is best to come up with a generalised solution. The I/O task handler listed below ensures that no exceptions are hidden by exceptions from Closeable objects.

public abstract class IOTaskHandler {

  private List<Throwable> errorList = null;

  private boolean runtimeErrors = false;

  private List<Closeable> resources = null;

  private boolean hasRun = false;

  private List<Throwable> getErrorList() {
    if (errorList == null) {
      errorList = new ArrayList<Throwable>();
    }
    return errorList;
  }

  private List<Closeable> getResourceList() {
    if (resources == null) {
      resources = new ArrayList<Closeable>();
    }
    return resources;
  }

  /**
   * Callers should invoke the handler via this method.
   */
  public final void run() throws IOException {
    if (hasRun) {
      throw new IllegalStateException("Closed");
    }
    hasRun = true;

    try {
      runCallbackAndClose();
    } finally {
      if (errorList != null) {
        if (runtimeErrors) {
          throw new CompositeRuntimeException(errorList);
        } else {
          throw new CompositeIOException(errorList);
        }

      }
    }
  }

  private void runCallbackAndClose() {
    try {
      // perform the task
      doCallback();
    } catch (IOException e) {
      getErrorList().add(e);
    } catch (RuntimeException e) {
      runtimeErrors = true;
      getErrorList().add(e);
    } finally {
      // do cleanup resources
      close();
    }
  }

  private void close() {
    if (resources == null) {
      return;
    }

    for (Closeable resource : resources) {
      try {
        resource.close();
      } catch (IOException e) {
        getErrorList().add(e);
      } catch (RuntimeException e) {
        runtimeErrors = true;
        getErrorList().add(e);
      }
    }
  }

  /**
   * All streams should be registered via this method. Streams will be closed
   * in reverse order to the one they were added.
   */
  protected void registerResource(Closeable resource) {
    if (resource == null) {
      throw new NullPointerException();
    }
    getResourceList().add(0, resource);
  }

  /**
   * I/O operations should be implemented in this method. This method should
   * only be invoked by IOTaskHandler.
   */
  protected abstract void doCallback() throws IOException;

}

All an implementer has to do is wrap I/O calls in the doCallback() method and register any streams with the handler. Streams should be registered with the handler in the call after they have been created to ensure they get closed. The handler will take care of closing streams and dealing with exceptions. Its usage can be seen below where IOTaskHandler is extended to implement an I/O task. You could easily use an anonymous class instead - extending it is clearer for demonstration purposes.

public class AllExceptionsDemo extends IOTaskHandler {

  private byte[] data;

  public AllExceptionsDemo(byte[] data) {
    this.data = data;
  }

  /**
   * return a stream that throws an exception on output and close
   */
  private OutputStream openStream() {
    return new OutputStream() {
      @Override
      public void write(int b) throws IOException {
        throw new RuntimeException("out:write - not even an IOException");
      }

      @Override
      public void close() throws IOException {
        throw new IOException("out:close");
      }
    };
  }

  @Override
  protected void doCallback() throws IOException {
    OutputStream stream = openStream();
    registerResource(stream);

    stream.write(data);
  }

  public static void main(String[] args) {
    byte[] data = { 1 };
    AllExceptionsDemo handler = new AllExceptionsDemo(data);
    try {
      handler.run();
      System.out.println("OK");
    } catch (IOException e) {
      System.err.println("ERROR");
      e.printStackTrace();
    } catch (RuntimeException e) {
      System.err.println("ERROR");
      e.printStackTrace();
    }
  }

}

Output from AllExceptionsDemo:

ERROR
streamdemos.io.CompositeRuntimeException
 at streamdemos.io.IOTaskHandler.run(IOTaskHandler.java:73)
 at streamdemos.AllExceptionsDemo.main(AllExceptionsDemo.java:73)
CAUSED BY:
java.lang.RuntimeException: out:write - not even an IOException
 at streamdemos.AllExceptionsDemo$1.write(AllExceptionsDemo.java:51)
 at java.io.OutputStream.write(OutputStream.java:99)
 at java.io.OutputStream.write(OutputStream.java:58)
 at streamdemos.AllExceptionsDemo.doCallback(AllExceptionsDemo.java:66)
 at streamdemos.io.IOTaskHandler.runCallbackAndClose(IOTaskHandler.java:85)
 at streamdemos.io.IOTaskHandler.run(IOTaskHandler.java:69)
 at streamdemos.AllExceptionsDemo.main(AllExceptionsDemo.java:73)
java.io.IOException: out:close
 at streamdemos.AllExceptionsDemo$1.close(AllExceptionsDemo.java:56)
 at streamdemos.io.IOTaskHandler.close(IOTaskHandler.java:104)
 at streamdemos.io.IOTaskHandler.runCallbackAndClose(IOTaskHandler.java:93)
 at streamdemos.io.IOTaskHandler.run(IOTaskHandler.java:69)
 at streamdemos.AllExceptionsDemo.main(AllExceptionsDemo.java:73)

As you can see in the above listing, the application reports all the exceptions encountered. Decisions need to be made in the IOHandler implementation, like whether the code should handle RuntimeExceptions. Catching all types of Exception or Throwable should normally be avoided. Distinctions need to be made between checked exceptions (which the caller may recover from) and runtime exceptions (indicating a programming error).

Although complicated, this type of error handling provides information that is normally hidden from you.

What Is The Best Way?

So, is it worth doing all this? The answer depends on the use case. I follow this procedure:

  • Use the code described in Imperfect, But A Good Default.
  • If an exception hiding bug is discovered, use code like that described in How To Get The First Exception (Round 2).
  • Under exceptional circumstances, use a class like the IOHandler described in Getting All The Exceptions.

Perhaps the automatic resource clean up proposal for Java 7 will alleviate the situation.

Source Code

All the sources are available in a public Subversion repository.

Repository: http://illegalargumentexception.googlecode.com/svn/trunk/code/java/
License: MIT
Project: StreamExceptionHandlingDemos

Posted by McDowell at Thursday, October 02, 2008
Labels: , ,

4 comments:

  1. Dave L14 June 2009 at 02:57

    Thanks for the excellent discussion and code. Here's one more alternative that always gets the first exception and calls close only once (unfortunately, I don't know how to make the code well formatted in a comment):

    public void sendToStream(byte[] data) throws IOException {
    OutputStream stream = openStream();
    try {
    stream.write(data);
    } catch (IOException e) {
    closeSilently(stream);
    throw e;
    }
    stream.close();
    }

    ReplyDelete
  2. Dave L14 June 2009 at 04:11

    Oops, the above doesn't handle unchecked exceptions in write.

    ReplyDelete
  3. Robert Frunzke30 January 2013 at 01:49

    Thanks, this a great analysis of the problem.

    But, would it be fair to offer this case as proof of superiority of RAII-capable languages?

    Since some days I am working on a Java project, and while striving for the most perfect solution, of course I coded something similar to your BadDemo. Leaking resources seems to be a huge problem in java, does this explain the typical recurrent issues I have with various java software (locked files, open pipes, sockets,...) ?

    With RAII it would all be so much simpler, but anyway, I am bound to Java 6 in my project.

    So, thanks a lot for your article.

    ReplyDelete
  4. McDowell30 January 2013 at 09:36

    I can only speculate on the problems your application is experiencing, but bad resource cleanup is a possibility. Consider using FindBugs to help you detect potential problems.

    RAII is nice though care is still required with destructors and exception handling.

    Java 7 introduces try-with-resources to make deallocation neater. The precise behaviour is described in the Java Language Specification.

    ReplyDelete

All comments are moderated

Subscribe to: Post Comments (Atom)