Java: a rough guide to character encoding

It can be tricky figuring out the difference between character handling code that works and code that just appears to work because testing did not encounter cases that exposed bugs. This is a post about some of the pitfalls of character handling in Java.

Topics:

• Unicode in source files
• Unicode and Java data types
• How long is a (piece of) String?
• Encodings
• Stream encoding
• Lossy conversions
• Unicode byte order marks
• Automatic detection of encoding
• The potential pitfalls of encoding and decoding using the String class
• Notes

I wrote a little bit about Unicode before.

This post might be exhausting, but it isn't exhaustive.

Unicode in source files

Java source files include support for Unicode. There are two common mechanisms for writing code that includes a range of Unicode characters.

One choice is to encode the source files as Unicode, write the characters and inform the compiler at compile time. javac provides the -encoding <encoding> option for this. The downside is that anyone who uses the source file needs to be aware of this. Otherwise, attempting to compile the file on a system that uses a different default encoding will produce unpredictable results. The bad thing is that it might appear to work.

Code saved as UTF-8, as might be written on an Ubuntu machine:

public class PrintCopyright {
  public static void main(String[] args) {
    System.out.println("© Acme, Inc.");
  }
}

1. Compiling the code using the correct encoding:

javac -encoding UTF-8 PrintCopyright.java

2. Simulating a straight compile on Western-locale Windows:

javac -encoding Cp1252 PrintCopyright.java

These compiler settings will produce different outputs; only the first one is correct.

Note: the JDK 1.6 javac compiler will not compile a UTF-8 source file starting with a byte order mark, failing with the error illegal character: \65279. Byte order marks are discussed further down the page.

The other way to handle Unicode in source files is to only use characters that are encoded with the same values in many character encodings and replace other characters with Unicode escape sequences. In the amended source, the copyright sign © is replaced with the escape sequence \u00A9:

public class PrintCopyright {
  public static void main(String[] args) {
    System.out.println("\u00A9 Acme, Inc.");
  }
}

The amended source, saved as UTF-8, will produce the same output whether processed as UTF-8 or Cp1252. The characters used have the same binary values in both encodings.

These escape sequences apply to the whole file, not just String/char literals. This can produce some funny edge cases. Try compiling this code:

public class WillNotCompile {
  // the path c:\udir is interpreted as an escape sequence
  // the escape \u000A is replaced with a new line
}

Unicode support is detailed in the Lexical Structure chapter of the Java Language Specification.

Unicode and Java data types

Before tackling the encoding API, it is a good idea to get a handle on how text is represented in Java strings.

Grapheme	Unicode Character Name(s)	Unicode Code Point(s)	Java char Literals	UTF-16BE (encoded bytes)	UTF-8 (encoded bytes)
A	LATIN_CAPITAL_LETTER_A	U+0041	'\u0041'	0041	41
©	COPYRIGHT_SIGN	U+00A9	'\u00A9'	00A9	C2A9
é	LATIN_SMALL_LETTER_E_WITH_ACUTE	U+00E9	'\u00E9'	00E9	C3A9
é	LATIN_SMALL_LETTER_E COMBINING_ACUTE_ACENT	U+0065 U+0301	'\u0065' '\u0301'	0065 0301	65 CC81
क्तु	DEVENAGARI_LETTER_KA DEVENAGARI_SIGN_VIRAMA DEVENAGARI_LETTER_TA DEVENAGARI_VOWEL_SIGN_U	U+0915 U+094D U+0924 U+0941	'\u0915' '\u094D' '\u0924' '\u0941'	0915 094D 0924 0941	E0A495 E0A58D E0A4A4 E0A581
𝔊	MATHEMATICAL_FRAKTUR_CAPITAL_G	U+1D50A	'\uD835' '\uDD0A'	D835DD0A	F09D948A
All values are hexadecimal. Some of the graphemes might not render (though they all seem to work on Firefox).
Table 1

The table above shows some of the things we have to look out for.

1. Stored characters can take up an inconsistent number of bytes. A UTF-8 encoded character might take between one (LATIN_CAPITAL_LETTER_A) and four (MATHEMATICAL_FRAKTUR_CAPITAL_G) bytes. Variable width encoding has implications for reading into and decoding from byte arrays.

2. Characters can be represented in multiple forms. As can be seen with e-acute (é), sometimes there is more than one way to store and render a grapheme. Sometimes they can be formed using combining sequences (as in the e-acute example); sometimes there are similar characters (Greek mu μ vs Mathematical micro µ). This may have implications for sorting, expression matching and capitalisation. It might raise compatibility issues for translating data between encodings. You can normalise strings using the Normalizer class, but be aware of any gotchas.

3. Not all code points can be stored in a char. The MATHEMATICAL_FRAKTUR_CAPITAL_G example lies in the supplementary range of characters and cannot be stored in 16 bits. It must be represented by two sequential char values, neither of which is meaningful by itself. The Character class provides methods for working with 32-bit code points.

// Unicode code point to char array     char[] math_fraktur_cap_g = Character.toChars(0x1D50A);

4. The relationship between the grapheme visible to the user and the code point type may not be 1:1. This can be seen in the combining character sequences (the e-acute example). As the Devenagari example shows, combining sequences can get quite complex.

How long is a (piece of) String?

So, if a "character" can span multiple char values, we need to know how that affects our ability to calculate string lengths.

• String.length() returns the number of chars in the String.
• String.codePointCount(int, int) returns the number of Unicode code points in the String.
• BreakIterator.getCharacterInstance() can be used to count the number of graphemes in a String.

The char count is likely to be most useful for low level data operations; the grapheme count is the one you would use to do your own line wrapping in a GUI. You can find out if a particular font can render your string using the Font class.

Code to get the various lengths:

/**    * @param type    *          the type of string length    * @param text    *          non-null text to inspect    * @return the length of the string according to the criteria    */   public static int getLength(CountType type, String text) {     switch (type) {     case CHARS:       return text.length();     case CODEPOINTS:       return text.codePointCount(0, text.length());     case GRAPHEMES:       int graphemeCount = 0;       BreakIterator graphemeCounter = BreakIterator           .getCharacterInstance();       graphemeCounter.setText(text);       while (graphemeCounter.next() != BreakIterator.DONE) {         graphemeCount++;       }       return graphemeCount;     default:       throw new IllegalArgumentException("" + type);     }   }   enum CountType {     CHARS, CODEPOINTS, GRAPHEMES   }

Sample strings and their lengths by the various criteria:

Unicode code points	graphemes	char count	code point count	grapheme count
U+0041	A	1	1	1
U+00E9	é	1	1	1
U+0065 U+0301	é	2	2	1
U+0915 U+094D U+0924 U+0941	क्तु	4	4	1
U+1D50A	𝔊	2	1	1
all of the above	Aééक्तु𝔊	10	9	5
Table 2

Encodings

Character encodings map characters to byte representations. The Unicode character set is mapped to bytes using Unicode transformation formats (UTF-8, UTF-16, UTF-32, etc.). Most encodings can represent only a subset of the characters supported by Unicode. Java strings use UTF-16.

Code that prints the byte representation of the pound sign (£) in different encodings:

String[] encodings = { "Cp1252", // Windows-1252         "UTF-8", // Unicode UTF-8         "UTF-16BE" // Unicode UTF-16, big endian     };     String poundSign = "\u00A3";     for (String encoding : encodings) {       System.out.format("%10s%3s  ", encoding, poundSign);       byte[] encoded = poundSign.getBytes(encoding);       for (byte b : encoded) {         System.out.format("%02x ", b);       }       System.out.println();     }

Output of the code:

    Cp1252  £  a3
     UTF8   £  c2 a3
  UTF-16BE  £  00 a3

Java uses two mechanisms to represent supported encodings. The initial mechanism was via string IDs. Java 1.4 introduce the type-safe Charset class. Note that the two mechanisms use different canonical names to represent the same encodings.

Java 6 implementations are only required to support six encodings (US-ASCII; ISO-8859-1; UTF-8; UTF-16BE; UTF-16LE; UTF-16). In practice, they tend to support many more:

• Sun Java 6 Supported Encodings
• Sun Java 5 Supported Encodings
• Sun Java 1.4 Supported Encodings

Other JREs (e.g. IBM; Oracle JRockit) may support different sets of encodings.

Every platform has a default encoding:

Charset charset = Charset.defaultCharset();     System.out.println("Default encoding: " + charset + " (Aliases: "         + charset.aliases() + ")");

If a standard Java API converts between byte and char data, there is a high probability that it is using the default encoding. Examples of such calls are: String(byte[]); String.getBytes(); InputStreamReader(InputStream); OutputStreamWriter(OutputStream); anything in FileReader or FileWriter.

The problem with using these calls is that it is that you cannot predict whether data written on one machine will be read correctly on another, even if you use the same application. An English Windows PC will use Windows-1252; a Russian Windows PC will use Windows-1251; an Ubuntu machine will use UTF-8. Prefer encoding methods like OutputStreamWriter(OutputStream, Charset) that let you set the encoding explicitly. Fall back on the default encoding only when you have no choice.

Portability is not the only problem with relying on the default encoding. As will be shown, using the old 8-bit encodings that many systems use for compatibility reasons can also result in data loss.

Encoding operations transform data. Do not try to pass "binary" data through character methods. A Java char is not a C char.

UPDATE (Dec 2009): A quick note on the file.encoding system property, which you may see mentioned in Java encoding discussions. Do not set or rely on this property. Sun says:

The "file.encoding" property is not required by the J2SE platform specification; it's an internal detail of Sun's implementations and should not be examined or modified by user code. It's also intended to be read-only; it's technically impossible to support the setting of this property to arbitrary values on the command line or at any other time during program execution.

Stream encoding

This code reliably writes and then reads character data using a file:

public static void writeToFile(File file,       Charset charset, String data) throws IOException {     OutputStream out = new FileOutputStream(file);     Closeable stream = out;     try {       Writer writer = new OutputStreamWriter(out, charset);       stream = writer;       writer.write(data);     } finally {       stream.close();     }   }   public static String readFromFile(File file,       Charset charset) throws IOException {     InputStream in = new FileInputStream(file);     Closeable stream = in;     try {       Reader reader = new InputStreamReader(in, charset);       stream = reader;       StringBuilder inputBuilder = new StringBuilder();       char[] buffer = new char[1024];       while (true) {         int readCount = reader.read(buffer);         if (readCount < 0) {           break;         }         inputBuilder.append(buffer, 0, readCount);       }       return inputBuilder.toString();     } finally {       stream.close();     }   }   public static void main(String[] args) {     String mathematical_fraktur_capital_g = "\uD835\uDD0A";     // use a Unicode encoding     Charset utf8 = Charset.forName("UTF-8");     File file = new File("test.txt");     // write the file     try {       writeToFile(file, utf8,           mathematical_fraktur_capital_g);       String input = readFromFile(file, utf8);       if (input.equals(mathematical_fraktur_capital_g)) {         System.out.println("OK");       } else {         System.err.println("DATA LOSS ERROR");       }     } catch (IOException e) {       System.err.println("IO ERROR");       e.printStackTrace();     }   }

The same encoding is used for both operations, so they are symmetrical. A Unicode encoding is used, so no data loss will occur.

One potential problem area is wrapping a stream with a writer when there data has already been written (or when appending data to a file). Be careful of byte order marks. This is an identical problem to the one discussed below using string encoding calls.

Lossy conversions

Saving to the operating system's default encoding can be a lossy process. This code simulates what happens when the mathematical expression x ≠ y is saved and loaded on an English Windows system using its default file encoding:

// \u2260 NOT EQUAL TO     String mathematicalExpression = "x \u2260 y";     // English Windows charset     Charset windows1252 = Charset.forName("windows-1252");     // Simulate save/load     byte[] encoded = mathematicalExpression         .getBytes(windows1252);     String decoded = new String(encoded, windows1252);     // exception will be thrown:     if (!mathematicalExpression.equals(decoded)) {       throw new Exception("Information was lost");     }

The symbol NOT_EQUAL_TO (≠) is replaced with a question mark during encoding. The Windows-1252 character set does not include that character.

Use non-Unicode encodings reluctantly. Prefer self-describing file formats that support Unicode (like XML [spec]) or formats that mandate Unicode (like JSON [spec]).

Note: normalisation and encoding are distinct operations; the combining sequence U+0065 U+0301 (é) will (using windows-1252) be encoded to the bytes 65 3F (e?). The API will not normalise text automatically.

Unicode byte order marks

Byte order marks (BOMs) are a sequence of bytes that are sometimes used to mark the start of the data with information about the encoding in use. Sometimes Unicode BOMs are mandatory; sometimes they must not be used. The Unicode BOM FAQ contains more details, including a section on how to deal with them.

This code generates the byte order marks for selected Unicode encodings:

// Encode this to get byte order mark     final String bomChar = "\uFEFF";     // Unicode encodings     String[] unicodeEncodings = { "UTF-8", "UTF-16BE", "UTF-16LE", "UTF-32BE",         "UTF-32LE" };     // Print the byte order marks     for (String encName : unicodeEncodings) {       Charset charset = Charset.forName(encName);       byte[] byteOrderMark = bomChar.getBytes(charset);       System.out.format("%10s BOM: ", charset.toString());       for (byte b : byteOrderMark) {         System.out.format("%02x ", b);       }       System.out.println();

The generated output:

     UTF-8 BOM: ef bb bf 
  UTF-16BE BOM: fe ff 
  UTF-16LE BOM: ff fe 
  UTF-32BE BOM: 00 00 fe ff 
  UTF-32LE BOM: ff fe 00 00 

Some encodings will automatically emit byte order marks on encode and read them on decode. Select the encoding scheme appropriate for your use case.

This code inserts a BOM automatically:

Charset charset = Charset.forName("UTF-16");     byte[] encodedBytes = "hello".getBytes(charset);     System.out.format("%10s encoded: ", charset.toString());     for (byte b : encodedBytes) {       System.out.format("%02x ", b);     }     System.out.println();

The output, prefixed with the fe ff big endian BOM:

    UTF-16 encoded: fe ff 00 68 00 65 00 6c 00 6c 00 6f 

Automatic detection of encoding

Automatic detection of encoding has been tried (notably in Windows), but there is no way to get it to work with one hundred percent reliability for an arbitrary text file. Notepad detects the windows-1252 encoded string this app can break as being CJK characters.

The International Components for Unicode project includes support for character set detection.

byte[] thisAppCanBreak = "this app can break"         .getBytes("ISO-8859-1");     CharsetDetector detector = new CharsetDetector();     detector.setText(thisAppCanBreak);     String tableTemplate = "%10s %10s %8s%n";     System.out.format(tableTemplate, "CONFIDENCE",         "CHARSET", "LANGUAGE");     for (CharsetMatch match : detector.detectAll()) {       System.out.format(tableTemplate, match           .getConfidence(), match.getName(), match           .getLanguage());     }

Output:

CONFIDENCE    CHARSET LANGUAGE
        63 ISO-8859-2       hu
        47 ISO-8859-9       tr
        47 ISO-8859-1       nl
        47 ISO-8859-1       en
        31 ISO-8859-2       ro
        31 ISO-8859-1       fr
        15 ISO-8859-1       sv
        15 ISO-8859-1       pt
        15 ISO-8859-1       es
        15 ISO-8859-1       da
        10       Big5       zh
        10     EUC-KR       ko
        10     EUC-JP       ja
        10    GB18030       zh
        10  Shift_JIS       ja
        10      UTF-8     null

The chosen text will decode correctly if interpreted as ISO-8859-2 Hungarian, so it looks like we will have to try harder in the selection of data that will cause character corruption. As the ICU project notes about detection:

This is, at best, an imprecise operation using statistics and heuristics.

The potential pitfalls of encoding and decoding using the String class

The java.lang.String class can be used to encode and decode data, though be aware of problem areas.

This code demonstrates how attempting to use string methods to decode multibyte-encoded text from a stream can lead to character corruption:

/** buggy decode method - do not use! */   private static String decodeFromStream(       InputStream stream, Charset encoding)       throws IOException {     StringBuilder builder = new StringBuilder();     byte[] buffer = new byte[4];     while (true) {       // read bytes into buffer       int r = stream.read(buffer);       if (r < 0) {         break;       }       // decode byte data into char data       String data = new String(buffer, 0, r, encoding);       builder.append(data);     }     return builder.toString();   }   public static void main(String[] args) throws IOException {     Charset utf8 = Charset.forName("UTF-8");     String original = "abc\u00A3";     byte[] encoded = original.getBytes(utf8);     // create mock file input stream     InputStream stream = new ByteArrayInputStream(encoded);     // decode the data     String decoded = decodeFromStream(stream, utf8);     if (!original.equals(decoded)) {       throw new IllegalStateException("Lost data");     }   }

The flaw in this code is that characters in the stream can span reads into the byte buffer. The character U+00A3 (£) becomes corrupted as one half of it ends up in the tail end of the buffer in one pass and the other half ends up at the start of the buffer in the next pass. Neither value is meaningful by itself. Because a UTF-8 encoded character may be between one and four bytes in length, there is no buffer size that will always read whole characters. The entire data would need to be read into a larger byte buffer and decoded in one go. It is safer to use an InputStreamReader.

This code demonstrates how data corruption can occur when encoding text incrementally:

/** buggy encoding method - do not use! */   private static void writeToStream(OutputStream out,       String text, Charset encoding) throws IOException {     byte[] encoded = text.getBytes(encoding);     out.write(encoded);   }   public static void main(String[] args) throws IOException {     Charset utf16 = Charset.forName("UTF-16");     String[] data = { "abc", "def" };     String concatenated = "";     // buffer standing in for a file     ByteArrayOutputStream buffer = new ByteArrayOutputStream();     for (String segment : data) {       concatenated += segment;       // encode and write the segment to a simulated file       writeToStream(buffer, segment, utf16);     }     byte[] encoded = buffer.toByteArray();     String decoded = new String(encoded, utf16);     if (!concatenated.equals(decoded)) {       throw new IllegalStateException("Data corrupted");     }   }

The problem here is that the UTF-16 encoding scheme used adds a byte order mark to encoded data. Every time a string is written, another BOM is added, littering the content with unwanted data. When the data is decoded, extra characters end up in the text. All text would need to be concatenated and encoded in one go. It is better to use an OutputStreamWriter.

There are encodings for which these operations are safe, but you would be tying your method implementations to specific encodings rather than making them general. Use string class encode/decode methods only on whole data.

Notes

System.out may not always print the results you expect. I posted about this in terms of the Windows console, but Windows is not the only system with this sort of behaviour.

If you feel you are on rocky ground with character encodings, Bob Balaban has a good presentation: Multi-Language Character Sets: What They Are, How to Use Them (2001; PDF) [alt link]. He worked on character issues for more than a little while. Chances are that you are not a Lotus Notes developer, so you can skip some of the specifics.

The ICU4J library is a good place to turn if you you're trying to find Unicode (and other I18N) features not included in the Java standard library.

Examples were developed using Java 6.