Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## outils
Class CharsetToolkit

    java.lang.Object
      outils.CharsetToolkit

    public class CharsetToolkit

extends java.lang.Object

Utility class to guess the encoding of a given byte array. The guess is unfortunately not 100% sure. Especially for 8-bit charsets. It's not possible to know which 8-bit charset is used. Except through statistical analysis. We will then infer that the charset encountered is the same as the default standard charset.

On the other hand, unicode files encoded in UTF-16 (low or big endian) or UTF-8 files with a Byte Order Marker are easy to find. For UTF-8 files with no BOM, if the buffer is wide enough, it's easy to guess.

Tested against a complicated UTF-8 file, Sun's implementation does not render bad UTF-8 constructs as expected by the specification. But with a buffer wide enough, the method guessEncoding() did behave correctly and recognized the UTF-8 charset.

A byte buffer of 4KB or 8KB is sufficient to be able to guess the encoding.

Usage:

     // guess the encoding
     Charset guessedCharset = com.glaforge.i18n.io.CharsetToolkit.guessEncoding(file, 4096);

     // create a reader with the charset we've just discovered
     FileInputStream fis = new FileInputStream(file);
     InputStreamReader isr = new InputStreamReader(fis, guessedCharset);
     BufferedReader br = new BufferedReader(isr);

     // read the file content
     String line;
     while ((line = br.readLine())!= null)
     {
         System.out.println(line);
     }

Date: 18 juil. 2002

Author:
  Guillaume LAFORGE

Field Summary
`private byte[]` | `buffer`
`private java.nio.charset.Charset` | `defaultCharset`
`private boolean` | `enforce8Bit`

Constructor Summary
`CharsetToolkit(byte[] buffer)` Constructor of the `com.glaforge.i18n.io.CharsetToolkit` utility class.
`CharsetToolkit(byte[] buffer, java.nio.charset.Charset defaultCharset)` Constructor of the `com.glaforge.i18n.io.CharsetToolkit` utility class.

Method Summary
`static java.nio.charset.Charset[]` | `getAvailableCharsets()` Retrieves all the available `Charset`s on the platform, among which the default `charset`.
`java.nio.charset.Charset` | `getDefaultCharset()` Retrieves the default Charset
`static java.nio.charset.Charset` | `getDefaultSystemCharset()` Retrieve the default charset of the system.
`boolean` | `getEnforce8Bit()` Gets the enforce8Bit flag, in case we do not want to ever get a US-ASCII encoding.
`java.nio.charset.Charset` | `guessEncoding()` Guess the encoding of the provided buffer.
`static java.nio.charset.Charset` | `guessEncoding(java.io.File f, int bufferLength)`
`static java.nio.charset.Charset` | `guessEncoding(java.io.File f, int bufferLength, java.nio.charset.Charset defaultCharset)`
`private static boolean` | `hasUTF16BEBom(byte[] bom)` Has a Byte Order Marker for UTF-16 Big Endian (utf-16 and ucs-2).
`private static boolean` | `hasUTF16LEBom(byte[] bom)` Has a Byte Order Marker for UTF-16 Low Endian (ucs-2le, ucs-4le, and ucs-16le).
`private static boolean` | `hasUTF8Bom(byte[] bom)` Has a Byte Order Marker for UTF-8 (Used by Microsoft's Notepad and other editors).
`private static boolean` | `isContinuationChar(byte b)` If the byte has the form 10xxxxx, then it's a continuation byte of a multiple byte character;
`private static boolean` | `isFiveBytesSequence(byte b)` If the byte has the form 11110xx, then it's the first byte of a five-bytes sequence character.
`private static boolean` | `isFourBytesSequence(byte b)` If the byte has the form 11110xx, then it's the first byte of a four-bytes sequence character.
`private static boolean` | `isSixBytesSequence(byte b)` If the byte has the form 1110xxx, then it's the first byte of a six-bytes sequence character.
`private static boolean` | `isThreeBytesSequence(byte b)` If the byte has the form 1110xxx, then it's the first byte of a three-bytes sequence character.
`private static boolean` | `isTwoBytesSequence(byte b)` If the byte has the form 110xxxx, then it's the first byte of a two-bytes sequence character.
`static void` | `main(java.lang.String[] args)`
`void` | `setDefaultCharset(java.nio.charset.Charset defaultCharset)` Defines the default `Charset` used in case the buffer represents an 8-bit `Charset`.
`void` | `setEnforce8Bit(boolean enforce)` If US-ASCII is recognized, enforce to return the default encoding, rather than US-ASCII.

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Field Detail

### buffer

    private byte[] buffer

### defaultCharset

    private java.nio.charset.Charset defaultCharset

### enforce8Bit

    private boolean enforce8Bit

Constructor Detail

### CharsetToolkit

    public CharsetToolkit(byte[] buffer)

  Constructor of the `com.glaforge.i18n.io.CharsetToolkit` utility class.

  Parameters:
    `buffer` - the byte buffer of which we want to know the encoding.

### CharsetToolkit

    public CharsetToolkit(byte[] buffer,
                          java.nio.charset.Charset defaultCharset)

  Constructor of the `com.glaforge.i18n.io.CharsetToolkit` utility class.

  Parameters:
    `buffer` - the byte buffer of which we want to know the encoding.
    `defaultCharset` - the default Charset to use in case an 8-bit charset is recognized.

Method Detail

### setDefaultCharset

    public void setDefaultCharset(java.nio.charset.Charset defaultCharset)

  Defines the default `Charset` used in case the buffer represents an 8-bit `Charset`.

  Parameters:
    `defaultCharset` - the default `Charset` to be returned by `guessEncoding()` if an 8-bit `Charset` is encountered.

### setEnforce8Bit

    public void setEnforce8Bit(boolean enforce)

  If US-ASCII is recognized, enforce to return the default encoding, rather than US-ASCII. It might be a file without any special character in the range 128-255, but that may be or become a file encoded with the default `charset` rather than US-ASCII.

  Parameters:
    `enforce` - a boolean specifying the use or not of US-ASCII.

### getEnforce8Bit

    public boolean getEnforce8Bit()

  Gets the enforce8Bit flag, in case we do not want to ever get a US-ASCII encoding.

  Returns:
    a boolean representing the flag of use of US-ASCII.

### getDefaultCharset

    public java.nio.charset.Charset getDefaultCharset()

  Retrieves the default Charset

  Returns:

### guessEncoding

    public java.nio.charset.Charset guessEncoding()

  Guess the encoding of the provided buffer.

  If Byte Order Markers are encountered at the beginning of the buffer, we immidiately return the charset implied by this BOM. Otherwise, the file would not be a human readable text file.

  If there is no BOM, this method tries to discern whether the file is UTF-8 or not. If it is not UTF-8, we assume the encoding is the default system encoding (of course, it might be any 8-bit charset, but usually, an 8-bit charset is the default one).

  It is possible to discern UTF-8 thanks to the pattern of characters with a multi-byte sequence.

       UCS-4 range (hex.)        UTF-8 octet sequence (binary)
       0000 0000-0000 007F       0xxxxxxx
       0000 0080-0000 07FF       110xxxxx 10xxxxxx
       0000 0800-0000 FFFF       1110xxxx 10xxxxxx 10xxxxxx
       0001 0000-001F FFFF       11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
       0020 0000-03FF FFFF       111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
       0400 0000-7FFF FFFF       1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

  With UTF-8, 0xFE and 0xFF never appear.

  Returns:
    the Charset recognized.

### guessEncoding

    public static java.nio.charset.Charset guessEncoding(java.io.File f,
                                                         int bufferLength)
                                                  throws java.io.FileNotFoundException,
                                                         java.io.IOException

  Throws:
    `java.io.FileNotFoundException`
    `java.io.IOException`

### guessEncoding

    public static java.nio.charset.Charset guessEncoding(java.io.File f,
                                                         int bufferLength,
                                                         java.nio.charset.Charset defaultCharset)
                                                  throws java.io.FileNotFoundException,
                                                         java.io.IOException

  Throws:
    `java.io.FileNotFoundException`
    `java.io.IOException`

### isContinuationChar

    private static boolean isContinuationChar(byte b)

  If the byte has the form 10xxxxx, then it's a continuation byte of a multiple byte character;

  Parameters:
    `b` - a byte.
  Returns:
    true if it's a continuation char.

### isTwoBytesSequence

    private static boolean isTwoBytesSequence(byte b)

  If the byte has the form 110xxxx, then it's the first byte of a two-bytes sequence character.

  Parameters:
    `b` - a byte.
  Returns:
    true if it's the first byte of a two-bytes sequence.

### isThreeBytesSequence

    private static boolean isThreeBytesSequence(byte b)

  If the byte has the form 1110xxx, then it's the first byte of a three-bytes sequence character.

  Parameters:
    `b` - a byte.
  Returns:
    true if it's the first byte of a three-bytes sequence.

### isFourBytesSequence

    private static boolean isFourBytesSequence(byte b)

  If the byte has the form 11110xx, then it's the first byte of a four-bytes sequence character.

  Parameters:
    `b` - a byte.
  Returns:
    true if it's the first byte of a four-bytes sequence.

### isFiveBytesSequence

    private static boolean isFiveBytesSequence(byte b)

  If the byte has the form 11110xx, then it's the first byte of a five-bytes sequence character.

  Parameters:
    `b` - a byte.
  Returns:
    true if it's the first byte of a five-bytes sequence.

### isSixBytesSequence

    private static boolean isSixBytesSequence(byte b)

  If the byte has the form 1110xxx, then it's the first byte of a six-bytes sequence character.

  Parameters:
    `b` - a byte.
  Returns:
    true if it's the first byte of a six-bytes sequence.

### getDefaultSystemCharset

    public static java.nio.charset.Charset getDefaultSystemCharset()

  Retrieve the default charset of the system.

  Returns:
    the default `Charset`.

### hasUTF8Bom

    private static boolean hasUTF8Bom(byte[] bom)

  Has a Byte Order Marker for UTF-8 (Used by Microsoft's Notepad and other editors).

  Parameters:
    `bom` - a buffer.
  Returns:
    true if the buffer has a BOM for UTF8.

### hasUTF16LEBom

    private static boolean hasUTF16LEBom(byte[] bom)

  Has a Byte Order Marker for UTF-16 Low Endian (ucs-2le, ucs-4le, and ucs-16le).

  Parameters:
    `bom` - a buffer.
  Returns:
    true if the buffer has a BOM for UTF-16 Low Endian.

### hasUTF16BEBom

    private static boolean hasUTF16BEBom(byte[] bom)

  Has a Byte Order Marker for UTF-16 Big Endian (utf-16 and ucs-2).

  Parameters:
    `bom` - a buffer.
  Returns:
    true if the buffer has a BOM for UTF-16 Big Endian.

### getAvailableCharsets

    public static java.nio.charset.Charset[] getAvailableCharsets()

  Retrieves all the available `Charset`s on the platform, among which the default `charset`.

  Returns:
    an array of `Charset`s.

### main

    public static void main(java.lang.String[] args)
                     throws java.io.FileNotFoundException,
                            java.io.IOException

  Throws:
    `java.io.FileNotFoundException`
    `java.io.IOException`

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD