Re: [Json] First two characters

[Nico Williams <[e-mail]>]

On Fri, Nov 22, 2013 at 07:59:36PM +0100, Bjoern Hoehrmann wrote:
>   Now that string literals can occur at the top level, the statement in
> 8.1 http://tools.ietf.org/html/draft-ietf-json-rfc4627bis-07 "the first
> two characters of a JSON text will always be ASCII characters" is
> incorrect and needs to be changed alongside changing the JSON-text
> grammar.

Good point.  Using 'xx' to mean 0x01..0x7F, XX to mean 0x01..0xFF, YY to
mean 0x80..0xFF, ZZ to mean 0x00..0xFF, and 00 to mean 0x00, we have
these possible BOM-less patterns:

   xx 00 00 00  UTF-32LE
   00 00 00 xx  UTF-32BE
   xx 00 ZZ ZZ  UTF-16LE
         but ZZ bytes cannot both be 00
   00 xx ZZ ZZ  UTF-16BE
         but ZZ bytes cannot both be 00
   xx XX        UTF-8

Simplified to assume sequential checking:

   00 00 00 xx  UTF-32BE
   00 xx        UTF-16BE
   xx 00 00 00  UTF-32LE
   xx 00        UTF-16LE
   xx           UTF-8

Optimized for the common cases:

   xx XX        UTF-8    (XX > 0)
   xx 00 ZZ ZZ  UTF-16LE (ZZ can be any value, but both ZZs can't be 0)
   00 xx        UTF-16BE
   00 00 00 xx  UTF-32BE
   xx 00 00 00  UTF-32LE

Nico
--