Re: [Json] Call for Consensus: Proposed Text for "8.1 Character Encoding"

[Peter Cordell <[e-mail]>]

On 13/03/2017 22:44, Peter Cordell wrote:
> On 13/03/2017 21:50, Julian Reschke wrote:
>> I personally would prefer the table that I posted, as directly
>> identifies byte ordering as well:
>>
>>>            00 00 00 xx  UTF-32BE
>>>            00 xx 00 xx  UTF-16BE
>>>            xx 00 00 00  UTF-32LE
>>>            xx 00 xx 00  UTF-16LE
>>>            xx xx xx xx  UTF-8
>
>
> However, the table is not correct because the second 00 in the UTF-16
> cases could be non-zero when we have a string that starts with a
> non-ASCII range character.
>
> Maybe a table in which the first n-bytes match the follow can be used:
>
>     00 00        UTF-32BE
>     00 xx        UTF-16BE
>     xx 00 00     UTF-32LE
>     xx 00 xx     UTF-16LE
>     xx xx        UTF-8

Seems I need to correct my correction to allow for 0100;LATIN CAPITAL 
LETTER A WITH MACRON and so on:

     00 00        UTF-32BE
     00 xx        UTF-16BE
     xx 00 00 00  UTF-32LE
     xx 00 00 xx  UTF-16LE
     xx 00 xx     UTF-16LE
     xx xx        UTF-8
     xx EOF       UTF-8 (For Carsten's recent comment)

I think that's roughly where we got last time, and sufficient doubt had 
crept in that the UTF-8 should be used if unless the system knows the 
encoding is otherwise.

Pete Cordell
Codalogic Ltd
Rules for Describing JSON Content, http://json-content-rules.org