Re: [I18ndir] Review volunteer needed (Fwd: [dispatch] WGLC of draft-ietf-dispatch-javascript-mjs-07)

[Asmus Freytag <[e-mail]>]

Don't have time to delve into this deeply, but am reading along the 
discussion here.

+1 on the text being too convoluted - I know my first reading was wrong.

+2 on seeing whether we can't use the revised spec to move to the 21st 
century in terms of character encoding.

(Their step 3 really attempts that, by making it the default / fallback 
when unlabeled / or without BOM signature, but it could be more explicit 
that you can only get legacy behavior if such is labeled).

+3 on stronger language about detecting mislabeled / inconsistent cases. 
Some conditions are clear cut enough to warrant rejection. Ideally, 
you'd always reject and not convert anything to U+FFFD. But, again, 
existing legacy may make that not possible.

+4 Aren't UTF-16 BOM versions are not legal UTF-8 byte sequences, or am 
I misremebering? If so, a UTF-16 BOM should override or flag as invalid 
any UTF-8 declaration.

A./

On 4/29/2020 3:30 PM, John C Klensin wrote:
>
> --On Wednesday, April 29, 2020 20:58 +0200 Patrik Fältström
> <[e-mail]> wrote:
>
>> On 29 Apr 2020, at 3:41, John C Klensin wrote:
>>
>>> Since some hours have gone by without a response to your
>>> message and I was in need of an excuse to delay getting to an
>>> unpleasant task...
>> Now I have read the draft as well.
>>
>>> Moreover, if I correctly understand what seems like
>>> unnecessary convoluted text (in both versions) a BOM is
>>> ignored in further processing if the character encoding
>>> scheme is determined to be UTF-8 in 4.2(2) or 4.2(3) but not
>>> ignored if charset="UTF-8" is present and the BOM occurs
>>> anyway (something clearly allowed by RFC 3629).  That doesn't
>>> appear to make sense.
>> I think I see the same thing as you, which is that even if the
>> charset parameter states the encoding is UTF-8, if the data
>> itself starts with a BOM, then the text is to be treated as
>> UTF-16.
> Actually, not what I noticed, and that reinforces my view that
> even if one ignores the specific i18n issues, the text is just
> too convoluted.  I had read the text as suggesting that, if the
> charset was labeled (part 1), that the checks in part 2 and 3
> just don't get made.  Whether that is smart or not -- whether
> the protocol or application should make a sanity check on
> whether something labeled as charset="UTF-8" is actually
> conforming UTF-8 and/or whether there is a BOF present if it is
> consistent with UTF-8 -- I don't really know except that it
> should probably be clear.  But, you are right: if the text is
> identified with charset-"UTF-8", then it really, really, better
> be UTF-8 and, if there is a BOF that suggests it is something
> else, then the spec should say something quite definite about
> that.
>
>> That is just so wrong.
> I was more concerned about something else (with the
> understanding that it isn't my only issue).  As I read the spec,
> the plan is, approximately:
>    (1) Apply step one, if there is no charset parameter present,
> go to step 2
>    (2) Apply step 2, i.e., go looking for a BOM fingerprint.  If
> there isn't one present, go to stem 3.
>    (3) Step 3: decide it is UTF-8.
>
> Now the (or at least one) problem with that is that, absent a
> rule that says "MUST use Unicode in some known encoding form",
> there is no practical and reliable way to distinguish UTF-8 from
> any part of ISO/IEC 8859 or, for that matter, any proprietary
> code page.  UTF-16, with or without the BOM heuristics of your
> choice is better, but not much better.
>     
>> I went to the ECMA spec and see they use UTF-16 all over the
>> place, and have to bend over backwards to get things right. It
>> feels like reading a BER encoding spec (again). :-)
> BER is at least precise about what it is talking about.  This
> spec isn't.
>   
>> This "problem" do already exist in RFC 4329...
>>
>> But, if they update RFC 4329 I think they should clean this
>> up, and my suggestion would be:
>>
>> The encoding must be what is actually labeled. If the encoding
>> is UTF-16 (which it seems it often is), then it should be
>> tagged as UTF-16, not UTF-8 with BOM.
> Absolutely.  The easiest way out of both the problem you saw and
> the one I saw is to get rid of steps 2 and 3 and insist on
> labeling and conformance to what is labeled.  If that is
> impractical for some reason, much more specificity is needed,
> starting with a firm Unicode requirement.  And, unless they
> intend to confine themselves to the BMP, they probably need to
> talk about surrogates and their implications (or include such an
> explanation by reference).   As to whether they "should" fix a
> problem left over from 4329, they have changed the text in that
> area and this is a Known Technical Omission or Defect.
>
> best,
>     john
>