Re: [I18ndir] Review volunteer needed (Fwd: [dispatch] WGLC of draft-ietf-dispatch-javascript-mjs-07)

[John C Klensin <[e-mail]>]

Asmus, good to hear from you.

Inline below.

--On Wednesday, April 29, 2020 16:54 -0700 Asmus Freytag
<[e-mail]> wrote:

>...
> +4 Aren't UTF-16 BOM versions are not legal UTF-8 byte
> sequences, or am I misremebering? If so, a UTF-16 BOM should
> override or flag as invalid any UTF-8 declaration.

Your memory is correct.  The question is whether, if
charset="UTF-8" is declared and then the first two octets are a
UTF-16 BOM, whether to let it override and treat the data as
UTF-16  -or- to figure out a way to say "this is nonsense".

Patrik interprets the document as requiring the override
behavior.  He and I both prefer the latter, I think for at least
three reasons:

(i) I think, but am not sure, that some valid UTF-16 sequences
are valid UTF-8, even if the BOMs aren't.  If so, a file that is
labeled as UTF-8 but contains a UTF-16 BOM because some idiot
(not necessarily a human one) put it there at the beginning,
could really get into a world of trouble.  For example, any
valid UTF-16 sequence, especially one that contains no zero
octets, could be a valid 8859 sequence or valid sequences in
most code pages.  So starting from "it was labeled as UTF-8 but
isn't" doesn't make it UTF-16, BOM or no BOM.

(ii) As you know, there are a number of sequences that look like
UTF-8 but are not valid,  ones that Unicode now prohibits (my
vague memory is that it took a while for that prohibition to be
clearly established).  If something is labeled as UTF-8 (and/or
starts with a UTF-8 BOM) but contains one of those sequences,
the choice is whether to say "this is nonsense" in some way or
to try to guess at what was intended.  My assumption is that
those two nonsense cases are more or less equivalent and that,
if systems are sent ill-formed files, they are better off not
trying to guess... especially if the spec does not require
Unicode (which goes to your 21st Century comment, I think).

(iii) Mislabeled content is just bad news for all sorts of
reasons as I have been discovering when one of the mail systems
of a very large vendor are identifying message bodies as
'text/plain; charset="UTF-8"' and then sending UTF-16, BOM and
all.  Anything that can be done to discourage that sort of
nonsense --or, if not, to describe carefully want should be done
when it is encountered would be, IMO, A Good Thing.

The I-D, in its present form, does none of that.

stay safe and well,
   john