Re: [I18ndir] Review volunteer needed (Fwd: [dispatch] WGLC of draft-ietf-dispatch-javascript-mjs-07)

[Asmus Freytag <[e-mail]>]

On 4/29/2020 6:45 PM, John Levine wrote:
> It looks like step 1 is saying that if the text starts with a BOM, you
> ignore the declared charset and sniff the BOM instead, which sounds to
> me like an ancient workaround that is perhaps no longer needed.

If data are preceded by ("start with")  a BOM, you do want to strip it; 
you never want to keep it (and the chance that a legitimate text starts 
with a BOM that has an actual function in the text is perfectly negligible).

If you have data that carries a UTF-16 bom it cannot be UTF-8, no matter 
what the charset declaration says (FF can't be a byte in UTF-8).

Therefore, you always want to look for all three.

If  it's UTF-8 you confirm you have the good character set and remove it.

If it's one of the UTF-16 ones, you switch the charset to UTF-16 (or 
proper endianness) and remove it. (Or reject the input?)

> Given that they are deprecating all of the existing javascript media
> types and reviving text/javascript which 4329 declared obsolete, this
> might be a good time to say if you're going to use our lovely new
> (old) media type, declare the correct character set so consumers can
> believe it and stop doing byte sniffing kludges.

There are two issues here. One centers around the fact that BOMs are 
"invisible". You can work hard at avoiding them, but they may be added 
by some "helpful" tool.

The other is that they serve as a useful consistency check when present.

You may write the spec to reject data with initial BOM, but then you'd 
still need to check for them. You definitely don't want to admit data 
without checking. Since you already need to check for them, you are 
better off giving clear instructions of how to make use of the fact that 
you've detected one.

A./