Re: [I18ndir] I18ndir early review of draft-ietf-dispatch-javascript-mjs-07

["Martin J. Dürst" <[e-mail]>]

On 09/05/2020 13:01, John C Klensin wrote:

>> Section 4.2 still includes step #3 to deal with the (in
>> practice quite common) case of a missing BOM and the media
>> type missing a charset parameter.  There are also too many
>> servers that set this to "ISO-8859-1" without otherwise
>> examining the sources being served. We'll make it clearer this
>> is a default/fallback case.
> 
> Noting (again) that, absent a BOM (or even with one) UTF-8
> cannot be reliably distinguished from ISO-8859-X (for any
> registered or unregistered value of X),

Sorry, not true. Please see 
https://www.sw.it.aoyama.ac.jp/2012/pub/IUC11-UTF-8.pdf, page 20.


> I don't know what the
> default/fallback case actually is or, more generally, what  the
> above paragraph means.  Unless I have misunderstood something
> important, the reality is that, if there is anywhere on the
> Internet that a web browser or server (including decades-old
> embedded servers) treat ISO/IEC 8859-1 as either the default or
> legitimate, then there are two possibilities: accurate labeling
> of the charset in use or use of heuristics that, by their nature
> and the nature of possible CCSs (not just encoding schemes), may
> fail.

For people not familiar with the details, the following should be 
pointed out clearly: The heuristic of detecting UTF-8 as UTF-8 based on 
its very specific bit patterns is an extremely strong heuristic. There 
are cases of specific character combinations in specific encodings that 
can look like UTF-8 bit combinations, but that is extremely rare and 
gets rarer and rarer the longer the data gets. This is already true if 
one looks just at the available characters/bit patterns, but even more 
so if one looks at the languages that use the respective encodings and 
their letter patterns.

Regards,   Martin.

> That calls for at least a health warning in the
> document, not proceeding as if the heuristics are foolproof.
> 
> best,
>     john