Re: [apps-discuss] Feedback about "Update to MIME regarding Charset Parameter Handling in Textual Media Types"

[Henri Sivonen <[e-mail]>]

On Thu, Feb 23, 2012 at 5:59 AM, Ned Freed <[e-mail]> wrote:
>> Additionally, media types should be able to define circumstances where
>> in-band indicators override the charset parameter even if the charset
>> parameter is present.
>
> That's a terrible way to do it

It's reality in WebKit and IE for text/html. So far, evidence suggests
that Gecko would serve users better if it changed to treat the BOM as
having higher precedence than the HTTP-level charset parameter for
text/html and text/javascript. (text/plain is loaded using the HTML
parser per the HTML spec and per reality, so it makes sense to do the
same for text/plain even though I don't have evidence to show about
compatibility issues either way.)

> - if the type is self-identifying in terms of
> charset, a charset parameter should simply not be defined for the type -
> exactly what the current specification says to do.

Logically, yes, but that's not how text/html, text/xml and text/css work.

>> In particular, media types should be allowed to override the charset
>> parameter if the first two or three bytes of the payload look like an
>> UTF-16 or UTF-8 BOM.
>
> There are quite a few charsets in existence where it is perfectly permissible
> for the first few bytes to match a BOM, except that it means something entirely
> different.

Seems to work for IE and WebKit in practice.

>> This seems wrong. If the charset parameter is present, it has an
>> effect for text/xml.
>
> That's only because the definition of test/xml did it incorrectly.

There's text/xml legacy by now, so chances are it's too late to make
HTTP charset have no effect for text/xml.

>> > or b) require explicit unconditional inclusion of the "charset" parameter
>> eliminating the need for a default value.
>
>> This seems naïve.

It's awesome that the IETF's own Web interface to the list archive
mangles the ï here. Yay dogfood.

>> Formats need to specify what happens when a charset
>> parameter is missing, since no matter how much the format says it's
>> "required", the party sending data can omit the charset parameter.
>
> Yep. They can also misspell the parameter name, misspell the charset name, use
> the wrong type/subtype, misspell the header field name, omit the field
> entirely, or any of a million other mistakes.
>
> And when (not if) this sort of thing happens, the receiver can elect to pursue
> whatever course of action it deems appropriate for invalid material.

No. To achieve interoperability, the behavior in error cases needs to
be well-defined. Broken content will depend on the behavior of
incumbent consumers. If that behavior isn't specified, new entrants to
the market face a barrier to competition when they need to reverse
engineer incumbents instead of just reading specs to see what needs to
be done.

>> > The default charset parameter value for text/plain is unchanged from
>> > [RFC2046] and remains as "US-ASCII".
>
> Note that this has the effect of rendering any content that contains 8bit
> as being invalid.

I'm ok with defining the absence of the charset parameter for
text/plain invalid if the BOM is also absent.

>> This is incompatible with reality. Web browsers, for instance, assume
>> a configuration-dependent default (which correlates with browser
>> localization) and may also (depending on configuration which, again,
>> correlates with localization by default) perform a heuristic analysis
>> on the payload.
>
> Well, on this one you'll have to argue with someone else. I don't especially
> like the approach in the draft, but I have been unable to come up with any
> reasonable alternative. Your proposed alternative below, is totally unworkable.

Why is it unworkable. It's what Gecko does with the modification that
the precedence of the BOM is as in WebKit and IE.

>> New text/* media types SHOULD use the following algorithm:
>
>> The character encoding is UTF-8. Terminate these steps.
>
> This approach is ridiculous when you consider how general the usage of some
> text types are and  the close similarity of many charsets in common use.

The UTF-8-only recommendation is for new text/* types which (by
definition of "new") don't have any charsets in common use previously.

> It is
> not at all common for there to only be one or two characters that in a large
> document that display incorrectly when the wrong charset is selected.

You mean like "naÃve" in the archive copy of your email? ;-)

> It is also
> unacceptable to ignore a valid label just because you don't support
> the encoding.

That's how browsers work.

> Finally, I'd say the odds of any implementor following such a complex
> set of rules is remote in the extreme.

As an implementor intending to implement the steps, I disagree. Gecko
already implements those steps with the BOM steps in a different order
relative to the charset parameter. I expect to patch Gecko to make the
BOM take precedence over the charset parameter in due course to match
IE and WebKit in text/html handling (and text/plain reuses the
text/html code path).

-- 
Henri Sivonen
[e-mail]
http://hsivonen.iki.fi/