Re: [EAI] UTF-8/MIME

[John C Klensin <[e-mail]>]

--On Thursday, August 19, 2010 14:43 +0100 Charles Lindsey
<[e-mail]> wrote:

> On Wed, 18 Aug 2010 19:31:52 +0100, Shawn Steele
> <[e-mail]> wrote:
> 
>>> my understanding is that headers are in utf-8,
>>> the body part might still use mime(gb2312).
>> 
>> I'm obviously really confused.  My understanding was that EAI
>> required   UTF-8 headers and MIME encoded bodies, which
>> should also be UTF-8, but   an appropriate MIME part.
>> Presumably the body could also be GB2312   MIME, though I'd
>> discourage that as much as possible.

The implication of draft-iab-idn-encoding is that we ought to be
gradually deprecating everything but ASCII and UTF-8 (I would
say "everything but UTF-8", but there is specific language in
the MIME specs stating a preference for coding body parts with
no non-ASCII characters as "us-ascii" rather than "utf-8").   So
I agree with Shawn's observation but don't think it is a matter
for EAI to discuss or offer advice about (see note 1 below).

> Now you've got me confused. An EAI message (i.e. one coming
> from an agent asserting a need for UTF8SMTP) might contain
> headers using UTF-8, but the body entirely in ASCII. So no
> MIME stuff anywhere in it. So you cannot say that EAI REQUIRES
> MIME, though for sure it would be stupid to implement it
> without MIME.

But there will always be "MIME stuff" in an EAI-conformant
message with non-ASCII header material.  Remember, UTF8SMTP[bis]
requires 8BITMIME, 8BITMIME requires MIME, and MIME requires at
least a MIME-version header field, so, even if there is only one
body part and the content type is allowed to default to
'text/plain; charset="us-ascii"' (i.e., no Content-Type header
field present), there is "MIME stuff" present.

> AIUI, if you assert UTF8SMTP and have UTF8 in the headers, and
> then want to use UTF-8 in the Body (a common situation,
> presumably), then you are supposed to include a Content-Type:
> specifying charset=utf8 and a suitable
> Content-Transfer-Encoding.

Yes.  Since 8BITMIME is needed to transport those UTF-8 headers,
the C-T-E can reasonably be "8bit"

> What we MIGHT do is to state that, for such EAI messages the
> default body charset was UTF-8 and the default CTE was 8bit
> (and most modern MUAs would likely display that correctly).
> That's just another extension to RFC 553[12], and we have
> extended those already. The downside would be that any attempt
> at downgrading would have to put those assumed-by-default
> headers back.

I think this would be certain to cause problems with other MIME
implementations.  Remember that messages are passed outside the
transport system and that making up other defaults --body parts
not requiring Content-Type or C-T-E fields-- would violate the
8BITMIME spec.   Now, if we were to say that use of EAI required
either the normal default Content-Type (text/plain;
charset="us-ascii") or that text types were required to be
charset="utf-8", I think that would be ok.  However, I think it
would accomplish very little in practice that a simple
recommendation to use UTF-8 in body parts where possible
(labeled as the MIME and 8BITMIME specs required) would not.

In particular, if someone were determined to use GB body parts,
I don't think we are going to be able to effectively prohibit
that.  We should just try to insist that they be properly
labeled and identified.


> In fact, I suspect that default practice is going to happen
> anyway within EAI-only communities, so we might as well make
> it official.

Partially because of the wave of MTAs that, in self-defense,
took very harsh measures to deal with unlabeled 8bit content,
there is, in my experience, little current practice of using
non-ASCII body parts without Content-type labeling.   I see no
reason at all to try to reintroduce that practice, especially
when we remember how common text/html and various binary
("application") and image type body parts have to be handled by
those same MTAs and MUAs.

    john