Re: [ietf-822] utf8 messages

[Brandon Long <[e-mail]>]

On Tue, Aug 12, 2014 at 7:54 AM, Ned Freed <[e-mail]> wrote:

> > > It is, or is supposed to be, a sealed system implemented as a set of
> > > interlocking extensions to existing email facilities.
>
> > So, if I have an "email" message, I can no longer just parse it.
> Instead,
> > there are actually two
> > types of email messages, and the only way to know how to parse it is to
> > know a priori which type it is.
> > Because all systems are "sealed" and there's never any leakage.
>
> Wrong on all counts, I'm afraid. First and foremost, in practice you have
> never
> been able to "just parse" email messages, for the simple reason that too
> many
> message creation agents don't follow the rules and create wildly
> incompliant
> messages. So there's always some heuristics involved, unless of course
> you're
> willing to only accept syntacticaly valid messages, in which case yes, you
> can "just parse" messages, including EAI messages.
>
> Where the lines are drawn has always been a tradeoff, and one which has
> changed over the years. It used to be the case that a lot more crap was
> generally tolerated. The spam problem has led to an overall tightening up
> what's tolerated.
>
> Second, because of overall lack of compliance, there have always been
> many types of messages.
>
> Third, of course there's leakage.
>

Ok, let me try to rephrase my point.  Before, there were two types of
messages, well specified / syntactically correct message, and not.  The not
well specified messages amount to a non-trivial number, but best
effort/heuristic results are fine.

If we imagine handling 1B messages/day, 2% being the "not well specified",
we have 20M messages handled by heuristic.  If we have only 1% heuristic
failures, that's 200k bad messages a day.  Eh, maybe ok.  Especially since
in practice, these tend to be spam messages or the 8bit chars in the header
are limited to some "content preview" header or other boneheadedness, or
best case, its only a character or two that's broken.

Adding under-specified 6532 messages to the mix means that we are now
generating messages that can easily slip into the second pool.  We try our
best to only generate well specified / syntactically correct messages.. "be
conservative in what you send" and all, but now we're being forced to take
steps that we know will increase the number of failures.  We're attempting
to adjust our heuristics to compensate, but that seems like a poor response
compared to making the messages be well specified.

Now, one can add some bit outside of the message to say its 6532... and
then modify everything that exchanges or stores messages to also exchange
and store that bit.  Passing messages to procmail?  Guess we need a new env
var.  Maildrop programs used by smtp servers to pass messages to imap
servers.  Mailing list software.  Mailing list archives.  How you call
spamassassin.  The thread about this during the discussion suggested adding
a new field to the From_ mbox separator.  Guess a new attribute field in
the maildir spec?

> > The problem we're having with 6532 messages, is that we moved from
> > > > explicitly identified charsets via 2047/etc mechanisms, to "its just
> > > > utf8"... and sometimes we mis-detect the utf8 as cp1250 or other
> > > encodings.
> > >
> > > No message created prior to the release of support for RFC 6532 can be
> > > assumed
> > > to be a RFC 6532 message. Now, if you want to vet those messages using
> some
> > > sort of process to insure such a message meets the syntax and at least
> > > looks
> > > semantically sensible, then I suppose you could set the flag in the
> > > metadata.
> > >
> > > But if you can't distinguish such messages from legitimate RFC 6532
> > > created and
> > > submitted by compliant clients, it sounds to me like you're not
> retaining a
> > > really critical piece of envelope/metadata in your implementation.
> > >
>
> > Yes.  And the most obvious place for that information, to me, is in the
> > headers of the message.  Assuming that every mechanism for exchanging
> email
> > messages needs an explicit external piece of data...
>
> Sorry, I'm not going to revisit or defend past design decisions. My point
> was
> and is that when you said that a piece of information was missing, that
> statement was incorrect.
>

I'm saying that requiring the information to be external grossly increases
the development required to support the standard.  I already pointed out
some common tools above.  For us, not having to add a new piece of external
metadata means that all we have to do is upgrade our parser... and validate
how we use addresses and fix hopefully minor issues.  If we have to use
external metadata, then we have 100s of data paths and data stores that
need to be upgraded.

We've also already agreed that leaks happen, which means separating the
metadata from the data itself guarantees that under specified data will
leak.  Making the message itself well specified means that
re-synchronization can take place, that messages can pass through agnostic
mechanisms (whether agnostic by choice or by happenstance) and be
understood on the other side.

You may not like or agree with how this was done. (For that matter, I never
> said I liked or agreed with how it was done.) But for better or worse,
> there's
> now a standard in place, and absent compelling evidence of there being a
> problem with implementing that standard - evidence which AFAICT you have
> not
> provided - it's not appropriate to propose competing mechanisms to that
> standard.
>

I thought implementation feedback was a useful thing, and generally desired
prior to something becoming a standard.  I also thought that internet
standards were generally considered a work in progress... 6532 is standards
track but not a STD.

I also wasn't proposing a competing mechanism, I was proposing that
implementation experience showed, to me, a strong possibility that a
clarification or change to the standard would be beneficial.  Also,
frankly, there is nothing in the standards that say we MUST not do this, so
thanks for the exhortation from authority, I'll take it under advisement.

Frankly, I don't understand this concept of refusing to revisit or defend.
 I tried to find and follow the discussion in the eai wg archives... and to
me, the decision seemed under explored.  The original section spent a lot
of real estate on the importance of such an explicit in message spec, only
to be removed.  I found two threads about it, about equal numbers of people
contributing to either side (like 1-2 on each side), a somewhat heavy
handed dismissal of the need, a call for consensus that passed due mostly
to non-contributors.  Perhaps more took place in meetings, or there were
other discussions that someone could point me to.

Perhaps you can explain to me what 'compelling evidence of there being a
problem implementing the standard' would mean in practice?

Brandon