[Extra] IMAP RFC 6855 (IMAP EAI) implementation experience: input to IMAP4rev2

["Chris Newman" <[e-mail]>]

Sometimes the philosophy behind a standard and what happens when it 
encounters the real-world result in implementations that willfully 
ignore some aspects of the standard. Since I wrote the first draft of 
RFC 6855, any such differences between the spec and real life are mostly 
my fault.

For better or worse, RFC 6530 took a very different approach from past 
IETF specifications when it comes to backwards compatibility. It choose 
not to introduce yet-another-ascii-encoding for email local parts and 
takes a "MUST bounce" position when a UTF-8 message encounters a 
5321/5322-only system. From a standards-purist viewpoint, this preserves 
interoperability but is a deployment deterrent to the new format. 
However, from an implementer viewpoint, I believe this is mostly a 
non-starter. Although we documented downgrading options (RFC 6857,6858) 
these options end up having poor cost/benefit in the real world. Ned's 
implementation experience from RFC 6531 was that the real world just 
can't usefully work the way the specification describes. That experience 
informed my work implementing RFC 6855.

My experience with RFC 6855 implementation is as follows:

1. Support to accept and validate UTF-8 in quoted-strings was 
straightforward to implement. I made the support unconditional as that 
was much simpler than adding ENABLE-conditional code branches that would 
require extra testing.

2. Support to append UTF-8 messages was medium complexity to implement 
because I decided to validate the header UTF-8 syntax and push any 
garbage-8-bit headers into the message body. I ended up just ignoring 
the <"UTF8" SP> protocol syntax when present and processing the message 
the same way regardless.

3. I choose not to implement downgrading or UTF-8-message hiding. If 
UTF-8 messages become common and legacy clients start choking on them, I 
might add a downgrading option, but I suspect it will be much easier for 
clients to just deal with UTF-8 where it wasn't previously allowed than 
to do anything useful with a downgraded UTF-8 email message or even 
fully implement RFC 6855 (which I suspect is far simpler for clients 
than for servers).

4. With my implementer hat on, I willfully ignored this requirement:
   The IMAP server MUST NOT send UTF-8 in quoted-strings to the client 
unless the client
   has indicated support for that syntax by using the "ENABLE 
UTF8=ACCEPT" command.
It's interesting that my implementer hat and standards hat result in 
diametrically opposed positions on this issue, but so be it. If there's 
a message with UTF-8 headers in the message store, IMAP clients will see 
it uncorrupted regardless of whether the client can handle it.

5. The requirement to eliminate modified-UTF-7 in mailbox names was 
medium complexity due to some past implementation choices. I was 
comfortable with the cost/benefit of the effort. MUTF-7 is ugly; good 
riddance. This code has to be conditional based on use of the ENABLE 
command -- however, if the implementation sees a client-provided UTF-8 
mailbox name, it treats that one name as if ENABLE had been set. It's 
only 7-bit mailboxes containing ampersand that have a behavior 
difference based on whether ENABLE was used.

6. The most difficult part of the project is supporting UTF-8 usernames 
throughout the server. IDN is actually the hardest part of this since 
it's necessary to support both A-labels and U-labels for the domain 
portion of a user name and treat them as synonyms. Luckily RFC 6855 does 
not mandate support for UTF-8 usernames so this doesn't impact the 
previous decision of whether to include RFC 6855 in IMAP4rev2. The 
U-label/A-label issue is also a problem for SMTP servers, milters, and 
Sieve.

Based on 1-5, I believe the text in the current IMAP4rev2 draft is 
correct, appropriate, and implementable.

Based on 6, we might want to consider some additional text with respect 
to UTF-8 usernames & passwords.

		- Chris