[Extra] IMAP RFC 6855 (IMAP EAI) implementation experience: input to IMAP4rev2

"Chris Newman" <chris.newman@oracle.com> Tue, 20 November 2018 23:28 UTC

Return-Path: <chris.newman@oracle.com>
X-Original-To: extra@ietfa.amsl.com
Delivered-To: extra@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 905AF126CC7 for <extra@ietfa.amsl.com>; Tue, 20 Nov 2018 15:28:12 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -4.771
X-Spam-Level:
X-Spam-Status: No, score=-4.771 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.47, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=oracle.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Os5asER2O_Ay for <extra@ietfa.amsl.com>; Tue, 20 Nov 2018 15:28:10 -0800 (PST)
Received: from aserp2120.oracle.com (aserp2120.oracle.com [141.146.126.78]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 6B77312007C for <extra@ietf.org>; Tue, 20 Nov 2018 15:28:10 -0800 (PST)
Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAKNPCUh181958 for <extra@ietf.org>; Tue, 20 Nov 2018 23:28:09 GMT
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : mime-version : content-type; s=corp-2018-07-02; bh=aOkWQ88QgKZSHrloc5r88cCvqyp3I29Q44ollk8p3Gs=; b=qe1lPpBUR7fhsYMj+4uCb4loqNhb0lii18Lc2Bu0XqUuWFl2bYOgViiFZ2d+oB1un16l lD1bdUqr4evuP5zftYqlbkRPmf9mhomDPjIKPld2AAQqlvTIrSWHJ7zD54C8SCzGWYN6 k/ZP0p46HsDxc//hXy7kjeYwkqu6Vz8r0tzO+AguPuAWKzoD9xzCpRwOr8EBHLD/ONjE zIiONyGYUjFJSLcRQzz3iyUmeQp4LVoiUjnl16cw0IUhWqbxqozjEhnbs5+MEb1+rxky SsXR89qkTVmGgM98IO2u7AMr1D39n58kkSMRFBMC9ohLZUWhYRStBVwAmNQLOvv9S5Tp +A==
Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by aserp2120.oracle.com with ESMTP id 2ntaxq6mnc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for <extra@ietf.org>; Tue, 20 Nov 2018 23:28:09 +0000
Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAKNS3Jj021293 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for <extra@ietf.org>; Tue, 20 Nov 2018 23:28:03 GMT
Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAKNS3IL002051 for <extra@ietf.org>; Tue, 20 Nov 2018 23:28:03 GMT
Received: from [192.168.15.59] (/47.41.1.180) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 20 Nov 2018 15:28:03 -0800
From: "Chris Newman" <chris.newman@oracle.com>
To: extra@ietf.org
Date: Tue, 20 Nov 2018 15:28:02 -0800
X-Mailer: MailMate (1.12.1r5552)
Message-ID: <F3CA8758-0897-4A5D-B275-E634CCA43207@oracle.com>
MIME-Version: 1.0
Content-Type: text/plain; format=flowed
X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9083 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=1 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=517 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1811200206
Archived-At: <https://mailarchive.ietf.org/arch/msg/extra/CR2clEvB6WfV0tlio2xBONRCRlA>
Subject: [Extra] IMAP RFC 6855 (IMAP EAI) implementation experience: input to IMAP4rev2
X-BeenThere: extra@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Email mailstore and eXtensions To Revise or Amend <extra.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/extra>, <mailto:extra-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/extra/>
List-Post: <mailto:extra@ietf.org>
List-Help: <mailto:extra-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/extra>, <mailto:extra-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 20 Nov 2018 23:28:13 -0000

Sometimes the philosophy behind a standard and what happens when it 
encounters the real-world result in implementations that willfully 
ignore some aspects of the standard. Since I wrote the first draft of 
RFC 6855, any such differences between the spec and real life are mostly 
my fault.

For better or worse, RFC 6530 took a very different approach from past 
IETF specifications when it comes to backwards compatibility. It choose 
not to introduce yet-another-ascii-encoding for email local parts and 
takes a "MUST bounce" position when a UTF-8 message encounters a 
5321/5322-only system. From a standards-purist viewpoint, this preserves 
interoperability but is a deployment deterrent to the new format. 
However, from an implementer viewpoint, I believe this is mostly a 
non-starter. Although we documented downgrading options (RFC 6857,6858) 
these options end up having poor cost/benefit in the real world. Ned's 
implementation experience from RFC 6531 was that the real world just 
can't usefully work the way the specification describes. That experience 
informed my work implementing RFC 6855.

My experience with RFC 6855 implementation is as follows:

1. Support to accept and validate UTF-8 in quoted-strings was 
straightforward to implement. I made the support unconditional as that 
was much simpler than adding ENABLE-conditional code branches that would 
require extra testing.

2. Support to append UTF-8 messages was medium complexity to implement 
because I decided to validate the header UTF-8 syntax and push any 
garbage-8-bit headers into the message body. I ended up just ignoring 
the <"UTF8" SP> protocol syntax when present and processing the message 
the same way regardless.

3. I choose not to implement downgrading or UTF-8-message hiding. If 
UTF-8 messages become common and legacy clients start choking on them, I 
might add a downgrading option, but I suspect it will be much easier for 
clients to just deal with UTF-8 where it wasn't previously allowed than 
to do anything useful with a downgraded UTF-8 email message or even 
fully implement RFC 6855 (which I suspect is far simpler for clients 
than for servers).

4. With my implementer hat on, I willfully ignored this requirement:
   The IMAP server MUST NOT send UTF-8 in quoted-strings to the client 
unless the client
   has indicated support for that syntax by using the "ENABLE 
UTF8=ACCEPT" command.
It's interesting that my implementer hat and standards hat result in 
diametrically opposed positions on this issue, but so be it. If there's 
a message with UTF-8 headers in the message store, IMAP clients will see 
it uncorrupted regardless of whether the client can handle it.

5. The requirement to eliminate modified-UTF-7 in mailbox names was 
medium complexity due to some past implementation choices. I was 
comfortable with the cost/benefit of the effort. MUTF-7 is ugly; good 
riddance. This code has to be conditional based on use of the ENABLE 
command -- however, if the implementation sees a client-provided UTF-8 
mailbox name, it treats that one name as if ENABLE had been set. It's 
only 7-bit mailboxes containing ampersand that have a behavior 
difference based on whether ENABLE was used.

6. The most difficult part of the project is supporting UTF-8 usernames 
throughout the server. IDN is actually the hardest part of this since 
it's necessary to support both A-labels and U-labels for the domain 
portion of a user name and treat them as synonyms. Luckily RFC 6855 does 
not mandate support for UTF-8 usernames so this doesn't impact the 
previous decision of whether to include RFC 6855 in IMAP4rev2. The 
U-label/A-label issue is also a problem for SMTP servers, milters, and 
Sieve.

Based on 1-5, I believe the text in the current IMAP4rev2 draft is 
correct, appropriate, and implementable.

Based on 6, we might want to consider some additional text with respect 
to UTF-8 usernames & passwords.

		- Chris