[EAI] [IETF] Relational Databases: UTF8 [was: Internationalized Email Internet Draft]

[<[e-mail]>]

John / Tony,
>(5) Section 2.1 asserts that email servers), implying all of them, store data (messages?) in relational databases.  That is simply false.  Some do; others don't.  Even for those that do, there may be a difference between Unicode-capable >data storage and Unicode-capable keys or indexes.  There is also absolutely no requirement that any such system store Unicode strings encoded in UTF-8; many do not.  
This actually came from my own operational experience in implementing an IEA capable email server.   Will definitely change the wording to not imply that ALL email servers store in relational databases or store in UTF-8.   I can see that is not a requirement.

But, the problem that I had was with storing the user name, domain name, matching, etc.   Not the messages.   It was totally a problem that I ran into that the back end (whatever knows about the mailbox) had a problem with storing and matching user names & doing the DNS queries.
Let me think about how to re-write this.   We may, in the end, want to do some Best Practices kind of things so that people do not struggle for quite as long as I did!
 Thanks,
Nalini ElkinsInside Products, Inc.www.insidethestack.com(831) 659-8360

      From: John C Klensin <[e-mail]>
 To: "HANSEN, TONY L" <[e-mail]>; [e-mail] 
 Sent: Sunday, October 9, 2016 7:57 PM
 Subject: Re: [EAI] [IETF] Internationalized Email Internet Draft
   


--On Thursday, October 06, 2016 4:39 PM +0000 "HANSEN, TONY L"
<[e-mail]> wrote:

> I think getting deployment feedback from EAI is important, and
> this draft is an excellent start.
> 
> I'm not convinced that section 1.2 describes a real problem.
> People do this all the time today with various combinations of
> languages. Why is the combination of Russian and Chinese any
> different? If you think it is, then please expand on the
> aspect that does make it more difficult.
> 
> I forwarded a number of nits to the authors.

Hi.  I was going to hold off until some later and more mature
version of this draft, but since Tony has commented, while I
believe the issues with EAI deployment are important, I see
several problems with this draft, some of which were actually
discussed in the WG but appear to be ignored here.  Perhaps more
important, it is seriously incomplete relative to issues that
have been discussed at great length in the EAI WG, at the APEC
meeting on internationalized email in Beijing in October 2014,
the May 2015 workshop in Thailand, and elsewhere.  I strongly
suggest that, if there is going to be a discussion in Seoul,
this document is in need of a great deal of work first.  Some of
those issues are:

(1) The so-called EAI standards, as listed in the Introduction,
are about email envelope and header information presented
directly (e.g., in UTF-8) as non-ASCII characters.  A good deal
of the document appears to address mail content information such
as textual message bodies, in other scripts.  With the possible
exception of language selection when a message is sent with the
same basic text in several languages (multipart/alternative was
designed with that case in mind but have been used in other
ways), we thought we solved that content problem with MIME in
1992.  If MIME is inadequate, the authors or others should
produce a document explaining the issues and not confuse them
with EAI / SMTPUTF8.  If it is adequate, then, like Tony
although perhaps for different reasons, I don't see what Section
1.2 is doing here, what the relevance of Section 3.2 is, and
several other statements should be examined carefully to be sure
they are talking about addresses and/or headers and not content.

(2) Within an address, there is, as the I-D points out and
consistent with RFC 5321, a local part and a domain part.  RFCs
6530 and 6531 make it quite clear (at least we thought they did)
that they are handled differently.  For the domain part, the
rules are laid out in the IDNA2008 specs (RFC 5890ff).  Issues
about look-alike characters have been extensively discussed and
written about (even though some of us have questioned the
quality of some of that work).  It does not seem useful to me to
revisit those issues here, especially without reference to the
prior work and discussions or if some of the discussion here is
wrong or contains obvious omissions.  As an example from the
first paragraph of Section 6.1, Latin "c" (U+0063) and Cyrillic
"c" (U+0441) are typically written with identical graphemes, but
are not on the list.    More important, while the "paypal"
example with U+0430 substituted for "a" (U+0061) has been used
repeatedly, including in a careful study in an article that is
not cited in this draft, it is possible to write "раура1"
with the first five characters in Cyrillic and the last one a
digit (which is script independent)
(\u'0440'\u'0430'\u'0443'\u'0440'\u'040'\u'0031' [1]), therefore
not even violating conventions prohibiting mixed-script labels.
There is, of course, no ambiguity in the A-label form, although
the authors quite properly point out that it is not
user-friendly.

By contrast, Section 1.1 talks about display of email addresses,
including the local part ("in Punycode" [2]).  While a mail
delivery server is free to create whatever aliases for a mailbox
local part it likes, including "xn-t2bmh3a" or "123456",
"george" or "example", in general converting a local part using
the Punycode algorithm and displaying the result is prohibited
by the EAI standards (and, incidentally, RFC5321).  More
important, it will often lose information and is potentially
very dangerous.

(3) Arabic should not be confused with a strictly right-to-left
writing system.  I am not aware of any such systems in wide use
for contemporary languages today.  The problem is that numerals,
whether written in European digits, Arabic or Arabic-Indic
digits, Chinese (Han) digits, or many others, have been written
left to right since that type of positional notation was
invented and became widely used.  As a result, the scripts are
referred to (in Unicode-speak) as "bidirectional" or "bidi" [3].
Their implications for domain names and IDNA are the subject of
RFC 5893.

(4) Multiple addresses for one user (and Section 4).  Keeping in
mind that many people maintain a number of identities, and even
multiple email addresses, for different purposes, I don't
understand what point you are trying to make with this section.
Many of us believe that users who have mailboxes whose names
involve non-ASCII local parts and who engage in communications
outside their primary language group will find it necessary to
maintain either separate all-ASCII mailboxes or all-ASCII
aliases to their primary mailboxes and to do so for a very long
time.  That issue has been extensively analyzed and discussed
but this document avoids that work, which is both a problem and
an opportunity.

(5) Section 2.1 asserts that email servers), implying all of
them, store data (messages?) in relational databases.  That is
simply false.  Some do; others don't.  Even for those that do,
there may be a difference between Unicode-capable data storage
and Unicode-capable keys or indexes.  There is also absolutely
no requirement that any such system store Unicode strings
encoded in UTF-8; many do not.  

(6) There is a necessary difficulty with SMTPUTF8, which is that
one cannot transmit a message with non-ASCII characters in
addresses or headers to a system that does not support them.
Final delivery systems should probably not accept messages
unless they have reason to predict that the mail store will
handle them _and_ that the user associated with the target
mailbox will be able to retrieve them.  Since a user with an
all-ASCII mailbox name might still receive a message with, e.g.,
a non-ASCII backward-pointing address in the envelope or
headers, making that decision is not straightforward.  That
leads to a strong case that, if one wants broad deployment of
SMTPUTF8, the place to start is with the MUAs (including the
Webmail systems) and associated POP and IMAP servers and
clients.  The "to various extents" list in the first part of
Section 3 is not particularly helpful in that regard.

(7) Finally, this is an internationalization (i18n) problem as
much as it is an email problem.  Terminology (and, where
characters or code points are referred to, their precise
identification) is very important because the alternative is
typically a good deal of user confusion about what you are
talking about and other impediments to making progress.  Saying
"English" were you mean "Basic Latin Script" or "ASCII" is not
helpful, especially given that 5321 local parts can include any
ASCII character and that ASCII is not sufficient to write
English.  Conversely, it appears that there are a few places
where, correctly or incorrectly, you really do mean "English"
when you say that.  Similarly, talking about one particular
encoding when you mean "Unicode" is confusing and may be
misleading.  RFC 6365 may give you a start on some of the issues.

regards,
    john


  -------------
[1] I recommend the authors have a look at RFC 5137.

[2] Punycode is an encoding method, not a display format.  See
RFC 5890, Section  2.3.4. 

[3] http://unicode.org/reports/tr9/

_______________________________________________
IMA mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/ima