Re: [xmpp] review of draft-ietf-xmpp-6122bis-12

[Peter Saint-Andre <[e-mail]>]

Hi Joe, thanks for the review and my apologies for taking a month to reply.

On 7/30/14, 3:25 PM, Joe Hildebrand (jhildebr) wrote:
> The reasons the precis group got a spate of questions from me today was I
> was prepping to do this review.  There are a couple of issues that the
> precis folk should pay more attention to.
>
>   > 1.  Introduction
> ...
>
>   >    Instead, this document builds upon the
>   >    internationalization framework defined by the IETF's PRECIS Working
>   >    Group [I-D.ietf-precis-framework], while attempting to ensure that
>   >    the characters allowed in Jabber IDs under stringprep are still
>   >    allowed and handled in the same way under PRECIS.
>
> "the same way" means more backward-compatibility to me than I think we
> intend here.

Yes, that is a bit vague, even though it does say "attempting". Here is 
one possible approach...

OLD

    Instead, this document builds upon the
    internationalization framework defined by the IETF's PRECIS Working
    Group [I-D.ietf-precis-framework], while attempting to ensure that
    the characters allowed in Jabber IDs under stringprep are still
    allowed and handled in the same way under PRECIS.

NEW

    Instead, this document builds upon the
    internationalization framework defined by the IETF's PRECIS Working
    Group [I-D.ietf-precis-framework].  Although every attempt has been
    made to ensure that the characters allowed in Jabber IDs under
    stringprep are still allowed and handled in the same way under
    PRECIS, there is no guarantee of strict backward compatibility
    because of changes in Unicode and the fact that PRECIS handling is
    based on Unicode properties, not a hardcoded table of characters.

>   > 3.1.  Fundamentals
>   >
>   >       jid           = [ localpart "@" ] domainpart [ "/" resourcepart ]
>   >       localpart     = 1*1023(localpoint)
>   >                       ;
>   >                       ; a "localpoint" is a UTF-8 encoded
>   >                       ; Unicode code point that conforms to
>   >                       ; the "JIDlocalIdentifierClass" profile
>   >                       ; of the PRECIS IdentifierClass
>   >                       ;
>
> This implies 1023 codepoints, not 1023 bytes to me. Same issue for ifqdn
> and resourcepart.  6122 just had 1*; I think going back to that would be
> fine since we have a rule below that captures the max size.

Your proposal seems fine to me, too. It's hard to capture these nuances 
in ABNF, at times. Although, from later in the thread, "localbyte" would 
work for me.

>   > 3.2.  Domainpart
>   >
>   >    The domainpart of a JID is that portion after the '@' character (if
>   >    any) and before the '/' character (if any); it is the primary
>
> I think it's often surprising to people that foo/@bar is a valid JID with
> "foo" as the domainpart and "@bar" as the resourcepart.  The text above,
> although pulled from 6122, might be better as:
>
> The domainpart of a JID is that portion after the first '@' character (if
> any) and before the first '/' character (if any);

That's acceptable to me.

> and possibly adding the example.

Examples are good. I'll add a few to Table 1.

>   >    In general, the content of a domainpart is an Internationalized
>   >    Domain Name ("IDN") as described in the specifications for
>   >    Internationalized Domain Names in Applications (commonly called
>   >    "IDNA2008"), and a domainpart is an "IDNA-aware domain name slot" as
>   >    defined in [RFC5890].  The following rules apply to a domainpart that
>   >    consists of a fully-qualified domain name and MUST be applied in the
>   >    following order:
>
> When do these rules need to be applied? Only before comparison or routing?

That is a very good question.

This might be a difference between the "preparation" and "comparison" of 
the PRECIS acronym.

You'll notice that the PRECIS nickname spec draws a sharper distinction 
between preparation and comparison than the others:

http://www.ietf.org/archive/id/draft-ietf-precis-nickname-09.txt

Section 2 there says in part:

    For preparation purposes (most commonly, when a chatroom client
    generates a nickname from user input for inclusion as a protocol
    element that represents a "nickname slot"), an application MUST at a
    minimum ensure that the string conforms to the "FreeformClass" string
    class defined in [I-D.ietf-precis-framework]; however, it MAY in
    addition perform the normalization and mapping operations specified
    below for comparison purposes.

    For comparison purposes (e.g., when a chatroom server determines if
    two nicknames are in conflict during the authorization process), an
    application MUST treat a nickname as specified below (these rules
    constitute the "NicknameFreeformClass" profile).  The operations
    specified MUST be completed in the order shown (in particular,
    normalization MUST be performed after the other mapping steps and
    before validity-checking against the definition of the PRECIS
    "FreeformClass", consistent with [I-D.ietf-precis-framework]).

    [various rules elided]

I wonder if we want to say, in general, that there is something of a 
lower bar for preparation than for comparison. For example, for an XMPP 
localpart we might say that an entity doing preparation just needs to 
ensure that it doesn't include any characters outside of the PRECIS 
IdentifierClass, whereas an entity doing comparison needs to apply the 
normalization and mapping rules. The primary reason we might do this is 
that it could ease the burden on XMPP clients or servers during certain 
operations, whereas at those times when comparison is truly needed 
(e.g., when user authentication or authorization are being made) the 
full set of rules would be applied.

Although I'm not entirely comfortable with this approach, pragmatically 
it might be more acceptable than saying that all entities must apply all 
of the rules all of the time.

This is related to text in Section 4:

    Enforcement of the XMPP address format rules is the responsibility of
    XMPP servers.  Although XMPP clients SHOULD prepare complete JIDs and
    parts of JIDs in accordance with this document before including them
    in protocol slots within XML streams (such that JIDs and parts of
    JIDs are in conformance), XMPP servers MUST enforce the rules
    wherever possible and reject stanzas and other XML elements that
    violate the rules (for stanzas, by returning a <jid-malformed/> error
    to the sender as described in Section 8.3.3.8 of [RFC6120]).

That text seems to imply the same principle: clients prepare and servers 
enforce (by mean of comparison?). But I think we could be clearer about 
the whole matter by explicitly saying that enforcement includes 
application of all the rules (just as comparison does - it's just that 
comparison involves applying all of the rules to two strings in order to 
determine if they are "equivalent", whereas enforcement involves apply 
the rules to a single string).

>   >    1.  The domainpart MUST contain only NR-LDH labels and U-labels as
>   >        defined in [RFC5890] and MUST consist only of Unicode code points
>   >        that conform to the rules specified in [RFC5892] (which includes
>   >        Unicode normalization).  This implies that the domainpart MUST
>   >        NOT include A-labels as defined in [RFC5890]; each A-label MUST
>   >        be converted to a U-label during preparation of a domainpart, and
>   >        comparison MUST be performed using U-labels, not A-labels.
>
> This seems like an always rule, including for dumb clients.

Things are a bit more clear-cut with regard to rules that are based on 
PRECIS, not IDNA, because the models are slightly different. In PRECIS 
we have base string classes (IdentifierClass and FreeformClass), so it 
might make sense to say that preparation involves ensuring that the 
preparing entity doesn't allow in any code points that are disallowed 
for that base string class. We don't have base string classes in IDNA. 
Although the foregoing rule is similar to the base string class idea, it 
goes beyond by including normalization. I'd almost prefer that we figure 
this out very clearly first for PRECIS-based identifiers (in XMPP, the 
localpart and resourcepart) and then see how the resulting text can be 
ported over to our use of IDNA-based identifiers (in XMPP, the domainpart).

>   >    2.  All uppercase and titlecase code points within the domainpart
>   >        MUST be mapped to their lowercase equivalents, preferably using
>   >        Unicode Default Case Folding as defined in Chapter 3 of the
>   >        Unicode Standard [UNICODE].
>
> Dumb clients might get away with this and the system would still work.
>
>   >    3.  Fullwidth and halfwidth characters within the domainpart MUST be
>   >        mapped to their decomposition mappings.
>
> Dumb clients have no shot at this one.

Right - in the emerging approach we're exploring here, the latter two 
rules would be a matter of enforcement and comparison only, not of 
preparation.

>   >       Implementation Note: The foregoing order is different from the
>   >       order for localparts and resourceparts as described below, to
>   >       maintain consistency with the IDNA methods in both [RFC5892] and
>   >       [RFC5895].
>   >
>   >    After any and all normalization, conversion, and mapping of code
>   >    points,
>
> as well as conversion to UTF-8.

True, although we kind of assume that in the XMPP world because all data 
sent over an XMPP stream is required to be UTF-8. Mentioning it seems 
useful, though.

>   >    a domainpart MUST NOT be zero octets in length and MUST NOT
>   >    be more than 1023 octets in length.  (Naturally, the length limits of
>   >    [RFC1034] apply, and nothing in this document is to be interpreted as
>   >    overriding those more fundamental limits.)
>   >
>   > 3.3.  Localpart
>   >
>   >    The localpart of a JID is an optional identifier placed before the
>   >    domainpart and separated from the latter by the '@' character.
>   >    Typically a localpart uniquely identifies the entity requesting and
>   >    using network access provided by a server (i.e., a local account),
>   >    although it can also represent other kinds of entities (e.g., a chat
>   >    room associated with a multi-user chat service [XEP-0045]).  The
>   >    entity represented by an XMPP localpart is addressed within the
>   >    context of a specific domain (i.e., <localpart@domainpart>).
>   >
>   >    A localpart MUST NOT be zero octets in length and MUST NOT be more
>   >    than 1023 octets in length.  This rule is to be enforced after any
>   >    normalization and mapping of code points.
>
> and conversion to UTF-8.

As above.

>   >    A localpart MUST consist only of Unicode code points that conform to
>   >    the "JIDlocalIdentifierClass" profile of the "IdentifierClass" base
>   >    string class defined in [I-D.ietf-precis-framework].  The
>   >    JIDlocalIdentifierClass profile includes all code points allowed by
>   >    the IdentifierClass base class, with the exception of the following
>   >    characters that are explicitly disallowed in XMPP localparts:
>
> (special precis focus)
> I would have expected this to be phrased more similarly to step 2 of
> http://tools.ietf.org/html/draft-ietf-precis-framework-17#section-5, or
> for section 5 to just have a step about codepoints forbidden in a given
> usage of the selected precis class.

Good point - I agree that more internal harmony would be helpful here 
between the framework and the various profiles.

>   >    The normalization and mapping rules for the JIDlocalIdentifierClass
>   >    are as follows, where the operations specified MUST be completed in
>   >    the order shown:
>
> Again, I think we need language about when these rules are applied.  The
> rest of the section is about what is allowed, not about how to compare.

As discussed above, I think we need to more clearly delineate what's 
required for preparation, what's required for enforcement, and what's 
required for comparison. And as mentioned seems to me right now that the 
same rules are involved in enforcement and comparison, except that 
applying those rules during enforcement is a way to determine if a 
single string conforms, whereas applying those rules during comparison 
is a way to determine if two strings are "equivalent". That said, your 
use of the phrase "about what is allowed, not about how to compare" 
might suggest that more is involved in comparison than in enforcement.

To choose a simple example, is the JID <[e-mail]> "allowed" if 
the jabber.org server enforces all the rules for a localpart? It seems 
to me not. We're saying that a client could send that (since both "S" 
and "P" are allowed by the category "Lu - Uppercase_Letter" and thus 
would pass the preparation test), but that a server which is enforcing 
the rules would map "S" to "s" and "P" to "p". However, the rules for 
comparison are the same as for enforcement: "StPeter" and "stpeter" 
would compare as equivalent.

>   >    1.  Fullwidth and halfwidth characters MUST be mapped to their
>   >        decomposition mappings.
>   >
>   >    2.  Uppercase and titlecase characters MUST be mapped to their
>   >        lowercase equivalents, preferably using Unicode Default Case
>   >        Folding as defined in Chapter 3 of the Unicode Standard
>   >        [UNICODE].
>
> Nothing about SpecialCasing?

That's a question for the WG. :-)

The PRECIS framework states:

    If case mapping is desired (instead of case preservation), it is
    RECOMMENDED to use Unicode Default Case Folding as defined in Chapter
    3 of the Unicode Standard [Unicode6.3].

       Note: Unicode Default Case Folding is not designed to handle
       various localization issues (such as so-called "dotless i" in
       several Turkic languages).  The PRECIS mappings document
       [I-D.ietf-precis-mappings] describes these issues in greater
       detail and defines a "local case mapping" method that handles some
       locale-dependent and context-dependent mappings.

Given the discussions in recent PRECIS WG meetings, I would shy away 
from applying locale-dependent and context-dependent mappings in XMPP 
localparts. However, I'm open to argument.

>   >    A resourcepart MUST NOT be zero octets in length and MUST NOT be more
>   >    than 1023 octets in length.  This rule is to be enforced after any
>   >    normalization and mapping of code points.
>   >
>   >    A resourcepart MUST consist only of Unicode code points that conform
>   >    to the "JIDresourceFreeformClass" profile of the "FreeformClass" base
>   >    string class defined in [I-D.ietf-precis-framework].
>   >
>   >    The normalization and mapping rules for the resourcepart of a JID are
>   >    as follows, where the operations specified MUST be completed in the
>   >    order shown:
>
> Again, when are the rules applied?

See above.

>   >    1.  Fullwidth and halfwidth characters MAY be mapped to their
>   >        decomposition mappings.
>
> (precis)
> I need a hint as to when do this.  "MAY" isn't nearly enough.

Do you mean "when" as "in what contexts is it smart to do width mapping 
on resourceparts" or as something else (e.g., "when" could mean "by 
which entities" such as clients, servers, and XMPP "components").

Later in this thread, you and Florian Zeitz seem to think that MUST NOT 
perform width mapping is the right approach.

However, resourceparts are used in multiple contexts (we could say that 
there are multiple "resourcepart slots").

For the JIDs of connected resources (user@domain/foo), I tend to agree.

For the JIDs of chatroom participants, the precis-nickname spec says to 
use NFKC, which handles width mapping as part of normalization (and thus 
might be taken to violate the proposed MUST NOT approach).

I haven't yet taken the time to find and think about other resourcepart 
slots in various XMPP extensions, but I hesitate to make a categorical 
statement in 6122bis since the applicability of width mapping might 
depend on the context in which a resourcepart is used.

>   >    2.  Map any instances of non-ASCII space to ASCII space (U+0020).
>
> (precis)
> I was hoping either the framework doc or the mappings doc would tell me
> more about which characters to map here.  RFC 3454 had table C.1.2, but I
> don't see any hints about what I'm supposed to do now.

Good catch.

> Is the rule "has a
> compatibility mapping to U+0020"?

BTW I count at least three kinds of compatibility mapping to 0020: 
<compat> (as in U+0384 GREEK TONOS), <noBreak> (as in U+2007 FIGURE 
SPACE)), and <wide> (as in U+3000 IDEOGRAPHIC SPACE).

> That doesn't hit U+200B which is in
> C.1.2,

Right. I am not sure whether ZERO WIDTH SPACE really ought to be mapped 
to U+0020. See Florian's comment later in this thread.

> nor does "has category Zs".

IMHO that is insufficient.

My intuition is that by "non-ASCII space" we mean anything that has a 
compatibility mapping of any kind of U-0020, since that seems safest (it 
casts a wider net) and is something we can apply in a programmatic way. 
However, my intuitions are not always correct and applying this rule 
this would result in a larger table than what we find in Appendix C.1.2 
of RFC 3454.

> draft-ietf-precis-mappings says
> "Therefore, the special mapping table should be based on a well-
>     defined mapping table for each protocol", which although I don't
> particularly like, I can live with - but we need the table here.

Do you feel that we need the table in 6122bis or in the framework? As 
you say, the mappings document implies that each specification that 
defines a rule like "map non-ASCII space to ASCII space" needs to define 
their own table, but that seems like a recipe for trouble. If, say, SASL 
and XMPP and LDAP each defines a different table, authentication might 
become confusing (especially since XMPP uses SASL and authentication 
might be based on an LDAP lookup).

>   >    3.  So-called additional mappings MAY be applied, such as mapping of
>   >        characters that are similar to common delimiters (such as '@',
>   >        ':', '/', '+', '-', and '.', e.g., mapping of IDEOGRAPHIC FULL
>   >        STOP (U+3002) to FULL STOP (U+002E)) and special handling of
>   >        certain characters or classes of characters (e.g., mapping of
>   >        non-ASCII spaces to ASCII space); the PRECIS mappings document
>   >        [I-D.ietf-precis-mappings] describes such mappings in more
>   >        detail.
>   >
>   >    4.  Uppercase and titlecase characters MAY be mapped to their
>   >        lowercase equivalents, preferably using Unicode Default Case
>   >        Folding as defined in Chapter 3 of the Unicode Standard
>   >        [UNICODE].
>
> Again, I need more about the MAY here.
>
>   > 6.  IANA Considerations
>   >
>   >    The following completed templates provide the information necessary
>   >    for the IANA to add 'JIDlocalIdentifierClass' and
>   >    'JIDresourceFreeformClass' to the PRECIS Profiles Registry.
>
> Should we also ask them to mark the status of nodeprep and resourceprep to
> deprecated in the stringprep profiles registry?

Yes.

>   > Appendix A.  Differences from RFC 6122
>   >
>   >    Based on consensus derived from working group discussion,
>   >    implementation and deployment experience, and formal interoperability
>   >    testing, the following substantive modifications were made from RFC
>   >    6122.
>
> I think it might be nice to point out that this may have made
> previously-valid JIDs no longer valid (or vice-versa), and that we suggest
> careful testing before migrating user data.

+1 to at least that text. Ideally we'd perform the kind of analysis that 
Takahiro Nemoto performed for SASLprep vs. SASLprepbis:

http://www.ietf.org/mail-archive/web/precis/current/msg00790.html

I haven't done that yet, though.

Thanks again to you and Florian for your careful reviews.

Peter