[idn] stringprep: existing profiles and string processing complexity

[Erik van der Poel <[e-mail]>]

Simon Josefsson wrote:

> The issue should be handled in an update of the StringPrep.  If the
> PR-29 fix is incorporated, there must a good transition discussion in
> the document.  The problem cannot only be discussed in the realm of
> IDNA, since StringPrep is used for other purposes as well.  For me,
> the most important use is SASLprep, because it is used to prepare
> username and passwords.  Hence, it is used in security critical
> application, where the requirements are different than from IDNA.  The
> discussion must be wider than for only IDNA.

I downloaded all the RFCs and Internet Drafts and grepped for 
Stringprep. Some of the profiles specified in RFCs are not registered at 
IANA. There are 3 profiles in an NFS4 spec (RFC 3530) that have not been 
registered. It is not clear to me that profiles must be registered. I 
have started a table of profiles that shows which parts of Stringprep 
are being used. So far, I have only listed profiles found in RFCs (not 
Internet Drafts). This table was far easier to create in an HTML editor, 
and I didn't want to send HTML to this list, so it's on my Web site:

http://nameprep.org/stringprep.html

It's still rough and doesn't indicate all the details, but it's a start. 
As I read the profiles and text in their vicinity, it seems as though 
some may not be fully aware of some of the complexities of string 
processing in Unicode. (This is not a flaw in Unicode. There are various 
written languages in the world, and some of them have complex rules.)

For example, RFC 3530 says:

    If the case_preserving attribute is present and set to false, then
    the NFS version 4 server MUST use table B.2 to map case when
    processing utf8str_cs strings.  Whether the server maps from lower to
    upper case or the upper to lower case is an implementation
    dependency.

Stringprep's case mapping table is intended to be used only to map upper 
case to lower case. For example, here is a special case:

    00DF; 0073 0073; Case map

You can't really use this entry to go "backwards".

There are other RFCs and Internet Drafts with various issues like this, 
or that are so vague as to make you wonder whether interoperability is 
truly assured. The IETF community is of course full of networking 
experts, and the Unicode community is full of internationalization 
experts. Stringprep finds itself in the rather special position of being 
wedged between networking (specifically, string input and matching) and 
Unicode. It might be a good idea to collect the issues and try to 
increase understanding of international string processing among 
networking community members.

Erik