Re: [I18nrp] [art] Use Unicode if Using Unicode?

[Shawn Steele <[e-mail]>]

The file Unicode generates can be used either with the UTS46 compatibility characters or without.  To use your terms, it both provides data that conforms to the IDNA2008 rules... and hybrid versions.  If you choose the appropriate information when reading the IdnaMappingTable.txt you should have exactly the table expected.  In particular the NV8 and XV8 tagged characters I believe you'd prefer to exclude.

I'm unaware of other differences.  I could also be in error, I haven't actually done an exhaustive comparison of the IndaMappingTable.txt data vs generated datasets via the rules.

The IdnaMappingTable.txt file also isn't "authoritative," however since it's generated from the Unicode data and the rules in the RFCs, the file should be as accurate as any other reference.  

Unicode is aware of the importance of the stability of the valid character set, so probably the best time to review and vet the Unicode changes for new Unicode versions would be during the Unicode process, prior to release.  After a Unicode version was released, any disagreements would lead to problematic confusion (as we've seen before.)

It seems to me that using the IdnaMappingTable.txt - perhaps with guidance as to proper usage - would reduce delays in adopting new characters.

-Shawn

-----Original Message-----
From: John C Klensin <[e-mail]> 
Sent: Wednesday, October 10, 2018 7:01 PM
To: Shawn Steele <[e-mail]>om>; [e-mail]; [e-mail]
Subject: Re: [art] Use Unicode if Using Unicode?

Shawn,

I'm confused about what you are suggesting, so let me clarify where I'm confused and then hope you can enlighten me...

--On Wednesday, October 10, 2018 19:02 +0000 Shawn Steele <Shawn.Steele=[e-mail]> wrote:

> The draft states "It further suggests for the IETF a path forward 
> regarding ensuring IDNA2008 follows the evolution of the Unicode 
> Standard" and "this document requests that IANA update the tables to 
> Unicode 11."

> Each Unicode version creates a data file with information from 
> applying the IDNA2008 rules that can be used for IDNA mapping 
> algorithms.  (Indeed, that's where Windows gets the data from).

Taking a half-step back, IDNA imposes two requirements for new versions of Unicode, both of which are addressed by the present draft.  One is that the changes in the new version be examined to ensure that nothing new has been done that requires changes to IDNA (most likely adding exceptions or rules for particular code points) or careful explanation somewhere.  No one I know of expected that provision would be exercised very often but, for a variety of reasons, the consensus was that it was an important
safeguard.   The other was that the IDNA ruleset be run against
the characters and properties in the Unicode Character Database to produce a table reflecting the combination of that version of Unicode and IDNA, as modified, at that time.  That table was (and is) to be stored with IANA but, while we expect it to be checked carefully for accuracy, it is not actually authoritative
-- only the rules and categories specified by IDNA (and the Unicode properties used to support them) are.

The discussion in RFC 5895 notwithstanding, one of the critically important properties of IDNA2008 is that U-labels and A-labels are duals: one can get from one to the other and back without any loss of information.  That reversibility is true in only some cases for Unicode normalization (especially with compatibility normalization) or case folding, much less for other mapping scenarios.  So I don't understand what sort of "mapping" you are talking about.

The only Unicode-created IDN data files I'm aware of are those associated with the UTS#46 effort.  Because UTS#46 makes recommendations that are inconsistent with IDNA2008, if Microsoft is using those tables, its usage is non-conforming to IDNA2008.  I certainly cannot prevent Microsoft from doing that (and wouldn't try), but it would certainly not be consistent with general interoperability of IDNs or what is known elsewhere as Universal Acceptance of those domain names.

> If the goal is to "follow the evolution of the Unicode Standard" and 
> the Unicode Standard is providing data that conforms to the IDNA 
> rules, then why not just point directly to the Unicode derived tables?

The simplest answer to your question is that, unless I've missed something, the conditional is false: the Unicode Standard is not providing data that conforms to the IDNA2008 rules.  Instead, the data they are providing is more like one of the creative
IDNA2003-IDNA2008 hybrids to which Patrik refers.

   best,
     john