Re: [Idna-update] Genart telechat review of draft-faltstrom-unicode11-08

[Asmus Freytag <[e-mail]>]

On 3/18/2019 11:47 PM, Patrik Fältström wrote:
> 6. The fact it was an issue was of course also brought up to the Unicode Consortium as I am (handy enough) also the liaison from IETF to Unicode Consortium. No response. They do push, as we know, TR#46 which is a different kind of animal than IDNA2008. Oh well. Things where fine, and we moved on. And IETF could focus on what IETF really should focus on, to ensure the algorithm itself was still "ok". I.e. without looking at individual code points, but more in general terms, was IDNA2008 still good enough? Lots of good feedback from people like Asmus that was fighting like mad (and still is) trying to a. convince registries they MUST come up with a subset of IDNA2008 permissible code points when they decide what can be used in their zones and b. come up with processes and rules for how that work should be done. Specifically work did go on for a long time regarding what code points can be used in the root zone, i.e. for TLDs. This is managed in the IAB document, and the Klensin draft about and more.
>
> 6. Now in version 11.0.0, the non-backward-compatible change happened again. Sigh.

I think its worth injecting some pragmatism here.

We know that only about 35K code points out of 130K total are actually 
used in someones orthography, where that someone is a speaker of a 
language used for daily activities and with good intergenerational 
transmission (= not likely to die out soon).

All the other 100K code points are either symbols or punctuation or, 
more often, code points that are recognized only by scholars.

Very little activity in Unicode affects that first set of 35K code 
points: future changes may be possible due to orthography reforms, as 
well as some population acquiring widespread literacy for the first 
time, but other than that, there's not much going on that needs new code 
points. There's the occasional case where the properties for some of the 
lesser used code points in this set may turn out to require corrections 
as result of software implementations for these becoming more mainstream 
and thus discovering issues. Generally, these affect languages where 
digital deployment itself is still nascent; and only a small subset 
potentially affects IDNA category.

The upshot is that the practical effect of such activity in the margins 
is effectively invisible.

The overwhelming part of the additions and other changes are within the 
remaining set of 100K specialized characters. These code points are 
either misused for malicious registrations, relying on users' 
unfamiliarity with these rare code points or of use only to specialized 
audiences.

The practical effect of any incompatibilities in that set is, again, 
effectively minuscule; even given the realistic expectation that the 
property data for these won't get rigorously tested in practical 
applications and therefore could be found incorrect at some future point.


>   I was at that time on my way to just give up, and recommend IETF that what we do today, to reference something that we thought was normative, stable etc (i.e. the Unicode Standard) did not work. That IETF instead would pick up and use some relationship with ISO that should be formalized, and instead of referencing the Unicode Consortium IETF should reference ISO, and then ISO should have in its rules that backward compatibility would be a requirement.


This is a complete nonstarter.

Nobody wins if we encourage the existence of two different encoding 
standards - and the current process of cooperation works precisely 
because it doesn't work the way it's sketched below.

> If then Unicode did ship something to ISO for "approval", then ISO would say "no, try again" or else ISO would make a process violation that members could appeal within ISO. But after sleeping on this for a bit, I felt the issue with 11.0.0 also happened with 7.0.0 and maybe one should first have a discussion within IETF again before the bolts are blown regarding relationship with Unicode Consortium. That just started when the I18N directorate was closed, and I ended up in a void. After long time, pushing and what not (lets discuss in the bar) this group finally was created were this VERY SERIOUS ISSUE could be discussed, and as John explained, there are many things to discuss, where we can boil it down to two:
>
> A. Should the non-backward compatible change in 11.0.0 result in a change in rule [G] in RFC 5892, or should we accept a non-backward compatible change? To trigger the discussing, I proposed the same result as for 7.0.0, to NOT update [G], and simply accept it. This also because the next issue is the important one.
>
> B. What should we do with IDNA2008? Obviously Unicode Standard is not stable enough. Or is it? What should we do with review? Should we have to start do what Martin just did with 12.0.0?

what Martin did with Unicode 12.0.0 is what I call a sanity check. I 
think it's a reasonable activity (if resources exist), but I would not 
expect to find any issues that are of enough practical importance to 
actually matter. (See above).


>   Do we IETF have the expertise? Can we rely on individuals like Martin, Asmus, myself and John and very few more to be around? Can we rely on Unicode Consortium? Is this the time to instead move to ISO? Like keeping IDNA2008 algorithm BUT tie it to ISO approved charset, and then ask ISO to protect the backward compatibility? Or are we done, so that we simply freeze IDN to a specific Unicode version, and simply ignore all added code points after that version? I.e. at 5.2.0 we knew some interesting code points where still to be added, but after 11.0.0?
>
> What I now know is that IESG have told me that we do have agreement in IETF on draft-faltstrom-unicode11-08.txt which implies moving to IDNA 11.0.0 without adding things to [G] is the path forward for now.
>
> What to do with 12.0.0 and future versions is still up in the air.


I believe the correct answer is to have a very serious discussion on how 
to help registries to make sure their support for the widely needed 35K 
is more robust and not to get sidetracked by the vast wasteland of 
obsolescent, obsolete, ancient or otherwise specialized use code points 
(whether for liturgical, phonetic, or poetic usage).

Continue to keep a watchful eye over the application of the algorithm 
for future versions, but don't expect that any cases will rise to the 
threshold where any exceptions need to be baked into the protocol - with 
all the attendant issues for updating software libraries.

But focus on solving issues that have at least practical impact.

A./