Re: [I18ndir] I18ndir last call review of draft-ietf-regext-dnrd-objects-mapping-06

[John C Klensin <[e-mail]>]

Asmus,

Sorry -- I read and responded to Marc's note before seeing yours
and Patrik's.   I think we are in complete agreement, but see
comments inline below.

--On Friday, March 6, 2020 00:40 -0800 "Asmus Freytag (c)"
<[e-mail]> wrote:

> On 3/5/2020 9:46 PM, John C Klensin wrote:
>> Asmus,
>> 
>> --On Thursday, March 5, 2020 16:14 -0800 Asmus Freytag
>> <[e-mail]> wrote:
>> 
>>> On 3/5/2020 12:47 PM, John C Klensin wrote:
>>> 
>>>> In addition, given increasing trend on the web 9at
>>>> least) to do exactly what TUS says to do, which is to
>>>> normalize only at comparison time rather than trying to
>>>> carry strings around in normalized form, the application of
>>>> that attribute all almost all text-type values may be
>>>> inappropriate. I can find no evidence in the I-D that those
>>>> issues were considered; the document should not progress
>>>> until they are.
>>> Right, there are many other contexts where it makes sense to
>>> keep the data as submitted and not to force normalization.
>>   
>>> However, this does not appear to be one of them.
>> For the many fields that do not appear to be IDN labels, why
>> not?  From skimming the document, I'd assume that the IDN
>> labels should be required to be in NFC but that a
>> normalization requirement is probably inappropriate for most
>> other fields, especially those fields that appear to be free
>> text descriptions.  Which ones probably requires a
>> field-by-field analysis.
>> 
>>> We may need a statement specific to IETF that captures the
>>> suggested policy.
>> Indeed.
>> 
> I did not do a full review, just checking some of the points
> that
> others had noted against the original.
> 
> That means I have not looked into the details of all the text
> fields. As a matter of policy, if we were to develop one to
> help
> us guide in our review, we may perhaps partition:

I think not just our reviews, but the advice we give to WGs
developing protocols and data formats.  That, of course, gets
back to the question as to whether (and when) it will be
appropriate to take the next steps beyond statements like "just
use UTF-8" and "just normalize everything" [1] and toward "don't
normalize (or tamper with case) unless you are actually about to
compare or there is a particular protocol-specific reason to do
so" and "don't use or allow anything other than UTF-8 unless
there is good reason to do so and you can document that reason"

> (1) IDN U-labels
> 
> (2) Free text (description, comment)
> 
> (3) Other, protocol relevant
 
> It makes sense to require (1) to be in the format specified by
> IDNA (NFC and lowercase), and likewise it does not make sense
> for (2) to be normalized.

Right.   See also my note to Marc.

To try to keep our language precise, for IDN U-labels, the
format is not only a matter of making sense.  By the definition
-- which was written very carefully and intentionally -- if a
string is not NFC and lowercase (and conformant to most other
IDNA requirements) it is simply not a U-label.  That is the
reason why parts of the IDNA2008 specs are sprinkled with terms
like "putative U-label".

But, as far as I can tell from another quick pass through this
document, your case (1) does not exist at all or may be
violated: domain objects, as described in Section 5.1.1.1, use
A-labels and not U-labels; "U-label" does not appear in the
document at all.  However, NNDN elements (there and elsewhere)
are Unicode strings and, after reading the definition of NNDN in
Section 1 a couple of times, I can't tell whether their
components are expected to be IDNA2008-valid U-labels or not.
IMO, it is another deficiency in this I-D that this is not clear
and that either U-labels must be required or there must be more
explanation why not.

> String that may be relevant as part of some protocol, but are
> not
> IDN U-labels, would be RECOMMENDED to be in whatever
> format the protocol sets out for them. (There's a small
> benefit to be able to use non-protocol aware tools to do
> things like search data sets, and therefore it's preferable to
> have the data in the expected form for easy comparison,
> even if the protocol describes its own preprocessing step.

Right.  However, there is a separate recommendation for the
protocols themselves.  We should we try to minimize the number
of different preferred forms floating around.  For your case (2)
there is no preferred form.  But for the third case and
especially because most of the people designing protocols and
implementing libraries that are not i18n-specific cannot be
expected to be i18n or Unicode experts and because we have a
great deal of experience with strings "leaking" between protocol
contexts, having many different preferred formats out there is
an invitation to confusion and bugs.  The tradeoffs may need to
be evaluated carefully on a case-by-case basis, but simplicity
in the number of options and formats may trump optimality for
the needs of a particular protocol.

> To give an example: Unicode allows for character and property
> names to elide spaces with or without camel case or to use -
> or _ as separators. However, in the context of the standard
> as much as possible only one of these formats will be used
> (single case with space as separator for character names and
> _ separator for properties).

Hmm.  Thanks, I just learned something because I had not studied
(or had forgotten) that set of rules.  [2]

 
> While everybody knows how to compare the values, it helps
> the reader to know that the text of the standard follows a
> uniform convention.
> 
> If a protocol allows mulitple forms, as Unicode does for its
> identifiers, it would still be good form to pick a canonical
> representation for such protocol-relevant items and as
> reviewers we should have a policy on that.

Yes.  And I think it is more than "as reviewers".  As discussed
in earlier notes, we may be getting to the point that it is time
for us to recommend, and the IETF to establish, some clear
guidance on these subjects so that we minimize the number of
situations in which WGs develop work and then we come along
during IETF Last Call and tell them they messed up.  Patrik may
be right that we need more reviews and responses before we are
ready for that, but my sense is that we are fairly close.

best,
   john


[1]  I am not familiar enough with the Regext set of documents
to know if "normalizedString" is defined somewhere.  Certainly,
draft-ietf-regext-dnrd-objects-mapping-06 uses it without
defining it locally or providing a specific normative reference
to such definitions.  At minimum, this suggests another problem
with the I-D that may need fixing.  But, if there is not a clear
definition -- if the instruction for what should go into those
fields is essentially "just normalize", than that is obviously
bad news.  It is bad news even for IDN U-labels because a string
in, e.g., NFC would not conform to the requirements for
U-labels.  Normal IETF and RFC practice requires a statement,
probably in the Terminology section and probably with a
normative reference, to where definitions of that and other
attributes can be found.

[2] It might be of interest that there are two reasons we don't
have "_" in LDH-style domain names because they were prohibited
in the original host name syntax.  And they were prohibited
there precisely because of a human factors argument: when
reading the handwriting of others, people cannot reliably
distinguish between "-" and "_", so it is best to allow one and
not the other.  And the hyphen was chosen in part because the
most common grapheme for the BCD character that evolved into "_"
in ASCII (and ECBDIC) was a left-pointing arrow, not a
horizontal bar at the baseline.  So the ARPANET decided to go
with a single form at the codepoint level while the Unicode
decision was to allow different forms and canonicalize later.
Of course, we have a different variation on the Unicode decision
with "FWS" and "CWFS" constructions in some well-known protocols.