Re: [I18ndir] I18ndir last call review of draft-ietf-regext-dnrd-objects-mapping-06

[John C Klensin <[e-mail]>]

--On Friday, March 6, 2020 08:42 -0800 "Asmus Freytag (c)"
<[e-mail]> wrote:

> On 3/6/2020 7:45 AM, John C Klensin wrote:
>> [2] It might be of interest that there are two reasons we
>> don't have "_" in LDH-style domain names because they were
>> prohibited in the original host name syntax.  And they were
>> prohibited there precisely because of a human factors
>> argument: when reading the handwriting of others, people
>> cannot reliably distinguish between "-" and "_", so it is
>> best to allow one and not the other.  And the hyphen was
>> chosen in part because the most common grapheme for the BCD
>> character that evolved into "_" in ASCII (and ECBDIC) was a
>> left-pointing arrow, not a horizontal bar at the baseline.
>> So the ARPANET decided to go with a single form at the
>> codepoint level while the Unicode decision was to allow
>> different forms and canonicalize later. Of course, we have a
>> different variation on the Unicode decision with "FWS" and
>> "CWFS" constructions in some well-known protocols.
> 
> Unicode's "loose matching" rules are designed to help these
> identifiers to be embedded as needed in other protocols, where
> there are native restrictions. Unicode does provide an
> exhaustive
> list of what those other protocols might be, but it's been
> clear
> that they include, for examples, programming languages.
> 
> Bu making sure identifiers stay unique without requiring
> literal
> hyphens, spaces or underlines, this can be accommodated.

Interesting and helpful.  Of course, the tradeoffs between
defining a canonical form and using and carrying it around as
much as possible (certainly between systems and in archival
formats) versus allowing local forms and converting to canonical
ones only when necessary is an ancient one that won't get
settled today.  In Multics, canonicalization was expected to
occur in the terminal drivers, so it was hard to even enter a
non-canonical form string. 
 
> I was simply speculating that for any 'new' protocol that
> collects data that have a defined format in other protocols,
> it may make sense to not force these data to be normalized
> or canonicalized different from what the external protocol
> expects.
> 
> If external protocols allow multiple formats, then it may make
> sense to pick one canonical form for the 'new' protocol, just
> so data in the new protocol are simplified.
> 
> It depends a bit on the use case. A protocol that's a transient
> container of data transmitted between implementations
> of an external protocol is a different matter from a database
> that collects information from potentially different sources,
> but could be expected to have some internal consistency.

Right.  There is another issue which might be relevant to a
record-keeping (or archival) storage format like this one and
would be different from our "on the wire" expectations.   It
takes us back to the reason the Punycode algorithm was
specified.   If one expects most of all of the code points one
will have to deal with will come out of the BMP, UTF-16 may
actually make more sense that UTF-8.  Part of the motivation for
the Punycode work is that UTF-8 takes up more space for East
Asian scripts than it does for ASCII.    For a file or stream
that is mostly ASCII but with some other characters, UTF-8 is
usually still a win.  But, for a file that is mostly in, say,
Han characters, UTF-16 will typically be much more compact and
will eliminate most or all of the decoding problems associated
with UTF-8.  I still don't think that says "use UTF-8 or
whatever you like" as the I-D appears to say, but it may be a
tad less obvious that UTF-8 is the right answer then for data
that are expected to be transferred regularly over the network.

> ps: if/when we get around to writing this down, the language
> will need to become much more precise than these musings.

Of course.  Always.

best,
  john