Re: [I18ndir] I-D on filesystem I18N

[Nico Williams <[e-mail]>]

On Tue, Jul 07, 2020 at 04:43:42PM -0700, Asmus Freytag wrote:
> On 7/6/2020 3:51 PM, Nico Williams wrote:
> > I've submitted draft-williams-filesystem-18n-00.
> 
> Here are my comments:

Thanks!

> >     This document describes requirements for internationalization (I18N)
> >     of filesystems specifically in the context of Internet protocols, the
> >     architecture for filesystems in most currently popular general
> >     purpose operating systems, and their implications for filesystem
> >     I18N.  From the I18N requirements for filesystems and the
> 
> The first sentence doesn't scan - the constructions joined by "and"
> seemingly are not parallel enough.

I think the "their" in "and their implications" is unclear as to
antecedent.  I'll wordsmith it.

> >     [TBD: Add references galore.  How to reference Unicode?  How to
> 
> If you go on the Unicode site and look for the page on the latest version
> (currently https://www.unicode.org/versions/Unicode13.0.0/), you can
> navigate to suggested ways to reference. I would cite the latest version as
> of the time of drafting of the I-D and then also write: "the latest version
> is available at" and give the URL as http:\\www.unicode.org\versions\latest.

The question was about bibxml specifically, though I should have made
that clear.  I failed to update that comment -- I did figure out how to
reference Unicode, and did reference it.

I do still need to add a number of other references.  And for the two
books I referenced I should add section numbers.

> If you need to reference specifically the properties, the character database
> is at https://www.unicode.org/Public/13.0.0/ or 
> https://www.unicode.org/Public/UCD/latest/ -- You could also link to UAX#44,
> which is the overview of the Unicode Character Database. The "latest" is at:
> https://www.unicode.org/reports/tr44/ and the current one, as of today, is
> at https://www.unicode.org/reports/tr44/tr44-26.html (see the "This version"
> link for the latest).

In XML terms... you're saying I should use <eref>, I think.

> >   To deal with the equivalence problem, Unicode defines Normal Forms
> Unicode calls these "Normalization Forms", see Section 3.11 in TUS (The
> Unicode Standard) so that is what should be used when capitalized. Ditto for
> the formal names for NFC and NFD.
> 
> I think you need to mention already here that NFC/NFD represent a semantic
> identity (and generally identical appearance) while NFKC and NFCD abstract
> away rather noticeable differences in appearance that may in some cases
> imply strong semantic differences to some users (e.g. math alphabets).

I chose to do so one page later because I wanted to keep the intro
brief:

|   Unicode compatibility equivalence allows equivalence between
|   different representations of the same abstract character that may
|   nonetheless have different visual appearance of behavior.  There are
|   two canonical forms that support compatibility equivalence: NFKC and
|   NFKD.  Using NoCL with NFKC or NFKD may be surprising to users in a
|   visual way.  While form-insensitivity with NFKC or NFKD may surprise
|   users who might consider two file names distinct even when Unicode
|   considers them equivalent under compatibility equivalence.  The
|   latter seems less likely and less surprising, though that is an
|   entirely subjective judgement.

Here "NoCL" == normalize-on-CREATE-and-LOOKUP, defined just above the
quoted text.

Does that work for you?

> >   Unicode compatibility equivalence allows equivalence between
> >     different representations of the same abstract character that may
> >     nonetheless have different visual appearance of behavior.  There are
> >     two canonical forms that support compatibility equivalence: NFKC and
        ^^^^^^^^^^^^^
> >     NFKD.  Using NoCL with NFKC or NFKD may be surprising to users in a
> >     visual way.  While form-insensitivity with NFKC or NFKD may surprise
> >     users who might consider two file names distinct even when Unicode
> >     considers them equivalent under compatibility equivalence.  The
> >     latter seems less likely and less surprising, though that is an
> >     entirely subjective judgement.
>
> You really MUST not use the term "canonical" with NFKC and NFKD because for
> Unicode, the two forms NFC and NFD are considered "canonical" and that term
> is used contrastively to "compatibility".

The word 'canonical' was preceded by 'two' -- I think that's pretty
clear!  :)

> There's an "of" that should be an "or", but it's not just behavior - it's
> also meaning. Mathematicians would object strenuously to having their math
> alphabets "normalized" to standard A-Z.

Ay, yes, thanks!

> NFK(C/D) is useful in a different way: if it used to disallow code points
> that aren't stable under these normalization forms, then one sidesteps the
> issue of whether the distinct appearance etc. is meaningful, but without
> ever changing a filename (which would happen if it was normalized when
> stored). (There may be no file-systems that take this approach, nevertheless
> it's worth discussing as it is used in other naming schemes).

ZFS (in Solaris and most ports, but not OS X) form-insensitivity, and
lets the user choose a form for use for this.

See https://docs.oracle.com/cd/E71909_01/html/E71919/gpssl.html which
shows that NFC, NFD, NFKC, and NFKC.

It doesn't make much sense to offer NFC or NFKC for this given that the
behavior is form-insensitive and form-preserving (NFC is specified as
requiring canonical decomposition as a first step, which means that NFC
is very likely slower than NFD, and the same applies to NFKC and NFKD).
But whatever.

In at least the original port to OS X, before Apple abandoned it, ZFS
was made to normalize-on-CREATE (and LOOKUP) instead of being
form-preserving.  This was probably done to match HFS+'s behavior.  I
remember a post by Linus Torvalds complaining bitterly about this and
blaming ZFS in general rather than Apple, when some user complained that
Git did not handle this case correctly.  The issue was that Git looked
for the file using the name it knew from the repository, and that was in
NFC because of course the input mode used when the file was committed
produced something NFC-ish, but on HFS+ this got converted to NFD, and
Git then failed to find the file by memcmp() in the directory listing.
Ironically, that very possibility had been my motivation for pushing for
form-insensitive and form-preserving behavior in ZFS in Solaris to begin
with, but Apple engineers insisted on the HFS+ behavior in their port.

Apple's choice of NFD makes sense for performance reasons since it's
faster to normalize to form D than to form C (see above).  But
normalizing on CREATE to NFD conflicts with input modes that tend to
produce pre-composed character codepoints.  Form-insensitivity is the
obvious compromise.  Until the Git incident, I had no evidence of that
actually causing problems.

Git nowadays contains normalization code to deal with this, whereas it
could have not had any if we'd had consensus on form-insensitivity a few
years before we added it to ZFS.  Bummer.  Still, form-insensitivity is
clearly the better behavior, though I'm not proposing that we mandate
it.

> >     foldings are defined by Unicode.  Generally, case-insensitive
> >     filesystems preserve original case just form-insensitive filesystems
> >     preserve original form.
>
> There's an "as" missing.

Thanks :)

> However, early case-insensitive file systems did not preserve case. Not sure
> how rare this has become.

So.. sentence-case?  I refer to that later:

|   The only way to implement I18N behaviors in the VFS layer rather than
|   at the filesystem is to abandon form- and case-preserving behaviors.
|   For case-insensitivity this would require using sentence-case, or all
|   lower-case, perhaps, and all such choices would surely be surprising
|   to users.  At any rate, that approach would also render much running
|   code "non-compliant" with any Internet filesystem protocol I18N
|   specification.

> >     listings that work the same way as on the server.  We do not specify
> >     any case foldings here.  Instead we will either create a registry of
>
> the "here" is unclear. Does it refer to recommendation in this ID relevant
> to caching clients? If so, link to section.

'here' == "this document".  I'll clarify.

> >    "just-use-8" or "just-use-16" (as in UTF-16 [UNICODE  <https://tools.ietf.org/html/draft-williams-filesystem-18n-00#ref-UNICODE>]), with no
> >     attempt at normalization or case folding done anywhere in between.
>
> In Unicode parlance: "just use strings of code units".

"just-use-8" means strings of 8-bit words that could be... any 8-bit
codeset, or UTF-8, or whatever.

"just-use-16" is almost certainly "as in UCS-2" or "as in UTF-16",
though for all I know there may be some non-Unicode 16-bit wchar_t
codesets in use as "just-use-16".

> Specifically for UTF-8, this would imply that there are also no guarantees
> of well-formedness of the UTF-8 strings (likewise for surrogate pairs in
> UTF-16).

Well, yes.  But that wasn't what I was alluding to.  I was referring to,
e.g., using Linux (or *BSD, or Solaris/Illumos) in, say, an ISO-8859-1
locale.  In that case you might have an application all `creat(2)` with
a `pathname` that contains non-ASCII that is also not UTF-8.

The C library system call stub will *not* convert this to UTF-8 or
anything else, nor will the kernel-land system call know anything about
the locale in use in user-land.  The given `pathname` string will just
be a pointer to a zero-terminated array of `char` in user-land, and will
be copied into the kernel as such by the kernel-side of the system call,
then it will be passed to the VFS.

The VFS will interpret just two byte values specially: 0x00 (ASCII NULL)
and 0x2F (ASCII '/').  The filesystems will generally only interpret
0x00 specially, though some (e.g., ZFS) may reject strings that are not
valid UTF-8.

We call this "just-use-8" in some circles (e.g., in the KITTEN WG).  I
don't recall who coined that.

I expect something similar happens in Windows for both, the *A()
(just-use-8?) and *W() functions (just-use-16?).  But I'm not
sufficiently experienced a Windows programmer to really know.

"just-use-16" is my coinage as an obvious variation of "just-use-8".

> -- reached end of section 1 and out of time slot --

Thanks so much!

Nico
--