Re: [apps-discuss] I-D Action: draft-ietf-appsawg-file-scheme-09.txt

[Matthew Kerwin <[e-mail]>]

Hi Mark, sorry for sitting on this for another week. I've been called up
for jury duty.

On 20 May 2016 at 18:55, Mark Nottingham <[e-mail]> wrote:

>
> > On 20 May 2016, at 4:37 PM, Matthew Kerwin <[e-mail]>
> wrote:
> >
> >> >    Without other encoding information, percent-encoded octets in a
> file
> >> >    URI ([RFC3986], Section 2.1) MAY be interpreted according to the
> >> >    preferred or configured encoding of the system on which the URI is
> >> >    being interpreted.
> >>
> >>
> >> Do the current implementations of file:// do this -- i.e., use the
> filesystem's encoding for the URI?
> >
> >
> > ​Apparently. I don't have a spare drive lying around where I can
> reformat a partition to test it for myself, though. A discussion I had with
> Dave Thaler back at the very start of this draft revolved around the fact
> that percent-encoded URIs are ambiguous (apparently a real issue for
> Windows), which was why for a very long time the draft contained advice to
> use an IRI​ instead, or at the least normalize.
>
> VMs are good for testing.
>
>
​But some operating systems don't come cheap, if you don't happen to
already have a Windows VM to hand. ;)​



> It appears that both Windows and OSX have used UTF-8 for file name
> encoding for some time (since NT for the former, 10.0 for the latter,
> AIUI). See: <
> https://docs.python.org/3/library/sys.html#sys.getfilesystemencoding>
>

​I thought NTFS used UCS2 for storing file and directory names; and the
Windows API functions I've seen (like PathCreateFromURL) return either UCS2
or Windows-1252 strings, depending on how you use them. I can't speak to
HFS+.​



>
> Linux uses whatever locale is set. However, it appears that both Gnome and
> Firefox (at least) try to be 'smart' and will recognise UTF-8 even if
> ISO-8859-1 is set as the locale. Having said that, it's not too smart; if I
> try to open a file with a UTF-8 encoded name, Firefox can't find it when
> the locale isn't UTF-8 (although the file chooser *does* see it).
>
> This is an important point; the advice above that they "MAY be interpreted
> according to the preferred or configured encoding of the system on which
> the URI is being interpreted" doesn't account for the fact that a single
> filesystem might have several users who have different encodings set*.
>
>
​Not explicitly, although I had it in mind when I wrote it. Maybe your
"heuristics" bit below covers it well enough.​



> Other encodings seem to just be percent-encoded straight into the file
> URI, and Firefox doesn't make any attempt to display them as IRIs.
>
> (This seems to mirror how most browsers handle non-ascii characters in
> HTTP headers, e.g., Location; they just percent-encode them, since that's
> an encoding of bytes, not characters).
>
> Can we say something more like this?
>
> ---%<---
> When a file URI is produced, characters not allowed by the ABNF MUST be
> percent-encoded as characters using UTF-8 encoding, as per
> ​​
> RFC3986 Section 2.5.
>
>
​That's strengthening the requirement. RFC3986 §2.5 only says that: if
characters are part of UCS they "should first be encoded" as UTF-8 before
being percent-encoded. The MUST-level requirement is just that octets not
allowed by the ABNF must be percent-encoded.



> However, encoding information for file and/or directory names might not be
> available. In these cases, implementations MAY use heuristics to determine
> the encoding. If that fails, they SHOULD percent-encode the raw bytes of
> the label directly.
>

​This says this part well.​



> --->%---
>
> Cheers,
>
>
> * Possible but unlikely, since most people are going to be using UTF-8.
> Still...
>
>
​Cheers
-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/