Re: [apps-discuss] I-D Action: draft-ietf-appsawg-file-scheme-09.txt

[Mark Nottingham <[e-mail]>]

> On 20 May 2016, at 4:37 PM, Matthew Kerwin <[e-mail]> wrote:
> 
>> >    Without other encoding information, percent-encoded octets in a file
>> >    URI ([RFC3986], Section 2.1) MAY be interpreted according to the
>> >    preferred or configured encoding of the system on which the URI is
>> >    being interpreted.
>> 
>> 
>> Do the current implementations of file:// do this -- i.e., use the filesystem's encoding for the URI?
> 
> 
> ​Apparently. I don't have a spare drive lying around where I can reformat a partition to test it for myself, though. A discussion I had with Dave Thaler back at the very start of this draft revolved around the fact that percent-encoded URIs are ambiguous (apparently a real issue for Windows), which was why for a very long time the draft contained advice to use an IRI​ instead, or at the least normalize.

VMs are good for testing.

It appears that both Windows and OSX have used UTF-8 for file name encoding for some time (since NT for the former, 10.0 for the latter, AIUI). See: <https://docs.python.org/3/library/sys.html#sys.getfilesystemencoding>

Linux uses whatever locale is set. However, it appears that both Gnome and Firefox (at least) try to be 'smart' and will recognise UTF-8 even if ISO-8859-1 is set as the locale. Having said that, it's not too smart; if I try to open a file with a UTF-8 encoded name, Firefox can't find it when the locale isn't UTF-8 (although the file chooser *does* see it).

This is an important point; the advice above that they "MAY be interpreted according to the preferred or configured encoding of the system on which the URI is being interpreted" doesn't account for the fact that a single filesystem might have several users who have different encodings set*.

Other encodings seem to just be percent-encoded straight into the file URI, and Firefox doesn't make any attempt to display them as IRIs. 

(This seems to mirror how most browsers handle non-ascii characters in HTTP headers, e.g., Location; they just percent-encode them, since that's an encoding of bytes, not characters).

Can we say something more like this?

---%<---
When a file URI is produced, characters not allowed by the ABNF MUST be percent-encoded as characters using UTF-8 encoding, as per RFC3986 Section 2.5. 

However, encoding information for file and/or directory names might not be available. In these cases, implementations MAY use heuristics to determine the encoding. If that fails, they SHOULD percent-encode the raw bytes of the label directly.
--->%---

Cheers,


* Possible but unlikely, since most people are going to be using UTF-8. Still...


--
Mark Nottingham   https://www.mnot.net/