Re: [I18ndir] Working Group Last Call: Structured Headers for HTTP

[John C Klensin <[e-mail]>]

--On Tuesday, February 4, 2020 23:35 +0100 Patrik Fältström
<[e-mail]> wrote:

> On 4 Feb 2020, at 20:24, John C Klensin wrote:
> 
>>> I think we agree, normalization is a level up from what
>>> they're describing here.
>> 
>> Ack.
> 
> Ok, I have tried to really understand what they are doing
> here, and it feels like if Unicode was pasted in after they
> had designed the whole thing.

Indeed.  It feels as if they started with "all ASCII" and then
someone said "well, how about cases that need non-ASCII
characters", and they responded with a lot of circling around,
much of which amounts to "just use UTF-8".  That is something we
know is almost always inadequate or worse and often a sign of
deeper problems and lack of thought.

> Look for example at B.1 when they compare with JSON. They say
> one advantage of their format is that JSON do allow Unicode
> data which gives interoperability issues, but they do allow it
> themselves as well.
> 
> Then this is 3.3.3:
> 
>> Unicode is not directly supported in strings, because it
>> causes a number of interoperability issues, and - with few
>> exceptions - header values do not require it.
> 
> How do they know header values do not need it -- with a few
> exceptions?
> 
>> When it is necessary for a field value to convey non-ASCII
>> content, a byte sequence (Section 3.3.5) SHOULD be specified,
>> along with a character encoding (preferably [UTF-8]).
> 
> I think that the default encoding MUST be UTF-8 OR specified
> explicitly.
> 
> I further think it should be noted comparison of parameter
> values is NOT specified in this base specification as
> normalization might create non-interoperability. If needed,
> the specification of the header must say how comparison is
> managed.
> 
> I also think 4.2.7 should mention the resulting sequence of
> the parsing of a binary structure might be a UTF-8 encoded
> string with a reference to 3.3.3.
> 
> I.e. I find it hard to read (and understand why I
> misunderstood this at first) that the only mentioning of
> Unicode and UTF-8 is in "string" but the only thing it does is
> to reference byte sequence, which in turn do never talk about
> it. Neither at serialization or deserialization. So where
> UTF-8 strings are described thet are not mentioned.
> 
> But ok...you are right.

Right about what?  We may be in agreement after all.  I think
there are at least three different issues here.

(1) I took Martin's note, especially the "in this day and age"
part, as suggesting that the document should encourage, rather
than discourage, non-ASCII strings in header field values.   I
suggested that, in most cases, discouraging such values was
entirely appropriate although it was reasonable to make
provision for them in special cases.

(2) The we got to the question of whether non-ASCII strings
should be allowed in header field names at all.   I think we (at
least you, John Levine, and me) are in agreement that it almost
certainly a bad idea.

(3) In the situations that non-ASCII characters are allowed in
header field values, we are in agreement that the description is
inadequate in multiple ways.  They include that the spec should
either require UTF-8 everywhere (e.g., "encoding MSUT be Unicode
in UTF=8") or the description / container syntax  must specify
how and in what syntax the encoding (or "charset') is specified.
They also include a requirement that any header files defined
according to this spec, at least ones that allow non-ASCII
characters  MUST be required to include in those definitions.
Those definitions would be required to have either an
"Internationalization Considerations" section or or equivalent.
And that section MUST either explain why comparison, searching,
or ordering of all or part of the header field value is never
going to be required  or describe the mechanisms or conventions
needed to carry out the operations.

For (3), I first understood you to be suggesting a discussion
of, e.g., normalization in the container/ format document that
would apply to all header fields defined under it.  I think that
would be wrong because different header fields and their values
might require different treatment.  However, I certainly agree
that information has to be provided somewhere and that this
document is deficient in saying as little as it does about
strings, byte sequences, and non-ASCII characters without
spelling out the requirement for that type of information to be
specified somewhere.   I also think that the IETF must never
again produce a technical specification or BCP that deal with or
allows non-ASCII characters by saying the equivalent of "just
use UTF-8".

Does that put us closer together?

   john