Re: [I18ndir] Working Group Last Call: Structured Headers for HTTP

[John C Klensin <[e-mail]>]

--On Tuesday, February 4, 2020 13:07 +0100 Patrik Fältström
<[e-mail]> wrote:

> On 4 Feb 2020, at 2:30, John C Klensin wrote:
> 
>> --On Monday, February 3, 2020 17:05 -0500 John R Levine
>> <[e-mail]> wrote:
>> 
>>> ...
>>> In section 3.3.3 it says that non-ASCII text is to be encoded
>>> as base64, which seems reasonable, "along with a character
>>> encoding (preferably UTF-8)" but it doesn't have any
>>> convention for where the encoding goes. Experience with MIME
>>> suggests that life is simpler if there's a standard form for
>>> an encoded text blob so libraries can decode and if need be
>>> transliterate in one go.
>> 
>> And omission of a clear model for specifying what is really
>> what we call a "charset" elsewhere (not just an encoding) is
>> the one clear defect in that particular discussion in the
>> document.
> 
> Don't forget normalization or otherwise how comparison is to
> be done of the UTF-8 encoded characters in the string. Nothing
> about that either.

The omission was deliberate, so I should explain.  Let me take
advantage of John Levine's observation about email headers, with
which we have much more experience.  Very loosely and
informally, there are two types of header field data - the stuff
that is highly structured like addresses (e.g., "From:", "To:",
and "Cc:") and maybe even "Date:" and the stuff, like "Subject:"
fields, that is basically free text.  For the latter, in most
cases one doesn't care whether it can be compared to other
strings or not -- the important thing is that it can be read by
the user and not be confusing.  Of course, someone might want to
sort or select on unstructured fields, but that may not impose
as strong a requirement.   For HTTP headers that contain
structured material, strict normalization and comparison rules
might be less important -- as I suggested to John, one place
where the analogy breaks down is that many or most HTTP headers
are closer to envelope information in email than they are to
user presentation information.  

What is more important is that, as The Unicode Standard has been
suggesting all along and more recent W3C work addresses in a
more refined way, in many or most cases (and even for more
structured fields), the right answer is to normalize (and apply
other rules as needed) only at comparison time, normalizing both
(all) strings to be compared, and not even bothering to store
the normalized strings after the comparison is completed.  I
still believe that we got IDN labels right by requiring that
both the stored form and any lookup keys be normalized and
structured early, but they are, for several reasons, an unusual
case.  

So, out of concern that anything that looked like "one size fits
all" might turn out to be closer to "...fits none", I decided to
avoid touching those issues in the context of this particular
document.  Maybe that is wrong but, if it is, I'd be inclined to
suggest that any document that specifies a new HTTP header field
type and that allows characters in the value/data that are not
natively ASCII MUST include an Internationalization
Considerations section that explicitly addresses these issues.

Does that make sense?  Do we think that sort of requirement is
needed and would be helpful?

   john