[art] Re: draft-bray-unichars-09 review

[Tim Bray <[e-mail]>]

On Oct 4, 2024 at 7:22:12 AM, Orie Steele <[e-mail]> wrote:

>
> ## Comments
>
> ### Lack of BCP14 Guidance
>

[This is just me, haven’t talked over Orie’s contributions with Paul yet.]

This is a really interesting issue.  Our vision of how this would be used
successfully was along the lines of:


   1. WG takes up a draft. That draft says "the value of this field is
   Unicode text." Maybe it even says "…encoded in UTF-8.”
   2. Someone (co-chair, WG member, AD, last-call commenter) says “Which
   Unicode characters? Go look at [Unichars].”
   3. The draft editor says “Oh, OK” and specifies one of the subsets from
   [Unichars] or maybe [PRECIS].


So, how can we use BCP14 to encourage this happening?  What would people’s
feelings be if we added a sentence, perhaps as a new paragraph in the
section 2 introduction, just before 2.1, reading as follows:

“It is RECOMMENDED that designers of protocols and data formats, for any
data field which contains textual data, consider the issues discussed in
this document. They SHOULD specify, for that data field, the use of one of
the subsets specified in this document or one of the profiles specified in
[PRECIS].”

I’ve never written BCP14 language that operates at the meta level like
this, aimed at designers of new protocols and data formats, rather than
describing a specific protocol/format. Is it even a thing? Are there other
examples?

### Recommendation to allow unassigned code points?
>
> ```
> 125   than 150,000 have been assigned to characters.  It is difficult to
> 126   specify that unassigned code points should be avoided, because they
> 127   regularly become assigned when new characters are added to Unicode.
> ```
>
> This is bordering on a recommendation, should it be strengthened to BCP14
> language?
>

Once again, I’m a little nervous about BCP14 at the meta level. That aside,
what about language like

“Designers of protocols and data formats SHOULD NOT attempt to forbid the
use of code points on the basis that they are unassigned at the time the
specification is created."

### What about CBOR or protocol buffers?
>
> ```
> 145   a single transformation format.  UTF-8 is widely used for
> 146   interoperable data formats such as JSON, YAML, and XML.
> ```
>

Hmm. Nothing against CBOR but the list could get pretty long with Protobufs
and Thrift and Avro etc etc. I want to avoid giving the impression that
this list is exhaustive rather than suggestive. Anyone got editorial
suggestions.

### Where is UTF-16 used?
>
> ```
> 178   A surrogate which occurs in text encoded in any transformation format
> 179   other than UTF-16 has no meaning and may cause malfunction in
> 180   software that encounters it.  In particular, it is impossible to
> 181   represent a surrogate in well-formed UTF-8.
> ```
>
> Some guidance here on where to expect UTF-16 could help explain where to
> expect surrogates.
>

UTF-16 is essentially never used on the wire. I for one have never seen it
in my decades of work in this space.  Does anyone have a counter-example?
Or even better, does anyone know of an RFC that says “you MUST NOT put
UTF-16 on the wire.”? Because thankfully, nobody does.

### Where are legacy codes still used in IETF protocols?
>
> ```
> 198   Aside from the useful controls, the control codes are mostly obsolete
> 199   and generally lack interoperable semantics.  This document uses the
> ```
>
> It would be good to call out any known places where these codes still have
> interoperable semantics.
>

I agree. I wouldn’t know where to look to find them. Does anyone have any
suggestions?

### Citation for this?
>
> ```
> 232   Surrogate code points have been observed to cause software failures.
> ```
>
Ideally in an RFC.
>

Sadly, I guess “Tim’s bitter personal experiences with legacy Java and C
code behind BigTech firewalls” doesn’t qualify.  I agree with Orie’s
implication that if we can’t back this up with a good reference we probably
shouldn’t say it.

### Clearer guidance on handling problematic input
>
> ```
> 260   Reasonable options for dealing with problematic input include, first,
> 261   rejecting text containing problematic code points, and second,
> 262   replacing them with placeholders.  (As an exception, [UNICODE] notes
> 263   that it may in some cases be appropriate, specifically for
> 264   noncharacters, to treat them as non-problematic unassigned code
> 265   points.)
> ```
>
> This could be framed with BCP14 language... The phrasing along with the
> parenthetical make this slightly hard to understand.
>

I was going to say we should strengthen the callout to [RFC9413] but I just
checked and, while it is very good, it also has no BCP14 language. I’d like
to say something like “Designers MUST consult [RFC9413] and [UNICODE]” but
that’s sort of meta-meta. I agree, the language about [UNICODE] should be
pulled out of parantheses and the reference should specify the [UNICODE]
section.

Oh, and thanks Orie, those are useful remarks.

 -T