Re: [Tools-discuss] [xml2rfc] [Rfc-markdown] New xml2rfc release: v3.16.0

[John C Klensin <[e-mail]>]

--On Thursday, January 19, 2023 21:00 +0000 Jay Daley
<[e-mail]> wrote:

> 
> 
>> On 19 Jan 2023, at 20:46, Marc Petit-Huguenin
>> <[e-mail]> wrote:
>> 
>> Signed PGP part
>> On 1/19/23 12:35, Salz, Rich wrote:
>>> Thank you for your reply.
>>> 
>>>> Obviously when I say add a new element to the language, it
>>>> means create a RFC7991bis that does that modification. And
>>>> then modify the xml2rfc tool to match the new spec.
>>> 
>>> Wasn't obvious, I appreciate the clarification. But if
>>> you're modifying the XML language, why not just remove the
>>> limitations on non-ASCII that have been found to be
>>> problematic?
>> 
>> Because what I find to be problematic is allowing Unicode
>> everywhere.  My proposal of following RFC 7997 to the letter
>> instead of sneakily circumvent it has, among others, the
>> advantage of not allowing emojis in a document generated by
>> xml2rfc.  I cannot think of a more important goal for the
>> future of humanity.
> 
> I can only repeat what I've said before in the hope that
> doing so leads to some recognition - all this change does is
> move the enforcement of RFC 7997 from the tool to the RPC
> editors and stream managers.  That same set of people already
> enforce many other rules quite successfully.  This is not
> opening the floodgates.

Jay,

I agree, but it is probably worth digging a bit further into
this.  Assume valid UTF-8 [1][2], just to make things a bit more
simple than it would be otherwise [3]. Because of issues with,
e.g., ordering of combining characters as well as issues
specific to particular scripts and use of some scripts by
particular languages, "valid UTF-8" is not sufficient to imply
strings that can be rendered in a particular or reasonable way.
Some strings that are valid UTF-8 are, because of different
sorts of peculiarities, going to render differently by different
engines.  Sometimes that makes a difference, sometimes it
doesn't.  Combining characters that modify preceding ones
appearing first in a string aside, the most commonly cited
troublesome examples involve 

 * mixtures of characters drawn from scripts
	characterized as right-to-left with characters that are
	not associated with inherent directionality (e.g., the
	digits associated with contemporary Latin-based scripts)
	and characters from scripts characterized as
	left-to-right and
	
 * rendering issues with Emoji and other graphemes that
	can look very different in different environments.

 * Emoji combining sequences, where they are and are not
	valid, and how they are interpreted and rendered.

I don't see the type of markup under discussion as having much
to do with the above and hence, again, agree with what you have
said as far as it goes.  At the same time, asking the RPC to
sort out that whole range of issues may require skill sets that
the RPC does not have today and, even then, additional markup
may be needed to give them adequate information.   In
particular, unless you / the RPC want to hire staff or
consultants whose skills extend across the full range of writing
systems and possible uses of Unicode, caution may be in order.
It might be relevant --if it is not already generally understood
on these lists-- that those skill sets are actually fairly rare
(possibly translating into "hard to obtain" and/or "expensive").
Especially if that is not the plan, it may be worth considering:

(1) Enhancing the Style Guide to indicate which scripts and
languages can be used freely and which ones require, e.g.,
advanced consultation with the RPC.  Maybe normalization
suggestions belong there too, maybe not.  If that list starts
with, e.g., European scripts derived from Greek or Latin plus
maybe Chinese script and gradually expands, I don't see much, or
any, harm.

(2) Retaining the <u> element, not as a requirement for anytime
Unicode is being used (or used in an unusual context) but to
allow, not just the existing attributes but specification of
direction and language.

The two obviously interact because one could then make a rule
that <u> is optional for language-script pairs enumerated in the
Style guide but must be specified for all other cases and any
additional cases where the author believes it is needed
(including as advice for the RPC).

And, btw, I do not quite understand what happens to information
now specified in the "format" attribute if <u> is dropped
generally.

    john

p.s. to John Levine and others who might have noticed: After
looking at the <u> section (3.62) of
draft-irse-draft-irse-xml2rfcv3-implemented-03, "Unicide" sounds
like a term that it might be useful to define and use, perhaps
for particular flavors of Unicode abuse such as, in other
contexts, deliberately using non-obvious Unicode strings to
create confusion or deception.  An alternate definition
involving committing, or wanting to commit, acts of violence
against Unicode designers would obviously be inappropriate.

_Notes_

[1] Note the difference between RFC 3629 and 2279 and, in turn
between 2279 and 2044.   Also note that 3269, despite its
Internet Standard status, should probably not be the last word
on the subject. 

[2] FWIW, the current text-form output from xml2rfc violates a
SHOULD in RFC 3629.  As we consider how and where Unicode code
sequences may be used, that should either be fixed or documented
and justified in some clear way and prominent place. 

[3]  Noting that the native Unicode encoding of some important
operating systems and tools is not UTF-8, so some caution is
required even there.