Re: [Tools-discuss] [xml2rfc] [Rfc-markdown] New xml2rfc release: v3.16.0 (details of Unicode)

["Martin J. Dürst" <[e-mail]>]

On 2023-01-20 12:43, John C Klensin wrote:

> I agree, but it is probably worth digging a bit further into
> this.  Assume valid UTF-8 [1][2], just to make things a bit more
> simple than it would be otherwise [3]. Because of issues with,
> e.g., ordering of combining characters as well as issues
> specific to particular scripts and use of some scripts by
> particular languages, "valid UTF-8" is not sufficient to imply
> strings that can be rendered in a particular or reasonable way.
> Some strings that are valid UTF-8 are, because of different
> sorts of peculiarities, going to render differently by different
> engines.

Can we please assume the following the following three points?

a) Like for all other aspects of draft/rfc-to-be creation, the
    authors/editors will use non-ASCII characters carefully and
    diligently to the best of their knowledge. They will be
    familiar with the characters and scripts in question e.g.
    because it's their name or address, or otherwise. They will
    also check these characters carefully in AUTH48, including
    the .pdf version.
b) Like all other aspects of draft/rfc-to-be review, the wide
    range of people (WG members, chairs, shepherds, ADs,...)
    who will review non-ASCII characters in a document will do
    so with as much care as for the other aspects of a document
    (and because of their number, they will do so using a wide
    range of tools such as plain text editors, browsers, MUAs,
    ..., on a wide range of OSes).
c) The RPC will check non-ASCII characters carefully the same
    way they check other aspects of an rfc-to-be, and will get
    back to editors/authors/WG/ADs/... if they see any need.

Given the above, I'm rather confident that in general, things will work 
out just fine. Still of course, very occasionally, something might go 
wrong. But that happens with other aspects of RFCs, too. For a clear 
example, check out RFC　7158/7159.


> Sometimes that makes a difference, sometimes it
> doesn't.  Combining characters that modify preceding ones
> appearing first in a string aside, the most commonly cited
> troublesome examples involve
> 
>   * mixtures of characters drawn from scripts
> 	characterized as right-to-left with characters that are
> 	not associated with inherent directionality (e.g., the
> 	digits associated with contemporary Latin-based scripts)
> 	and characters from scripts characterized as
> 	left-to-right and
> 	
>   * rendering issues with Emoji and other graphemes that
> 	can look very different in different environments.
> 
>   * Emoji combining sequences, where they are and are not
> 	valid, and how they are interpreted and rendered.
> 
> I don't see the type of markup under discussion as having much
> to do with the above and hence, again, agree with what you have
> said as far as it goes.  At the same time, asking the RPC to
> sort out that whole range of issues may require skill sets that
> the RPC does not have today and, even then, additional markup
> may be needed to give them adequate information.   In
> particular, unless you / the RPC want to hire staff or
> consultants whose skills extend across the full range of writing
> systems and possible uses of Unicode, caution may be in order.
> It might be relevant --if it is not already generally understood
> on these lists-- that those skill sets are actually fairly rare
> (possibly translating into "hard to obtain" and/or "expensive").

Finding a consultant whose' skills extends across the full range of 
writing systems and possible uses of Unicode may simply be impossible. 
I'd know quite a few people who come close, but only close.

But I know that in the community as a whole, we have many people who 
have expertise in specific scripts, languages, and encoding mechanisms. 
And I hope we and the RPC can leverage this expertise. I for one 
volunteer to help with any specific questions the RPC has with respect 
to Unicode.


> Especially if that is not the plan, it may be worth considering:
> 
> (1) Enhancing the Style Guide to indicate which scripts and
> languages can be used freely and which ones require, e.g.,
> advanced consultation with the RPC.  Maybe normalization
> suggestions belong there too, maybe not.  If that list starts
> with, e.g., European scripts derived from Greek or Latin plus
> maybe Chinese script and gradually expands, I don't see much, or
> any, harm.

Such a list probably would create more harm than good. The average 
draft/rfc-to-be author/editor doesn't use Unicode characters because 
they want to cheat the system, but because they need them for specific 
names/addresses/examples/... If it so happened that they needed Egyptian 
Hieroglyphs (very rare case, but let's just assume), it won't help them 
if the RPC says "we currently have Greek and Chinese, why don't you try 
one of these" or "at our current pace of adding scripts to our 'allowed' 
list, come back in 20 years".

Please note that RFC 9290, which initiated the current discussion, 
needed an RTL example, and chose Hebrew שלום because that avoided any 
potential ligaturing problems with Arabic. This shows both how authors 
are careful and how your idea of an "allowed" list would have failed.


> (2) Retaining the <u> element, not as a requirement for anytime
> Unicode is being used (or used in an unusual context) but to
> allow, not just the existing attributes but specification of
> direction and language.
> 
> The two obviously interact because one could then make a rule
> that <u> is optional for language-script pairs enumerated in the
> Style guide but must be specified for all other cases and any
> additional cases where the author believes it is needed
> (including as advice for the RPC).
> 
> And, btw, I do not quite understand what happens to information
> now specified in the "format" attribute if <u> is dropped
> generally.
> 
>      john
> 
> p.s. to John Levine and others who might have noticed: After
> looking at the <u> section (3.62) of
> draft-irse-draft-irse-xml2rfcv3-implemented-03, "Unicide" sounds
> like a term that it might be useful to define and use, perhaps
> for particular flavors of Unicode abuse such as, in other
> contexts, deliberately using non-obvious Unicode strings to
> create confusion or deception.  An alternate definition
> involving committing, or wanting to commit, acts of violence
> against Unicode designers would obviously be inappropriate.
> 
> _Notes_
> 
> [1] Note the difference between RFC 3629 and 2279 and, in turn
> between 2279 and 2044.   Also note that 3269, despite its
> Internet Standard status, should probably not be the last word
> on the subject.

[RFC 3269 is "Author Guidelines for Reliable Multicast Transport (RMT) 
Building Blocks and Protocol Instantiation documents". I assume you 
meant RFC　3629.]

> [2] FWIW, the current text-form output from xml2rfc violates a
> SHOULD in RFC 3629.  As we consider how and where Unicode code
> sequences may be used, that should either be fixed or documented
> and justified in some clear way and prominent place.

RFC　3269 contains four SHOULDs (one of them a SHOULD NOT). Could you 
indicate which one you think is violated?

All the SHOULDs revolve around the use of U+FEFF as a signature. I just 
carefully re-read them, but can't find any way in which xml2rfc 
currently would violate any of these SHOULDs.


> [3]  Noting that the native Unicode encoding of some important
> operating systems and tools is not UTF-8, so some caution is
> required even there.

The relevant operating systems actually apply the necessary caution. 
It's extremely rare these days that some action is needed by users.


Regards,   Martin.