Re: [Rpat] [Xml-sg-cmt] [AD] <u>: Re: AUTH48: RFC-to-be 9290 <draft-ietf-core-problem-details-08> for your review

[Jay Daley <[e-mail]>]

> Allowing non-ASCII text everywhere might be construed as a policy topic...

I’ve reread RFC 7997 and to my surprise I agree with Carsten that non-ASCII characters are allowed in the body text with some constraints and in this specific case the constraints do not apply.  

My analysis begins with this statement, which, while frustratingly written in terms of what is not allowed rather than what is allowed, does imply that non-ASCII characters used as part of an example can appear anywhere without special handling:

> 3.1. General Usage throughout a Document
> 
> Where the use of non-ASCII characters is purely part of an example and not otherwise required for correct protocol operation, escaping the non-ASCII character is not required.

There is a qualifier that immediately follows that statement that is again frustratingly cryptic and I don’t understand how/why it is presented as a qualifier, however my best guess is that it does not apply where the non-ASCII characters are not being used in a word that would normally be spelt in English without them (e.g. café).

> Note, however, that as the language of the RFC Series is English, the use of non-ASCII characters is based on the spelling of words commonly used in the English language following the guidance in the Merriam-Webster dictionary [MerrWeb].

Later on in the document there is a discussion about code points, which again I do not think applies because this is specifically about when the individual characters need to be identified (which is what I interpret as the use of the word "mention" in this text), and in Carsten’s case all that matters is that people are away this is a RTL string in Hebrew script.  In other words, if this RFC were specifying what Hebrew characters could be used in a particular place then they would have to be specified as below, but not if they are just any Hebrew characters used as an example.

> 3.4. Body of the Document
> 
> When the mention of non-ASCII characters is required for correct protocol operation and understanding, the characters' Unicode code points must be used in the text. The addition of each character name is encouraged.

Then we get to the current position, which is based on changes made after the publication of RFC 7991/7997 and which in my view are based on a mistaken reading of 3.4 above because 3.4 only applies where the actual characters matter (and therefore need to be mentioned) as well as being based on an entirely novel concept of being able to "distinguish between intentional and non-intentional inclusion of non-ASCII characters".

From https://www.ietf.org/archive/id/draft-levkowetz-xml2rfc-v3-implementation-notes-13.html#name-new-section-2x-u 

> 3.1.24. New Section 2.X, <u>
> 
> Thinking about being able to issue warnings both during xml2rfc processing and when running idnits, it seems very hard to distinguish between intentional and non-intentional inclusion of non-ASCII characters in document text.
> 
> In addition to the problem of correctly detecting non-intentional use of Unicode characters, there is also the issue (for authors) of correctly converting given Unicode characters to one of the forms recommended in [RFC7997], and the issue (for idnits) of verifying that any Unicode characters or strings are correctly represented as Unicode code-point values next to the literal character or string.
> 
> One solution to this could be to not try to guess, or establish heuristics, but instead use a v3 schema element with preptool validation to ensure a straightforward solution to all the issues, as follows:
> 
> Proposal: Limit the arbitrary placement of Unicode characters and strings in the body of a document, and control the expansion of the Unicode code-points by requiring that Unicode characters and strings be placed within a specific element if they are to occur in the body of a document. Such an expansion is already mandated by Section 3.4 of [RFC7997]; but without schema support, it would be very hard for tools to enforce this. The text in Appendix A.1 is proposed for inclusion in RFC 7991-bis as a new section.
> 
> Proposal:
> Limit the arbitrary placement of Unicode characters and strings in the body of a document, and control the expansion of the Unicode code-points by requiring that Unicode characters and strings be placed within a specific element if they are to occur in the body of a document. Such an expansion is already mandated by Section 3.4 of [RFC7997]; but without schema support, it would be very hard for tools to enforce this. The text in Appendix A.1 is proposed for inclusion in RFC 7991-bis as a new section.
> 
> Implementation: Implemented as described in Appendix A.1.
> 
> Heather's indication 20 Jul 2019: Isn't this already required by 7997??

I don’t actually want to undo the addition of <u> because I can see it is useful when section 3.4 above applies, but I do agree with Carsten that xml2rfc should allow non-ASCII characters in the body of a document without being wrapped in <u> tags (or <name> or any other workaround) and it is up to the RPC to decide if that usage is acceptable, not the tool.  All the tool needs to do is provide a clear warning that they exist so that the RPC knows they are there.

In this case my advice to the RPC would be that this usage is acceptable because the text is just an example of RTL text using Hebrew and the choice of what characters are displayed is largely irrelevant.

Jay

-- 
Jay Daley
IETF Executive Director
[e-mail]