Re: [Rfc-markdown] [Tools-discuss] New xml2rfc release: v3.18.0

[Carsten Bormann <[e-mail]>]

> See https://github.com/ietf-tools/xml2rfc/releases/tag/v3.18.0 for
> release details.
> 
> This release allows the use of Unicode characters everywhere.

Wonderful!

(This release allows the use of non-ASCII Unicode characters everywhere;
Xml2rfc already allowed Unicode characters that were in its “ASCII” subset — which included a few select non-ASCII characters.)

This update should not require any updates in kramdown-rfc, but of course the need for workarounds like {{{}}{{🤦‍♂️}}} is gone.

> The  `--warn-bare-unicode` command line option will warn if Unicode
> characters are present in any element except artwork, city, cityarea,
> code, country, email, extaddr, organization, pobox, postalLine,
> refcontent, region, sortingcode, sourcecode, street, title and u.
> See https://github.com/ietf-tools/xml2rfc/pull/1017 for more details.

We generally want a soft transition to using the full Unicode repertoire, not the least because xml2rfc’s PDF generator may need attention with new character blocks coming into use — Gurmukhi may not quite work just yet.
RFCXML's <u element stays useful as an easy way to fulfil RFC 7997’s requirement to fully explain non-ASCII characters when that may be needed for interchange.

Non-ASCII characters sometimes sneak into drafts via copy-paste from sources that use full Unicode as a matter of course.
Not just typographic quotes, which can be jarring when mixed with typewriter quotes, but also various invisible characters such as zero-width space and word joiners which were already part of xml2rfc’s “ASCII” repertoire.

Kramdown-rfc comes with an analysis tool called “echars” (explain characters).

Running this on a markdown (or XML or TXT!) file generates output such as:

$ echars draft-bormann-restatement.md
*** Latin-1 Supplement
»: U+00BB    1 RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK (Common)
ä: U+00E4    1 LATIN SMALL LETTER A WITH DIAERESIS (Latin)
*** General Punctuation (Common)
—: U+2014    5 EM DASH
’: U+2019    1 RIGHT SINGLE QUOTATION MARK
”: U+201D    2 RIGHT DOUBLE QUOTATION MARK
…: U+2026    1 HORIZONTAL ELLIPSIS
⁠: U+2060    1 WORD JOINER

So there is nothing strange in this document, but it is still worth knowing where these characters outside the LF + %x40-7e space are (in this case: mostly in the titles of references), so I check this now and then for my documents (*).
Your editor might help with that, e.g. in Emacs use:

M-C-s [^^J-~]

(where ^J is a newline character, entered as ctrl-j, while the ^ preceding it is a caret.)

Of course, in XML you might be hiding beyond-ASCII by using entity references such as &nbsp; or character references such as &#x20AC; or &#8364; — echars doesn’t show these, but then the intent should be quite obvious in the manuscript.

> Report any issues on https://github.com/ietf-tools/xml2rfc/issues

… and any issues with kramdown-rfc on [e-mail] and/or as issues in https://rfc.space

Grüße, Carsten

(*) I contemplate generating this report with each kramdown-rfc run, possibly modulated by declarations in the YAML header that say which characters the document author already expects.  But I’m going on a couple of vacations first now…