Re: [Rfc-markdown] [Tools-discuss] New xml2rfc release: v3.18.0

[Kesara Rathnayake <[e-mail]>]

Hi all,

Output from kramdown-rfc's echars and `xml2rfc --warn-bare-unicode`
option is now listed as separate sections in the validation report on
https://author-tools.ietf.org/
Select the file and click on the "Validate (idnits)" button for this.

Cheers,
Kesara

On Fri, 4 Aug 2023 at 16:26, Carsten Bormann <[e-mail]> wrote:
>
> > See https://github.com/ietf-tools/xml2rfc/releases/tag/v3.18.0 for
> > release details.
> >
> > This release allows the use of Unicode characters everywhere.
>
> Wonderful!
>
> (This release allows the use of non-ASCII Unicode characters everywhere;
> Xml2rfc already allowed Unicode characters that were in its “ASCII” subset — which included a few select non-ASCII characters.)
>
> This update should not require any updates in kramdown-rfc, but of course the need for workarounds like {{{}}{{🤦‍♂️}}} is gone.
>
> > The  `--warn-bare-unicode` command line option will warn if Unicode
> > characters are present in any element except artwork, city, cityarea,
> > code, country, email, extaddr, organization, pobox, postalLine,
> > refcontent, region, sortingcode, sourcecode, street, title and u.
> > See https://github.com/ietf-tools/xml2rfc/pull/1017 for more details.
>
> We generally want a soft transition to using the full Unicode repertoire, not the least because xml2rfc’s PDF generator may need attention with new character blocks coming into use — Gurmukhi may not quite work just yet.
> RFCXML's <u element stays useful as an easy way to fulfil RFC 7997’s requirement to fully explain non-ASCII characters when that may be needed for interchange.
>
> Non-ASCII characters sometimes sneak into drafts via copy-paste from sources that use full Unicode as a matter of course.
> Not just typographic quotes, which can be jarring when mixed with typewriter quotes, but also various invisible characters such as zero-width space and word joiners which were already part of xml2rfc’s “ASCII” repertoire.
>
> Kramdown-rfc comes with an analysis tool called “echars” (explain characters).
>
> Running this on a markdown (or XML or TXT!) file generates output such as:
>
> $ echars draft-bormann-restatement.md
> *** Latin-1 Supplement
> »: U+00BB    1 RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK (Common)
> ä: U+00E4    1 LATIN SMALL LETTER A WITH DIAERESIS (Latin)
> *** General Punctuation (Common)
> —: U+2014    5 EM DASH
> ’: U+2019    1 RIGHT SINGLE QUOTATION MARK
> ”: U+201D    2 RIGHT DOUBLE QUOTATION MARK
> …: U+2026    1 HORIZONTAL ELLIPSIS
> ⁠: U+2060    1 WORD JOINER
>
> So there is nothing strange in this document, but it is still worth knowing where these characters outside the LF + %x40-7e space are (in this case: mostly in the titles of references), so I check this now and then for my documents (*).
> Your editor might help with that, e.g. in Emacs use:
>
> M-C-s [^^J-~]
>
> (where ^J is a newline character, entered as ctrl-j, while the ^ preceding it is a caret.)
>
> Of course, in XML you might be hiding beyond-ASCII by using entity references such as &nbsp; or character references such as &#x20AC; or &#8364; — echars doesn’t show these, but then the intent should be quite obvious in the manuscript.
>
> > Report any issues on https://github.com/ietf-tools/xml2rfc/issues
>
> … and any issues with kramdown-rfc on [e-mail] and/or as issues in https://rfc.space
>
> Grüße, Carsten
>
> (*) I contemplate generating this report with each kramdown-rfc run, possibly modulated by declarations in the YAML header that say which characters the document author already expects.  But I’m going on a couple of vacations first now…
>


-- 
Kesara Rathnayake
Senior Software Development Engineer - IETF Administration LLC
[e-mail]