Re: [Tools-discuss] xml2rfc in --v2 mode -- bug report?

[John C Klensin <[e-mail]>]

Jay,

This is very helpful even thought I think, for the person who is
trying to get work done rather than being steeped in the theory,
the vocabulary - schema and PI distinctions (including the
additional distinction in Carsten's recent note) might be
considered fussy details.  I don't believe (but have not gone
back and checked) that any of the definitions say something
equivalent to "don't bother to try to understand this without
first becoming very familiar with the details of XML terminology
and distinctions".

As to the transition, you may reasonably disagree either about
the principle or about the boundary point but, from my
perspective, the purpose of tools like the RFCXML definition and
xml2rfc is to let people contributing to the IETF to get work
done without unreasonable barriers (including sorting out of
working around significant bugs).  From that perspective, it
would be a grave disservice to the community to declare v2
support at an end until xml2rfc is actually stable and
relatively problem-free.  As long as people, such as Carsten,
who have been paying far more attention to this than I have, are
mentioning hundreds of "issues" as significant or making
comments about keeping the newer versions of things alive, we
are not near "relatively problem-free" yet.

Finally and most important, we should all remember that there
has never been a publicly stated expectation that everyone in
the IETF who might need to write a document or use other tools
will be on this list.  I have no way to know, but I assume that
only a very small fraction of those contributors are watching
the list carefully.   Information such as that below --if you
and others think it is as important as it appears to be-- should
be on an easily found web page somewhere and the community
pointed to it, including in a normative reference from the
revised vocabulary document.

thanks,
   john


--On Monday, June 13, 2022 15:21 +0100 Jay Daley
<[e-mail]> wrote:

> Hi John
> 
> From reading your message I think I need to start with a clear
> taxonomy of the various moving parts here because this is
> still not clear to most participants, afaict:
> 
> 1.  All XML languages define what elements and attributes are
> acceptable within that language and what element can appear
> inside what other element.  This is what we call RFCXML and
> what was previously called the xml2rfc vocabulary.  This is
> more generally called the *grammar*.  The grammar is defined
> in RFCs.
> 
> 2.  We have chosen to use another language to formally define
> the grammar above.  For v1, defined in RFC 2629, the formal
> definition used a DTD.  For v2, defined in RFC 7749, this was
> specified in a RelaxNG schema and not a DTD.  For v2, defined
> in RFC 7991, this was also defined in a RelaxNG schema.  This
> is more generally called the *schema*. At some point work was
> put into changing rfc2629.dtd to make it compliant with v2
> (possibly even v3) but I believe it does not (and cannot)
> because of the limitations of DTDs, correctly define the v2
> grammar.
> 
> 3.  There is an XML construct called a *processing
> instruction* (aka a PI), which is embedded inside an XML
> document and provides instructions that are to be interpreted
> by any XML processor.  These are not part of the grammar and
> therefore cannot be part of the formal definition.  To repeat
> myself - PIs sit outside of the grammar, they are conceptually
> similar to escape codes in that respect.  DOCTYPE is a
> processing instruction, as are <?xml-model…> and
> <?xml-stylesheet…>..  None of the above RFCs have
> comprehensively covered PIs - RFC 2629 does not mention them
> at all, RFC 7749 notes that certain things are set by PIs but
> does not define them and  RFC 7991 notes that certain grammar
> changes are intended to deprecate some PIs but doesn't
> formally define those.
> 
> With that in mind ...
> 
>> On 12 Jun 2022, at 18:05, John C Klensin <[e-mail]>
>> wrote:
>> 
>> 
>> 
>> --On Sunday, June 12, 2022 16:43 +0100 Jay Daley
>> <[e-mail]> wrote:
>> 
>>> Hi John
>>> 
>>>> On 11 Jun 2022, at 20:29, John C Klensin <[e-mail]>
>>>> wrote:
>>>> 
>>>> Hi. I have an old document, in xml2rfc v2 format, whose
>>>> content I'm trying to upgrade. When I run
>>>> xml2rfc DocName.xml --v2
>>>> (with version 3.12.7) I get a series of messages that look
>>>> like
>>>> 
>>>> Warning:
>>>> 	file:/C:/Users/Klensin/AppData/Local/Programs/Python/Pyt
>>>> 	hon37/lib/site-packages/xml2rfc/templates/rfc2629-xhtml.
>>>> 	ent is no longer needed as the special processing of
>>>> 	non-ASCII characters has been superseded by direct
>>>> 	support for non-ASCII characters in RFCXML.
>>>> 
>>>> The source does not contain any references to
>>>> rfc2629-xhtml.ent. It does, of course, contain 
>>>> 	<?xml-stylesheet type='text/xsl' href='rfc2629.xslt' ?>
>>>> as well as the DOCTYPE statement that specifies the DTD
>>>> 
>>>> Are these warnings actually addressed to something in the
>>>> DTD or stylesheet file than should be cleaned out and, if
>>>> so, how does that get done?
>>> 
>>> The DOCTYPE that you have included I am guessing is for
>>> rfc2629.dtd which includes by reference rfc2629-xhtml.ent,
>>> 
>>> The DOCTYPE is superfluous for document validation and has
>>> been since v2 of the grammar because v2 is when the change
>>> was made from XML Schema (as specified in a DOCTYPE) to
>>> RelaxNG. However, it continued to be used partly I think
>>> because the templates were never updated to be RelaxNG-aware
>>> and partly because it was a convenient way to incorporate
>>> the character entities.
>> 
>> But this is a v2 document whose first version dates to early
>> 2017, when v3 was still in its infancy and, IIR, the online
>> version of xml2rfc would not yet handle v3. RelaxNG was, again
>> IIR, only introduced with RFC 7749 in February 2016, bringing
>> the change from a DTD-based definition to a Schema-based one
>> with it. Most of the changes described in its Appendix B had
>> (as RFC 7749 says) been adopted some years earlier and
>> templates adjusted, so that the actual changes needed in 2016
>> were very small and many documents and templates did not
>> require changes at all.
> 
> That "most of the changes … had been adopted some years
> earlier" doesn't change the fact that v2 is defined in RFC
> 7749 and that uses a RelaxNG schema.  
> 
>> I also note that DOCTYPE appears in Section 4 of RFC
>> 7749 with language that implies to me that it is required.
> 
> No, it's a PI not part of the grammar.
> 
>> 
>>> I normally recommend that authors replace the DOCTYPE
>>> statement with this:
>>> 
>>> 	<?xml-model href="rfc7991bis.rnc"?>
>>> 
>>> (The file referenced can be found at
>>> https://raw.githubusercontent.com/ietf-tools/rfcxml-template
>>> s- and-schemas/main/rfc7991bis.rnc)
>> 
>> But that is a piece of version 3 vocabulary (see below).
> 
> No, it's a PI and therefore outside of the v3 vocabulary.  
> 
> However, I did make a mistake here, forgetting that you are a
> v2 user.  The correct PI for a v2 document would be 
> 
> 	<?xml-model href="rfc7749.rnc"?>
> 
> Where that file can be found at
> https://raw.githubusercontent.com/ietf-tools/legacy-templates-
> and-schemas/main/rfc7749.rnc
> 
>>> Doing this tells any XML processor that this uses the
>>> referenced RelaxNG schema and a RelaxNG aware editor will
>>> then both validate against this schema and provide
>>> schema-aware editing support (such as auto-suggestion and
>>> autoi-completion). 
>> 
>>> However afaict your editor, Epsilon, does not appear to do
>>> schema validation of any sort and so neither the statement
>>> above nor a DOCTYPE will result in any validation.
>> 
>> Epsilon (and emacs) are "just" editors. Their modes for
>> handling XML are not aware of schema, just such fundamental
>> --and essentially lexical-- issues as formatting, element
>> matching, and so on. 
> 
> As explained by Carsten that is not correct for Emacs.  From
> my research Epsilon is almost unique in not doing any form of
> schema validation.  The reason all the others do it appears to
> be because they use the same underlying open source XML
> libraries that provide this functionality  
> 
>> 
>> So I am confused by your explanation and suggestions:
>> 
>> (1) They would seem to lead to documents that are v2-v3
>> hybrids. I don't know how the current versions of xml2rfc
>> would deal with that but, given assorted v2 elements and
>> constructions that were deprecated in v3, I'd guess it would
>> be very hard to get right and that going in that direction
>> would be a bad idea.
> 
> As noted above, PIs are not part of the grammar and so you can
> have a v2 document that uses new PIs and it is still a v2
> document.
> 
>> 
>> (2) If retaining DOCTYPE, or at least DOCTYPE with those
>> definitions, in a v3 document is, as you suggest, obsolete
>> and a bad practice, then that should be reflected in the v2v3
>> conversion process. However, when I did the conversion
>> yesterday, that definition (straight out of the v2 document
>> and RFC 7749) is retained unchanged. I presume that should go
>> onto the list of bugs in the converter.
> 
> Except that it's not a bug.  Having a DOCTYPE in a v3
> document doesn't stop it from being a v3 document.  What it
> does is provide an instruction to any XML processor that may
> be wrong and which it may choose to ignore.  I agree however
> that a warning from xml2rfc would be helpful.
>> 
>> 
>> More generally, I have no idea what happens behind the scenes
>> when I invoke xml2rfc v3.12.7 with "--v2"
> 
> https://authors.ietf.org/en/upgrading-from-v2
> 
>> but, given the very
>> large number of documents in the RFC Editor's collection in v2
>> format that, I assume, have not been converted to v3 and
>> tested for consistency with the output produced, a decision
>> to retire support for v2 should be taken only with great care
>> (and, IMO, given the risks and tradeoffs, made only with IESG
>> signoff after a community Last Call). Until then, I believe
>> the tools team and IETF staff have considerable
>> responsibility for keeping version 2 supported. I don't think
>> a few spurious warning messages are a big deal unless they
>> are a sign of things to come, but, when the answer to a
>> problem with constructions that are valid and well-documented
>> under v2 is "convert to v3", that is not supporting v2
>> properly and as the community has a reasonable right to
>> expect. 
> 
> v2 was officially obsoleted when RFC 7991 was published and
> RFC 7991 is explicit about that.  Yes a transition process
> should be supported and it has for six years now, but I
> disagree that the community has any right to expect that to
> continue.  It will inevitably get more expensive and complex
> to support v2.  Having said that, the transition certainly has
> taken longer than many might expect, which I attribute to the
> templates that were provided until recently, which still used
> a number of v2 idioms and did not showcase v3.  The new
> templates should ease the transition considerably.
> 
>> 
>> Similar comments apply to your comment about epsilon: my
>> expectation is that I can continue to use a text editor to
>> work with what are now called RFCXML files in both v2 and v3,
>> expecting that validation will come out of the xml2rfc program
>> (or, at worst, a competent, LINT-like, validator will be
>> provided by the IETF). I also expect those validation
>> processes will produce clear, correct, and, where possible,
>> actionable warning and error messages. If that ever becomes
>> not the case, e.g., if it were expected that people creating
>> documents will use an editor with XML Schema validating
>> capability and validation responsibility will lie there, that
>> would essentially require document authors who wish to work
>> in XML to use such an editor. I would assume that, too, would
>> require IESG signoff after an IETF Last Call if only because
>> it would increase the barriers to entry and participation in
>> the IETF and hence reduce the diversity of its active,
>> document-writing, participants.
> 
> There's no expectation that you use a schema-aware editor,
> but if you do then with the appropriate processing
> instructions, it will make your work considerably easier.
> Your choice though,
> 
> Jay
> 
>> 
>> best,
>> john