[TOOLS-DEVELOPMENT] Fwd: Preview release of Text Submission Converter, id2xml

[Megan Ferguson <[e-mail]>]

Hi Henrik,

Thank you for your reply and explanations.  Inline below with MF.

Megan

Begin forwarded message:

> From: Henrik Levkowetz <[e-mail]>
> Subject: Re: [TOOLS-DEVELOPMENT] Preview release of Text Submission Converter, id2xml
> Date: May 19, 2017 at 2:20:25 PM PDT
> To: Megan Ferguson <[e-mail]>
> Cc: [e-mail]
> 
> Hi Megan,
> 
> On 2017-05-19 21:55, Megan Ferguson wrote:
>> Hi Henrik,
>> 
>> Some notes on our initial test pass. Please let me know if you would
>> like any further information on any of the points below.
> 
> In general, I'd very much appreciate the relevant draft name when you
> point out an issue; that will let me use that in testing and fixing
> the issue.  As you'll see below, I believe that many of the issues you
> mention below are fixed in the latest release (1.0.0-rc1), but there
> are some cases below where the draft name would be good to have.

MF- Ack.  I used draft-ietf-isis-mi-bis-03 originally.  I have rerun this same file with the newer version and found 
most of the items previously discussed to be resolved.

I also tested on draft-ietf-rtgwg-yang-key-chain-24.

Latest Release Notes:
---------------------

1) Missing references:

When running id2xml (v1.0.0-rc1) on draft-ietf-isis-mi-bis-03, I received the following warning:

id2xml draft-ietf-isis-mi-bis-03test.txt
Converting 'draft-ietf-isis-mi-bis-03test.txt'

draft-ietf-isis-mi-bis-03test.txt(660): Warning: Failed parsing a reference:
   [ISO10589]
              "Intermediate system to Intermediate system intra-domain
              routeing information exchange protocol for use in
              conjunction with the protocol for providing the
              connectionless-mode Network Service (ISO 8473), ISO/IEC
              10589:2002, Second Edition.", Nov 2002.
Written to 'draft-ietf-isis-mi-bis-03test.xml’

With the reference being removed but the reference element still included (causing v2 not to parse until the empty 
element was removed).

Note that I also got a similar error running draft-ietf-rtgwg-yang-key-chain-24:

id2xml draft-ietf-rtgwg-yang-key-chain-24test.txt
Converting 'draft-ietf-rtgwg-yang-key-chain-24test.txt'

draft-ietf-rtgwg-yang-key-chain-24test.txt(983): Warning: Failed parsing a reference:
   [Dobb96b]  Dobbertin, H., "The Status of MD5 After a Recent Attack",
              CryptoBytes Vol. 2, No. 2, Summer 1996.

*This second instance was more interesting to me based on the fact that the document contains a [Dobb96a] reference 
that is used very similarly, but was picked up just fine.

2) I was curious about how a reference to a BCP that included more than one RFC would be handled (for example, BCP 9 below).  


In text, it would appear as:

   [BCP9]     Bradner, S., "The Internet Standards Process -- Revision
              3", BCP 9, RFC 2026, October 1996.

              Kolkman, O., Bradner, S., and S. Turner, "Characterization
              of Proposed Standards", BCP 9, RFC 7127, January 2014.

              Dusseault, L. and R. Sparks, "Guidance on Interoperation
              and Implementation Reports for Advancement to Draft
              Standard", BCP 9, RFC 5657, September 2009.

              Housley, R., Crocker, D., and E. Burger, "Reducing the
              Standards Track to Two Maturity Levels", BCP 9, RFC 6410,
              October 2011.

              Resnick, P., "Retirement of the "Internet Official
              Protocol Standards" Summary Document", BCP 9, RFC 7100,
              December 2013.

              Dawkins, S., "Increasing the Number of Area Directors in
              an IETF Area", BCP 9, RFC 7475, March 2015.

              <http://www.rfc-editor.org/info/bcp9>


I don’t have a draft in queue to test this on, so I just manually added a reference to BCP 9 and corresponding citation 
to draft-ietf-rtgwg-yang-key-chain-24.txt. Below is the resulting XML from id2xml:

<reference anchor="BCP9"><front><title>The Internet Standards Process -- Revision 3</title>
	<author fullname="S. Bradner" initials="S." surname="Bradner"/>
	<date month="October" year="1996"/>
	</front>
	<seriesInfo name="BCP" value="9"/>
	<seriesInfo name="RFC" value="2026"/>
	</reference>
	<reference><front/>
	</reference>
	<reference><front/>
	</reference>
	<reference><front/>
	</reference>
	<reference><front/>
	</reference>
	<reference><front/>
	</reference>
	<reference><front/>
	</reference>

*Note - references to multi-RFC BCPs and STDs are rare and require manual post-xml2rfcv2 updates with our current tools. 
The current possible workaround in xml is to include an <annotation> element.  However, I *believe* only one such element is allowed.


3) There are XML snippets in the appendices of draft-ietf-rtgwg-yang-key-chain-24.  id2xml put the first two them into lists, 
which broke the alignment (the original XML by the authors used <artwork>), but the third snippet in the last appendix was put into <artwork>.


Original text:

Appendix A.  Examples

A.1.  Simple Key Chain with Always Valid Single Key

   <?xml version="1.0" encoding="utf-8"?>
   <data xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
     <key-chains xmlns="urn:ietf:params:xml:ns:yang:ietf-key-chain">
       <key-chain>
         <name>keychain-no-end-time</name>
         <description>
           A key chain with a single key that is always valid for
           transmission and reception.
         </description>
         <key>
           <key-id>100</key-id>
           <lifetime>
             <send-accept-lifetime>
               <always/>
             </send-accept-lifetime>
           </lifetime>
           <crypto-algorithm>hmac-sha-256</crypto-algorithm>
           <key-string>
             <keystring>keystring_in_ascii_100</keystring>
           </key-string>
         </key>
       </key-chain>
     </key-chains>
   </data>



id2xml text:
Appendix A.  Examples

A.1.  Simple Key Chain with Always Valid Single Key

   <?xml version="1.0" encoding="utf-8"?>
     <data xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"> <key-chains
     xmlns="urn:ietf:params:xml:ns:yang:ietf-key-chain"> <key-chain>
     <name>keychain-no-end-time</name> <description> A key chain with a
     single key that is always valid for transmission and reception.

       </description> <key> <key-id>100</key-id> <lifetime> <send-
       accept-lifetime> <always/>

          </send-accept-lifetime>

          </lifetime>
            <crypto-algorithm>hmac-sha-256</crypto-algorithm> <key-
            string> <keystring>keystring_in_ascii_100</keystring>

          </key-string>

       </key>

          </key-chain>

          </key-chains>

   </data>






[snip]
>> 
>> 1) We were surprised to see symrefs set to no as most reference
>> entries we see use the RFC number as citation tag.
> 
> The generated xml should reflect the symrefs setting appropriate for
> the input.  If the input file uses symrefs="no", the xml will try to
> detect that; and likewise for symrefs="yes".  Could you send me the
> input source which gave symrefs="no" when you did not expect it, please?

MF  - This was in the XML output from draft-ietf-isis-mi-bis-03.txt: it appears to be set to “yes” in the current version of id2xml.

> 
> 
[snip]

>> Artwork tags vs. lists 
>> ———————
>> 4) The following list was converted to a hanging list, which really changed the output.  
>> The author-submitted version had this list in artwork.
>> 
>> 
>> Original text:
>> 
>>   An MI-RTR MUST NOT support [RFC5120] multi-topology within a non-zero
>>   instance when any non-zero ITID is supported.  The following TLVs
>>   MUST NOT be sent in an LSP associated with a non-zero instance which
>>   supports a non-zero ITID and such an LSP MUST be ignored when
>>   received:
>> 
>>    TLV 222 - MT IS Neighbors
>>    TLV 235 - MT IP Reachability
>>    TLV 237 - MT IPv6 Reachability
>> 
>> id2xml text output:
>> 
>>   An MI-RTR MUST NOT support [RFC5120] multi-topology within a non-zero
>>   instance when any non-zero ITID is supported.  The following TLVs
>>   MUST NOT be sent in an LSP associated with a non-zero instance which
>>   supports a non-zero ITID and such an LSP MUST be ignored when
>>   received:
>> 
>>   TLV 222 - MT IS Neighbors
>> 
>>      TLV 235 - MT IP Reachability TLV 237 - MT IPv6 Reachability
> 
> Yes.  A definite flaw.  This is much improved in version 1.0.0-rc1, which
> I just released.  There will, however, still be cases where id2xml tries
> to make lists out of something that originally was artwork.  This is for
> good reasons: Artwork is the last fallback if nothing else will fit, because
> it doesn't map well to html elements -- it's just a fixed-width-font blob.
> 
> I'll snip the other examples you give -- please try v1.0.0-rc1 on the
> relevant documents, and see if it does better.  I believe it will.  If not,
> please let me know the document names I should use to reproduce the issue.
> 
MF - This is greatly improved in the current version when running this same file.
> 
>> 5) The text following this figure was originally included in an
>> <artwork> tag and appeared as slightly indented in the text.
>> 
>> original text:
> 
> ...
> 
>> id2xml text output:
>> 
>>     Type:   7
>>     Length: 2 - 254
>>     Value:
>>                                            No. of octets
>>                 +-------------------------+
>>                 | IID (0 - 65535)         |     2
>>                 +-------------------------+
>>                 | Supported ITID          |     2
>>                 +-------------------------+
>>                 :                         :
>>                 +-------------------------+
>>                 | Supported ITID          |     2
>>                 +-------------------------+
>> 
>>   When the IID = 0, the list of supported ITIDs MUST NOT be present.
>> 
>>   An IID-TLV with IID = 0 MUST NOT appear in an SNP or LSP.  When
>> 
>>      the TLV appears (with a non-zero IID) in an SNP or LSP, exactly
>>      one ITID MUST be present indicating the instance-specific topology
>>      with which the PDU is associated.  If no ITIDs or multiple ITIDs
>>      are present or the IID is zero, then the PDU MUST be ignored.
> 
> 
> ( <<snip>> a number of similar cases )
> 
> Yes.  This was a recurring phenomenon for some documents in 0.9.3.
> I believe it will be better handled by 1.0.0-rc1.  Could you check?
> 
MF - Yes, this seems to be resolved.

> If not, could you please provide the name of the input document, so
> I can add it to my test suite and work in improving the handling?
> 
> 
>> 
>> Citations throughout text
>> —————————
>> 
>> 6) It appears that citations were not converted to xref tags
>> throughout. For example:
> 
> This should also be much improved in 1.0.0-rc1.  Still missing is
> conversion of citations within tables; I've found that the chances
> of messing up there are higher.  I'll have a look at addressing that
> case, too.
> 
MF - Yes it is.
> 
>> Reference Entries in the References section
>> ————————
>> 7) We note that <reference> elements were not used and the references
>> section was created using a <section> element instead of <references>
>> and appears in the <middle> as opposed to <back>.
> 
> The detection of reference sections was very mixed in version 0.9.3.  If
> the reference sections were numbered, they would be missed, if they were
> not, they would be processed.  Version 1.0.0-rc1 attempts to recognize also
> numbered reference sections, and generate both inline <reference/> entries
> and entity definitions at the top.

MF - This seems to be the case when rerunning draft-ietf-isis-mi-bis-03.  
(Please note the 2 exceptions I encountered above.)


> 
>> Is it possible to make these be includes (e.g., 
>> <?rfc include='reference.RFC.XXXX’?>)? That way, the RPC will know the
>> entries are being pulled from the citation library and contain the
>> correct information. We understand that v3 will have to contain the
>> expanded references.
> 
> It is possible, but as you indicate, the <?rfc include=""?> construct is
> deprecated.  The best (I believe) alternative when working with both v2
> and v3 xml sources is to use entity definitions, and have those expanded by
> xml2rfc (in v2) and by the preptool (in v3) during final processing.  In
> order to make it easy to use entities, I insert entity definitions at the
> start of the generated xml both in 0.9.3 and in 1.0.0-rc1.
> 
> The choice of inserting both entity definitions at the start, and expanded
> xml refereces in the <references/> sections was done to make it easier for
> you to choose one or the other.  If you'd rather I inserted the appropriate
> entity at the point of reference, I could do so.  You would then instead
> of:
> 
>  <back>
>    <references title="Normative References">
>      <?rfc include="reference.RFC.2119"?>
>      <?rfc include='reference.RFC.5304'?>
>      ...
> 
> see:
> 
>  <back>
>    <references title="Normative References">
>      &RFC2119;
>      &RFC5304;
>      ...
> 
> and it would pull the reference entries from the citation library in the
> same way as with the <?rfc include ?> pi.
> 
MF - We believe it is more desirable for our purposes to have the entity definitions at the start and the &RFC2119;-style calls in the References section.  
> 
>> 8) In v2, there is currently a setting to keep DOIs, draft strings,
>> and URLs on the same line (if possible for the latter).
> 
> Ok; which setting are you thinking of?  I can certainly insert it if
> desired.

MF - Sorry - I don’t know of a specific setting.  I simply remember them implementing it into one of the v2 versions. 
> 
>> 9) We generally see a single space after the author initials instead
>> of two spaces there.
>> original text:
>> 
>>   [RFC5310]  Bhatia, M., Manral, V., Li, T., Atkinson, R., White, R.,
>>              and M. Fanto, "IS-IS Generic Cryptographic
>>              Authentication", RFC 5310, DOI 10.17487/RFC5310, February
>>              2009, <http://www.rfc-editor.org/info/rfc5310>.
>> 
>> id2xml text output:
>> 
>>   [RFC5310]  Bhatia, M., Manral, V., Li, T., Atkinson, R., White, R.,
>>              and M.  Fanto, "IS-IS Generic Cryptographic
>>              Authentication", RFC 5310, DOI 10.17487/RFC5310, February
>>              2009, <http://www.rfc-editor.org/info/rfc5310>.
> 
> There was an issue in 0.9.2 where extra spaces were not stripped from the
> initials string; that was fixed in 0.9.3, and should also be fixed in 
> 1.0.0-rc1.  Please let me know if you find that it's not the case.
> 
MF - This is resolved.
> 
> Thank you so much for the feedback!
> 
> 
> Best regards,
> 
> 	Henrik
> 
>