Re: [TOOLS-DEVELOPMENT] Preview release of Text Submission Converter, id2xml

[Megan Ferguson <[e-mail]>]

Hi Henrik,

Input file: draft-ietf-trill-directory-assist-mechanisms-12
Version: id2xml 1.0.0
Issues: File not originally generated with XML
Files available: 
https://www.rfc-editor.org/rfc/v3test/draft-ietf-trill-directory-assist-mechanisms-12v3.original 
https://www.rfc-editor.org/rfc/v3test/draft-ietf-trill-directory-assist-mechanisms-12v3.txt
https://www.rfc-editor.org/rfc/v3test/draft-ietf-trill-directory-assist-mechanisms-12v3-rfcdiff.html

(The .txt version above includes our manual updates and the rfcdiff highlights these from the original.)

Note - This file is very representative of the types of issues we receive when getting a file not originally 
created with xml2rfc and the case where we feel the id2xml tool would be most useful for the RPC.  That is, 
we believe that this is a good example of a common use case of id2xml.

Testing this file raised a number of questions/errors.

1) Header

a) The status “Proposed Standard” does not seem to be recognized.  This is a common value we see in I-Ds not created with xml2rfc.
For our purposes, Proposed Standard, Draft Standard, Internet Standard, Standard Track, and Standards Track should all map to 
“Standards Track” handling.

 File "/usr/lib/python2.7/site-packages/id2xml/parser.py", line 1403, in get_category
   self.warn(line, "Expected a recognized status name, found '%s'" % (line.txt, ))

b) WG name: This document did not include a working group name in the header.  We receive documents with no working 
group indicated (as with this one), with the full WG name, or with the abbreviation.  Is this required for parsing?

c) Does the use of the authors’ full names in the header (or if they included more than one initial) have any affect on output?

We are getting the following error *whether or not* the initials or full first names appear:

Failure converting: 'NoneType' object does not support item assignment
Traceback (most recent call last):
  File "/usr/bin/id2xml", line 9, in <module>
    load_entry_point('id2xml==1.0.0', 'console_scripts', 'id2xml')()
  File "/usr/lib/python2.7/site-packages/id2xml/run.py", line 222, in run
    xml = parser.parse_to_xml()
  File "/usr/lib/python2.7/site-packages/id2xml/parser.py", line 953, in parse_to_xml
    doc = self.document()
  File "/usr/lib/python2.7/site-packages/id2xml/parser.py", line 978, in document
    self.root.append(self.back())
  File "<string>", line 2, in back
  File "/usr/lib/python2.7/site-packages/id2xml/parser.py", line 560, in wrap
    ret = fn(self, *params,**kwargs)
  File "/usr/lib/python2.7/site-packages/id2xml/parser.py", line 2669, in back
    self.read_authors_addresses()
  File "<string>", line 2, in read_authors_addresses
  File "/usr/lib/python2.7/site-packages/id2xml/parser.py", line 560, in wrap
    ret = fn(self, *params,**kwargs)
  File "/usr/lib/python2.7/site-packages/id2xml/parser.py", line 1693, in read_authors_addresses
    self.maybe_author_address(item)
  File "<string>", line 2, in maybe_author_address
  File "/usr/lib/python2.7/site-packages/id2xml/parser.py", line 560, in wrap
    ret = fn(self, *params,**kwargs)
  File "/usr/lib/python2.7/site-packages/id2xml/parser.py", line 1793, in maybe_author_address
    item['address'] = {}
TypeError: 'NoneType' object does not support item assignment


*However*, we get the following similar error on another file that previously parsed correctly when we 
solely updated to use the authors’ full names in the header (instead of just initials):

Failure converting: 'NoneType' object does not support item assignment
Traceback (most recent call last):
  File "/usr/bin/id2xml", line 9, in <module>
    load_entry_point('id2xml==1.0.0', 'console_scripts', 'id2xml')()
  File "/usr/lib/python2.7/site-packages/id2xml/run.py", line 222, in run
    xml = parser.parse_to_xml()
  File "/usr/lib/python2.7/site-packages/id2xml/parser.py", line 953, in parse_to_xml
    doc = self.document()
  File "/usr/lib/python2.7/site-packages/id2xml/parser.py", line 978, in document
    self.root.append(self.back())
  File "<string>", line 2, in back
  File "/usr/lib/python2.7/site-packages/id2xml/parser.py", line 560, in wrap
    ret = fn(self, *params,**kwargs)
  File "/usr/lib/python2.7/site-packages/id2xml/parser.py", line 2669, in back
    self.read_authors_addresses()
  File "<string>", line 2, in read_authors_addresses
  File "/usr/lib/python2.7/site-packages/id2xml/parser.py", line 560, in wrap
    ret = fn(self, *params,**kwargs)
  File "/usr/lib/python2.7/site-packages/id2xml/parser.py", line 1693, in read_authors_addresses
    self.maybe_author_address(item)
  File "<string>", line 2, in maybe_author_address
  File "/usr/lib/python2.7/site-packages/id2xml/parser.py", line 560, in wrap
    ret = fn(self, *params,**kwargs)
  File "/usr/lib/python2.7/site-packages/id2xml/parser.py", line 1793, in maybe_author_address
    item['address'] = {}
TypeError: 'NoneType' object does not support item assignment

We tried (to no avail):
-updating full first names to initials only
-matching 1:1 the affiliation of each author with what appears in the Authors’ Addresses section 
(original uses Huawei for two authors and Huawei Technologies was used in the Addresses section).
-updating the header to include Internet-Draft
-removing blank space in the left side of the header

Follow ups on header-related issues:

i) What regulations/limitations exist on how items must appear in the header?  We are also curious about documents 
from other streams such as IRTF or Independent (which we have yet to test).

ii) We are unsure about the meaning of some of the header-related errors.  For example:

Warning: Expected a label indication top left, found none

Perhaps getting a bit more information on this would help us troubleshoot.

iii) Perhaps "Intended status" and the like should be case insensitive?

2) The Status of This Memo did not match the current version.  This caused an error:

Error: Unexpected text: expected 'Internet-Drafts are working documents of the Internet
  Engineering Task Force (IETF).  Note that other groups may also distribute working documents as Internet-Drafts.  The
  list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.', found 'Distribution of this
  document is unlimited. Comments should be sent’

3) Older documents frequently have Copyright notices at the end of the document (such as this one).  This caused an error:

Error: Unexpected text: expected 'Copyright Notice', found 'D. Eastlake, et al
  [Page 1]’

4) The running header (perhaps due to the period after D?) was a problem and needed to be manually stripped.

D. Eastlake, et al                                              [Page 2]
INTERNET-DRAFT                       TRILL: Directory Service Mechanisms

Error: Unexpected section number; expected '1' or a subsection, found 'D. Eastlake, et al
  [Page 1]'

5) The references section is unnumbered.  This (maybe)creates an error.  

Error: Expected an Authors' Addresses section, found 'Normative References’

Thank you in advance for your help!

Megan