[xml2rfc] Non-determinism (Re: Unable to run xml2rfc --help)

[Carsten Bormann <[e-mail]>]

While it might be a special need to actually compare generated HTML between different runs, but when testing an entire authoring chain, it is a bit unnerving to have changes like

-          <li class="toc ulEmpty compact" id="section-toc.1-1.12">
+          <li class="ulEmpty toc compact" id="section-toc.1-1.12">

between runs that are meant to produce identical output.

My python-fu is not sufficient to suggest a way to hide the randomized Python behavior from this output (replace Dictionary by OrderedDict?, but I thought that was the default now since Python 3.7); could somebody else please step in.

Grüße, Carsten



> On 2020-12-29, at 16:44, Carsten Bormann <[e-mail]> wrote:
> 
> That appears to be a rather funny bug.
> Apparently, the ASCII neutral minus hyphen is in the list of keys. Depending on how the Python hashing random number generator is seeded, that hyphen is going to a position where it can’t be in the regexp or where it can. True Heisenbug. 
> (The hyphen needs to be protected to go into a character class, even when the resulting character range seems to be syntactically acceptable.)
> 
> Sent from mobile, sorry for terse
> 
>> On 29. Dec 2020, at 11:22, Roger Price <[e-mail]> wrote:
>> 
>> On Mon, 28 Dec 2020, Carsten Bormann wrote:
>> 
>>> On 2020-12-25, at 15:24, Roger Price <[e-mail]> wrote:
>>>> python3.5
>>> but this got my attention: Python 3.5 is end-of-life; did you try a newer Python?
>> 
>> I changed the xml2rfc shebang to #!/usr/bin/python3.8 and ran Python 3.8.1 but it made no difference.
>> 
>>> I’m out of educated guesses (at least until you are showing more of the traceback),
>> 
>> The full Python error message is:
>> 
>> Traceback (most recent call last):
>>  File "/usr/bin/xml2rfc", line 7, in <module>
>>    from xml2rfc.run import main
>>  File "/mnt/home/rprice/.local/lib/python3.5/site-packages/xml2rfc/__init__.py", line
>>    14, in <module>
>>    from xml2rfc.parser import  XmlRfcError, CachingResolver, XmlRfcParser, XmlRfc
>>  File "/mnt/home/rprice/.local/lib/python3.5/site-packages/xml2rfc/parser.py",
>>    line 20, in <module> from xml2rfc.writers import base
>>  File "/mnt/home/rprice/.local/lib/python3.5/site-packages/xml2rfc/writers/__init__.py",
>>    line 2, in <module> from xml2rfc.writers.base import RfcWriterError
>>  File "/mnt/home/rprice/.local/lib/python3.5/site-packages/xml2rfc/writers/base.py",
>>    line 30, in <module>
>>    from xml2rfc.util.unicode import ( punctuation, unicode_replacements,
>>    unicode_content_tags, bare_unicode_tags,
>>  File "/mnt/home/rprice/.local/lib/python3.5/site-packages/xml2rfc/util/unicode.py",
>>    line 260, in <module>
>>    punctuation_re = re.compile(r'[%s]'%''.join(list(punctuation.keys())))
>>  File "/usr/lib/python3.5/re.py", line 224, in compile
>>    return _compile(pattern, flags)
>>  File "/usr/lib/python3.5/re.py", line 293, in _compile
>>    p = sre_compile.compile(pattern, flags)
>>  File "/usr/lib/python3.5/sre_compile.py", line 536, in compile
>>    p = sre_parse.parse(p, flags)
>>  File "/usr/lib/python3.5/sre_parse.py", line 829, in parse
>>    p = _parse_sub(source, pattern, 0)
>>  File "/usr/lib/python3.5/sre_parse.py", line 437, in _parse_sub
>>    itemsappend(_parse(source, state))
>>  File "/usr/lib/python3.5/sre_parse.py", line 575, in _parse
>>    raise source.error(msg, len(this) + 1 + len(that))
>> sre_constants.error: bad character range …-‚ at position 3
>> 
>> In file unicode.py at line 260
>> 
>> punctuation_re = re.compile(r'[%s]'%''.join(list(punctuation.keys())))
>> 
>> I added the two lines
>> 
>> import sys
>> print("unicode.py: list(punctuation.keys())) = {}".format(list(punctuation.keys())))
>> 
>> On runs when Python crashes I see outputs:
>> 
>> unicode.py: list(punctuation.keys()) = ['−', '‐', '′', '\u2002', '’', '´',
>>                 '\u2003', '‚', '–', '\u2009', '”', '-', '“', '‘', '…', '„', '—']
>> unicode.py: list(punctuation.keys()) = ['„', '”', '—', '–', '…', '-', '\u2009',
>>                 '‘', '’', '\u2003', '−', '´', '\u2002', '‚', '′', '‐', '“']
>> 
>> When the runs succeed I see outputs:
>> 
>> unicode.py: list(punctuation.keys()) = ['−', '\u2003', '\u2009', '‘', '“', '‚',
>>                 '-', '”', '—', '’', '′', '\u2002', '‐', '–', '…', '´', '„']
>> unicode.py: list(punctuation.keys()) = ['\u2002', '‐', '′', '\u2003', '−', '´',
>>                 '“', '”', '„', '–', '-', '‘', '’', '…', '‚', '\u2009', '—']
>> 
>> I do not understand why such a list of constants has to be so random. I'm not sure where to look next, but if you would like to see further traces of other functions, just ask. Roger_______________________________________________
>> xml2rfc mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/xml2rfc
> _______________________________________________
> xml2rfc mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/xml2rfc