Re: [xml2rfc] assuming that period (.) ends a sentence is sometimes wrong

[Carsten Bormann <[e-mail]>]

> 
> Two things are being muddled here:

Indeed.


> 1) two spaces at end of sentences in .txt output;

And 1a) how to render sentence spacing in the HTML (which, AFAIK, is not helping here).

A sentence spacing of two character positions has been the traditional rendering in plain text.  
That is suboptimal, of course (as many things are in monospaced environments).  


> 2) how two distinguish sentence endings by xml2rfc in xml input.
> 
> There has been *some* discussion of using two spaces in the input for (2), but it doesn't work that way now and there are many issues in changing it to work that way. It isn't evident to me that it is a serious proposal.

So this is about sentence detection, not about sentence spacing.
(Which in turn can make use of sentence spacing in the input, but that is orthogonal.)

I’m trying to understand why the traditional method doesn’t work.
There is never any ambiguity with traditional keyboarding (a new line starts after a sentence); except that one has to be careful not to do an input line-break after [.!?] that is within a sentence.

The question is whether there is a need to accommodate multiple sentences per input line for proper sentence detection.
People differ in their style here.
The newline after a sentence rule helps with version control as well, so I have a strong preference for that style.
But for people who like running on on the same input line, the two-space convention has been working well.

XML may not "preserve whitespace”, but what that exactly means here is not clear to me.  
Double spaces in XML input are copied verbatim into the HTML (where they then are swallowed by the HTML processor), so it is not like the processor is not seeing them.

An easy fallback position is to no longer recognize sentence ends within an input line; that certainly solves the Philip R. Zimmermann issue (which shouldn’t be there as we have a <contact> element for natural names, but I digress).

> *If* we had a reliable method for (2) then I doubt there would be much issue with (1). The problem is that the existing method for (2) isn't reliable.

I’d say get rid of the heuristics.

But then, the thinking in the python textwrap module that is being used here to do the heavy lifting is extremely confused, so I don’t know whether that can be parameterized to be sane.  
Xml2rfc certainly tries...

> I haven't checked, but I presume the current problems (2) are also exhibited in html output.

There is no attempt in xml2rfc that I can recognize to have sentence spacing in HTML output.

> ISTM that the real question is whether authors will be willing to manually annotate the xml input to indicate sentence endings.

The answer is: Absolutely not.

> I haven't seen any proposal mentioned that I would willingly use on a regular basis. I would rather suffer with the existing heuristic.

A heuristic causes people to want to work around its limitations.  
Adding a zero-width space after the end of a middle name is a simple workaround for the current heuristic, but, ugh.  
As long as the future of the <contact> element is unclear (and the grammar bugs around it aren’t fixed), I’d also not want to steer people towards that.

Grüße, Carsten