Re: [Tools-discuss] BOMs and rfcmarkup

[Henrik Levkowetz <[e-mail]>]

Hi Adam,

On 2017-09-20 23:44, Adam Roach wrote:
> On 9/20/17 1:34 PM, Henrik Levkowetz wrote:
>> Hi Brian,
>>
>> On 2017-09-20 01:47, Brian E Carpenter wrote:
>>> Just a note that https://tools.ietf.org/html/rfc8187 seems
>>> to be fine, iff your browser is set to assume UTF-8. But it
>>> does include the BOM.
>>>
>>> If your browser is set to assume what Firefox calls "Western",
>>> the document starts with  (which is the IS8859 interpretation
>>> of a UTF-8 BOM, expressed in UTF-8, if that doesn't make your head
>>> hurt and if it survives the email system). Naturally, the £, € and
>>> ü do not display correctly.
>>>
>>> The BOM is in the generated HTML immediately after the <pre>.
>>> However, with Firefox it performs no useful function; all that matters
>>> is the View/Text Encoding setting. Exactly the same applies to Internet
>>> Explorer.
>>>
>>> Thus, at least for these two browsers, including the BOM in the HTML
>>> file seems to be pointless. I'd vote for removing it.
>> I've updated rfcmarkup to strip out BOMs; the link above should now
>> give you a document without a BOM after the <pre>.
> 
> Wait. WAIT! No. This is the wrong answer. If you are in a tool that 
> makes the BOM visible at all, the problem isn't the presence of the BOM 
> (which was well-defined rendering semantics in UTF8 that basically say 
> "this shouldn't be visible in any way at all"); the problem is that you 
> are loading a UTF-8 document and treating it as _some_ _other_ _encoding_.

I'm sorry, but I disagree (at least in one sense).  Earlier, rfcmarkup
left a BOM in the middle of the html, which I believe is wrong.  You
could say that this is the result of not reading a UTF-8 file with BOM
correctly, and I would agree.  So now I'm reading the content as UTF-8,
but not keeping the BOM, and not placing it in the middle of a html file
(which is served with Content-Type: text/html; charset=UTF-8).  I think
that's right.

> It does no good to paper over the presence of a BOM, since that just 
> defers the problem to elsewhere (namely, any other non-ASCII codepoints 
> in the document).

Nope.  I've been handling UTF-8 correctly in rfcmarkup for several years;
it has behaved correctly for draft-ietf-httpbis-rfc5987bis for every
version which has had non-ascii characters.

> The proper behavior here, and especially for rfcdiff, is to make sure 
> everything is using UTF-8. Leaving the BOM in place serves as a useful 
> canary in this coal mine, and I would suggest *not* stripping it, at 
> least until we're sure the tools *otherwise* handle UTF-8 correctly. 
> That is, the BOM provides a good litmus test: if you can see it, you're 
> doing it wrong.

I think my experience is that rfcmarkup has bean dealing just fine with
utf-8 in general, but the BOM (which I think is nonsense in UTF-8, but
that's a different discussion) has been visible, which I've now fixed.


Best regards,

	Henrik