Re: [Tools-discuss] non-ASCII, non-UTF-8 in rsync'ed html

[Henrik Levkowetz <[e-mail]>]

On 2018-11-20 17:16, Daniel Kahn Gillmor wrote:
> On Tue 2018-11-20 14:45:28 +0100, Henrik Levkowetz wrote:
>> On 2018-11-20 00:53, Daniel Kahn Gillmor wrote:
>>
>>> i found that there were several files in that repo that contained
>>> non-ASCII and non-UTF-8 octet sequences.
>>
>> Yes.  Many of these (possibly all, but I haven't inspected each file
>> individually) are due to control characters in the text documents.
>> SUB (^Z) seems to be the most common one, but there are BS, DC2, SI,
>> ACK, FS, and more.
> 
> Thanks!  Filtering those out for the HTML would be good.  But that's
> clearly not the only thing remaining, since some files contain high
> bytes.
> 
> I did a review of all of the documents that contain octets above 0x7f:

I've looked through the equivalent .txt files, and I believe that common
for all except draft-ietf-2000-issue is that they use unknown code pages;
not latin-1 and certainly not utf-8.  An attempt to check some of them
against other less code pages failed to find a match.

Any fixup of these would have to be built into the htmlizer as custom
tweaks for each specific document (except for draft-ietf-2000-issue,
where some other solution would be needed).

All of these are old, I think the latest from 2005; two from 2004; and
the rest earlier.  I doubt it's meaningful to add custom fixes...


Best regards,

	Henrik

>  * draft-ietf-2000-issue-05.html is the most troubling one -- it appears
>    to contain fragments of an embedded pdf document, if i'm reading it
>    correctly.
> 
>  * draft-felton-universal-language-00.html contains what appears to be
>    attempted embedded unicode Chinese, but it has been garbled.
> 
>  * draft-ietf-dnsop-interim-signed-root-01.html and
>    draft-ihren-dnsop-interim-signed-root-01.html contain more mangled
>    unicode in the author names.
> 
>  * draft-ietf-pkix-cmmf-02.html contains some kind of mangled em dashes
>    and smartquotes
> 
>  * draft-ietf-rmonmib-pi-ipv6-01.html has a lot of weirdness in the very
>    end of the document
> 
>  * draft-ietf-smime-domsec-00.html appears to have mangled smartquotes
> 
>  * draft-klensin-dns-role-01.html has some breakage around "Montréal"
>    and 'German glyph "ö"'
> 
>  * draft-loughney-sctp-sig-prot-00.html contains some mangled characters
>    in the organization addresses
> 
>  * draft-rabbat-ccamp-carrier-survey-01.html contains a weird character
>    in the page headers between "Expires" and "April 2005"
> 
>  * draft-vasseur-mpls-backup-computation-02.html mangles "España"
> 
>  * draft-worster-mpls-in-ip-02.html has a damaged Fax number for Rick
>    Wilder
> 
> Regards,
> 
>         --dkg
>