Re: [Tools-discuss] non-ASCII, non-UTF-8 in rsync'ed html

[Daniel Kahn Gillmor <[e-mail]>]

On Tue 2018-11-20 14:45:28 +0100, Henrik Levkowetz wrote:
> On 2018-11-20 00:53, Daniel Kahn Gillmor wrote:
>
>> i found that there were several files in that repo that contained
>> non-ASCII and non-UTF-8 octet sequences.
>
> Yes.  Many of these (possibly all, but I haven't inspected each file
> individually) are due to control characters in the text documents.
> SUB (^Z) seems to be the most common one, but there are BS, DC2, SI,
> ACK, FS, and more.

Thanks!  Filtering those out for the HTML would be good.  But that's
clearly not the only thing remaining, since some files contain high
bytes.

I did a review of all of the documents that contain octets above 0x7f:

 * draft-ietf-2000-issue-05.html is the most troubling one -- it appears
   to contain fragments of an embedded pdf document, if i'm reading it
   correctly.

 * draft-felton-universal-language-00.html contains what appears to be
   attempted embedded unicode Chinese, but it has been garbled.

 * draft-ietf-dnsop-interim-signed-root-01.html and
   draft-ihren-dnsop-interim-signed-root-01.html contain more mangled
   unicode in the author names.

 * draft-ietf-pkix-cmmf-02.html contains some kind of mangled em dashes
   and smartquotes

 * draft-ietf-rmonmib-pi-ipv6-01.html has a lot of weirdness in the very
   end of the document

 * draft-ietf-smime-domsec-00.html appears to have mangled smartquotes

 * draft-klensin-dns-role-01.html has some breakage around "Montréal"
   and 'German glyph "ö"'

 * draft-loughney-sctp-sig-prot-00.html contains some mangled characters
   in the organization addresses

 * draft-rabbat-ccamp-carrier-survey-01.html contains a weird character
   in the page headers between "Expires" and "April 2005"

 * draft-vasseur-mpls-backup-computation-02.html mangles "España"

 * draft-worster-mpls-in-ip-02.html has a damaged Fax number for Rick
   Wilder

Regards,

        --dkg