Re: [Tools-discuss] non-ASCII, non-UTF-8 in rsync'ed html

[Henrik Levkowetz <[e-mail]>]

Hi Daniel,

On 2018-11-20 00:53, Daniel Kahn Gillmor wrote:
> I ran the following to get all html files:
> 
>      rsync -az rsync.tools.ietf.org::tools.html/ html/
> 
> But i found that there were several files in that repo that contained
> non-ASCII and non-UTF-8 octet sequences.

Yes.  Many of these (possibly all, but I haven't inspected each file
individually) are due to control characters in the text documents.
SUB (^Z) seems to be the most common one, but there are BS, DC2, SI,
ACK, FS, and more.

Yes, I could have a go at filtering out those at the same time as I
generate the htmlized versions.  I'll do so for control characters,
then we can revisit and see if additional action is needed.


Best regards,

	Henrik

> In particular, i ran:
> 
>     find html/ -iname '*.html' -print0 | xargs -0 file | grep -v 'ASCII text' | grep -v 'UTF-8 Unicode text'
> 
> and it produced this output:
> 
> html/draft-calabrese-requir-logprot-04.html:                     data
> html/draft-dewinter-queue-start-00.html:                         data
> html/draft-dewinter-queue-start-01.html:                         data
> html/draft-duerst-iri-05.html:                                   data
> html/draft-dusse-smime-msg-03.html:                              data
> html/draft-felton-universal-language-00.html:                    data
> html/draft-fujiwara-dnsop-resolver-update-00.html:               data
> html/draft-iana-special-ipv4-03.html:                              data
> html/draft-iana-special-ipv4-04.html:                              data
> html/draft-ietf-2000-issue-00.html:                                data
> html/draft-ietf-2000-issue-03.html:                                data
> html/draft-ietf-2000-issue-04.html:                                data
> html/draft-ietf-2000-issue-05.html:                                data
> html/draft-ietf-2000-issue-06.html:                                data
> html/draft-ietf-ccamp-mpls-tp-rsvpte-ext-associated-lsp-08.html: data
> html/draft-ietf-dhc-mdhcp-00.html:                               data
> html/draft-ietf-dhc-multopt-00.html:                             data
> html/draft-ietf-disman-remops-mib-00.html:                       data
> html/draft-ietf-disman-remops-mib-01.html:                       data
> html/draft-ietf-dnsop-interim-signed-root-01.html:               data
> html/draft-ietf-hubmib-mau-mib-03.html:                          data
> html/draft-ietf-hubmib-mau-mib-04.html:                          data
> html/draft-ietf-hubmib-repeater-dev-02.html:                     data
> html/draft-ietf-hubmib-repeater-dev-03.html:                     data
> html/draft-ietf-ipcdn-pktc-signaling-07.html:                   data
> html/draft-ietf-ippm-ipdv-01.html:                              data
> html/draft-ietf-isis-opexp-01.html:                             data
> html/draft-ietf-ldapext-matchedval-06.html:                       data
> html/draft-ietf-mpls-p2mp-loose-path-reopt-01.html:              data
> html/draft-ietf-ospf-hmac-sha-01.html:                           data
> html/draft-ietf-pkix-cmmf-02.html:                               data
> html/draft-ietf-pkix-scvp-15.html:                               data
> html/draft-ietf-rmonmib-pi-ipv6-01.html:                         data
> html/draft-ietf-rtfm-new-traffic-flow-00.html:                   data
> html/draft-ietf-smime-domsec-00.html:                                data
> html/draft-ietf-spki-cert-req-00.html:                               data
> html/draft-ietf-spki-cert-structure-00.html:                         data
> html/draft-ihren-dnsop-interim-signed-root-01.html:                   data
> html/draft-klensin-dns-role-01.html:                             data
> html/draft-lear-ietf-rfc2026bis-00.html:                         data
> html/draft-leung-sigtran-stream-sctp-00.html:                    data
> html/draft-loughney-sctp-sig-prot-00.html:                        data
> html/draft-murray-auth-ftp-ssl-03.html:                              data
> html/draft-newnan-isomib-internet-00.html:                           data
> html/draft-rabbat-ccamp-carrier-survey-01.html:                      data
> html/draft-rfced-info-dudley-01.html:                                               data
> html/draft-vasseur-mpls-backup-computation-02.html:               data
> html/draft-vrancken-oauth-redelegation-01.html:                   data
> html/draft-wenzel-cctld-bcp-00.html:                              data
> html/draft-white-slapm-mib-00.html:                               data
> html/draft-worster-mpls-in-ip-02.html:                            data
> html/draft-zhu-rmcat-nada-02.html:                              data
> html/draft-zhu-rmcat-nada-03.html:                              data
> html/rfc1142.html:                                              data
> html/rfc542.html:                                            data
> html/rfc652.html:                                            data
> html/rfc674.html:                                            data
> html/rfc684.html:                                            data
> html/rfc731.html:                                            data
> html/rfc734.html:                                            data
> html/rfc736.html:                                            data
> html/rfc752.html:                                            data
> html/rfc774.html:                                            data
> html/rfc776.html:                                            data
> html/rfc783.html:                                            data
> 
> Are these things that can be cleaned up by a future html re-rendering?
> they're likely to break html parsers or other transformations that work
> from the html source.
> 
> Regards,
> 
>     --dkg
> 
> 
> 
> ___________________________________________________________
> Tools-discuss mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/tools-discuss
> 
> Please report datatracker.ietf.org and mailarchive.ietf.org
> bugs at http://tools.ietf.org/tools/ietfdb
> or send email to [e-mail]
> 
> Please report tools.ietf.org bugs at
> http://tools.ietf.org/tools/issues
> or send email to [e-mail]
>