[rfc-i] archiving outlinks in RFCs

[Alexis Rossi <[e-mail]>]

Hi all,

I wanted to let the community know about something I’ve been working on. As you might know, one of my previous jobs was running the Wayback Machine, so when I started working with with this collection of RFCs one of my first thoughts was, “I wonder how many broken links are in these RFCs from the past few decades?” 

In general, the average lifespan of a URL before the content changes or disappears is on the order of 100 days. Fortunately for us, the links used in RFC references seem to be much more stable than that. For instance, so far I’ve only found one broken link in an RFC from the past 6 months [1].

Even though we favor these more stable URLs, some of them will eventually change or go 404 and having archival documents with link rot is something we can take steps to avoid in the future.

The first thing I wanted to do was just make sure we were archiving these outlinks somewhere. This won’t fix a broken link in the RFC, but at least the resource can be saved elsewhere for someone curious enough to go look (and potentially we could fix links in some version of the RFC in future).

The main services that are well qualified for this purpose are Archive-It.org <http://archive-it.org/> (run by the Internet Archive) and Perma.cc <http://perma.cc/> (run by Harvard Law School Library). I chose Archive-It, and when I approached them they offered us an account [2] with enough free data storage for our needs. Yay for non-profits supporting each other! 

So far I have used Archive-It to:
Archive rfc-editor.org <http://rfc-editor.org/>, iab.org <http://iab.org/>, irtf.org <http://irtf.org/>, and ietf.org <http://ietf.org/> (minus datatracker and the mail archive)
There are lots of references to these sites in RFCs, but I also wanted to preserve the contents for their own sake. I plan to revisit these sites once per year.
I am avoiding datatracker (except for outlinks from RFCs) because of concern about the extra traffic causing problems for the team that maintains the site.
I have not concentrated on archiving the mail archive yet, though I know some of it has been saved incidentally. 
Archive outlinks from RFCs
About once per quarter I’ll grab the outlinks from newly published RFCs and get them crawled. 
I am also going backwards through the entire series - I’ve started with the most recent RFCs (links are more likely to still be live) and am working my way back in time. 

There may be more room for improvements here, for example including archived links in RFCs from the start w here appropriate, or potentially defining a way for links to be self-healing in published RFCs. 

Please let me know if you have ideas or feedback on this.

Thanks,
Alexis


[1] RFC9311 published in September 2022, in Section 11 (Informative References) this link is 404: https://www.ietf.org/how/meetings/98/bits-n-bites/ <https://www.ietf.org/how/meetings/98/bits-n-bites/>[2] https://archive-it.org/organizations/2540 <https://archive-it.org/organizations/2540>