[Sidrops] RPKIViews 2021-2025 Amalgamations: a near-complete capture of the entire RPKI

[Job Snijders <[e-mail]>]

Dear all,

The RPKI is an infrastructure which helps secure the global Internet
routing system. The RPKI is a distributed database. RPKIViews.org is a
global multi-perspective data collection initiative. I've normalized,
deduplicated, merged, sorted, and recompressed tens of terabytes of RPKI
data, resulting in handy datasets named the "RPKIViews Amalgamations".
These compact datasets together contain basically every ROA, ASPA, CRL,
Manifest, and Certificate that were issued in the last 5 years.

  RPKIViews 2021 Amalgamation, 46,507,392 RPKI objects
  doi: https://doi.org/10.5281/zenodo.18082494
  compressed: 33.8 gigabytes (uncompressed: 78.1 gigabytes).

  RPKIViews 2022 Amalgamation, 51,850,445 RPKI objects
  doi: https://doi.org/10.5281/zenodo.18327144
  compressed: 41.7 gigabytes (uncompressed: 110.2 gigabytes).

  RPKIViews 2023 Amalgamation, 57,286,485 RPKI objects
  doi: https://doi.org/10.5281/zenodo.18327429
  compressed: 48.7 gigabytes (uncompressed: 111.8 gigabytes).

  RPKIViews 2024 Amalgamation, 56,586,149 RPKI objects
  doi: https://doi.org/10.5281/zenodo.18328474
  compressed: 51.4 gigabytes (uncompressed: 115.5 gigabytes).

  RPKIViews 2025 Amalgamation, 61,524,413 RPKI objects
  doi: https://doi.org/10.5281/zenodo.18332099
  compressed: 58.4 gigabytes (uncompressed: 145.0 gigabytes)

The DER-encoded objects originally were fetched from publication servers
which were discovered using the Trust Anchor Locators of the following
Regional Internet Registries: AfriNIC, APNIC, ARIN, LACNIC, and RIPE
NCC. Not all objects contained within this dataset comply with the
requirements of modern day RPKI relying party implementations.

The following file naming scheme is used for the tar archive members:
every filename is the URL-safe Base64-encoded SHA-256 message digest of
the object's ASN.1 encoded content. For performance reasons, the
directory hierarchy is constructed using the last few bytes of the
filename. The file's recorded last-modification timestamp is constructed
following the procedure outlined in RFC 9589. The tar achive adheres to
the IEEE Std 100.2 POSIX.2 "USTAR" interchange format and is compressed
in "--long -19" Zstandard (RFC 8878) form. Individual objects can be
inspected as JSON with "rpki-client -jf ./path/to/object" (filemode).

Happy hacking & sleuthing! :-)

Kind regards,

Job