[Sidrops] Publication Point -> RP synchronization in bandwidth constrained environments (note for RRDP v2)

[Job Snijders <[e-mail]>]

Dear all,

I wanted to share a note that if work ever starts on RRDP v2, that
design team should consider conservation of bandwidth a priority.

Recently one of the RIRs ran into a network congestion issue noticable
on intercontinental transfers: their 36 megabyte RRDP snapshot was being
served at a rate of ~ 25 kilobytes per second, and worse the TCP
connections often timed out.

As some RP implementations expect a minimum delivery rate (for example
100 kB/sec), some implementations allocate a timebox, and all
implementations will switch to RSYNC if RRDP is interrupted for whatever
reason, (causing a RRDP snapshot fetch to be queued up for the future).
The more RPs try to fetch the RRDP snapshot for one reason or another,
the more bandwidth is demanded, which in turn increases congestion.

Some back of the napkin math: with 2500 RRDP clients interested in a 36
megabyte RRDP snapshot, timeboxed into 15 minutes, the RRDP server would
need to be able to sustain 1 Gb/sec for 15 minutes to send that data to
to all RPs. If such a server for whatever reason ends up moving behind a
100 Mb/sec link - in the steady state serving mostly just deltas the
available bandwidth still might seem good enough. However, the moment an
event like a RRDP Session restart happens, things go awry and perpetual
congestion could be the result.

In this instance a (temporary) solution was to disable RRDP [1] (by
serving a HTTP 204 error for the /notification.xml file), steering all
RPs to use RSYNC, which seems to perform far better in bandwidth
constrained environments.

Yes, while "make the pipes bigger!" or "Just use a CDN!" might seem a
logical and conclusive answer, I think there is room for optimization in
the RPKI synchronization protocols. Bigger pipes cost bigger money, CDNs
usually aren't free either, plus an additional layer of caching in the
CDN also introduces additional complexity in operations.

Making RPKI PP/RP synchronization more efficient is advantageous to all
stakeholders in this ecosystem.

Why does RSYNC perform better in this type of situation?
=========================================================

1) The Base64 encoding in RRDP imposes a non-negligible overhead of 33%
   which doesn't exist in RSYNC (where the files are transferred in
their 'bare' form).
2) The XML formatting of the resulting document (elements, attributes &
   newlines) adds another single digit percentage of overhead compared
   to transfer of the bare files in bare form.
3) RRDP snapshot downloads are hard to resume: if the session is
   interrupted and then the RRDP serial increases, RPs will try to fetch
   the new snapshot as per instructions from the RRDP server. RSYNC on
   the other hand provides a far more fine-grained and seamless
   resumption mechanism.

Of course there are well-understood downsides to RSYNC too, what we're
looking at is a set of trade-offs: mostly centered on the trade-off
between bandwidth conservation in exchange for cpu cycles. The point of
this message is not to promote RSYNC but to take a look at the upsides
of RSYNC compared to RRDP v1.

In this particular event (measured at a distance of 182 ms latency),
disabling RRDP caused the time it takes to do a full synchronization
from 30+ minutes down to 17 seconds, subsequent resyncs now take 6
seconds, and congestion seems to have cleared.

Takeaways for RRDP v2
=====================

* The publication point operator should be able to signal in the
  (equivalent of) notification.xml file a set of recommended timers
  for fetching: it would be super nice if during congestion events the
  PP operator can inform the clients to please check back at at slower
  pace (say every 20 or 40 minutes) instead of whatever the RP's timers
  are. This would give the PP operator some tools to ameliorate the
  situation when lacking bandwidth or other resources.

* Replace the concept of distributing signed objects, certs & crls
  inside XML documents using Base64-encoding, with something more
  efficient. Perhaps packing all to-be-transferred files in a DER
  SEQUENCE, optionally wrapped in a self-signed CMS container for easy
  checksumming. Another feasible approach would be to design an
  RPKI-application specific compression algorithm : there is a ton of
  duplicity in for example AuthorityInfoAccess fields or the
  policyQualifier fields.

* Rsync offers the community individually addressable URIs for
  individual files, while RRDP v1 only offers 'the whole thing'.
  Individually fetchable files are fantastic for debugging, and makes it
  easier for CA/RP developers to reason and communicate about events.
  What APNIC is doing here by offering all files on the RSYNC server
  also as HTTPS download is a step in the right direction:
  https://rpki.apnic.net/member_repository/A91A0D9C/9E28300657A811E8B4AC0877C4F9AE02/

* Reconsider compression. I am aware of issues like CVE-2021-43174
  which prompted some developers to remove support for gzip compression
  in implementations, yet at the same time there is a lot to be gained
  from compression:

    $ ls -lahtrS afrinic.*
    -rw-r--r--  1 job  wheel   5.8M May 29 12:15 afrinic.tar.lzma
    -rw-r--r--  1 job  wheel   6.0M May 29 12:15 afrinic.tar.zst
    -rw-r--r--  1 job  wheel   6.1M May 29 12:15 afrinic.tar.bz2
    -rw-r--r--  1 job  wheel   6.3M May 29 12:15 afrinic.tar.gz
    -rw-r--r--  1 job  wheel  13.0M May 20 12:51 afrinic.rrdp.xml.gz
    -rw-r--r--  1 job  wheel  17.8M May 29 12:15 afrinic.tar
    -rw-r--r--  1 job  wheel  35.5M May 20 12:51 afrinic.rrdp.xml

  In the above terminal transcript the 'afrinic.tar' file simply is a
  tar archive of the validated cache directory specific to AfriNIC. This
  tar file contains all currently valid objects. The tar file is half
  the size of the RRDP snapshot, and compressing the tar with lzma,
  zstd, bz2, or gzip yields a considerable smaller outcome than gzip
  compressing the RRDP v1 XML snapshot. As noted above, an
  RPKI-application-aware compression algorithm would yield even smaller
  snapshots.

I'm sure other people have learned other lessons over the years working
with both RSYNC and RRDP: your and their input could perhaps contribute
to a RRDP v2 requirements document.

When we have RRDP v2, perhaps either RSYNC or RRDP v1 can be deprecated.

Kind regards,

Job

[1]: AfriNIC disabled RRDP https://status.afrinic.net/notices/caelaru4q1vhqbv2-rrdp-service-availability