Re: [Sidrops] Publication Point -> RP synchronization in bandwidth constrained environments (note for RRDP v2)

[Di Ma <[e-mail]>]

Job,

You are making sense here.

I have been there when operating RPSTIR2.

And I am in favor of your proposed solution especially “Reconsider compression” after evaluating the running codes of RPSTIR2 .

Di


> 2023年5月30日 23:39，Job Snijders <job=[e-mail]> 写道：
> 
> Dear all,
> 
> I wanted to share a note that if work ever starts on RRDP v2, that
> design team should consider conservation of bandwidth a priority.
> 
> Recently one of the RIRs ran into a network congestion issue noticable
> on intercontinental transfers: their 36 megabyte RRDP snapshot was being
> served at a rate of ~ 25 kilobytes per second, and worse the TCP
> connections often timed out.
> 
> As some RP implementations expect a minimum delivery rate (for example
> 100 kB/sec), some implementations allocate a timebox, and all
> implementations will switch to RSYNC if RRDP is interrupted for whatever
> reason, (causing a RRDP snapshot fetch to be queued up for the future).
> The more RPs try to fetch the RRDP snapshot for one reason or another,
> the more bandwidth is demanded, which in turn increases congestion.
> 
> Some back of the napkin math: with 2500 RRDP clients interested in a 36
> megabyte RRDP snapshot, timeboxed into 15 minutes, the RRDP server would
> need to be able to sustain 1 Gb/sec for 15 minutes to send that data to
> to all RPs. If such a server for whatever reason ends up moving behind a
> 100 Mb/sec link - in the steady state serving mostly just deltas the
> available bandwidth still might seem good enough. However, the moment an
> event like a RRDP Session restart happens, things go awry and perpetual
> congestion could be the result.
> 
> In this instance a (temporary) solution was to disable RRDP [1] (by
> serving a HTTP 204 error for the /notification.xml file), steering all
> RPs to use RSYNC, which seems to perform far better in bandwidth
> constrained environments.
> 
> Yes, while "make the pipes bigger!" or "Just use a CDN!" might seem a
> logical and conclusive answer, I think there is room for optimization in
> the RPKI synchronization protocols. Bigger pipes cost bigger money, CDNs
> usually aren't free either, plus an additional layer of caching in the
> CDN also introduces additional complexity in operations.
> 
> Making RPKI PP/RP synchronization more efficient is advantageous to all
> stakeholders in this ecosystem.
> 
> Why does RSYNC perform better in this type of situation?
> =========================================================
> 
> 1) The Base64 encoding in RRDP imposes a non-negligible overhead of 33%
>   which doesn't exist in RSYNC (where the files are transferred in
> their 'bare' form).
> 2) The XML formatting of the resulting document (elements, attributes &
>   newlines) adds another single digit percentage of overhead compared
>   to transfer of the bare files in bare form.
> 3) RRDP snapshot downloads are hard to resume: if the session is
>   interrupted and then the RRDP serial increases, RPs will try to fetch
>   the new snapshot as per instructions from the RRDP server. RSYNC on
>   the other hand provides a far more fine-grained and seamless
>   resumption mechanism.
> 
> Of course there are well-understood downsides to RSYNC too, what we're
> looking at is a set of trade-offs: mostly centered on the trade-off
> between bandwidth conservation in exchange for cpu cycles. The point of
> this message is not to promote RSYNC but to take a look at the upsides
> of RSYNC compared to RRDP v1.
> 
> In this particular event (measured at a distance of 182 ms latency),
> disabling RRDP caused the time it takes to do a full synchronization
> from 30+ minutes down to 17 seconds, subsequent resyncs now take 6
> seconds, and congestion seems to have cleared.
> 
> Takeaways for RRDP v2
> =====================
> 
> * The publication point operator should be able to signal in the
>  (equivalent of) notification.xml file a set of recommended timers
>  for fetching: it would be super nice if during congestion events the
>  PP operator can inform the clients to please check back at at slower
>  pace (say every 20 or 40 minutes) instead of whatever the RP's timers
>  are. This would give the PP operator some tools to ameliorate the
>  situation when lacking bandwidth or other resources.
> 
> * Replace the concept of distributing signed objects, certs & crls
>  inside XML documents using Base64-encoding, with something more
>  efficient. Perhaps packing all to-be-transferred files in a DER
>  SEQUENCE, optionally wrapped in a self-signed CMS container for easy
>  checksumming. Another feasible approach would be to design an
>  RPKI-application specific compression algorithm : there is a ton of
>  duplicity in for example AuthorityInfoAccess fields or the
>  policyQualifier fields.
> 
> * Rsync offers the community individually addressable URIs for
>  individual files, while RRDP v1 only offers 'the whole thing'.
>  Individually fetchable files are fantastic for debugging, and makes it
>  easier for CA/RP developers to reason and communicate about events.
>  What APNIC is doing here by offering all files on the RSYNC server
>  also as HTTPS download is a step in the right direction:
>  https://rpki.apnic.net/member_repository/A91A0D9C/9E28300657A811E8B4AC0877C4F9AE02/
> 
> * Reconsider compression. I am aware of issues like CVE-2021-43174
>  which prompted some developers to remove support for gzip compression
>  in implementations, yet at the same time there is a lot to be gained
>  from compression:
> 
>    $ ls -lahtrS afrinic.*
>    -rw-r--r--  1 job  wheel   5.8M May 29 12:15 afrinic.tar.lzma
>    -rw-r--r--  1 job  wheel   6.0M May 29 12:15 afrinic.tar.zst
>    -rw-r--r--  1 job  wheel   6.1M May 29 12:15 afrinic.tar.bz2
>    -rw-r--r--  1 job  wheel   6.3M May 29 12:15 afrinic.tar.gz
>    -rw-r--r--  1 job  wheel  13.0M May 20 12:51 afrinic.rrdp.xml.gz
>    -rw-r--r--  1 job  wheel  17.8M May 29 12:15 afrinic.tar
>    -rw-r--r--  1 job  wheel  35.5M May 20 12:51 afrinic.rrdp.xml
> 
>  In the above terminal transcript the 'afrinic.tar' file simply is a
>  tar archive of the validated cache directory specific to AfriNIC. This
>  tar file contains all currently valid objects. The tar file is half
>  the size of the RRDP snapshot, and compressing the tar with lzma,
>  zstd, bz2, or gzip yields a considerable smaller outcome than gzip
>  compressing the RRDP v1 XML snapshot. As noted above, an
>  RPKI-application-aware compression algorithm would yield even smaller
>  snapshots.
> 
> I'm sure other people have learned other lessons over the years working
> with both RSYNC and RRDP: your and their input could perhaps contribute
> to a RRDP v2 requirements document.
> 
> When we have RRDP v2, perhaps either RSYNC or RRDP v1 can be deprecated.
> 
> Kind regards,
> 
> Job
> 
> [1]: AfriNIC disabled RRDP https://status.afrinic.net/notices/caelaru4q1vhqbv2-rrdp-service-availability
> 
> _______________________________________________
> Sidrops mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/sidrops
>