Re: [Sidrops] Publication Point -> RP synchronization in bandwidth constrained environments (note for RRDP v2)

Hi Job, all,

I think compression is probably the quickest way of mitigating the size problem:

Repeating some of your experiments:

tmp> ls -lh snapshot.xml*
-rw-r--r--  1 mpuzanov  staff   202M Jun  8 12:44 snapshot.xml
-rw-r--r--  1 mpuzanov  staff    89M Jun  8 12:58 snapshot.xml.bz2
-rw-r--r--  1 mpuzanov  staff    90M Jun  8 13:01 snapshot.xml.gz
-rw-r--r--  1 mpuzanov  staff   124M Jun  8 12:57 snapshot.xml.lz4
-rw-r--r--  1 mpuzanov  staff    83M Jun  8 12:58 snapshot.xml.lzma

Even the good ol’ gzip can shrink RIPE NCC’s snapshot more than twofold.
Also compression should take care of the repetitive parts, i.e. 
XML tags, newlines, etc.

Extra 2 cents as an RP implementer: rpki-prover fetches every repository 
in a separate OS process with constrained heap, constrained download size 
and a timeout, so it pretty much ignores all CVE-2021-43174-related 
considerations and compression was never disabled in it. If the fetching 
process crashes or runs out of some resource the impact is limited to 
the specific repository. So there is a way to have compression and 
tolerate crashes.

--
Misha

> 
> 1) The Base64 encoding in RRDP imposes a non-negligible overhead of 33%
>   which doesn't exist in RSYNC (where the files are transferred in
> their 'bare' form).
> 2) The XML formatting of the resulting document (elements, attributes &
>   newlines) adds another single digit percentage of overhead compared
>   to transfer of the bare files in bare form.
> 3) RRDP snapshot downloads are hard to resume: if the session is
>   interrupted and then the RRDP serial increases, RPs will try to fetch
>   the new snapshot as per instructions from the RRDP server. RSYNC on
>   the other hand provides a far more fine-grained and seamless
>   resumption mechanism.
> 
> Of course there are well-understood downsides to RSYNC too, what we're
> looking at is a set of trade-offs: mostly centered on the trade-off
> between bandwidth conservation in exchange for cpu cycles. The point of
> this message is not to promote RSYNC but to take a look at the upsides
> of RSYNC compared to RRDP v1.
> 
> In this particular event (measured at a distance of 182 ms latency),
> disabling RRDP caused the time it takes to do a full synchronization
> from 30+ minutes down to 17 seconds, subsequent resyncs now take 6
> seconds, and congestion seems to have cleared.
> 
> Takeaways for RRDP v2
> =====================
> 
> * The publication point operator should be able to signal in the
>  (equivalent of) notification.xml file a set of recommended timers
>  for fetching: it would be super nice if during congestion events the
>  PP operator can inform the clients to please check back at at slower
>  pace (say every 20 or 40 minutes) instead of whatever the RP's timers
>  are. This would give the PP operator some tools to ameliorate the
>  situation when lacking bandwidth or other resources.
> 
> * Replace the concept of distributing signed objects, certs & crls
>  inside XML documents using Base64-encoding, with something more
>  efficient. Perhaps packing all to-be-transferred files in a DER
>  SEQUENCE, optionally wrapped in a self-signed CMS container for easy
>  checksumming. Another feasible approach would be to design an
>  RPKI-application specific compression algorithm : there is a ton of
>  duplicity in for example AuthorityInfoAccess fields or the
>  policyQualifier fields.
> 
> * Rsync offers the community individually addressable URIs for
>  individual files, while RRDP v1 only offers 'the whole thing'.
>  Individually fetchable files are fantastic for debugging, and makes it
>  easier for CA/RP developers to reason and communicate about events.
>  What APNIC is doing here by offering all files on the RSYNC server
>  also as HTTPS download is a step in the right direction:
>  https://rpki.apnic.net/member_repository/A91A0D9C/9E28300657A811E8B4AC0877C4F9AE02/
> 
> * Reconsider compression. I am aware of issues like CVE-2021-43174
>  which prompted some developers to remove support for gzip compression
>  in implementations, yet at the same time there is a lot to be gained
>  from compression:
> 
>    $ ls -lahtrS afrinic.*
>    -rw-r--r--  1 job  wheel   5.8M May 29 12:15 afrinic.tar.lzma
>    -rw-r--r--  1 job  wheel   6.0M May 29 12:15 afrinic.tar.zst
>    -rw-r--r--  1 job  wheel   6.1M May 29 12:15 afrinic.tar.bz2
>    -rw-r--r--  1 job  wheel   6.3M May 29 12:15 afrinic.tar.gz
>    -rw-r--r--  1 job  wheel  13.0M May 20 12:51 afrinic.rrdp.xml.gz
>    -rw-r--r--  1 job  wheel  17.8M May 29 12:15 afrinic.tar
>    -rw-r--r--  1 job  wheel  35.5M May 20 12:51 afrinic.rrdp.xml
> 
>  In the above terminal transcript the 'afrinic.tar' file simply is a
>  tar archive of the validated cache directory specific to AfriNIC. This
>  tar file contains all currently valid objects. The tar file is half
>  the size of the RRDP snapshot, and compressing the tar with lzma,
>  zstd, bz2, or gzip yields a considerable smaller outcome than gzip
>  compressing the RRDP v1 XML snapshot. As noted above, an
>  RPKI-application-aware compression algorithm would yield even smaller
>  snapshots.
> 
> I'm sure other people have learned other lessons over the years working
> with both RSYNC and RRDP: your and their input could perhaps contribute
> to a RRDP v2 requirements document.
> 
> When we have RRDP v2, perhaps either RSYNC or RRDP v1 can be deprecated.
> 
> Kind regards,
> 
> Job
> 
> [1]: AfriNIC disabled RRDP https://status.afrinic.net/notices/caelaru4q1vhqbv2-rrdp-service-availability
> 
> _______________________________________________
> Sidrops mailing list
> Sidrops@ietf.org
> https://www.ietf.org/mailman/listinfo/sidrops