Re: [Sidrops] Publication Point -> RP synchronization in bandwidth constrained environments (note for RRDP v2)

[Tim Bruijnzeels <[e-mail]>]

> On 8 Jun 2023, at 14:55, Ties de Kock <[e-mail]> wrote:
> 
> 
> 
>> On 8 Jun 2023, at 14:35, Claudio Jeker <[e-mail]> wrote:
>> 
>> On Thu, Jun 08, 2023 at 01:45:46PM +0200, Mikhail Puzanov wrote:
>>> Hi Job, all,
>>> 
>>> I think compression is probably the quickest way of mitigating the size problem:
>>> 
>>> Repeating some of your experiments:
>>> 
>>> tmp> ls -lh snapshot.xml*
>>> -rw-r--r--  1 mpuzanov  staff   202M Jun  8 12:44 snapshot.xml
>>> -rw-r--r--  1 mpuzanov  staff    89M Jun  8 12:58 snapshot.xml.bz2
>>> -rw-r--r--  1 mpuzanov  staff    90M Jun  8 13:01 snapshot.xml.gz
>>> -rw-r--r--  1 mpuzanov  staff   124M Jun  8 12:57 snapshot.xml.lz4
>>> -rw-r--r--  1 mpuzanov  staff    83M Jun  8 12:58 snapshot.xml.lzma
>>> 
>>> Even the good ol’ gzip can shrink RIPE NCC’s snapshot more than twofold.
>>> Also compression should take care of the repetitive parts, i.e. 
>>> XML tags, newlines, etc.
>>> 
>>> Extra 2 cents as an RP implementer: rpki-prover fetches every repository 
>>> in a separate OS process with constrained heap, constrained download size 
>>> and a timeout, so it pretty much ignores all CVE-2021-43174-related 
>>> considerations and compression was never disabled in it. If the fetching 
>>> process crashes or runs out of some resource the impact is limited to 
>>> the specific repository. So there is a way to have compression and 
>>> tolerate crashes.
>> 
>> The inherent problem of RRDP is that when a CA is forced to issue a new
>> RRDP session

Technically it's the Publication Server that may perform a session reset,
but you probably meant that.

Publication Servers work hard to avoid session resets. It's really just
the escape hatch in case a server can no longer provide deltas. E.g. they
have had to do a disaster-recovery.

>> (or all RP lose their RRDP state for the CA) every RP system
>> will grab a snapshot.

RRDP state for the repository, again you probably meant that.

Yes, this puts a burden on the RP. See below.

>> So you end up with a thundering herd and I doubt a
>> reduction by half is enough to make that effect go away.
> 
> The 45-70% reduction in traffic and costs is significant.

Definitely helps I would think.

>> The worst case RRDP bandwith requirements are significantly bigger then
>> what is needed in the steady state. This is an inherently bad design.
>> The only fix is to overprovision by a lot.
> 
> The statement on over-provisioning holds for rsync.

Rsync minimises bandwidth, but it requires that the server keeps an open
session with each client and invests CPU and memory for recursive fetches.

This is not a big issue for small repos, but requires significant resources
for big repositories. RP to server numbers are highly asymmetrical. The server
can be DoS'ed trivially.

RRDP is designed to leverage CDNs for delivery of mostly static files. I.e.
the state of the repository (snapshot) and changes between given states (deltas)
are files that will never change. It puts the CPU and download burden on the
RP where it can scale, in parallel.

But, yes there are definitely lessons learned and things that can be improved in
a future version.


> In practice a RP needs to be out of sync for a significant amount of time before
> a fallback to snapshot is needed (> 8h for RIPE NCC repo). In that period, >60%
> of objects in the repository has churned because of manifest+crl rotation.

First off: there is a just an awful lot of churn in the RPKI caused by manifest
and crl size and renewal. We could think about other designs, but it's a much
bigger discussion.

The bandwidth issues that I have seen are caused by a combination of very large
notification files that are served from a slow link (no CDN or other distribution
used).

It will help, a *lot*, if files are kept small. At the time that RRDP (RFC 8182)
was written we lacked the operational experience to make better recommendations
than the 'MUST' that is in there that says that the size of the deltas MUST NOT
exceed the snapshot size.

In practice this can lead to huge notification files. If you try to serve huge
files to every RP over a slow network.. then we get the issues we have seen.

There are a number of things that can be done in the short term, within the
current RRDP standards. We finally (yes, should have done this earlier) wrote
down recommendations in draft-timbru-sidrops-publication-server-bcp-00.

As for lessons learned for a future RRDP. There are a number of things that
I think could be done. To just name a few:

- No new snapshot on every update

This is hard for the server. Writing a full snapshot of multiple MBs for
every change adds up. I think it would be better to have snapshots every
so often and use deltas to get to the current state

- Combine deltas

Let the server pre-calculate combined deltas from serial X to current.
This saves the RPs downloading many files. It saves bandwidth in cases
where files are updated multiple times (e.g. a CA published multiple
ROAs as separate events).

- Binary format

We can have a DER encoded structures (since we do DER anyway) that can
contain all the data. This saves a lot of XML and base64 overhead. I have
not done the analysis but I think that the actual DER objects don't compress
well - so we may not need compression if we do this.

- Pointers for stragglers?

Notification files should be short. This means that RPs that have not
synced for a long time will have to do a full re-sync. What may help
is that there are pointers included to longer notification files for
those that need it.


Tim





> 
> Kind regards,
> Ties
> _______________________________________________
> Sidrops mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/sidrops