Re: [Sidrops] Publication Point -> RP synchronization in bandwidth constrained environments (note for RRDP v2)

[Lukas Tribus <[e-mail]>]

On Tue, 30 May 2023 at 17:39, Job Snijders
<job=[e-mail]> wrote:
>
> Dear all,
>
> I wanted to share a note that if work ever starts on RRDP v2, that
> design team should consider conservation of bandwidth a priority.
>
> Recently one of the RIRs ran into a network congestion issue noticable
> on intercontinental transfers: their 36 megabyte RRDP snapshot was being
> served at a rate of ~ 25 kilobytes per second, and worse the TCP
> connections often timed out.
>
> As some RP implementations expect a minimum delivery rate (for example
> 100 kB/sec), some implementations allocate a timebox, and all
> implementations will switch to RSYNC if RRDP is interrupted for whatever
> reason, (causing a RRDP snapshot fetch to be queued up for the future).
> The more RPs try to fetch the RRDP snapshot for one reason or another,
> the more bandwidth is demanded, which in turn increases congestion.
>
> Some back of the napkin math: with 2500 RRDP clients interested in a 36
> megabyte RRDP snapshot, timeboxed into 15 minutes, the RRDP server would
> need to be able to sustain 1 Gb/sec for 15 minutes to send that data to
> to all RPs. If such a server for whatever reason ends up moving behind a
> 100 Mb/sec link - in the steady state serving mostly just deltas the
> available bandwidth still might seem good enough. However, the moment an
> event like a RRDP Session restart happens, things go awry and perpetual
> congestion could be the result.
>
> In this instance a (temporary) solution was to disable RRDP [1] (by
> serving a HTTP 204 error for the /notification.xml file), steering all
> RPs to use RSYNC, which seems to perform far better in bandwidth
> constrained environments.
>
> Yes, while "make the pipes bigger!" or "Just use a CDN!" might seem a
> logical and conclusive answer, I think there is room for optimization in
> the RPKI synchronization protocols. Bigger pipes cost bigger money, CDNs
> usually aren't free either, plus an additional layer of caching in the
> CDN also introduces additional complexity in operations.

I agree with protocol optimizations.

I would expect PP operators (or at the very least RIR based PPs) to do
a little more than just consider 2500 well behaving/ RFC compliant
RRDP clients though. An attacker shouldn't be able to knock out a PP
of a RIR in a few minutes, with a few dollars of budget, be it with
volumetric or application level attacks.

Now if organization XYZ decides it needs to host its own PP on a
single 1x Gbit/s or 100 mbit/s pipe, that is their choice of course.
But at RIR PP level this would be rather problematic.

I'm sure Afrinic is on top of this now, I'm just saying, we should
probably be careful about creating the perception that it's a good
idea to host a PP on infrastructure that is not up to the task.



Lukas