Re: [Sidrops] [WG ADOPTION] Adoption call: draft-timbru-sidrops-publication-server-bcp - ENDS 02/08/2024

[Tim Bruijnzeels <[e-mail]>]

Hi Job, all,

> On Feb 6, 2024, at 18:03, Job Snijders <[e-mail]> wrote:
> 
> On Tue, Feb 06, 2024 at 05:19:11PM +0100, Tim Bruijnzeels wrote:
>>> The notification file (by specification) indeed is mutable.
>>> 
>>> I think it is helpful to point out that the RRDP deltas really only
>>> need to be generated once, as some implementers seem to have gotten
>>> this wrong in the past.
>> 
>> To the best of my knowledge no RRDP implementation has intentionally
>> regenerated deltas.
> 
>> There have been issues where a server was restored from backup to an
>> earlier state and it was unaware of the state changes since the
>> backup.
>> 
>> So, I would like to include text in this BCP that instructs the
>> Publication Server to perform a full RRDP session reset in case they
>> restored from backup.
> 
> Yes, the 'reset session' after backup restore is a good point.
> 
>>> The situation I had in mind was an example from last year, when for
>>> one of the regional internet registries all my alerts cleared within
>>> 2 hours after RRDP was disabled.
>>> 
>>> You are right to point out that the RRDP notification file usually
>>> is smaller than the rsync filename transfer list, but in turn, the
>>> rsync transfer list is way smaller than the RRDP snapshot.
>> 
>> I am in two minds about this.
>> 
>> I think the BCP should not recommend disabling RRDP, it should
>> recommend that enough bandwidth is available and/or a CDN is used.
>> 
>> The problem happens when this cannot be done. In that case RRDP
>> degrades badly, as described in the document, because all RPs fall
>> back to full snapshots. This makes the bandwidth load worse.
>> 
>> In the case of your example there were higher layer (non-technical)
>> reasons why the bandwidth capacity could not be increased and a CDN
>> could not be used at the time.
>> 
>> In this specific case disabling RRDP helped because even though there
>> could still be bandwidth issues affecting RPs this allowed enough of
>> them to perform a sync so that subsequent data usage was reduced.
>> 
>> So, yes disabling RRDP helped here, but no, I don’t think it should be
>> recommended best practice. I want to think about wording that captures
>> this...
> 
> Or phrased differently: the best practise is to ensure there is
> sufficient bandwidth available, while disabling RRDP is a dirty hack :-)
> 
> Perhaps a some kind of guidance can be included as to what 'sufficient
> bandwidth' is? Here is a starter:
> 
>  "Size of snapshot" times "Number of deployed RP instances" should
>  comfortably fit in 15 minute delivery window.
> 
> (I picked 15 minutes, because rpki-client by default timeboxes
> synchronizing individual repositories into 15 minutes.)

I think Routinator syncs every 10 mins *after* the last run completed, so
it should be close to this as well.

> So, for an RRDP server serving a 207 megabyte snapshot to 3000 RPs, the
> operator would need *at least* (207*8*3000)/900 = 5520 megabit/sec after
> a session reset. To make it fit comfortably, double or triple this number.

That approach sounds about right. And indeed, session resets are where
the full re-syncs happen in the absence of issues. So, servers should plan
their capacity to be able to handle this.

Provided that the server is confident about this, they may wish to initiate
a session reset from time to time to ensure that they have operational
experience.

If the resulting number is sufficiently high, using a CDN is highly
recommended.

But note that this is a worse case “thundering herd” issue, and
steady-state data usage will be lower.

> Given the affiliations of the authors, I'm sure that group can do a
> better job and speak from experience how capacity planning is done. :-)

We can dig up some numbers for our own session resets, but it’s a point
in space and time data point only. The numbers will change over time as
the snapshot grows, more RPs are deployed and/or their update frequency
increases. So, this is something that servers should monitor for themselves.

For small (e.g. single CA) repositories this may not be an issue at all as their
snapshot is tiny, and they are able to serve RPs on a “normal” link. We talk
about this a bit in the document, but we "RECOMMEND that CAs use a
publication service provided by their RIR, NIR or other parent as much as
possible, because availability issues with such [single CA] repositories are
frequent, and can negatively impact Relying Party software."

The way this degrades in the absence of a CDN can vary. If the capacity is
way too low, then RPs simply won’t be able to get the snapshots. Or they
get them once, and then fallback to rsync on a later sync, and then try a
full RRDP snapshot again, etc. ad nauseam. But this depends on how quickly
RPs  decide to fall back to rsync and retry (full) RRDP. If the fallback and
retries are spread out over time, then it may still recover.

Of course, servers should plan capacity such that this does not happen.
But, less aggressive fall-back and retries by RPs can help. Rate limiting
HTTP headers were not considered when RRDP was discussed, but
potentially could perhaps help a bit. The thinking at the time was that data
would not be an issue as long as CDNs were used.

Tim



> 
> Kind regards,
> 
> Job
>