Re: [Sidrops] [WG ADOPTION] Adoption call: draft-timbru-sidrops-publication-server-bcp - ENDS 02/08/2024

[Job Snijders <[e-mail]>]

On Tue, Feb 06, 2024 at 08:57:59AM +0100, Ties de Kock wrote:
> > On 5 Feb 2024, at 22:03, Job Snijders <job=[e-mail]> wrote:
> > Some suggestions for draft-timbru-sidrops-publication-server-bcp-02:
> > 
> > - Make a recommendation to Publishers to not regenerate RRDP content for
> >  every request, nor regenerate previous RRDP deltas when creating a new
> >  snapshot. This might help implementers of RRDP publication software
> >  perceive and treat older RRDP content as immutable. See
> >  https://datatracker.ietf.org/doc/draft-spaghetti-sidrops-rrdp-desynchronization/
> >  for more details about mutations in previously published RRDP content.
> 
> I think that it is clear to all that the content of a delta should not
> change after being written for the first time. I think specifying the
> "how" for that process is too prescriptive. For example, your proposed
> method would rule out symlink pivoting for RRDP while you still need
> an atomic mechanism to prevent partial reads of some files - e.g.
> notification.xml.

The notification file (by specification) indeed is mutable.

I think it is helpful to point out that the RRDP deltas really only need
to be generated once, as some implementers seem to have gotten this
wrong in the past.

> In practice, RRDP distribution uses multiple layers of caching
> (generating software -> origin servers -> CDN internal caching nodes
> -> edge). Effectively all write a copy. All can corrupt it. I think
> the hashes in RRDP are the perfect machanism for detecting corruption.
> Personally, I have found a memory corruption issue due to those
> hashes...

Hah, memtest86+ implemented in RRDP ;-)

> > - Perhaps a note could be added about temporarily disabling RRDP
> > being a viable option to conserve bandwidth in (rare) scenarios
> > where the Publication Point operator's network is congested. In
> > low-bandwidth scenarios rsync performs far better than RRDP (a
> > particular race condition in RRDP is avoided, rsync synchronization
> > trivially is resumed). Notwithstanding that both PP and RP side of
> > the house would do well to implement bandwidth conservation
> > strategies such as implementation of HTTP content encoding
> > compression. I do think it is helpful to remind interested readers
> > that ultimately the objective is to /somehow/ serve RPKI data (be it
> > via RRDP or RSYNC) in a timely fashion.
> 
> That is a recommendation that comes with big caveats. While the
> behaviour of RRDP degrades badly (we describe the negative feedback
> loop in the document), in a steady-state situation, rsync uses
> significantly more bandwidth (and IO).

Right, which is why it might be counter-intuitive for Publication Point
operators to realize that temporarily disabling RRDP can help when
bandwidth is scarce, and as such perhaps worth documenting.

> I wonder what the success rate of rsync clients is during fallback in
> a bandwidth-limited scenario. More data on this would help make a
> recommendation.  The fixed cost of transferring file names is high
> (e.g. 640KB for rpki.afrinic.net, 263KB+1.6MB for rpki.apnic.net,
> 5.2MB for rpki.arin.net, 1.2MB for rpki.lacnic.net, 5.5MB for
> rpki.ripe.net) Based on that I would expect many rsync clients to hit
> timeouts as well.

The situation I had in mind was an example from last year, when for one
of the regional internet registries all my alerts cleared within 2 hours
after RRDP was disabled.

You are right to point out that the RRDP notification file usually is
smaller than the rsync filename transfer list, but in turn, the rsync
transfer list is way smaller than the RRDP snapshot.

Kind regards,

Job