Re: [Sidrops] [WG ADOPTION] Adoption call: draft-timbru-sidrops-publication-server-bcp - ENDS 02/08/2024

[Ties de Kock <[e-mail]>]

Hi Tim,

> On 6 Feb 2024, at 17:19, Tim Bruijnzeels <[e-mail]> wrote:
> 
> Hi,
> 
>> On Feb 6, 2024, at 14:03, Job Snijders <job=[e-mail]> wrote:
>> 
>> On Tue, Feb 06, 2024 at 08:57:59AM +0100, Ties de Kock wrote:
>>>> On 5 Feb 2024, at 22:03, Job Snijders <job=[e-mail]> wrote:
>>>> Some suggestions for draft-timbru-sidrops-publication-server-bcp-02:
>>>> 
>>>> - Make a recommendation to Publishers to not regenerate RRDP content for
>>>> every request, nor regenerate previous RRDP deltas when creating a new
>>>> snapshot. This might help implementers of RRDP publication software
>>>> perceive and treat older RRDP content as immutable. See
>>>> https://datatracker.ietf.org/doc/draft-spaghetti-sidrops-rrdp-desynchronization/
>>>> for more details about mutations in previously published RRDP content.
>>> 
>>> I think that it is clear to all that the content of a delta should not
>>> change after being written for the first time. I think specifying the
>>> "how" for that process is too prescriptive. For example, your proposed
>>> method would rule out symlink pivoting for RRDP while you still need
>>> an atomic mechanism to prevent partial reads of some files - e.g.
>>> notification.xml.
>> 
>> The notification file (by specification) indeed is mutable.
>> 
>> I think it is helpful to point out that the RRDP deltas really only need
>> to be generated once, as some implementers seem to have gotten this
>> wrong in the past.
> 
> To the best of my knowledge no RRDP implementation has intentionally
> regenerated deltas. There have been issues where a server was restored
> from backup to an earlier state and it was unaware of the state changes
> since the backup.

I know of two RRDP implementations that re-generate deltas. However they
re-generate the deltas with identical content. If we add phrasing for this,
let’s make sure that it means that the content needs to be identical. How to
get there is up to implementers?

> So, I would like to include text in this BCP that instructs the Publication
> Server to perform a full RRDP session reset in case they restored from
> backup.


A session reset as a fallback mechanism is a idea that is generally safe I think.
An operator _needs_ to have infrastructure that can survive the switch back from
rsync to RRDP. This session reset would cause identical load and is much easier
to handle if it's a prepared scenario.

>>> In practice, RRDP distribution uses multiple layers of caching
>>> (generating software -> origin servers -> CDN internal caching nodes
>>> -> edge). Effectively all write a copy. All can corrupt it. I think
>>> the hashes in RRDP are the perfect machanism for detecting corruption.
>>> Personally, I have found a memory corruption issue due to those
>>> hashes...
>> 
>> Hah, memtest86+ implemented in RRDP ;-)
>> 
>>>> - Perhaps a note could be added about temporarily disabling RRDP
>>>> being a viable option to conserve bandwidth in (rare) scenarios
>>>> where the Publication Point operator's network is congested. In
>>>> low-bandwidth scenarios rsync performs far better than RRDP (a
>>>> particular race condition in RRDP is avoided, rsync synchronization
>>>> trivially is resumed). Notwithstanding that both PP and RP side of
>>>> the house would do well to implement bandwidth conservation
>>>> strategies such as implementation of HTTP content encoding
>>>> compression. I do think it is helpful to remind interested readers
>>>> that ultimately the objective is to /somehow/ serve RPKI data (be it
>>>> via RRDP or RSYNC) in a timely fashion.
>>> 
>>> That is a recommendation that comes with big caveats. While the
>>> behaviour of RRDP degrades badly (we describe the negative feedback
>>> loop in the document), in a steady-state situation, rsync uses
>>> significantly more bandwidth (and IO).
>> 
>> Right, which is why it might be counter-intuitive for Publication Point
>> operators to realize that temporarily disabling RRDP can help when
>> bandwidth is scarce, and as such perhaps worth documenting.
>> 
>>> I wonder what the success rate of rsync clients is during fallback in
>>> a bandwidth-limited scenario. More data on this would help make a
>>> recommendation.  The fixed cost of transferring file names is high
>>> (e.g. 640KB for rpki.afrinic.net, 263KB+1.6MB for rpki.apnic.net,
>>> 5.2MB for rpki.arin.net, 1.2MB for rpki.lacnic.net, 5.5MB for
>>> rpki.ripe.net) Based on that I would expect many rsync clients to hit
>>> timeouts as well.
>> 
>> The situation I had in mind was an example from last year, when for one
>> of the regional internet registries all my alerts cleared within 2 hours
>> after RRDP was disabled.
>> 
>> You are right to point out that the RRDP notification file usually is
>> smaller than the rsync filename transfer list, but in turn, the rsync
>> transfer list is way smaller than the RRDP snapshot.
> 
> I am in two minds about this.
> 
> I think the BCP should not recommend disabling rsync, it should
> recommend that enough bandwidth is available and/or a CDN is
> used.
> 
> The problem happens when this cannot be done. In that case RRDP
> degrades badly, as described in the document, because all RPs
> fall back to full snapshots. This makes the bandwidth load worse.

Another factor that would be good to know: How often do RPs fetch from rsync?
If the routinator interval for RRDP is 10 minutes, but 60 minutes for rsync,
that would make a material difference.

Kind regards,
Ties
> 
> In the case of your example there were higher layer (non-technical)
> reasons why the bandwidth capacity could not be increased and
> a CDN could not be used at the time.
> 
> In this specific case disabling RRDP helped because even though
> there could still be bandwidth issues affecting RPs this allowed
> enough of them to perform a sync so that subsequent data usage
> was reduced.
> 
> So, yes disabling RRDP helped here, but no, I don’t think it should
> be recommended best practice. I want to think about wording that
> captures this...
> 
> Kind regards,
> Tim
> 
>> 
>> Kind regards,
>> 
>> Job
>> 
>> _______________________________________________
>> Sidrops mailing list
>> [e-mail] <mailto:[e-mail]>
>> https://www.ietf.org/mailman/listinfo/sidrops