Re: [Sidrops] [WG ADOPTION] Adoption call: draft-timbru-sidrops-publication-server-bcp - ENDS 02/08/2024

[Tim Bruijnzeels <[e-mail]>]

Hi,

> On Feb 6, 2024, at 14:03, Job Snijders <job=[e-mail]> wrote:
> 
> On Tue, Feb 06, 2024 at 08:57:59AM +0100, Ties de Kock wrote:
>>> On 5 Feb 2024, at 22:03, Job Snijders <job=[e-mail]> wrote:
>>> Some suggestions for draft-timbru-sidrops-publication-server-bcp-02:
>>> 
>>> - Make a recommendation to Publishers to not regenerate RRDP content for
>>> every request, nor regenerate previous RRDP deltas when creating a new
>>> snapshot. This might help implementers of RRDP publication software
>>> perceive and treat older RRDP content as immutable. See
>>> https://datatracker.ietf.org/doc/draft-spaghetti-sidrops-rrdp-desynchronization/
>>> for more details about mutations in previously published RRDP content.
>> 
>> I think that it is clear to all that the content of a delta should not
>> change after being written for the first time. I think specifying the
>> "how" for that process is too prescriptive. For example, your proposed
>> method would rule out symlink pivoting for RRDP while you still need
>> an atomic mechanism to prevent partial reads of some files - e.g.
>> notification.xml.
> 
> The notification file (by specification) indeed is mutable.
> 
> I think it is helpful to point out that the RRDP deltas really only need
> to be generated once, as some implementers seem to have gotten this
> wrong in the past.

To the best of my knowledge no RRDP implementation has intentionally
regenerated deltas. There have been issues where a server was restored
from backup to an earlier state and it was unaware of the state changes
since the backup.

So, I would like to include text in this BCP that instructs the Publication
Server to perform a full RRDP session reset in case they restored from
backup.

>> In practice, RRDP distribution uses multiple layers of caching
>> (generating software -> origin servers -> CDN internal caching nodes
>> -> edge). Effectively all write a copy. All can corrupt it. I think
>> the hashes in RRDP are the perfect machanism for detecting corruption.
>> Personally, I have found a memory corruption issue due to those
>> hashes...
> 
> Hah, memtest86+ implemented in RRDP ;-)
> 
>>> - Perhaps a note could be added about temporarily disabling RRDP
>>> being a viable option to conserve bandwidth in (rare) scenarios
>>> where the Publication Point operator's network is congested. In
>>> low-bandwidth scenarios rsync performs far better than RRDP (a
>>> particular race condition in RRDP is avoided, rsync synchronization
>>> trivially is resumed). Notwithstanding that both PP and RP side of
>>> the house would do well to implement bandwidth conservation
>>> strategies such as implementation of HTTP content encoding
>>> compression. I do think it is helpful to remind interested readers
>>> that ultimately the objective is to /somehow/ serve RPKI data (be it
>>> via RRDP or RSYNC) in a timely fashion.
>> 
>> That is a recommendation that comes with big caveats. While the
>> behaviour of RRDP degrades badly (we describe the negative feedback
>> loop in the document), in a steady-state situation, rsync uses
>> significantly more bandwidth (and IO).
> 
> Right, which is why it might be counter-intuitive for Publication Point
> operators to realize that temporarily disabling RRDP can help when
> bandwidth is scarce, and as such perhaps worth documenting.
> 
>> I wonder what the success rate of rsync clients is during fallback in
>> a bandwidth-limited scenario. More data on this would help make a
>> recommendation.  The fixed cost of transferring file names is high
>> (e.g. 640KB for rpki.afrinic.net, 263KB+1.6MB for rpki.apnic.net,
>> 5.2MB for rpki.arin.net, 1.2MB for rpki.lacnic.net, 5.5MB for
>> rpki.ripe.net) Based on that I would expect many rsync clients to hit
>> timeouts as well.
> 
> The situation I had in mind was an example from last year, when for one
> of the regional internet registries all my alerts cleared within 2 hours
> after RRDP was disabled.
> 
> You are right to point out that the RRDP notification file usually is
> smaller than the rsync filename transfer list, but in turn, the rsync
> transfer list is way smaller than the RRDP snapshot.

I am in two minds about this.

I think the BCP should not recommend disabling rsync, it should
recommend that enough bandwidth is available and/or a CDN is
used.

The problem happens when this cannot be done. In that case RRDP
degrades badly, as described in the document, because all RPs
fall back to full snapshots. This makes the bandwidth load worse.

In the case of your example there were higher layer (non-technical)
reasons why the bandwidth capacity could not be increased and
a CDN could not be used at the time.

In this specific case disabling RRDP helped because even though
there could still be bandwidth issues affecting RPs this allowed
enough of them to perform a sync so that subsequent data usage
was reduced.

So, yes disabling RRDP helped here, but no, I don’t think it should
be recommended best practice. I want to think about wording that
captures this...

Kind regards,
Tim

> 
> Kind regards,
> 
> Job
> 
> _______________________________________________
> Sidrops mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/sidrops
>