Re: [Sidrops] [WG ADOPTION] Adoption call: draft-timbru-sidrops-publication-server-bcp - ENDS 02/08/2024

[Ties de Kock <[e-mail]>]

Hi Job,

Thanks for your feedback. I will comment on two because I think they deserve
more discussion in public.

> On 5 Feb 2024, at 22:03, Job Snijders <job=[e-mail]> wrote:
> 
> Dear all,
> 
> On Fri, Jan 26, 2024 at 02:55:55PM -0500, Russ Housley wrote:
>> <no_hats>
>> 
>> I this the WG should adopt this document.
> 
> I am of the same opinion.
> 
> In a previous message on this topic, Di Ma seemed to suggest that it
> would be mighty helpful to have a guide capturing operational lessons
> learned in recent years for the publication side of the house, and I
> concur. Let's blamelessly look back at all 'RPKI incidents' in the last
> few years and see if there were lessons learned to distribute via this
> proposed BCP.
> 
> In some sections the document repeats what's already specified in
> Standards Tracks RFCs, but I don't necessarily see that as a bad: the
> proposed BCP is well positioned to elaborate on what might happen if the
> suggestions in original specifications are not followed.
> 
> Some suggestions for draft-timbru-sidrops-publication-server-bcp-02:
> 
> - Make a recommendation to Publishers to not regenerate RRDP content for
>  every request, nor regenerate previous RRDP deltas when creating a new
>  snapshot. This might help implementers of RRDP publication software
>  perceive and treat older RRDP content as immutable. See
>  https://datatracker.ietf.org/doc/draft-spaghetti-sidrops-rrdp-desynchronization/
>  for more details about mutations in previously published RRDP content.

I think that it is clear to all that the content of a delta should not change
after being written for the first time. I think specifying the "how" for that
process is too prescriptive. For example, your proposed method would rule out
symlink pivoting for RRDP while you still need an atomic mechanism to prevent
partial reads of some files - e.g. notification.xml.

In practice, RRDP distribution uses multiple layers of caching (generating
software -> origin servers -> CDN internal caching nodes -> edge). Effectively
all write a copy. All can corrupt it. I think the hashes in RRDP are the
perfect machanism for detecting corruption. Personally, I have found a memory
corruption issue due to those hashes...

> 
> - Section 6 could perhaps (in addition to the phrase 'consistent')
>  describe the desired outcome using the phrase 'atomic'. I think within
>  our community we often describe the cause of 'failed fetches' as the
>  repository being updated in a non-atomic fashion. 
> 
> - The draft might benefit from a "Recommended Reading" section (in
>  addition to the Glossary) to help the reader understand the context.
> 
> - The proposed BCP could perhaps include a section on "Monitoring
>  Publication Points", because I am not sure all stakeholders are aware
>  that it's massively beneficial to deploy multiple instances of
>  different RPKI Cache implementations at a given publication point, and
>  have monitoring software compare various entrypoint permutations:
>    - rsync only
>    - rrdp first, rsync second
>    - rrdp snapshot only
>    - rrdp only (following deltas)
>  The above of course for multiple distinct implementations and versions
>  of implementations.
> 
> - Continuing in context of the previous suggestion - especially at the
>  RIR-level - publication point operators should strive to support all
>  commonly deployed validator implementations (within reason) by making
>  those part of smoketests before deployment. No big website got big by
>  *only* testing with Internet Explorer. I imagine Publication Point
>  operators want to be compatible with a wide range of potential RP
>  implementations, so testing with a wide range is helpful to help find
>  the lowest common denominator: if one RP implementation is stricter
>  than other RP implementations, the Publication Point adhering to the
>  higher standard helps all deployed clients.
> 
> - Perhaps also a section about common hosting problems: if you offer
>  your publication point over IPv6, make sure that port 443 + 873 are
>  open on IPv6. Monitor the validity of TLS HTTPS certificates (for both
>  address families).


> - Perhaps a note could be added about temporarily disabling RRDP being a
>  viable option to conserve bandwidth in (rare) scenarios where the
>  Publication Point operator's network is congested. In low-bandwidth
>  scenarios rsync performs far better than RRDP (a particular race
>  condition in RRDP is avoided, rsync synchronization trivially is
>  resumed). Notwithstanding that both PP and RP side of the house would
>  do well to implement bandwidth conservation strategies such as
>  implementation of HTTP content encoding compression. I do think it is
>  helpful to remind interested readers that ultimately the objective is
>  to /somehow/ serve RPKI data (be it via RRDP or RSYNC) in a timely
>  fashion.

That is a recommendation that comes with big caveats. While the behaviour of
RRDP degrades badly (we describe the negative feedback loop in the document),
in a steady-state situation, rsync uses significantly more bandwidth (and IO).

I wonder what the success rate of rsync clients is during fallback in a
bandwidth-limited scenario. More data on this would help make a recommendation.
The fixed cost of transferring file names is high (e.g. 640KB for
rpki.afrinic.net, 263KB+1.6MB for rpki.apnic.net, 5.2MB for rpki.arin.net,
1.2MB for rpki.lacnic.net, 5.5MB for rpki.ripe.net) Based on that I would
expect many rsync clients to hit timeouts as well.

Kind regards,
Ties

> 
> Tiny tiny nits:
> 
> - s/Certificate Authorities/Certification Authorities/
> - s/next update time/nextUpdate time/
> - s/Unique Hostname/Distinct Hostname/I

Thanks! Will incorporate this into the main branch.