Re: [Sidrops] RRDP / RSYNC / Other?

[Tim Bruijnzeels <[e-mail]>]

Dear all,

> On Jan 12, 2024, at 13:03, Job Snijders <job=[e-mail]> wrote:
> 
> Dear all,
> 
> On Thu, Jan 11, 2024 at 10:52:55PM -0500, Christopher Morrow wrote:
>> The 'prefer rrdp' draft has been sitting(expired)  for a while, I had
>> thought when the document started life there was a set of good reasons
>> for this plan. I think I still want to prefer 'not rsync' on my RP
>> deployment :)
> 

Chris, to be clear: the prefer-rrdp draft does not attempt to remove rsync. It
formulated a plan to prefer RRDP. I think it is largely overtaken by reality,
except for a number of  suggestions that it’s making to update various
documents to ensure that RRDP is no longer optional and must get the
same 24/7 attention as rsync.

I think those suggestions are still useful even if deprecating rsync (as the
draft was called in an earlier stage) is a separate discussion.

> =
>> I believe that a longer term goal for rrdp (and the draft in question)
>> was to eventually decommission/deprecate the RSYNC code paths, as
>> well.
> 
> Well, that is not my goal. Before we even consider deprecation of rsync,
> there first needs to be a viable alternative to both RSYNC and RRDP v1.
> 
> As it stands - RRDP has numerous inefficiencies, implementation
> inconsistencies, curious pitfalls, unforeseen operational failure modes,
> and performs extremely poor in low bandwidth situations. But of course,
> rsync is not without its own issues and complications. Fortunately, not
> a lot of the failure modes are shared between the transport protocols,
> meaning that one is a good backup for the other.
> 
> I think some people in this working group underappreciate the benefits
> of rsync as a (backup) transport, especially in the face of instability
> in the RRDP transport. As long as people are stuck in just mindlessly
> repeating "rsync bad" and refuse to see and articulate the benefits that
> rsync offers, it'll be hard to copy those benefits over to a next gen
> transport protocol.

I agree that we also have seen issues with RRDP. There are lessons 
learned on different levels. Just like we have also learned issues with
rsync. With regards to current operational challenges I believe that
draft-timbru-sidrops-publication-server-bcp-01 that I and others have
been working on is a good start - not just for RRDP (bandwidth, load
balancing), but also for rsync (e.g. the symlinking and mtime fixing). We
will renew it soon and ask for working group adoption.

> 
>> In light of the original reasonings for RRDP development:
>>  * standard(ized) transport protocol (http)
>>  * scalability and cacheability of the content
>>  * easier to inspect/debug service(s)
>>  (there may be others,  don't think right now its important that there are)
> 
> I agree HTTP is a standardized protocol and I hope ultimately something
> HTTP-based is where we'll end up. But, the other arguments can be
> challenged: RRDP 'scales so well' that we've seen RPs become an
> unrecoverable denial-of-service against the publication point. In
> practise rsync is easier to debug and inspect with out-of-the-box
> tooling, whereas RRDP requires a quite specialized knowledge and tooling
> to meaningfully debug.

I appreciate the direct access to objects for debugging that rsync offers. I
would prefer some future where we can have direct access over HTTP (and
use a CDN).

> 
>> and the original arguments against RSYNC as the protocol for long term use:
>>  * how could this possibly scale!
>>  * i can ship random bits  to all RPs and they can't NOT accept them
>>  * wow is this a bear to deal with in a programmatic manner!
>>  (there may be others, not important right now what they are)
>> 
>> Is it time that we fresh-slate thought about this problem?
>> I believe we CAN swap in 'anything' that has a URL type naming/pointing form
>>  (perhaps we would need to define new url protocols, but...)
>> I think if we had a solid option, with running code we could talk
>> about removing rsync, preferring rrdp and testing the 'new thing'...
>> or some form of that ordering.
> 
> When you actually remove rsync, RRDP no longer becomes a matter of
> 'preference'. :-)
> 
>> Should we take up a discussion and work to investigate a RRDP
>> replacement?

I still think that first step is to prefer RRDP and make it mandatory. In practice
this is the reality today, but it’s not reflected in standards where RRDP is still
optional and 24/7 commitments are not clearly formulated.

The main design goal of RRDP was to allow the use of CDNs and ensure that
servers can create cacheable data once, rather than having the server needing
to invest memory and CPU on every connection. There are many RPs to
repositories. This is a skewed ratio. Sadly, *recursive* rsync servers with
a lot of data are trivial to DoS because of that.

RRDP makes the tradeoff to use more bandwidth in order to reduce the direct
impact on a back-end server in terms of CPU and memory needs to maintain
sessions. Sadly, this means that if the notification file is too large, and the
bandwidth too low (no, do not use a single 100 Mbit node), this results in
issues.

Where bandwidth is not an issue, and the load-balancing/timing of notification
file updates is addressed, I do believe that RRDP offers a big improvement today.
It’s not perfect, but it allows RPs to be more aggressive in their fetching.

> The timing is not optimal from my perspective, I don't mind the status
> quo. I think we should first finish:
> 
> * ASPA signed object: draft-ietf-sidrops-aspa-profile
> * ASPA-based BGP message verification: draft-ietf-sidrops-aspa-verification
> * ASPA RTR: draft-ietf-sidrops-8210bis
> * draft-ietf-sidrops-signed-tal (has multiple implementations, authors
>  requested WGLC multiple times now...)
> * draft-ietf-sidrops-cms-signing-time
> * draft-ietf-sidrops-rpki-prefixlist
> * And there also is a recent interest to revisit the validation
>  algorithm, this will all take up a fair chunk of working group
>  bandwidth.
> 
> In short - deprecating rsync before having a viable alternative to RRDP
> v1 is horse behind cart. A deprecation of rsync won't solve immediate
> operational issues, but instead will add problems. The community now
> also is aware of various optimizations that can be done to make
> RRDP-to-RSYNC switchovers far less resource intensive, so scalability
> concerns are not as strong as they once were.

Job, as you are saying (I think), I agree the we should only consider deprecating
rsync altogether until we have an improvement over RRDP, or at least ensure
that fallbacks are possible somehow. I believe that a discussion about a next
gen protocol is useful and can occur in parelel to other work. There is no reason
why e.g. the ASPA work should be stalled because of such a discussion. 

For what it's worth, some initial thoughts regarding an RRDP++/bis/replacement:

It would be good if a future iteration can reduce the bandwidth. Other signaling
than HTTP polling of potentially large files may be useful to consider here.

There is no easy direct access to files in RRDP. Furthermore, URIs can be
anything - there is no consistent scoping. Writing a full snapshot on every change
is heavy (I agree with draft-ietf-grow-nrtm-v4 on this one). The encoding is
convoluted. We may want to think about mirroring.

Etc. This is not complete. Let’s first establish if we want to have the discussion
now, then we can talk more details..

Kind regards,
Tim


> 
> Kind regards,
> 
> Job
> 
> _______________________________________________
> Sidrops mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/sidrops
>