Re: [Sidrops] RRDP / RSYNC / Other?

[Christopher Morrow <[e-mail]>]

(sorry this is a bit of an omnibus reply, to tim/job/ties)

On Fri, Jan 12, 2024 at 9:58 AM Tim Bruijnzeels <[e-mail]> wrote:
>
> Dear all,
>
> > On Jan 12, 2024, at 13:03, Job Snijders <job=[e-mail]> wrote:
> >
> > Dear all,
> >
> > On Thu, Jan 11, 2024 at 10:52:55PM -0500, Christopher Morrow wrote:
> >> The 'prefer rrdp' draft has been sitting(expired)  for a while, I had
> >> thought when the document started life there was a set of good reasons
> >> for this plan. I think I still want to prefer 'not rsync' on my RP
> >> deployment :)
> >
>
> Chris, to be clear: the prefer-rrdp draft does not attempt to remove rsync. It
> formulated a plan to prefer RRDP. I think it is largely overtaken by reality,

sure, understood... Maybe I misunderstood that the longer term plan
(perhaps not articulated in the draft)
was to deprecate rsync. Before deprecation we'd certainly want to
prefer 'other' (the draft's point).

> except for a number of  suggestions that it’s making to update various
> documents to ensure that RRDP is no longer optional and must get the
> same 24/7 attention as rsync.
>
> I think those suggestions are still useful even if deprecating rsync (as the
> draft was called in an earlier stage) is a separate discussion.

fair!

>
> > =
> >> I believe that a longer term goal for rrdp (and the draft in question)
> >> was to eventually decommission/deprecate the RSYNC code paths, as
> >> well.
> >
> > Well, that is not my goal. Before we even consider deprecation of rsync,
> > there first needs to be a viable alternative to both RSYNC and RRDP v1.
> >

Sure, I thought I was making a sort of clear point that:
  "we have a draft that says: 'prefer RRDP'"

I was attempting to also make a point that probably if a goal is:
  "prefer rrdp, and then later deprecate rsync"

we should answer a couple other questions:
  1) do we want to maintain 2 ways to do this thing
      (move bits around relative to rpki)?

  2) do we want a better answer than rrdp?
     (are there problems/issues with RRDP that should be addressed?
can we do that in the exisitng binaries/protocol?)

  3) do we want a better answer than rsync?

I don't know the correct answer(s) to the above, but it sure seems to me like:
  "one way seems simpler to support in code and on the systems"
  "there have been issues raise with both rsync and rrdp"

> > As it stands - RRDP has numerous inefficiencies, implementation
> > inconsistencies, curious pitfalls, unforeseen operational failure modes,
> > and performs extremely poor in low bandwidth situations. But of course,

Some of this could be the implementation(s) in particular and not the
underlying transport,
by this I mean either the clients or the servers may be behaving
poorly in certain enviroments/conditions.

> > rsync is not without its own issues and complications. Fortunately, not
> > a lot of the failure modes are shared between the transport protocols,
> > meaning that one is a good backup for the other.
> >
> > I think some people in this working group underappreciate the benefits
> > of rsync as a (backup) transport, especially in the face of instability
> > in the RRDP transport. As long as people are stuck in just mindlessly
> > repeating "rsync bad" and refuse to see and articulate the benefits that
> > rsync offers, it'll be hard to copy those benefits over to a next gen
> > transport protocol.
>

This could be, again I don't know the answer here i was attempting to raise
some questions and ask if the questions were headed in a good direction AND
if the WG was interested in pursuing the conversation.

> I agree that we also have seen issues with RRDP. There are lessons
> learned on different levels. Just like we have also learned issues with
> rsync. With regards to current operational challenges I believe that
> draft-timbru-sidrops-publication-server-bcp-01 that I and others have
> been working on is a good start - not just for RRDP (bandwidth, load
> balancing), but also for rsync (e.g. the symlinking and mtime fixing). We
> will renew it soon and ask for working group adoption.

ok, sounds good.

> >> In light of the original reasonings for RRDP development:
> >>  * standard(ized) transport protocol (http)
> >>  * scalability and cacheability of the content
> >>  * easier to inspect/debug service(s)
> >>  (there may be others,  don't think right now its important that there are)
> >
> > I agree HTTP is a standardized protocol and I hope ultimately something
> > HTTP-based is where we'll end up. But, the other arguments can be
> > challenged: RRDP 'scales so well' that we've seen RPs become an
> > unrecoverable denial-of-service against the publication point. In
> > practise rsync is easier to debug and inspect with out-of-the-box
> > tooling, whereas RRDP requires a quite specialized knowledge and tooling
> > to meaningfully debug.

This seems (the last couple sentences really) to be a side
conversation about how
the tooling (current or future) used by the RP and publication point
could use a bunch
more thought about their use in normal operations and in debugging scenarios.

The tooling built around inspecting the collected repository data has
been a boon to
debugging of the system at large, I think. More of that work, and
assisting that work
with better collection protocol(s) would certainly be beneficial to
the ops community.

> I appreciate the direct access to objects for debugging that rsync offers. I
> would prefer some future where we can have direct access over HTTP (and
> use a CDN).
>
> >
> >> and the original arguments against RSYNC as the protocol for long term use:
> >>  * how could this possibly scale!
> >>  * i can ship random bits  to all RPs and they can't NOT accept them
> >>  * wow is this a bear to deal with in a programmatic manner!
> >>  (there may be others, not important right now what they are)
> >>
> >> Is it time that we fresh-slate thought about this problem?
> >> I believe we CAN swap in 'anything' that has a URL type naming/pointing form
> >>  (perhaps we would need to define new url protocols, but...)
> >> I think if we had a solid option, with running code we could talk
> >> about removing rsync, preferring rrdp and testing the 'new thing'...
> >> or some form of that ordering.
> >
> > When you actually remove rsync, RRDP no longer becomes a matter of
> > 'preference'. :-)

yes, clearly a plan isn't: "prefer rrdp, kill rsync, now plan on a
replacement rrdp!" :)
a reasonable plan (it seemed to me) was: "discuss/define some requirements for
the system (client/server), and operations of the system."

that plan may end up deprecating rsync, or rrdp or both over some time period.

> >> Should we take up a discussion and work to investigate a RRDP
> >> replacement?
>
> I still think that first step is to prefer RRDP and make it mandatory. In practice
> this is the reality today, but it’s not reflected in standards where RRDP is still
> optional and 24/7 commitments are not clearly formulated.
>

I, for selfish reasons, really want rrdp to be supported across the
board, so yes this
seems ok to me. it'd still be good to get consensus on this though :)
and disregard
my selfish reason(s).

> The main design goal of RRDP was to allow the use of CDNs and ensure that
> servers can create cacheable data once, rather than having the server needing
> to invest memory and CPU on every connection. There are many RPs to
> repositories. This is a skewed ratio. Sadly, *recursive* rsync servers with
> a lot of data are trivial to DoS because of that.

this sounds like 2 (at least) requirements for the server and client
infra, almost certainly
some / one of those was included in the original rrdp
docs/requirements/planning.

This also highlights a possible reason to deprefer rsync.

> RRDP makes the tradeoff to use more bandwidth in order to reduce the direct
> impact on a back-end server in terms of CPU and memory needs to maintain
> sessions. Sadly, this means that if the notification file is too large, and the
> bandwidth too low (no, do not use a single 100 Mbit node), this results in
> issues.

sure, I would bet this is also addressable in the server config...
  'please do not accept enough requests that let you kill yourself'
but.. sure, also maybe this is a failure mode we learned and can attempt to
address in a replacement protocol suite?

> Where bandwidth is not an issue, and the load-balancing/timing of notification
> file updates is addressed, I do believe that RRDP offers a big improvement today.
> It’s not perfect, but it allows RPs to be more aggressive in their fetching.
>
> > The timing is not optimal from my perspective, I don't mind the status
> > quo. I think we should first finish:
> >
> > * ASPA signed object: draft-ietf-sidrops-aspa-profile
> > * ASPA-based BGP message verification: draft-ietf-sidrops-aspa-verification
> > * ASPA RTR: draft-ietf-sidrops-8210bis
> > * draft-ietf-sidrops-signed-tal (has multiple implementations, authors
> >  requested WGLC multiple times now...)
> > * draft-ietf-sidrops-cms-signing-time
> > * draft-ietf-sidrops-rpki-prefixlist
> > * And there also is a recent interest to revisit the validation
> >  algorithm, this will all take up a fair chunk of working group
> >  bandwidth.
> >
> > In short - deprecating rsync before having a viable alternative to RRDP
> > v1 is horse behind cart. A deprecation of rsync won't solve immediate

don't disagree, my intent was absolutely NOT to 'ok, turn off rsync now'.

> > operational issues, but instead will add problems. The community now
> > also is aware of various optimizations that can be done to make
> > RRDP-to-RSYNC switchovers far less resource intensive, so scalability
> > concerns are not as strong as they once were.
>
> Job, as you are saying (I think), I agree the we should only consider deprecating
> rsync altogether until we have an improvement over RRDP, or at least ensure
> that fallbacks are possible somehow. I believe that a discussion about a next
> gen protocol is useful and can occur in parelel to other work. There is no reason
> why e.g. the ASPA work should be stalled because of such a discussion.
>
> For what it's worth, some initial thoughts regarding an RRDP++/bis/replacement:
>
> It would be good if a future iteration can reduce the bandwidth. Other signaling
> than HTTP polling of potentially large files may be useful to consider here.
>
> There is no easy direct access to files in RRDP. Furthermore, URIs can be
> anything - there is no consistent scoping. Writing a full snapshot on every change
> is heavy (I agree with draft-ietf-grow-nrtm-v4 on this one). The encoding is
> convoluted. We may want to think about mirroring.
>
> Etc. This is not complete. Let’s first establish if we want to have the discussion
> now, then we can talk more details..

thanks! (ties/job/tim all) for the conversation. I suspect we have
another little chat time
in email, then maybe we could work out some of the question(s) /
answers and do that
in a document.

-chris

>
> >
> > Kind regards,
> >
> > Job
> >
> > _______________________________________________
> > Sidrops mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/sidrops
> >
>