From nobody Thu Mar 9 07:48:37 2023 Return-Path: X-Original-To: idr@ietfa.amsl.com Delivered-To: idr@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id EF21CC169528 for ; Thu, 9 Mar 2023 07:48:36 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -7.096 X-Spam-Level: X-Spam-Status: No, score=-7.096 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-5, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=raszuk.net Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id l8ZJJYC845RL for ; Thu, 9 Mar 2023 07:48:32 -0800 (PST) Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id B884CC169524 for ; Thu, 9 Mar 2023 07:48:32 -0800 (PST) Received: by mail-wr1-x42d.google.com with SMTP id q16so2350638wrw.2 for ; Thu, 09 Mar 2023 07:48:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raszuk.net; s=google; t=1678376911; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=EqGkwqxMNm8mrARS4wikV88liJ9ASqmLdTA/t0BRGXs=; b=ZonxBEJWxo+gg1ZLbyTkMukvl9tI8zJKeR0QH7QN/oJJ+xRMR3sRPNJJBb4HNhKkTm TZMi831nWasVjvvDQiJkf9bsRATdKaZUyFnXU0uM+b91vuPCs1YStQW6DLkLGoIEwooy FhX+xWjcNW8RrPU2HteDBlTBxTjS6STeHsUWM2lIZacaytIIOGND3AkhqPEd89XcV7Hn YchBXZR7VUBngc5nwBeb2Twfnvg1GH9opmjw5BZn1gJbIQ6sD6kCI4GrxLLzQmk8dwzZ 0TbhULUSaMIBD/3Gpg0IJ6zy6IlPcKvcY9p5Ao4UhWrK1O16JVbdTHNnVZDD+zKFmsb9 qnag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678376911; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=EqGkwqxMNm8mrARS4wikV88liJ9ASqmLdTA/t0BRGXs=; b=LsAnbWFWRO/54BSsjcpeHmgoQmyrofAtgzQLzXjnG3FxAJwPdY9PulzyWm1bDVLwhY kv22E6EGAR5jumC0R/YMtQlJns10lM1pVy6qfS2BsNVGj6ZBSn4PSJFoCl2Uu3Tw/PBx 0t6wBdxc9zOiLEGgH6/FWLWxlSveAQ8LxO2EZwFpclxHwo8FZxBpwal2zszYORr7/tje J1IJ9lTJ2gmw6jFrzLJjHiiWhBHaA1EQ296Dmp/jj0lg5IGuw7B9TJcbhelrLMHc9T/S En5OTkyx90iUF5qUhvdN9PjkrrtsG+95WkSbAeAM9mfcC7WsPK7/E3rWwx6CheXUvy0K 2zvg== X-Gm-Message-State: AO0yUKV6WjsDq1Ez9MfXhVXV4XrFvcREVF67CHCVIQMYvTe/s2howZ5g 108s+TpMU6VnIVMFd9CMyo37In1OV3v9MC5oc3JiFQ== X-Google-Smtp-Source: AK7set+51heXc47ei67O02ixsKQPpp7A28nNprlSnEC5VtVXGD3Qc2fGa4aHXA1Z5bu0B63X2OJpR6V9U66BCfgX0ls= X-Received: by 2002:adf:dcd0:0:b0:2c9:b9bf:e211 with SMTP id x16-20020adfdcd0000000b002c9b9bfe211mr4834618wrm.2.1678376911056; Thu, 09 Mar 2023 07:48:31 -0800 (PST) MIME-Version: 1.0 References: <45816498-6F09-4135-ACB4-6E51683001C7@pfrc.org> In-Reply-To: From: Robert Raszuk Date: Thu, 9 Mar 2023 16:48:20 +0100 Message-ID: To: Tony Przygienda Cc: Jeffrey Haas , "idr@ietf. org" Content-Type: multipart/alternative; boundary="0000000000009542b005f67993ac" Archived-At: Subject: Re: [Idr] BGP Message level ACKs X-BeenThere: idr@ietf.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: Inter-Domain Routing List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Mar 2023 15:48:37 -0000 --0000000000009542b005f67993ac Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Tony, > But I lost track frankly WHAT we're solving here with this ACK you suggest ... The reported problem is: "The remote peer is not reading my BGP messages while TCP works fine." So if a remote peer can ACK that it got messages I send to it (last 10 or last 100 etc ..) it solves the problem as defined. In fact it solves it better then current proposal of using single side socket buffer full timer. I should have not thrown into this UDP as it changes the objective - from informational nice to have ACK to something much more complicated to provide transport reliability. That was not the original aim. Best, R. PS. If you like lets talk more about it face to face :) On Thu, Mar 9, 2023 at 4:36=E2=80=AFPM Tony Przygienda wrote: > simple update message "counter" does not change much of anything really > since the sender has to keep this "counter" on the NLRI at which an updat= e > was sent to understand which prefixes have been processed by the peer. an= d > all very amusing if you start to overlap with newer updates containing th= e > newer version of the same prefix. Unless you think of "keeping all the > updates stored on the outbound until they 'acked" which would be a > fantastic attack vector by intentional or unintentional slow peers + doub= le > with the TCP buffering already. With the interesting stuff on top of when > you give up on the peer if it doesn't ACK in timely manner/at all ... > > So, if you want to know what the peer on the other side "processed" you > will have no other choice but keep per NLRI _something_ and possibly per > peer (as I explained with lots cleverness you can get away without per pe= er > possibly). Unless we're talking unreliable transport under BGP which as I > said seems to be solving wrong problems in wrong layers. But I lost track > frankly WHAT we're solving here with this ACK you suggest ... > > I still don't understand what "application consuymed the data" _really_ > means in terms of assurances the remote system is giving you in terms of > forwarding behavior/convergence with other peers/whatever and hence this > "assurance" seems to be to have no semantics. > > So, I chime out. Either this all has no real semantics and possibly > conflates multiple issues together or I don't grok what it's all about. > > motivation for having seq# on enhanced GR draft I get. we do mark/sweepin= g > kind of thingies on session re-establish (which GR really is) and we have > to sync up on versions of state held (as optimization vs. re-sending > e'thing). And having the other side "confirming" really means somethjing, > namely FIB/RIB is programmed to this version so no updates are needed. It= 's > kind of simplified version of the refresh-options (since it basically > assumes there is no GR-over-GR or endless updating on sender that makes t= he > process never stop?) but it burns basically 8 bits per NLRI as well > (assuming most likely there is never more than one peer GR'ing either or > maybe just one version for all peers work, I didn't think that through > yet). But yeah, no free lunch ... > > --- tony > > On Thu, Mar 9, 2023 at 9:48=E2=80=AFAM Robert Raszuk = wrote: > >> Tony, >> >> I think you are looking at this in a different way. >> >> So what you are likely considering is a very fine and detailed sequence >> number like approach. That's a lot of work indeed to accommodate it into >> current BGP outbound code - with tons of optimizations. >> >> I on the other hand was just thinking of reusing simple UPDATE message >> counter per peer. Some (I would say most) implementations I think have i= t >> already. >> >> The only sort of open issue is to synchronize the counters or at least >> clocks. Alternative would be to inject periodically a sequenced BGP MARK= ER >> message to tag local counter with remote counter and have the ACKs being >> MARKER triggered. Quite popular solution in completely asynchronous syst= ems >> :) >> >> So to spec this out should be pretty straightforward. But again this >> email is to just probe the topic not volunteer to do it. >> >> Many thx, >> Robert. >> >> >> On Thu, Mar 9, 2023 at 8:45=E2=80=AFAM Tony Przygienda >> wrote: >> >>> The problem is on the sender side. it has to keep track of whether an >>> update has been ack'ed by a peer and that's in simple terms per peer pe= r >>> NLRI (modulo group discussion as usual). It can be tricked into per NLR= I >>> only but since NLRIs get packed differently, in different sequence towa= rds >>> different peers a high scale implementation will end up with at least p= er >>> prefix. This is _very_ expensive, particularly memory wise. Tricks can = be >>> played with rollovers like in >>> https://datatracker.ietf.org/doc/draft-idr-bgp-route-refresh-options/ >>> but the logic isn't simple by any means and ultimately any solution may >>> force implementations into structures to support this that are very >>> inefficient for other parts of the implementation. >>> >>> Generally, shoving reliability of the transport into BGP is basically >>> solving the wrong problem at the wrong layer just like shortcutting BGP= to >>> run on a user transport protocol brought nothing but lots of technical = debt >>> grief over the years. Real problem is lack of control session layer th= at >>> realizes underneath that transport went away and notifies up and/or all= ows >>> transport for the session to come up again. It's only when a new sessio= n >>> needs be established AFAIS that we need state sync which lands us in >>> possibly GR corners or refresh corners again rather than trying to rein= vent >>> TCP on top of UDP in BGP by ACKs (I wonder when the first window size >>> proposal starts to come in ;-) >>> >>> -- tony >>> >>> On Wed, Mar 8, 2023 at 11:52 PM Robert Raszuk wrote= : >>> >>>> Hi Jeff, >>>> >>>> >>>>> A fundamental challenge for bgp implementations is efficiently keepin= g >>>>> versioning across peer-groups.* Some implementations are friendlier = to >>>>> that idea than others, but the inconsistency in the ability to do tha= t is >>>>> one of the things that prevented this idea from progressing. >>>>> >>>> >>>> I did not mean that it should touch update generation and replication. >>>> The ACKs which I had in mind would be operating way above this and sim= ply >>>> periodically indicate to the sender that last N number of UPDATE messa= ges >>>> got received by remote BGP process. >>>> >>>> N-being configured and possibly negotiated on the session bootstrap. >>>> >>>> Critical part of this is the time scale. If we are talking about the >>>> last 10 min or last 20 min this would not be sufficient for helping to >>>> achieve reliable transport, but could be IMO more than sufficient to m= ake >>>> sure UPDATE got to remote BGP I/O. >>>> >>>> So I am not sure if this is real distraction to the topic. I would >>>> rather call it exploration of alternative approaches. Just like puntin= g the >>>> "issue" to telemetry ... avoiding need for any BGP code or FSM state >>>> machine change needed. >>>> >>>> Many thx, >>>> R. >>>> >>>> _______________________________________________ >>>> Idr mailing list >>>> Idr@ietf.org >>>> https://www.ietf.org/mailman/listinfo/idr >>>> >>> --0000000000009542b005f67993ac Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Tony,

> But I lost track frank= ly WHAT we're solving here with this ACK you suggest ...

The reported problem is:=C2=A0

"The = remote peer is not reading my BGP messages while TCP works fine."

So if a remote peer can ACK that it got messages I sen= d to it (last 10 or last 100 etc ..) it solves the problem as defined. In f= act it solves it better then current proposal of using single side socket b= uffer full timer.=C2=A0

I should have not thrown= =C2=A0into this UDP as it changes the objective - from informational nice t= o have ACK to something much more complicated to provide transport reliabil= ity. That was not the original aim.=C2=A0

Best,
R.

PS. If you like lets talk more abou= t it face to face :)






On Thu, Mar 9, 2023 at 4:36=E2=80=AFPM Tony Przyg= ienda <tonysietf@gmail.com>= ; wrote:
simple update message "counter" does not change mu= ch of anything really since the sender has to keep this "counter"= on the NLRI at which an update was sent to understand which prefixes have = been processed by the peer. and all very amusing if you start to overlap wi= th newer updates containing the newer version of the same prefix. Unless yo= u think of "keeping all the updates stored on the outbound until they = 'acked" which would be a fantastic attack vector by intentional or= unintentional slow peers + double with the TCP buffering already. With the= interesting stuff on top of when you give up on the peer if it doesn't= ACK in timely manner/at all ...

So, if = you want to know what the peer on the other side "processed" you will have no other choice but keep per NLRI _something_ and=20 possibly per peer (as I explained with lots cleverness you can get away=20 without per peer possibly). Unless we're talking unreliable transport u= nder BGP which as I said seems to be solving wrong problems in wrong layers= . But I lost track frankly WHAT we're solving here with this ACK you su= ggest ...

I still don't understand = what "application consuymed the data" _really_ means in terms of = assurances the remote system is giving you in terms of forwarding behavior/= convergence with other peers/whatever and hence this "assurance" = seems to be to have no semantics.

So, I chime= out. Either this all has no real semantics and possibly conflates multiple= issues together or I don't grok what it's all about.

motivation for having seq# on enhanced GR draft I get. we = do mark/sweeping kind of thingies on session re-establish (which GR really = is) and we have to sync up on versions of state held (as optimization vs. r= e-sending e'thing). And having the other side "confirming" re= ally means somethjing, namely FIB/RIB is programmed to this version so no u= pdates are needed. It's kind of simplified version of the refresh-optio= ns (since it basically assumes there is no GR-over-GR or endless updating o= n sender that makes the process never stop?) but it burns basically 8 bits = per NLRI as well (assuming most likely there is never more than one peer GR= 'ing either or maybe just one version for all peers work, I didn't = think that through yet). But yeah, no free lunch ...=C2=A0

--- tony

On Thu, Mar 9, 2023 at 9:48=E2=80=AFAM Robert Raszuk = <robert@raszuk.ne= t> wrote:
Tony,

I think you are looki= ng at this in a different way.=C2=A0

So what = you are likely considering is a very fine and detailed sequence number=C2= =A0like approach. That's a lot of work indeed to accommodate=C2=A0it in= to current BGP outbound code - with tons of optimizations.=C2=A0
<= div>
I on the other hand was just thinking of reusing simple= =C2=A0UPDATE message counter per peer. Some (I would say most) implementati= ons I think have it already.=C2=A0

The only sort o= f open issue is to synchronize the counters or at least clocks. Alternative= would be to inject periodically=C2=A0a sequenced BGP MARKER message to tag= local counter with remote counter and have the ACKs being MARKER triggered= . Quite popular solution in completely asynchronous=C2=A0systems :)=C2=A0

So to spec this out should be pretty=C2=A0straightf= orward. But again this email=C2=A0is to just probe the topic not volunteer = to do it.=C2=A0

Many thx,
Robert.
<= /div>


On Thu, Mar 9, 2023 at 8:45=E2=80=AFAM Tony Przygi= enda <tonysietf= @gmail.com> wrote:
The problem is on the sender side. it has = to keep track of whether an update has been ack'ed by a peer and that&#= 39;s in simple terms per peer per NLRI (modulo group discussion as usual). = It can be tricked into per NLRI only but since NLRIs get packed differently= , in different sequence towards different peers a high scale implementation= will end up with at least per prefix. This is _very_ expensive, particular= ly memory wise. Tricks can be played with rollovers like in https://datatracker.ietf.org/doc/draft-idr-bgp-route-refresh-op= tions/ but the logic isn't simple by any means and ultimately any s= olution may force implementations into structures to support this that are = very inefficient for other parts of the implementation.

=
Generally, shoving reliability of the transport into BGP is basi= cally solving the wrong problem at the wrong layer just like shortcutting B= GP to run on a user transport protocol brought nothing but lots of technica= l debt grief over the years.=C2=A0 Real problem is lack of control session = layer that realizes underneath that transport went away and notifies up and= /or allows transport for the session to come up again. It's only when a= new session needs be established AFAIS that we need state sync which lands= us in possibly GR corners or refresh corners again rather than trying to r= einvent TCP on top of UDP in BGP by ACKs (I wonder when the first window si= ze proposal starts to come in ;-)

-- tony

On Wed, Mar 8, 2023 at 11:52 PM Robert Raszuk <robert@raszuk.net> wrote:
Hi Jeff,
=C2=A0
A fundamental challenge for bgp implement= ations is efficiently keeping versioning across peer-groups.*=C2=A0 Some im= plementations are friendlier to that idea than others, but the inconsistenc= y in the ability to do that is one of the things that prevented this idea f= rom progressing.=C2=A0

I did not mean t= hat it should touch update generation and replication. The ACKs which I had= in mind would be operating way above this and simply periodically indicate= to the sender that last N number of UPDATE messages got received by remote= BGP process.=C2=A0

N-being configured and possibl= y negotiated on the session bootstrap.=C2=A0

Criti= cal part of this is the time scale. If we are talking about the last 10 min= or last 20 min this would not be sufficient for helping to achieve reliabl= e transport, but could be IMO more than sufficient to make sure UPDATE got = to remote BGP I/O.=C2=A0

So I am not sure if this = is real distraction to the topic. I would rather call it exploration of alt= ernative approaches. Just like punting the "issue" to telemetry .= .. avoiding need for any BGP code or FSM state machine change needed.=C2=A0=

Many thx,
R.

=
_______________________________________________
Idr mailing list
Idr@ietf.org
https://www.ietf.org/mailman/listinfo/idr
--0000000000009542b005f67993ac--