Re: [iccrg] [tsvwg] New Internet Draft: Congestion Signaling (CSIG)

[Sebastian Moeller <[e-mail]>]

Hi Tom,


> On 20. Feb 2024, at 18:04, Tom Herbert <tom=[e-mail]> wrote:
> 
> 
> 
> On Mon, Feb 19, 2024 at 11:59 PM Sebastian Moeller <moeller0=[e-mail]> wrote:
> >
> > Hi Tom,
> >
> >
> > > On 19. Feb 2024, at 21:09, Tom Herbert <tom=[e-mail]> wrote:
> > >
> > > On Mon, Feb 19, 2024 at 11:31 AM Sebastian Moeller
> > > <moeller0=[e-mail]> wrote:
> > >>
> > >> Hi Tom,
> > >>
> > >>
> > >>> On 19. Feb 2024, at 18:53, Tom Herbert <tom=[e-mail]> wrote:
> > >>>
> > >>> On Sun, Feb 18, 2024 at 11:34 PM Sebastian Moeller
> > >>> <moeller0=[e-mail]> wrote:
> > >>>>
> > >>>> Hi Matt,
> > >>>>
> > >>>>> On 17. Feb 2024, at 20:17, Matt Mathis <[e-mail]> wrote:
> > >>>>>
> > >>>>> I think the L2/L4 split is brilliant.
> > >>>>
> > >>>> [SM] Respectfully, the brilliance depends very much on the goal/gamer plan. Is this purely aimed at data center traffic this looks like a sweet solution that is 'organically' confined to the domain with appropriately capable L2 elements? Or is the end-game here an (overdue) improvement of end-to-end loads/congestion information? In the former case L2/L4 seems a decent solution, in the latter case less so (not that getting a common L3 solution would be guaranteed or easy).
> > >>>>
> > >>>>
> > >>>>> Putting the forward instrumentation as low as possible in the stack permits easy processing in HW w/o parsing any L3.
> > >>>>
> > >>>> [SM] Sweet, but that really means that this solution is unlikely to survive over a full internet path.
> > >>>>
> > >>>>> Putting the replies in L4 only requires a handful of implementations to cover all possible paths,
> > >>>>
> > >>>> [SM] Mmmh, that might be but partly because the L2 solution noticeably restricts the set of possible paths, no?
> > >>>>
> > >>>>> and piggybacks on existing solutions to session layer issues, such as authentication and authorization.
> > >>>>
> > >>>> [SM] What is the threat model here? I would guess an attacker that knows the full path might just as well probe the congestion level and an attacker that does not know the path might not be able to do much with the congestion information? (Any attacker that can modify the congestion information might as well drop the packet directly).
> > >>>>
> > >>>>>
> > >>>>> I would consider mentioning but then temporarily excluding alternet placements: either as a shim at the top of L2, sort of like VLAN tags, or within an L3 option.   Both of these have their own challenges, but might be extremely valuable in some environments.
> > >>>>
> > >>>> [SM] Some environments, like the internet? I know that the I in IETF is not a strict limiter of scope, but still it would be nice if drafts would have a viable path of being implemented over the internet... That said, well possible that the current state does not merit use-over-the-internet yet and so maybe starting with an L2/L4 solution might be considered a safety back-stop?
> > >>>
> > >>> Sebastian,
> > >>>
> > >>> There's no reason to believe that Congestion Signaling isn't of
> > >>> interest to use on an internet (lower case 'i' is explicit here).
> > >>
> > >> [SM] Well, in my mind that still would keep this out of scope for the capital I ETCopy of Copy of Enfabrica-SiPandaF,
> >
> > [SM2] Not sure how that 'Copy of Copy of Enfabrica-SiPanda' string ended up in that paragraph, I do not recall writing that... either autocompletion run wild or some other issue.
> >
> >
> > >> I am interested in improving end to end congestion signalling over the Internet so I desire these signals to sink and source at my endpoints... Again, I understand that my position is in the rough regarding what the IETF should care about.
> > >>
> > >>> This is almost certainly beneficial for 6G for instance which is an
> > >>> internet composed for various link layer technologies.
> > >>> Neither is this
> > >>> the only protocol of this nature there are and will be others-- from
> > >>> an IETF POV I believe we want a extensible protocol solution that
> > >>> benefits multiple use cases and works in different environments.
> > >>
> > >> [SM] That way lies madness IMHO. Getting enough routers/switches support one signal and hence make it useful is already almost a Sisyphus task expecting them to support multiple signals selected individually per packet seems like a recipe of never getting this to work end to end (which is my motivator here). If we do not know which single signal to use here, I guess keeping this private and do more research seems like a productive way forward.
> > >
> > > Sebastian,
> > >
> > > I would agree with that if this was the first protocol ever trying to
> > > do something like this, but it's not. IOAM is already a published RFC.
> >
> 
> Hi Sebastian,
> 
> > [SM2] There are a number of points that are against IOAM being a suitable encapsulation here:
> 
> Yes, I wasn't suggesting that IOAM could be used without modification. I was suggesting that a compressed format could be derived where we can leverage the underlying mechanisms and implementation.

[SM3] But the same could be achieved outside of the IOAM constraints, no?


> 
> >
> > https://datatracker.ietf.org/doc/html/rfc9378 :
> > "IOAM is focused on "limited domains", as defined in [RFC8799]. IOAM is not targeted for a deployment on the global Internet."
> 
> That's more of a statement of security and not feasibility. There's simply no security in the Internet, so we cannot trust or validate that anonymous intermediate nodes are going to write correct information. Any plain text in a packet on the Internet is subject to inspection and modification if the data isn't authenticated, and in the worst case this could be a DoS vector by writing bad information.

[SM3] Indeed, but e.g. for TCP you would need to know a lot about the most recent packet to be able to play games, no? So either you are on path and already can drop/duplicate packets at will or you are off path but still need a recent enough veridical packet to be able cause mischief, no? (I might be insufficiently creative in attack vectors)

> 
> >
> > IMHO that already disqualifies IOAM here as the goal needs to be a fully end to end method having the Internet as its scope.
> >
> 
> Because of the fundamental security issues on the global Internet, it's unlikely we'll ever have a usable network-thost signaling protocol on the Internet.

[SM3] But we already do, ECN, CSIG really is just ECN on steroids so will not open qualitatively new attack avenues, no? I am not looking fore a generic network to host protocol for any kind of information, my goal is really just to fix "ECN"...

> 
> > [SM2] This is already handled in the ' IOAM Proof of Transit Type 0' field, the facrt that you did not notice that supports my argument that IOAM is too complicated an RFC for this simple use-case, no?
> >
> >
> > > If that constraint is
> > > removed then the only remaining argument against IOAM seems to be that
> > > it's easier for hardware to handle L3 rather than L2 in hardware.
> >
> > [SM2] No, it is still way to complicated (offering options all intermediary nodes will need to check before setting values) and way way to overheady...
> 
> We can define a fixed length L3 format for the fast path that would be quite efficient. See below.

[SM3] Sure but then we might drop all of the IOAM dressings... at which point doing this inside of IOAM might not be a good fit anymore?

> 
> >
> > > I don't believe there is currently consensus that that is generally
> > > true.
> >
> > [SM2] Unclear what the consensus is. IMHO is is not really a question of consensus, if we want this to be a first class end 2 end signal, L2 is simply not an option independent of our wishes.
> 
> I agree, but we have seen these sort of statements from vendors before that processing L3 in hardware is overly complex and difficult to make efficient.

[SM3] I am guessing here, but the more conditions need to be checked before reaching a location to modify the more vendors are going to hate it... and with at least some justification...


> Instead of blindly accepting that, or just dismissing the point, I believe it's in everyone's best interest to openly discuss this, and note there's already a lot of work to address known inefficiencies in IP processing see draft-ietf-6man-hbh-processing for instance.
> 
> >
> > > And, if this is why IOAM "has such a sparse (or no) support from
> > > switch vendors" as Jai claims then it seems like this is maybe
> > > something that should be discussed instead of just arbitrarily
> > > dismissing IOAM. Why exactly is IOAM in HW such a problem and can it
> > > be fixed? (a quick look at ippm archives didn't reveal any
> > > discussions).
> >
> > [SM2] As shown above IOAM seems to be one of these everything and the kitchensink solution, probably great for its intended use cases, but efficient end/network to end signalling IM;HO is not one of these use cases. Sure it could be bent into shape to support that use case, but I would rather see a meaner and leaner signal design for the congestion information aggregation along a path use case. If I had control, I would propose a new IPv6 header type (with a new next header number) mutually exclusive with hop by hop (so that confirming the next header number is the only check required before updating data at a known offset). There is value in simplicity... (I guess this scheme will fail due to firewalls only permitting a limited set of next header values...)
> 
> We don't need to define a new extension header (and realistically we'd never get a new EH through IETF anyway),

[SM3] Then we should close down IETF for good... because the whole justification for the 8 bit for this field is that we might end up with 256 different instances... if we declare this to be a complete set we will have failed. That said, just because we could define a new number/type here does not mean it would be the best way forward, but it would mean reading the primary next header value would give us an unambiguous offset to the to be RMW word...



> CSIG can be efficiently done as a Hop-by-Hop Option. For example:
> 
> 
>                                                                       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>                                    |                               |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               +
>       |                                                   Src Mac address                                                  |     
>       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |                         Dest Mac address                      |
>    +                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+    |                               |             0x86dd            |
>       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |   6   | Traffic Class |              Flow Label               |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |         Payload Length        |      0        |   Hop Limit   |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |                                                               |
>    +                                                               +
>    |                                                               |
>    +                         Source Address                        +
>    |                                                               |
>    +                                                               +
>    |                                                               |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |                                                               |
>    +                                                               +
>    |                                                               |
>    +                      Destination Address                      +
>    |                                                               |
>    +                                                               +
>    |                                                               |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |      0        |  Next header  |      50       |       4       |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    | Type  |         LM            |               S               |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |                                                               |
>    |              ULP (e.g. TCP header and payload                 |
> 
> This assumes 0x32 HBH Option type is assigned to CSIG.

[SM3] Now we need to first read primary next-hop field and then check the HBH Option type, sure we can solve a lot of things by adding layers of indirection, but it will not get cheaper that way...

> 
> * Is this encoding efficient? Yes. Compared to the compact format in the CSIG draft there is an additional four bytes of on-the-wire overhead, and it's the same overhead as the Expanded format which is eight bytes.

[SM3] Not sure that is fair... we just increased the minimal encoding from 4 to 8 bytes... this might be OK if the 4 byte format simply was a crutch to make CSIG masquerade as a VLAN tag, but if the 54 byte version was supposed to be the bread and butter encoding we just doubled the overhead (sure we also increased fidelity).

> * Is this friendly to hardware processing in the fast path? Yes. The Ethernet header, IPv6 header, and Hop-by-Hop Options header with a single option for CSIG form a fixed header of 42 bytes which will fit well within a router's parsing buffer. that can be quickly processed. The match of this format is Bytes 12-31 == 0x86dd && Byte 20 == 0 && Byte 34 ==0 Bytes 36-37 == 0x3204. Upon receiving a packet, these fields can be extracted and run through a TCAM.

[SM3] My limited experience tells me that TCAM is not cheap...

> If there's a hit that this is a packet with CSIG then it can be processed. Note that the CSIG option can be processed in parallel with processing the Ethernet header and IP header.

[SM3] Playing devils advocate, this preocessing, parallel or not will come with some power cost, better to keep it simple.

> 
> * Is this format extensible? Yes. The Type field allows CSIG formats with different semantics and structure (the CSIG draft defines three types, so thirteen are remaining for extensibility). 

[SM3] This is against making this efficient for hardware, IMHO CSIG needs to be whittled down to a single unambiguous piece of information (or at worst two) extensibility sounds great in theory, but in practise... (see above about getting a new next header ID through the IETF ;) ).

> 
> * Is the format routable? Yes, The format is contained in the Network layer so it is properly end-to-end data that is routable over a limited domain internet and the Internet (modulo that packets with extension headers might be dropped in the Internet)

[SM3] That last one should be a fixable issue... 

> 
> * Can the format be authenticated? Yes. AH header can be used. AH would authenticate the option but not the data since the intent is for routers along the path to write CSIG data in the packet.

[SM3] Yes aggregating information from all nodes along a network path is the whole reason for this exercise...


> AH prevents someone from inserting or removing the option in flight,

[SM3] Mmmh, removing such an option would be quite noticeable and adding it would probably require playing some MTU/MSS games to avoid creating packets that wiuld need fragmentation, no?


> the data written by routers isn't authenticated and there's really no way to do that since we'd have to establish and SA between routers and hosts; the typical answer to this is that we should restrict protocols like this to trusted limited domains (for instance, using CSIG over the Internet is pretty pointless since nothing prevents intermediate devices from writing bad information into a packet-- this could even be a DoS vector).

[SM3] I disagree, if end to end congestion control really really wants timely information for the current network path, and we already do this pretty successfully with the ECN bits. Plus most routers that could fudge a CSIG value might as well just drop the packet, and if someone accidentally/purposefully modifies the true signal to no-congestion (to cause the sender to send too much the only real DoS vector I can see) there is always the 'the congested node simply drops packets' remedy...

> * Can this format be encrypted? No. We'd need an SA established network and hosts which doesn't seem plausible.

[SM3] We would need ALL nodes along a path have the key, at which point we loose only a little security by not encrypting at all...

> However, the return path for reflecting the signal could be done in a Destination Option which could be encrypted by an IPsec header.

[SM3] That is a interesting question, currently everyone and their pony packs the return path into L4 or higher, but in theory that information might also be put into an L3 header... I guess the horse has left that barn long ago so this is purely theoretical...

Regards
	Sebastian

> 
> Tom
> 
> >
> >
> > Regards
> >         Sebastian
> >
> >
> > >
> > > Tom
> > >
> > >>
> > >> Regards
> > >>        Sebastian
> > >>
> > >>
> > >>>
> > >>> Tom
> > >>>
> > >>>>
> > >>>> Regards
> > >>>>       Sebastian
> > >>>>
> > >>>>>
> > >>>>> On Sat, Feb 10, 2024 at 7:42 AM Tom Herbert <tom=[e-mail]> wrote:
> > >>>>> On Fri, Feb 9, 2024 at 10:53 PM Nandita Dukkipati <[e-mail]> wrote:
> > >>>>>>
> > >>>>>> Hi Tom,
> > >>>>>>
> > >>>>>> We updated the draft, correcting some nit errata, and to not let the draft expire. It's not discussed in any other mailing lists.
> > >>>>>
> > >>>>> Thanks Nandita.
> > >>>>>
> > >>>>> I still have fundamental concerns about the protocol layering in this
> > >>>>> draft, please see my previous comments on that. The draft defines a
> > >>>>> protocol for end-to-end network to host signaling and IMO, such a
> > >>>>> protocol belongs in the network layer but the draft puts the protocol
> > >>>>> in L2 and L4 and seems to avoid L3 without explanation. IOAM defines a
> > >>>>> very similar method of signaling and RFC9486 is a good model for
> > >>>>> network layer protocol that provides network to host signaling.
> > >>>>>
> > >>>>> Tom
> > >>>>>
> > >>>>>>
> > >>>>>> Nandita
> > >>>>>>
> > >>>>>> On Thu, Feb 8, 2024 at 3:53 PM Tom Herbert <[e-mail]> wrote:
> > >>>>>>>
> > >>>>>>> Hi,
> > >>>>>>>
> > >>>>>>> I noticed there is now an -01 version of the draft posted on Feb. 2.
> > >>>>>>> Is this draft being discussed on some other list?
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>> Tom
> > >>>>>>>
> > >>>>>>> On Sat, Sep 9, 2023 at 9:09 AM Tom Herbert <[e-mail]> wrote:
> > >>>>>>>>
> > >>>>>>>> Hi, thanks for draft!
> > >>>>>>>>
> > >>>>>>>> The first thing that stands out to me is the carrier of the new packet headers. In the forward path it would be in L2 and in reflection it would be L4. As the draft describes, this would entail having to support the protocol in multiple L2 and multiple L4 protocols-- that's going to be a pretty big lift! Also, L2 is not really an end-to-end protocol (would legacy switches in the path also forward the header)l?).
> > >>>>>>>>
> > >>>>>>>> The signaling being described in the draft is network layer information, and hence IMO should be conveyed in network layer headers. That's is L3 which conveniently is the average of L2+L4 :-)
> > >>>>>>>>
> > >>>>>>>> IMO, the proper carrier of the signal data is Hop-by-Hop Options. This is end-to-end and allows modification of data in-flight. The typical concern with Hop-by-Hop Options is high drop rates on the Internet, however in this case the protocol is explicitly confined to a limited domain so I don't see that as a blocking issue for this use case.
> > >>>>>>>>
> > >>>>>>>> The information being carried seems very similar to that of IOAM (IOAM uses Hop-by-Hop Options and supports reflection). I suppose the differences are that this protocol is meant to be consumed by the transport Layer and the data is a condensed summary of path characteristics. IOAM seems pretty extensible, so maybe it could be adapted to carry the signals of this draft?
> > >>>>>>>>
> > >>>>>>>> A related proposal might be FAST draft-herbert-fast. Where the CSIG is network to host signaling, FAST is host to network signaling for the purposes of requesting network services. These might be complementary and options for both may be in the same packet. FAST also uses reflection, so we might be able to leverage some common implementation at a destination.
> > >>>>>>>>
> > >>>>>>>> Tom
> > >>>>>>>>
> > >>>>>>>> On Fri, Sep 8, 2023, 7:43 PM Abhiram Ravi <abhiramr=[e-mail]> wrote:
> > >>>>>>>>>
> > >>>>>>>>> Hi IPPM folks,
> > >>>>>>>>>
> > >>>>>>>>> I am pleased to announce the publication of a new internet draft, Congestion Signaling (CSIG): https://datatracker.ietf.org/doc/draft-ravi-ippm-csig/
> > >>>>>>>>>
> > >>>>>>>>> CSIG is a new end-to-end packet header mechanism for in-band signaling that is simple, efficient, deployable, and grounded in concrete use cases of congestion control, traffic management, and network debuggability. We believe that CSIG is an important new protocol that builds on top of existing in-band network telemetry protocols.
> > >>>>>>>>>
> > >>>>>>>>> We encourage you to read the CSIG draft and provide your feedback and comments. We have also cc'd the TSVWG, CCWG, and ICCRG mailing lists, as we believe that this work may be of interest to their members as well.
> > >>>>>>>>>
> > >>>>>>>>> Thank you for your time and consideration.
> > >>>>>>>>>
> > >>>>>>>>> Sincerely,
> > >>>>>>>>> Abhiram Ravi
> > >>>>>>>>> On behalf of the CSIG authors
> > >>>>>
> > >>>>> _______________________________________________
> > >>>>> iccrg mailing list
> > >>>>> [e-mail]
> > >>>>> https://mailman.irtf.org/mailman/listinfo/iccrg
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> Thanks,
> > >>>>> --MM--
> > >>>>> Evil is defined by mortals who think they know "The Truth" and use force to apply it to others.
> >
> >