Re: [CCWG] [tsvwg] [iccrg] New Internet Draft: Congestion Signaling (CSIG)

[Sebastian Moeller <[e-mail]>]

Hi Tom,


> On 19. Feb 2024, at 21:09, Tom Herbert <tom=[e-mail]> wrote:
> 
> On Mon, Feb 19, 2024 at 11:31 AM Sebastian Moeller
> <moeller0=[e-mail]> wrote:
>> 
>> Hi Tom,
>> 
>> 
>>> On 19. Feb 2024, at 18:53, Tom Herbert <tom=[e-mail]> wrote:
>>> 
>>> On Sun, Feb 18, 2024 at 11:34 PM Sebastian Moeller
>>> <moeller0=[e-mail]> wrote:
>>>> 
>>>> Hi Matt,
>>>> 
>>>>> On 17. Feb 2024, at 20:17, Matt Mathis <[e-mail]> wrote:
>>>>> 
>>>>> I think the L2/L4 split is brilliant.
>>>> 
>>>> [SM] Respectfully, the brilliance depends very much on the goal/gamer plan. Is this purely aimed at data center traffic this looks like a sweet solution that is 'organically' confined to the domain with appropriately capable L2 elements? Or is the end-game here an (overdue) improvement of end-to-end loads/congestion information? In the former case L2/L4 seems a decent solution, in the latter case less so (not that getting a common L3 solution would be guaranteed or easy).
>>>> 
>>>> 
>>>>> Putting the forward instrumentation as low as possible in the stack permits easy processing in HW w/o parsing any L3.
>>>> 
>>>> [SM] Sweet, but that really means that this solution is unlikely to survive over a full internet path.
>>>> 
>>>>> Putting the replies in L4 only requires a handful of implementations to cover all possible paths,
>>>> 
>>>> [SM] Mmmh, that might be but partly because the L2 solution noticeably restricts the set of possible paths, no?
>>>> 
>>>>> and piggybacks on existing solutions to session layer issues, such as authentication and authorization.
>>>> 
>>>> [SM] What is the threat model here? I would guess an attacker that knows the full path might just as well probe the congestion level and an attacker that does not know the path might not be able to do much with the congestion information? (Any attacker that can modify the congestion information might as well drop the packet directly).
>>>> 
>>>>> 
>>>>> I would consider mentioning but then temporarily excluding alternet placements: either as a shim at the top of L2, sort of like VLAN tags, or within an L3 option.   Both of these have their own challenges, but might be extremely valuable in some environments.
>>>> 
>>>> [SM] Some environments, like the internet? I know that the I in IETF is not a strict limiter of scope, but still it would be nice if drafts would have a viable path of being implemented over the internet... That said, well possible that the current state does not merit use-over-the-internet yet and so maybe starting with an L2/L4 solution might be considered a safety back-stop?
>>> 
>>> Sebastian,
>>> 
>>> There's no reason to believe that Congestion Signaling isn't of
>>> interest to use on an internet (lower case 'i' is explicit here).
>> 
>> [SM] Well, in my mind that still would keep this out of scope for the capital I ETCopy of Copy of Enfabrica-SiPandaF,

[SM2] Not sure how that 'Copy of Copy of Enfabrica-SiPanda' string ended up in that paragraph, I do not recall writing that... either autocompletion run wild or some other issue.


>> I am interested in improving end to end congestion signalling over the Internet so I desire these signals to sink and source at my endpoints... Again, I understand that my position is in the rough regarding what the IETF should care about.
>> 
>>> This is almost certainly beneficial for 6G for instance which is an
>>> internet composed for various link layer technologies.
>>> Neither is this
>>> the only protocol of this nature there are and will be others-- from
>>> an IETF POV I believe we want a extensible protocol solution that
>>> benefits multiple use cases and works in different environments.
>> 
>> [SM] That way lies madness IMHO. Getting enough routers/switches support one signal and hence make it useful is already almost a Sisyphus task expecting them to support multiple signals selected individually per packet seems like a recipe of never getting this to work end to end (which is my motivator here). If we do not know which single signal to use here, I guess keeping this private and do more research seems like a productive way forward.
> 
> Sebastian,
> 
> I would agree with that if this was the first protocol ever trying to
> do something like this, but it's not. IOAM is already a published RFC.

[SM2] There are a number of points that are against IOAM being a suitable encapsulation here:

https://datatracker.ietf.org/doc/html/rfc9378 :
"IOAM is focused on "limited domains", as defined in [RFC8799]. IOAM is not targeted for a deployment on the global Internet."

IMHO that already disqualifies IOAM here as the goal needs to be a fully end to end method having the Internet as its scope.

Sure one will be able to define a specific type of message that will only be added/removed by end points (aka Edge-to-Edge (E2E)), but now our intermediate nodes will need to check for a specific IOAM record type (not sure whether that is the proper IOAM nomenclature, but you get my idea, I hope). Except as currently defined E2E does not allow intermediate nodes to modify the information... so E2E is out and we are down to IOAM Proof of Transit Type 0... this is pretty complicated and obscure already.

But let's run with it, OK, IOAM Proof of Transit Type 0 it is:
4.5.1. IOAM Proof of Transit Type 0
IOAM Proof of Transit Option of IOAM POT-Type 0:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Namespace-ID |IOAM POT-Type=0|R R R R R R R R|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+
| PktID | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ P
| PktID (contd) | O
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ T
| Cumulative | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| Cumulative (contd) | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+

Namespace-ID:16-bit identifier of an IOAM-Namespace (see Section 4.3 above).IOAM POT-Type:8-bit identifier of a particular POT variant that specifies the POT data that is included (see Section 4.5 above). For this case here, IOAM POT-Type is set to the value 0.Bit 0-7:Undefined (see Section 4.5 above).PktID:64-bit packet identifier.Cumulative:64-bit Cumulative that is updated at specific nodes by processing per packet PktID field and configured parameters.Note: Larger or smaller sizes of "PktID" and "Cumulative" data are feasible and could be required for certain deployments, e.g., in case of space constraints in the encapsulation protocols used. Future documents could introduce different sizes of data for "Proof of Transit".


That comes in at a hefty 5 * 4 = 20 octets/bytes already, 12/20  of which carry no useful information for the congestion use case...
Sure it argues, one could define a smaller version, maybe getting rid of the PktID completely and reducing the Cumulative data to something smaller, but then we already likely down to 8 bytes out of which only 4 carry the relevant information (assuming we can dump PktID completely)

I also ignore that we still need a way to actually add this 20bytes to our data packets, which in turn will result in even more overhead. E.g. IPv6 Hop-byHop adds 2 bytes, but comes with an 8 byte granularity, so we need to add 4. But now we are at 16/24 or 2/3s pure overhead already...*


If we look how to fill the Cumulative information we get e.g.:

4.4.2.12. Buffer Occupancy
The "buffer occupancy" field is a 4-octet unsigned integer field. This field indicates the current status of the occupancy of the common buffer pool used by a set of queues. The units of this field are implementation specific. Hence, the units are interpreted within the context of an IOAM-Namespace and/or node identifier if used. The authors acknowledge that, in some operational cases, there is a need for the units to be consistent across a packet path through the network; hence, it is recommended for implementations to use standard units, such as bytes.
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| buffer occupancy |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Which in turn states its own unsuitability in its current form explicitly "r pool used by a set of queues. The units of this field are implementation specific. Hence, the units are interpreted within the context of an IOAM-Namespace and/or node identifier if used. The authors acknowledge that, in some operational cases, there is a need for the units to be consistent across a packet path through the network; hence, it is recommended for implementations to use standard units, such as bytes." Indeed, but a spec needs to make this clear and define that standard unit to use....

All of this then points to IOAM's high complexity, surely warranted for IOAM's intended use-cases but for a full end/network to end signal this complexity is not required.



*) According to Arslan, Serhat, and Nick McKeown. “Switches Know the Exact Amount of Congestion.” In Proceedings of the 2019 Workshop on Buffer Sizing, 1–6, 2019. 4 bits might already be enough to meaningfully encode buffer occupancy, so if we aim for 16 to 32 bits of actual information we should have plenty of bits...


> The problem with IOAM, according to the draft, is "they all commonly
> stack up multiple per-switch telemetry data per-hop in the path of a
> packet". I don't see why IOAM can't be adapted to contain fixed
> length CSIG data without requiring be packets.

[SM2] This is already handled in the ' IOAM Proof of Transit Type 0' field, the facrt that you did not notice that supports my argument that IOAM is too complicated an RFC for this simple use-case, no?


> If that constraint is
> removed then the only remaining argument against IOAM seems to be that
> it's easier for hardware to handle L3 rather than L2 in hardware.

[SM2] No, it is still way to complicated (offering options all intermediary nodes will need to check before setting values) and way way to overheady...

> I don't believe there is currently consensus that that is generally
> true.

[SM2] Unclear what the consensus is. IMHO is is not really a question of consensus, if we want this to be a first class end 2 end signal, L2 is simply not an option independent of our wishes.

> And, if this is why IOAM "has such a sparse (or no) support from
> switch vendors" as Jai claims then it seems like this is maybe
> something that should be discussed instead of just arbitrarily
> dismissing IOAM. Why exactly is IOAM in HW such a problem and can it
> be fixed? (a quick look at ippm archives didn't reveal any
> discussions).

[SM2] As shown above IOAM seems to be one of these everything and the kitchensink solution, probably great for its intended use cases, but efficient end/network to end signalling IM;HO is not one of these use cases. Sure it could be bent into shape to support that use case, but I would rather see a meaner and leaner signal design for the congestion information aggregation along a path use case. If I had control, I would propose a new IPv6 header type (with a new next header number) mutually exclusive with hop by hop (so that confirming the next header number is the only check required before updating data at a known offset). There is value in simplicity... (I guess this scheme will fail due to firewalls only permitting a limited set of next header values...)


Regards
	Sebastian


> 
> Tom
> 
>> 
>> Regards
>>        Sebastian
>> 
>> 
>>> 
>>> Tom
>>> 
>>>> 
>>>> Regards
>>>>       Sebastian
>>>> 
>>>>> 
>>>>> On Sat, Feb 10, 2024 at 7:42 AM Tom Herbert <tom=[e-mail]> wrote:
>>>>> On Fri, Feb 9, 2024 at 10:53 PM Nandita Dukkipati <[e-mail]> wrote:
>>>>>> 
>>>>>> Hi Tom,
>>>>>> 
>>>>>> We updated the draft, correcting some nit errata, and to not let the draft expire. It's not discussed in any other mailing lists.
>>>>> 
>>>>> Thanks Nandita.
>>>>> 
>>>>> I still have fundamental concerns about the protocol layering in this
>>>>> draft, please see my previous comments on that. The draft defines a
>>>>> protocol for end-to-end network to host signaling and IMO, such a
>>>>> protocol belongs in the network layer but the draft puts the protocol
>>>>> in L2 and L4 and seems to avoid L3 without explanation. IOAM defines a
>>>>> very similar method of signaling and RFC9486 is a good model for
>>>>> network layer protocol that provides network to host signaling.
>>>>> 
>>>>> Tom
>>>>> 
>>>>>> 
>>>>>> Nandita
>>>>>> 
>>>>>> On Thu, Feb 8, 2024 at 3:53 PM Tom Herbert <[e-mail]> wrote:
>>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I noticed there is now an -01 version of the draft posted on Feb. 2.
>>>>>>> Is this draft being discussed on some other list?
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Tom
>>>>>>> 
>>>>>>> On Sat, Sep 9, 2023 at 9:09 AM Tom Herbert <[e-mail]> wrote:
>>>>>>>> 
>>>>>>>> Hi, thanks for draft!
>>>>>>>> 
>>>>>>>> The first thing that stands out to me is the carrier of the new packet headers. In the forward path it would be in L2 and in reflection it would be L4. As the draft describes, this would entail having to support the protocol in multiple L2 and multiple L4 protocols-- that's going to be a pretty big lift! Also, L2 is not really an end-to-end protocol (would legacy switches in the path also forward the header)l?).
>>>>>>>> 
>>>>>>>> The signaling being described in the draft is network layer information, and hence IMO should be conveyed in network layer headers. That's is L3 which conveniently is the average of L2+L4 :-)
>>>>>>>> 
>>>>>>>> IMO, the proper carrier of the signal data is Hop-by-Hop Options. This is end-to-end and allows modification of data in-flight. The typical concern with Hop-by-Hop Options is high drop rates on the Internet, however in this case the protocol is explicitly confined to a limited domain so I don't see that as a blocking issue for this use case.
>>>>>>>> 
>>>>>>>> The information being carried seems very similar to that of IOAM (IOAM uses Hop-by-Hop Options and supports reflection). I suppose the differences are that this protocol is meant to be consumed by the transport Layer and the data is a condensed summary of path characteristics. IOAM seems pretty extensible, so maybe it could be adapted to carry the signals of this draft?
>>>>>>>> 
>>>>>>>> A related proposal might be FAST draft-herbert-fast. Where the CSIG is network to host signaling, FAST is host to network signaling for the purposes of requesting network services. These might be complementary and options for both may be in the same packet. FAST also uses reflection, so we might be able to leverage some common implementation at a destination.
>>>>>>>> 
>>>>>>>> Tom
>>>>>>>> 
>>>>>>>> On Fri, Sep 8, 2023, 7:43 PM Abhiram Ravi <abhiramr=[e-mail]> wrote:
>>>>>>>>> 
>>>>>>>>> Hi IPPM folks,
>>>>>>>>> 
>>>>>>>>> I am pleased to announce the publication of a new internet draft, Congestion Signaling (CSIG): https://datatracker.ietf.org/doc/draft-ravi-ippm-csig/
>>>>>>>>> 
>>>>>>>>> CSIG is a new end-to-end packet header mechanism for in-band signaling that is simple, efficient, deployable, and grounded in concrete use cases of congestion control, traffic management, and network debuggability. We believe that CSIG is an important new protocol that builds on top of existing in-band network telemetry protocols.
>>>>>>>>> 
>>>>>>>>> We encourage you to read the CSIG draft and provide your feedback and comments. We have also cc'd the TSVWG, CCWG, and ICCRG mailing lists, as we believe that this work may be of interest to their members as well.
>>>>>>>>> 
>>>>>>>>> Thank you for your time and consideration.
>>>>>>>>> 
>>>>>>>>> Sincerely,
>>>>>>>>> Abhiram Ravi
>>>>>>>>> On behalf of the CSIG authors
>>>>> 
>>>>> _______________________________________________
>>>>> iccrg mailing list
>>>>> [e-mail]
>>>>> https://mailman.irtf.org/mailman/listinfo/iccrg
>>>>> 
>>>>> 
>>>>> --
>>>>> Thanks,
>>>>> --MM--
>>>>> Evil is defined by mortals who think they know "The Truth" and use force to apply it to others.