Re: [ippm] [tsvwg] [iccrg] New Internet Draft: Congestion Signaling (CSIG)

[Tom Herbert <[e-mail]>]

On Wed, Sep 13, 2023 at 5:08 PM Abhiram Ravi <[e-mail]> wrote:
>
> Thank you for your comments. Responses inline.
>
> > Tom: The signaling being described in the draft is network layer information, and hence IMO should be conveyed in network layer headers. That's is L3 which conveniently is the average of L2+L4 :-)
> > Rachel:  Implementing in L2 may be difficult to be used in e2e transport.
> > Hang: I agree L2 may not be the best choice to carry the congestion signaling end-to-end and more bits are needed.
>
> The choice of using L2+L4 is an intentional and important design choice, one that is driven by need, for practicality and ease of deployment in data center networks. We acknowledge that previous approaches have attempted to extend L3 or L4, but these approaches do not meet the requirements for deployability in a heterogeneous network, including interoperability and backward compatibility. CSIG targets specific yet important use cases while meeting these requirements.

Hi Abhiram,

I'm not seeing how pushing this into layer 2 is a simplification for
deployment. When you say that you want to deploy in a heterogeneous
network, is this a network with different L2s but still IP over it, or
is it a network where there are non-IP links? If it's the former, then
if you put the information in the network layer then the fact that
it's a "heterogeneous network" is irrelevant-- there is immense value
in leveraging a common network layer across heterogeneous link layer
technologies, that's fundamental to the architecture of the Internet
:-)

>
> Section 7.1 of the draft explicitly addresses why such a design was chosen.
>
> The key reasons are:
>
> Simplicity of device implementations of CSIG (both at NICs and switches), given the location of the CSIG-tag in the packet header is not affected by any header at layer 3 and above. CSIG-tag being the last tag before the ethertype also makes it simple for ASICs to implement it inside the L2 header.
>

I'm skeptical that this is any easier to implement than Hop-by-Hop
Options. HBH is at a fixed location in packets, this can be
implemented as the first option for cut through. A programmable
device, switch or NIC, would have no problem with this.

> Support for delivering congestion signals even in the presence of in-network tunneling and encryption, which are commonly used in data centers, especially for Cloud traffic and in Traffic engineered domains. CSIG tags are carried through Layer 3 tunnels e.g., IP-in-IP, VxLAN, Geneve, at a fixed offset in the packet header. This avoids the need to copy and relocate CSIG tags across inner / outer headers during encapsulation and decapsulation of packets, which is a complex hardware operation and would be necessary if implemented instead at layers 3 or higher.

How does IP-IP carry CSIG tags if they're not part of the network
layer. It seems like it would only work if the L2 packet is tunneled.

>
> > Tom: The signaling being described in the draft is network layer information, and hence IMO should be conveyed in network layer headers.
>
> The information carried in L2 CSIG-tags are device signals or port signals from the bottleneck, and should not be viewed as network-layer signals. Although the bottleneck signal is also a property of the packet path in aggregate, the signal carried on the tag has a scope of one hop i.e., the measurement on one specific link or device on the path. The tag gets conditionally swapped on each device in a compare-and-replace manner as described in Section 4.3

RIght, what you're describing is an end to end protocol that can cross
and be forwarded across multiple links and the data can be consumed or
modified in flight by intermediate nodes. I think that pretty much
fits the definition of network layer information.

>
> Reflection is a property of the transport as described in Section 4.2, and hence CSIG reflection is implemented at layer 4 and is specific to the transport. This allows a clean separation of the mechanism used to collect signals from the network from the mechanism used for leveraging those signals at the transport.

"Clean separation" is relative. This protocol is being implemented in
both L2 and L4. I suppose ECN has similar characteristics but at least
that's L3 and L4 so just one layer difference.

>
> > Tom: Also, L2 is not really an end-to-end protocol (would legacy switches in the path also forward the header)
>
> Legacy devices can not only forward the packet, but also participate in the CSIG protocol via a software-assisted approach by treating CSIG-tags as VLAN-tags and leveraging VLAN resources to perform CSIG updates. This L2 approach is the only solution we’ve found which allows immediate deployment for certain signals with legacy devices in data centers,

I  don't understand how that could be the only possible solution. As
already mentioned, IOAM is already providing similar functionality,
and AFAIK, it works perfectly well with legacy devices. Please provide
more explanation why this protocol MUST be L2 and can't be in the
network layer.

Tom


Tom





>
> > Tom: As the draft describes, this would entail having to support the protocol in multiple L2 and multiple L4 protocols
>
> Section 4.1 lists how CSIG-tag operates with each relevant L2 encapsulation and Section 4.2 addresses implementation of reflection in relevant transports.
>
> > Tom: IMO, the proper carrier of the signal data is Hop-by-Hop Options. IOAM seems pretty extensible, so maybe it could be adapted to carry the signals of this draft?
> > Rachel: Of course it can work well in limited domain, like DC or HPC clusters. However, I also look for some solutions that could be able to go through internet
>
> CSIG is targeted at specific use cases in limited domains, primarily data centers, for congestion control, traffic management and network debuggability. The requirements for such use cases vary significantly from those of internet deployments, which is not in scope for CSIG.
> IOAM and hop-by-hop options in general are tuned for a broader set of use cases and do not meet all the requirements for the specific use cases that CSIG addresses, including interop with legacy devices and in-network tunneling, due to the placement of the header.  Section 2 speaks to these requirements.
>
> > Rachel: For L2 solution, it should be developed together with IEEE 802.1.
>
> Acknowledging that IEEE will be involved as well.
>
> > Sebastian: What happens if bottlenecks hand out immediate capacity/rate estimates and all greedy flows start increasing their cwin really fast?
>
> CSIG supports both min(ABW) and min(ABW/C). Using min(ABW/C), the each flow can increase its rate in proportion to the % available bandwidth, so that in aggregate the flows respect the available bottleneck bandwidth with reduced risk of overshoot. See Section 8.1.2.
>
> > Ingemar: CSIG sounds to me like something that belongs more in ICCRG
>
> CSIG is applicable to use cases beyond Congestion control (which is an important one), such as traffic management and network debugging. Section 8 describes the use cases in more detail.
>
> > Tom: A related proposal might be FAST draft-herbert-fast. These might be complementary and options for both may be in the same packet
> > Hang: We call it Advanced ECN. See draft-shi-ccwg-advanced-ecn
> > Ingemar: L4S is recently standardised and it is definitely gaining traction also in 3GPP
>
> Thank you for the pointers. We agree that these proposals are all complementary to CSIG.
>
>
>
> On Wed, Sep 13, 2023 at 5:20 AM <[e-mail]> wrote:
>>
>> Hi Ingemar, hi Sebastian,
>>
>> Skimming over the draft only, isn't CSIG about Ethernet-domains, while L4S is E2E?
>>
>> Regards,
>>
>> Ruediger
>>
>> -----Ursprüngliche Nachricht-----
>> Von: tsvwg <[e-mail]> Im Auftrag von Sebastian Moeller
>> Gesendet: Mittwoch, 13. September 2023 13:42
>> An: Ingemar Johansson S <ingemar.s.johansson=[e-mail]>
>> Cc: Vidhi Goel <vidhi_goel=[e-mail]>; Shihang(Vincent) <shihang9=[e-mail]>; tsvwg <[e-mail]>; Jai Kumar <[e-mail]>; IETF IPPM WG <[e-mail]>; Tom Herbert <tom=[e-mail]>; [e-mail]; Abhiram Ravi <abhiramr=[e-mail]>; Nandita Dukkipati <[e-mail]>; [e-mail]; Huangyihong (Rachel) <rachel.huang=[e-mail]>; Naoshad Mehta <[e-mail]>
>> Betreff: Re: [tsvwg] [iccrg] New Internet Draft: Congestion Signaling (CSIG)
>>
>> Hi Ingemar,
>>
>>
>> > On Sep 13, 2023, at 12:30, Ingemar Johansson S <ingemar.s.johansson=[e-mail]> wrote:
>> >
>> > Hi
>> >
>> > I agree with Vihdi
>> >
>> > L4S is recently standardised
>>
>>         [SM] In experimental track, the goal is currently to test whether it can/should be deployed at scale....
>>
>> > and it is definitely gaining traction also in 3GPP. We have an echo system that is looking forward to having L4S widely deployed.
>> > Still, the congestion control aspects are not fully explored yet. One interesting topic is if L4S allows to more safely deviate from additive increase to make congestion control algorithms more quickly converge to higher link capacity. There are a number of study topics around L4S congestion control that are listed in e.g the TCP Prague draft.
>> >
>> > I cannot dictate what others should do with their time and money but personally I'd prefer that the IETF explores L4S and its possibilities and downsides before jumping on the next idea.
>>
>>         [SM] L4S can be described as taking the ideas behind DCTCP and making them fit for use over the internet*. Yet the signaling discussed here is to be used in e.g. data center contexts where DCTCP is already used and found lacking compared to newer methods operating on richer congestion information (HPCC, Swift, Poseidon, ...).
>>         Given that L4S essentially uses a multi-packet signal(**) to report the "queue filling state" that is then stochastically distributed over all concurrent flows, it seems obvious to me that reconstructing a reliable estimate of on-path queueing will take some time and averaging for each individual flow, I would guess that in some environments this delay simply is too costly.
>>         So L4S and CSIG seem complementary and in no way mutually exclusive.
>>
>>
>> Regards
>>         Sebastian
>>
>>
>>
>> *) I will not further discuss whether that is achieved or not as it seems irrelevant here.
>> **) In essence transmission of congestion state via a 1-bit serial channel, clocked at the (variable***) packet rate at the bottleneck.
>> ***) as packets are not of uniform size
>>
>> >
>> > CSIG sounds to me like something that belongs more in ICCRG or ?
>> >
>> > /Ingemar
>> >
>> > From: tsvwg <[e-mail]> On Behalf Of Vidhi Goel
>> > Sent: Wednesday, 13 September 2023 00:59
>> > To: Shihang(Vincent) <shihang9=[e-mail]>
>> > Cc: Huangyihong (Rachel) <rachel.huang=[e-mail]>; Tom Herbert <tom=[e-mail]>; Abhiram Ravi <abhiramr=[e-mail]>; IETF IPPM WG <[e-mail]>; tsvwg <[e-mail]>; [e-mail]; [e-mail]; Nandita Dukkipati <[e-mail]>; Naoshad Mehta <[e-mail]>; Jai Kumar <[e-mail]>
>> > Subject: Re: [tsvwg] [iccrg] New Internet Draft: Congestion Signaling (CSIG)
>> >
>> > Not sure why we are coming up with so many new techniques when ECN just works fine.
>> > ECN is a 2 bit field (not 1 bit) and seems to be sufficient to indicate extent of congestion by marking it per packet. Adding more complexity to any layer whether it is L2 or L3 doesn’t work well in deployments. Our goal should be to simplify things and only add new headers if absolutely necessary.
>> >
>> > Vidhi
>> >
>> >
>> > On Sep 12, 2023, at 3:12 AM, Shihang(Vincent) <shihang9=[e-mail]> wrote:
>> >
>> > Hi,
>> > I agree L2 may not be the best choice to carry the congestion signaling end-to-end and more bits are needed. We have submitted a draft to carry the multi-bits congestion signaling in L3. We call it Advanced ECN. See https://datatracker.ietf.org/doc/draft-shi-ccwg-advanced-ecn/.
>> >
>> > Thanks,
>> > Hang
>> >
>> > From: CCWG <[e-mail]> On Behalf Of Huangyihong (Rachel)
>> > Sent: Tuesday, September 12, 2023 5:41 PM
>> > To: Tom Herbert <tom=[e-mail]>; Abhiram Ravi <abhiramr=[e-mail]>
>> > Cc: IETF IPPM WG <[e-mail]>; tsvwg <[e-mail]>; [e-mail]; [e-mail]; Nandita Dukkipati <[e-mail]>; Naoshad Mehta <[e-mail]>; Jai Kumar <[e-mail]>
>> > Subject: Re: [CCWG] [iccrg] [tsvwg] New Internet Draft: Congestion Signaling (CSIG)
>> >
>> > Hi,
>> >
>> > I also have the same feeling. Implementing in L2 may be difficult to be used in e2e transport. Of course it can work well in limited domain, like DC or HPC clusters. However, I also look for some solutions that could be able to go through internet. We have submitted a draft to describe the transport challenges. See https://datatracker.ietf.org/doc/html/draft-huang-tsvwg-transport-challenges.
>> >
>> > I share the same opinion that the congestion signal is useful and current 1-bit ECN solution is not fully sufficient. But I also feel like the more straight way is to extend L3, or l4, like update IOAM, to carry the information. For L2 solution, it should be developed together with IEEE 802.1.
>> >
>> > BR，
>> > Rachel
>> >
>> > 发件人: iccrg <[e-mail]> 代表 Tom Herbert
>> > 发送时间: 2023年9月10日 0:10
>> > 收件人: Abhiram Ravi <abhiramr=[e-mail]>
>> > 抄送: IETF IPPM WG <[e-mail]>; tsvwg <[e-mail]>; [e-mail]; [e-mail]; Nandita Dukkipati <[e-mail]>; Naoshad Mehta <[e-mail]>; Jai Kumar <[e-mail]>
>> > 主题: Re: [iccrg] [tsvwg] New Internet Draft: Congestion Signaling (CSIG)
>> >
>> > Hi, thanks for draft!
>> >
>> > The first thing that stands out to me is the carrier of the new packet headers. In the forward path it would be in L2 and in reflection it would be L4. As the draft describes, this would entail having to support the protocol in multiple L2 and multiple L4 protocols-- that's going to be a pretty big lift! Also, L2 is not really an end-to-end protocol (would legacy switches in the path also forward the header)l?).
>> >
>> > The signaling being described in the draft is network layer information, and hence IMO should be conveyed in network layer headers. That's is L3 which conveniently is the average of L2+L4 :-)
>> >
>> > IMO, the proper carrier of the signal data is Hop-by-Hop Options. This is end-to-end and allows modification of data in-flight. The typical concern with Hop-by-Hop Options is high drop rates on the Internet, however in this case the protocol is explicitly confined to a limited domain so I don't see that as a blocking issue for this use case.
>> >
>> > The information being carried seems very similar to that of IOAM (IOAM uses Hop-by-Hop Options and supports reflection). I suppose the differences are that this protocol is meant to be consumed by the transport Layer and the data is a condensed summary of path characteristics. IOAM seems pretty extensible, so maybe it could be adapted to carry the signals of this draft?
>> >
>> > A related proposal might be FAST draft-herbert-fast. Where the CSIG is network to host signaling, FAST is host to network signaling for the purposes of requesting network services. These might be complementary and options for both may be in the same packet. FAST also uses reflection, so we might be able to leverage some common implementation at a destination.
>> >
>> > Tom
>> >
>> > On Fri, Sep 8, 2023, 7:43 PM Abhiram Ravi <abhiramr=[e-mail]> wrote:
>> > Hi IPPM folks,
>> >
>> > I am pleased to announce the publication of a new internet draft, Congestion Signaling (CSIG): https://datatracker.ietf.org/doc/draft-ravi-ippm-csig/
>> >
>> > CSIG is a new end-to-end packet header mechanism for in-band signaling that is simple, efficient, deployable, and grounded in concrete use cases of congestion control, traffic management, and network debuggability. We believe that CSIG is an important new protocol that builds on top of existing in-band network telemetry protocols.
>> >
>> > We encourage you to read the CSIG draft and provide your feedback and comments. We have also cc'd the TSVWG, CCWG, and ICCRG mailing lists, as we believe that this work may be of interest to their members as well.
>> >
>> > Thank you for your time and consideration.
>> >
>> > Sincerely,
>> > Abhiram Ravi
>> > On behalf of the CSIG authors
>>
>
>
> --
> Abhiram Ravi