Re: [iccrg] [tsvwg] [CCWG] New Internet Draft: Congestion Signaling (CSIG)

["Rodney W. Grimes" <[e-mail]>]

Hesham,
	While I found a read of some of the IEEE 802.1Qau papers
interesting, I have NOT been able to find anyone who has real
world deployment data.  In a datacenter or otherwise.

	Do you know of any or does anyone here have data to
point at?

Regards,
Rod Grimes

> Hi Abhiram,
> IEEE 802.1 had a data center bridging TG which developed sometime ago the
> spec for quantized congestion control (QCN). The spec is IEEE 802.1Qau
> which is now part of 802.1Q.
> 
> QCN provide a means for a bridge to notify a source of congestion causing
> the source to reduce the flow rate.
> 
> QCN is independent of upper layer protocol. It coexists with TCP  (there is
> a nested control loops of QCN and TCP flow control).
> 
> QCN only works for unicast traffic, and operates over a domain where all
> bridges and end stations support it.
> 
> QCN does not require per flow state or per flow queuing in bridge.
> 
> Over the years there have been proposals to enhance it, but the basic idea
> stayed the same.
> 
> How you compare your congestion signaling proposal with QCN. One obvious
> difference is that yours is end-2end while QCN is not. But is end-end
> congestion control the best way to go?
> 
> Comments?
> 
> Hesham
> PS. I removed IPPM but kept other WG who may have an opinion on this issue.
> 
> 
> On Wed, Sep 13, 2023, 5:09 PM Abhiram Ravi <abhiramr=
> [e-mail]> wrote:
> 
> > Thank you for your comments. Responses inline.
> >
> > > Tom: The signaling being described in the draft is network layer
> > information, and hence IMO should be conveyed in network layer headers.
> > That's is L3 which conveniently is the average of L2+L4 :-)
> > > Rachel:  Implementing in L2 may be difficult to be used in e2e transport.
> > > Hang: I agree L2 may not be the best choice to carry the congestion
> > signaling end-to-end and more bits are needed.
> >
> > The choice of using L2+L4 is an intentional and important design choice,
> > one that is driven by need, for practicality and ease of deployment in data
> > center networks. We acknowledge that previous approaches have attempted to
> > extend L3 or L4, but these approaches do not meet the requirements for
> > deployability in a heterogeneous network, including interoperability and
> > backward compatibility. CSIG targets specific yet important use cases while
> > meeting these requirements.
> >
> > Section 7.1
> > <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-choice-of-layer-2>
> > of the draft explicitly addresses why such a design was chosen.
> >
> > The key reasons are:
> >
> >    -
> >
> >    Simplicity of device implementations of CSIG (both at NICs and
> >    switches), given the location of the CSIG-tag in the packet header is not
> >    affected by any header at layer 3 and above. CSIG-tag being the last tag
> >    before the ethertype also makes it simple for ASICs to implement it inside
> >    the L2 header.
> >    -
> >
> >    Support for delivering congestion signals even in the presence of
> >    in-network tunneling and encryption, which are commonly used in data
> >    centers, especially for Cloud traffic and in Traffic engineered domains.
> >    CSIG tags are carried through Layer 3 tunnels e.g., IP-in-IP, VxLAN,
> >    Geneve, at a fixed offset in the packet header. This avoids the need to
> >    copy and relocate CSIG tags across inner / outer headers during
> >    encapsulation and decapsulation of packets, which is a complex hardware
> >    operation and would be necessary if implemented instead at layers 3 or
> >    higher.
> >
> > > Tom: The signaling being described in the draft is network layer
> > information, and hence IMO should be conveyed in network layer headers.
> >
> > The information carried in L2 CSIG-tags are device signals or port signals
> > from the bottleneck, and should not be viewed as network-layer signals.
> > Although the bottleneck signal is also a property of the packet path in
> > aggregate, the signal carried on the tag has a scope of one hop i.e., the
> > measurement on one specific link or device on the path. The tag gets
> > conditionally swapped on each device in a compare-and-replace manner as
> > described in Section 4.3
> > <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-csig-operation-life-of-a-pa>
> >
> > Reflection is a property of the transport as described in Section 4.2, and
> > hence CSIG reflection is implemented at layer 4 and is specific to the
> > transport. This allows a clean separation of the mechanism used to collect
> > signals from the network from the mechanism used for leveraging those
> > signals at the transport.
> >
> > > Tom: Also, L2 is not really an end-to-end protocol (would legacy
> > switches in the path also forward the header)
> >
> > Legacy devices can not only forward the packet, but also participate in
> > the CSIG protocol via a software-assisted approach by treating CSIG-tags as
> > VLAN-tags and leveraging VLAN resources to perform CSIG updates. This L2
> > approach is the only solution we?ve found which allows immediate deployment
> > for certain signals with legacy devices in data centers, while native
> > support for CSIG is easily achievable in future devices. CSIG stripping can
> > also be exercised for interop when needed - see Section 6
> > <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-incremental-deployment-of-c>
> > for interoperability.
> >
> > > Tom: As the draft describes, this would entail having to support the
> > protocol in multiple L2 and multiple L4 protocols
> >
> > Section 4.1
> > <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-csig-tag-header-format>
> > lists how CSIG-tag operates with each relevant L2 encapsulation and Section
> > 4.2
> > <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-csig-reflection-header-form>
> > addresses implementation of reflection in relevant transports.
> >
> > > Tom: IMO, the proper carrier of the signal data is Hop-by-Hop Options.
> > IOAM seems pretty extensible, so maybe it could be adapted to carry the
> > signals of this draft?
> > > Rachel: Of course it can work well in limited domain, like DC or HPC
> > clusters. However, I also look for some solutions that could be able to go
> > through internet
> >
> > CSIG is targeted at specific use cases in limited domains, primarily data
> > centers, for congestion control, traffic management and network
> > debuggability. The requirements for such use cases vary significantly from
> > those of internet deployments, which is not in scope for CSIG.
> > IOAM and hop-by-hop options in general are tuned for a broader set of use
> > cases and do not meet all the requirements for the specific use cases that
> > CSIG addresses, including interop with legacy devices and in-network
> > tunneling, due to the placement of the header.  Section 2
> > <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-design-principles>
> > speaks to these requirements.
> >
> > > Rachel: For L2 solution, it should be developed together with IEEE 802.1.
> >
> > Acknowledging that IEEE will be involved as well.
> >
> > > Sebastian: What happens if bottlenecks hand out immediate capacity/rate
> > estimates and all greedy flows start increasing their cwin really fast?
> >
> > CSIG supports both min(ABW) and min(ABW/C). Using min(ABW/C), the each
> > flow can increase its rate in proportion to the % available bandwidth, so
> > that in aggregate the flows respect the available bottleneck bandwidth with
> > reduced risk of overshoot. See Section 8.1.2
> > <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-using-maximum-link-utilizat>
> > .
> >
> > > Ingemar: CSIG sounds to me like something that belongs more in ICCRG
> >
> > CSIG is applicable to use cases beyond Congestion control (which is an
> > important one), such as traffic management and network debugging. Section
> > 8
> > <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-use-cases-defined-by-bottle>
> > describes the use cases in more detail.
> >
> > > Tom: A related proposal might be FAST draft-herbert-fast. These might be
> > complementary and options for both may be in the same packet
> > > Hang: We call it Advanced ECN. See draft-shi-ccwg-advanced-ecn
> > > Ingemar: L4S is recently standardised and it is definitely gaining
> > traction also in 3GPP
> >
> > Thank you for the pointers. We agree that these proposals are all
> > complementary to CSIG.
> >
> >
> > On Wed, Sep 13, 2023 at 5:20?AM <[e-mail]> wrote:
> >
> >> Hi Ingemar, hi Sebastian,
> >>
> >> Skimming over the draft only, isn't CSIG about Ethernet-domains, while
> >> L4S is E2E?
> >>
> >> Regards,
> >>
> >> Ruediger
> >>
> >> -----Urspr?ngliche Nachricht-----
> >> Von: tsvwg <[e-mail]> Im Auftrag von Sebastian Moeller
> >> Gesendet: Mittwoch, 13. September 2023 13:42
> >> An: Ingemar Johansson S <ingemar.s.johansson=
> >> [e-mail]>
> >> Cc: Vidhi Goel <vidhi_goel=[e-mail]>; Shihang(Vincent)
> >> <shihang9=[e-mail]>; tsvwg <[e-mail]>; Jai
> >> Kumar <[e-mail]>; IETF IPPM WG <[e-mail]>; Tom
> >> Herbert <tom=[e-mail]>; [e-mail]; Abhiram
> >> Ravi <abhiramr=[e-mail]>; Nandita Dukkipati <
> >> [e-mail]>; [e-mail]; Huangyihong (Rachel) <rachel.huang=
> >> [e-mail]>; Naoshad Mehta <[e-mail]>
> >> Betreff: Re: [tsvwg] [iccrg] New Internet Draft: Congestion Signaling
> >> (CSIG)
> >>
> >> Hi Ingemar,
> >>
> >>
> >> > On Sep 13, 2023, at 12:30, Ingemar Johansson S <ingemar.s.johansson=
> >> [e-mail]> wrote:
> >> >
> >> > Hi
> >> >
> >> > I agree with Vihdi
> >> >
> >> > L4S is recently standardised
> >>
> >>         [SM] In experimental track, the goal is currently to test whether
> >> it can/should be deployed at scale....
> >>
> >> > and it is definitely gaining traction also in 3GPP. We have an echo
> >> system that is looking forward to having L4S widely deployed.
> >> > Still, the congestion control aspects are not fully explored yet. One
> >> interesting topic is if L4S allows to more safely deviate from additive
> >> increase to make congestion control algorithms more quickly converge to
> >> higher link capacity. There are a number of study topics around L4S
> >> congestion control that are listed in e.g the TCP Prague draft.
> >> >
> >> > I cannot dictate what others should do with their time and money but
> >> personally I'd prefer that the IETF explores L4S and its possibilities and
> >> downsides before jumping on the next idea.
> >>
> >>         [SM] L4S can be described as taking the ideas behind DCTCP and
> >> making them fit for use over the internet*. Yet the signaling discussed
> >> here is to be used in e.g. data center contexts where DCTCP is already used
> >> and found lacking compared to newer methods operating on richer congestion
> >> information (HPCC, Swift, Poseidon, ...).
> >>         Given that L4S essentially uses a multi-packet signal(**) to
> >> report the "queue filling state" that is then stochastically distributed
> >> over all concurrent flows, it seems obvious to me that reconstructing a
> >> reliable estimate of on-path queueing will take some time and averaging for
> >> each individual flow, I would guess that in some environments this delay
> >> simply is too costly.
> >>         So L4S and CSIG seem complementary and in no way mutually
> >> exclusive.
> >>
> >>
> >> Regards
> >>         Sebastian
> >>
> >>
> >>
> >> *) I will not further discuss whether that is achieved or not as it seems
> >> irrelevant here.
> >> **) In essence transmission of congestion state via a 1-bit serial
> >> channel, clocked at the (variable***) packet rate at the bottleneck.
> >> ***) as packets are not of uniform size
> >>
> >> >
> >> > CSIG sounds to me like something that belongs more in ICCRG or ?
> >> >
> >> > /Ingemar
> >> >
> >> > From: tsvwg <[e-mail]> On Behalf Of Vidhi Goel
> >> > Sent: Wednesday, 13 September 2023 00:59
> >> > To: Shihang(Vincent) <shihang9=[e-mail]>
> >> > Cc: Huangyihong (Rachel) <rachel.huang=[e-mail]>;
> >> Tom Herbert <tom=[e-mail]>; Abhiram Ravi
> >> <abhiramr=[e-mail]>; IETF IPPM WG <[e-mail]>;
> >> tsvwg <[e-mail]>; [e-mail]; [e-mail]; Nandita Dukkipati
> >> <[e-mail]>; Naoshad Mehta <[e-mail]>; Jai Kumar <
> >> [e-mail]>
> >> > Subject: Re: [tsvwg] [iccrg] New Internet Draft: Congestion Signaling
> >> (CSIG)
> >> >
> >> > Not sure why we are coming up with so many new techniques when ECN just
> >> works fine.
> >> > ECN is a 2 bit field (not 1 bit) and seems to be sufficient to indicate
> >> extent of congestion by marking it per packet. Adding more complexity to
> >> any layer whether it is L2 or L3 doesn?t work well in deployments. Our goal
> >> should be to simplify things and only add new headers if absolutely
> >> necessary.
> >> >
> >> > Vidhi
> >> >
> >> >
> >> > On Sep 12, 2023, at 3:12 AM, Shihang(Vincent) <shihang9=
> >> [e-mail]> wrote:
> >> >
> >> > Hi,
> >> > I agree L2 may not be the best choice to carry the congestion signaling
> >> end-to-end and more bits are needed. We have submitted a draft to carry the
> >> multi-bits congestion signaling in L3. We call it Advanced ECN. See
> >> https://datatracker.ietf.org/doc/draft-shi-ccwg-advanced-ecn/.
> >> >
> >> > Thanks,
> >> > Hang
> >> >
> >> > From: CCWG <[e-mail]> On Behalf Of Huangyihong (Rachel)
> >> > Sent: Tuesday, September 12, 2023 5:41 PM
> >> > To: Tom Herbert <tom=[e-mail]>; Abhiram Ravi
> >> <abhiramr=[e-mail]>
> >> > Cc: IETF IPPM WG <[e-mail]>; tsvwg <[e-mail]>; [e-mail];
> >> [e-mail]; Nandita Dukkipati <[e-mail]>; Naoshad Mehta <
> >> [e-mail]>; Jai Kumar <[e-mail]>
> >> > Subject: Re: [CCWG] [iccrg] [tsvwg] New Internet Draft: Congestion
> >> Signaling (CSIG)
> >> >
> >> > Hi,
> >> >
> >> > I also have the same feeling. Implementing in L2 may be difficult to be
> >> used in e2e transport. Of course it can work well in limited domain, like
> >> DC or HPC clusters. However, I also look for some solutions that could be
> >> able to go through internet. We have submitted a draft to describe the
> >> transport challenges. See
> >> https://datatracker.ietf.org/doc/html/draft-huang-tsvwg-transport-challenges
> >> .
> >> >
> >> > I share the same opinion that the congestion signal is useful and
> >> current 1-bit ECN solution is not fully sufficient. But I also feel like
> >> the more straight way is to extend L3, or l4, like update IOAM, to carry
> >> the information. For L2 solution, it should be developed together with IEEE
> >> 802.1.
> >> >
> >> > BR?
> >> > Rachel
> >> >
> >> > ???: iccrg <[e-mail]> ?? Tom Herbert
> >> > ????: 2023?9?10? 0:10
> >> > ???: Abhiram Ravi <abhiramr=[e-mail]>
> >> > ??: IETF IPPM WG <[e-mail]>; tsvwg <[e-mail]>; [e-mail];
> >> [e-mail]; Nandita Dukkipati <[e-mail]>; Naoshad Mehta <
> >> [e-mail]>; Jai Kumar <[e-mail]>
> >> > ??: Re: [iccrg] [tsvwg] New Internet Draft: Congestion Signaling (CSIG)
> >> >
> >> > Hi, thanks for draft!
> >> >
> >> > The first thing that stands out to me is the carrier of the new packet
> >> headers. In the forward path it would be in L2 and in reflection it would
> >> be L4. As the draft describes, this would entail having to support the
> >> protocol in multiple L2 and multiple L4 protocols-- that's going to be a
> >> pretty big lift! Also, L2 is not really an end-to-end protocol (would
> >> legacy switches in the path also forward the header)l?).
> >> >
> >> > The signaling being described in the draft is network layer
> >> information, and hence IMO should be conveyed in network layer headers.
> >> That's is L3 which conveniently is the average of L2+L4 :-)
> >> >
> >> > IMO, the proper carrier of the signal data is Hop-by-Hop Options. This
> >> is end-to-end and allows modification of data in-flight. The typical
> >> concern with Hop-by-Hop Options is high drop rates on the Internet, however
> >> in this case the protocol is explicitly confined to a limited domain so I
> >> don't see that as a blocking issue for this use case.
> >> >
> >> > The information being carried seems very similar to that of IOAM (IOAM
> >> uses Hop-by-Hop Options and supports reflection). I suppose the differences
> >> are that this protocol is meant to be consumed by the transport Layer and
> >> the data is a condensed summary of path characteristics. IOAM seems pretty
> >> extensible, so maybe it could be adapted to carry the signals of this draft?
> >> >
> >> > A related proposal might be FAST draft-herbert-fast. Where the CSIG is
> >> network to host signaling, FAST is host to network signaling for the
> >> purposes of requesting network services. These might be complementary and
> >> options for both may be in the same packet. FAST also uses reflection, so
> >> we might be able to leverage some common implementation at a destination.
> >> >
> >> > Tom
> >> >
> >> > On Fri, Sep 8, 2023, 7:43 PM Abhiram Ravi <abhiramr=
> >> [e-mail]> wrote:
> >> > Hi IPPM folks,
> >> >
> >> > I am pleased to announce the publication of a new internet draft,
> >> Congestion Signaling (CSIG):
> >> https://datatracker.ietf.org/doc/draft-ravi-ippm-csig/
> >> >
> >> > CSIG is a new end-to-end packet header mechanism for in-band signaling
> >> that is simple, efficient, deployable, and grounded in concrete use cases
> >> of congestion control, traffic management, and network debuggability. We
> >> believe that CSIG is an important new protocol that builds on top of
> >> existing in-band network telemetry protocols.
> >> >
> >> > We encourage you to read the CSIG draft and provide your feedback and
> >> comments. We have also cc'd the TSVWG, CCWG, and ICCRG mailing lists, as we
> >> believe that this work may be of interest to their members as well.
> >> >
> >> > Thank you for your time and consideration.
> >> >
> >> > Sincerely,
> >> > Abhiram Ravi
> >> > On behalf of the CSIG authors
> >>
> >>
> >
> > --
> > Abhiram Ravi
> > --
> > CCWG mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/ccwg
> >

-- 
Rod Grimes                                                 [e-mail]