Re: [iccrg] [tsvwg] [CCWG] New Internet Draft: Congestion Signaling (CSIG)

["Rodney W. Grimes" <[e-mail]>]

Hesahm,
	I think you missed my point, indeed there are lots of
vendors and equipment around that has support for 802.1Qau, that
is not hard to find.  What is hard to find is anyone who has ever
actually deployed this or anyone who even has test lab data
showing how it behaves when put to the test.


> I think Broadcom Thor ASIC supports 802.1Qau [
> https://docs.broadcom.com/doc/NCC-WP1XX] which is used by Arista [
> https://www.arista.com/assets/data/pdf/Broadcom-RoCE-Deployment-Guide.pdf]
The QCN in this guide is NOT IEEE 802.11Qua.

> 
> NVIDIA uses QCN [
> https://docs.nvidia.com/networking/plugins/servlet/mobile?contentId=12013417#content/view/12013417
> ]
> 
> Cisco used 802.1Qau in their earlier data center switches [
> https://www.cisco.com/c/dam/global/en_sg/training-events/datacentertechbyte/assets/pdfs/data_center_ethernet_qa.pdf
> ]
> I am not sure about their current ones. It is worth noting that Cisco was
> the main contributor to the standardization of IEEE 802.1Qau in the DCB TG.
> 
> Marvell FastLinQ 41000 Series support 802.1Qau [
> https://www.marvell.com/content/dam/marvell/en/public-collateral/ethernet-adaptersandcontrollers/marvell-ethernet-controllers-fastlinq-41000-product-brief.pdf].
> FastLinQ is used by HPE servers [
> https://www.marvell.com/content/dam/marvell/en/public-collateral/hpe/hpe-marvell-pb-fastlinq-41000-series-adapters.pdf
> ]
> 
> To address congestion due to incast problem in DC IEEE 802.1 TSN TG
> recently standardized IEEE 802.1Qcz [https://1.ieee802.org/tsn/802-1qcz/].
> 
> Although it is useful to know about products which use QCN, the main issue
> which I want to discuss is how we compare the new congestion signaling
> proposal with QCN.

That would be greatly assisted by actual DATA showing how 802.11Qua
behaves under various test conditions, which I can find NO data on.

Regards,
Rod Grimes

> Hesham
> 
> On Sat, Sep 16, 2023, 10:22 AM Rodney W. Grimes <[e-mail]>
> wrote:
> 
> > Hesham,
> >         While I found a read of some of the IEEE 802.1Qau papers
> > interesting, I have NOT been able to find anyone who has real
> > world deployment data.  In a datacenter or otherwise.
> >
> >         Do you know of any or does anyone here have data to
> > point at?
> >
> > Regards,
> > Rod Grimes
> >
> > > Hi Abhiram,
> > > IEEE 802.1 had a data center bridging TG which developed sometime ago the
> > > spec for quantized congestion control (QCN). The spec is IEEE 802.1Qau
> > > which is now part of 802.1Q.
> > >
> > > QCN provide a means for a bridge to notify a source of congestion causing
> > > the source to reduce the flow rate.
> > >
> > > QCN is independent of upper layer protocol. It coexists with TCP  (there
> > is
> > > a nested control loops of QCN and TCP flow control).
> > >
> > > QCN only works for unicast traffic, and operates over a domain where all
> > > bridges and end stations support it.
> > >
> > > QCN does not require per flow state or per flow queuing in bridge.
> > >
> > > Over the years there have been proposals to enhance it, but the basic
> > idea
> > > stayed the same.
> > >
> > > How you compare your congestion signaling proposal with QCN. One obvious
> > > difference is that yours is end-2end while QCN is not. But is end-end
> > > congestion control the best way to go?
> > >
> > > Comments?
> > >
> > > Hesham
> > > PS. I removed IPPM but kept other WG who may have an opinion on this
> > issue.
> > >
> > >
> > > On Wed, Sep 13, 2023, 5:09 PM Abhiram Ravi <abhiramr=
> > > [e-mail]> wrote:
> > >
> > > > Thank you for your comments. Responses inline.
> > > >
> > > > > Tom: The signaling being described in the draft is network layer
> > > > information, and hence IMO should be conveyed in network layer headers.
> > > > That's is L3 which conveniently is the average of L2+L4 :-)
> > > > > Rachel:  Implementing in L2 may be difficult to be used in e2e
> > transport.
> > > > > Hang: I agree L2 may not be the best choice to carry the congestion
> > > > signaling end-to-end and more bits are needed.
> > > >
> > > > The choice of using L2+L4 is an intentional and important design
> > choice,
> > > > one that is driven by need, for practicality and ease of deployment in
> > data
> > > > center networks. We acknowledge that previous approaches have
> > attempted to
> > > > extend L3 or L4, but these approaches do not meet the requirements for
> > > > deployability in a heterogeneous network, including interoperability
> > and
> > > > backward compatibility. CSIG targets specific yet important use cases
> > while
> > > > meeting these requirements.
> > > >
> > > > Section 7.1
> > > > <
> > https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-choice-of-layer-2
> > >
> > > > of the draft explicitly addresses why such a design was chosen.
> > > >
> > > > The key reasons are:
> > > >
> > > >    -
> > > >
> > > >    Simplicity of device implementations of CSIG (both at NICs and
> > > >    switches), given the location of the CSIG-tag in the packet header
> > is not
> > > >    affected by any header at layer 3 and above. CSIG-tag being the
> > last tag
> > > >    before the ethertype also makes it simple for ASICs to implement it
> > inside
> > > >    the L2 header.
> > > >    -
> > > >
> > > >    Support for delivering congestion signals even in the presence of
> > > >    in-network tunneling and encryption, which are commonly used in data
> > > >    centers, especially for Cloud traffic and in Traffic engineered
> > domains.
> > > >    CSIG tags are carried through Layer 3 tunnels e.g., IP-in-IP, VxLAN,
> > > >    Geneve, at a fixed offset in the packet header. This avoids the
> > need to
> > > >    copy and relocate CSIG tags across inner / outer headers during
> > > >    encapsulation and decapsulation of packets, which is a complex
> > hardware
> > > >    operation and would be necessary if implemented instead at layers 3
> > or
> > > >    higher.
> > > >
> > > > > Tom: The signaling being described in the draft is network layer
> > > > information, and hence IMO should be conveyed in network layer headers.
> > > >
> > > > The information carried in L2 CSIG-tags are device signals or port
> > signals
> > > > from the bottleneck, and should not be viewed as network-layer signals.
> > > > Although the bottleneck signal is also a property of the packet path in
> > > > aggregate, the signal carried on the tag has a scope of one hop i.e.,
> > the
> > > > measurement on one specific link or device on the path. The tag gets
> > > > conditionally swapped on each device in a compare-and-replace manner as
> > > > described in Section 4.3
> > > > <
> > https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-csig-operation-life-of-a-pa
> > >
> > > >
> > > > Reflection is a property of the transport as described in Section 4.2,
> > and
> > > > hence CSIG reflection is implemented at layer 4 and is specific to the
> > > > transport. This allows a clean separation of the mechanism used to
> > collect
> > > > signals from the network from the mechanism used for leveraging those
> > > > signals at the transport.
> > > >
> > > > > Tom: Also, L2 is not really an end-to-end protocol (would legacy
> > > > switches in the path also forward the header)
> > > >
> > > > Legacy devices can not only forward the packet, but also participate in
> > > > the CSIG protocol via a software-assisted approach by treating
> > CSIG-tags as
> > > > VLAN-tags and leveraging VLAN resources to perform CSIG updates. This
> > L2
> > > > approach is the only solution we?ve found which allows immediate
> > deployment
> > > > for certain signals with legacy devices in data centers, while native
> > > > support for CSIG is easily achievable in future devices. CSIG
> > stripping can
> > > > also be exercised for interop when needed - see Section 6
> > > > <
> > https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-incremental-deployment-of-c
> > >
> > > > for interoperability.
> > > >
> > > > > Tom: As the draft describes, this would entail having to support the
> > > > protocol in multiple L2 and multiple L4 protocols
> > > >
> > > > Section 4.1
> > > > <
> > https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-csig-tag-header-format
> > >
> > > > lists how CSIG-tag operates with each relevant L2 encapsulation and
> > Section
> > > > 4.2
> > > > <
> > https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-csig-reflection-header-form
> > >
> > > > addresses implementation of reflection in relevant transports.
> > > >
> > > > > Tom: IMO, the proper carrier of the signal data is Hop-by-Hop
> > Options.
> > > > IOAM seems pretty extensible, so maybe it could be adapted to carry the
> > > > signals of this draft?
> > > > > Rachel: Of course it can work well in limited domain, like DC or HPC
> > > > clusters. However, I also look for some solutions that could be able
> > to go
> > > > through internet
> > > >
> > > > CSIG is targeted at specific use cases in limited domains, primarily
> > data
> > > > centers, for congestion control, traffic management and network
> > > > debuggability. The requirements for such use cases vary significantly
> > from
> > > > those of internet deployments, which is not in scope for CSIG.
> > > > IOAM and hop-by-hop options in general are tuned for a broader set of
> > use
> > > > cases and do not meet all the requirements for the specific use cases
> > that
> > > > CSIG addresses, including interop with legacy devices and in-network
> > > > tunneling, due to the placement of the header.  Section 2
> > > > <
> > https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-design-principles
> > >
> > > > speaks to these requirements.
> > > >
> > > > > Rachel: For L2 solution, it should be developed together with IEEE
> > 802.1.
> > > >
> > > > Acknowledging that IEEE will be involved as well.
> > > >
> > > > > Sebastian: What happens if bottlenecks hand out immediate
> > capacity/rate
> > > > estimates and all greedy flows start increasing their cwin really fast?
> > > >
> > > > CSIG supports both min(ABW) and min(ABW/C). Using min(ABW/C), the each
> > > > flow can increase its rate in proportion to the % available bandwidth,
> > so
> > > > that in aggregate the flows respect the available bottleneck bandwidth
> > with
> > > > reduced risk of overshoot. See Section 8.1.2
> > > > <
> > https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-using-maximum-link-utilizat
> > >
> > > > .
> > > >
> > > > > Ingemar: CSIG sounds to me like something that belongs more in ICCRG
> > > >
> > > > CSIG is applicable to use cases beyond Congestion control (which is an
> > > > important one), such as traffic management and network debugging.
> > Section
> > > > 8
> > > > <
> > https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-use-cases-defined-by-bottle
> > >
> > > > describes the use cases in more detail.
> > > >
> > > > > Tom: A related proposal might be FAST draft-herbert-fast. These
> > might be
> > > > complementary and options for both may be in the same packet
> > > > > Hang: We call it Advanced ECN. See draft-shi-ccwg-advanced-ecn
> > > > > Ingemar: L4S is recently standardised and it is definitely gaining
> > > > traction also in 3GPP
> > > >
> > > > Thank you for the pointers. We agree that these proposals are all
> > > > complementary to CSIG.
> > > >
> > > >
> > > > On Wed, Sep 13, 2023 at 5:20?AM <[e-mail]> wrote:
> > > >
> > > >> Hi Ingemar, hi Sebastian,
> > > >>
> > > >> Skimming over the draft only, isn't CSIG about Ethernet-domains, while
> > > >> L4S is E2E?
> > > >>
> > > >> Regards,
> > > >>
> > > >> Ruediger
> > > >>
> > > >> -----Urspr?ngliche Nachricht-----
> > > >> Von: tsvwg <[e-mail]> Im Auftrag von Sebastian Moeller
> > > >> Gesendet: Mittwoch, 13. September 2023 13:42
> > > >> An: Ingemar Johansson S <ingemar.s.johansson=
> > > >> [e-mail]>
> > > >> Cc: Vidhi Goel <vidhi_goel=[e-mail]>;
> > Shihang(Vincent)
> > > >> <shihang9=[e-mail]>; tsvwg <[e-mail]>; Jai
> > > >> Kumar <[e-mail]>; IETF IPPM WG <[e-mail]>; Tom
> > > >> Herbert <tom=[e-mail]>; [e-mail];
> > Abhiram
> > > >> Ravi <abhiramr=[e-mail]>; Nandita Dukkipati <
> > > >> [e-mail]>; [e-mail]; Huangyihong (Rachel)
> > <rachel.huang=
> > > >> [e-mail]>; Naoshad Mehta <[e-mail]>
> > > >> Betreff: Re: [tsvwg] [iccrg] New Internet Draft: Congestion Signaling
> > > >> (CSIG)
> > > >>
> > > >> Hi Ingemar,
> > > >>
> > > >>
> > > >> > On Sep 13, 2023, at 12:30, Ingemar Johansson S <ingemar.s.johansson=
> > > >> [e-mail]> wrote:
> > > >> >
> > > >> > Hi
> > > >> >
> > > >> > I agree with Vihdi
> > > >> >
> > > >> > L4S is recently standardised
> > > >>
> > > >>         [SM] In experimental track, the goal is currently to test
> > whether
> > > >> it can/should be deployed at scale....
> > > >>
> > > >> > and it is definitely gaining traction also in 3GPP. We have an echo
> > > >> system that is looking forward to having L4S widely deployed.
> > > >> > Still, the congestion control aspects are not fully explored yet.
> > One
> > > >> interesting topic is if L4S allows to more safely deviate from
> > additive
> > > >> increase to make congestion control algorithms more quickly converge
> > to
> > > >> higher link capacity. There are a number of study topics around L4S
> > > >> congestion control that are listed in e.g the TCP Prague draft.
> > > >> >
> > > >> > I cannot dictate what others should do with their time and money but
> > > >> personally I'd prefer that the IETF explores L4S and its
> > possibilities and
> > > >> downsides before jumping on the next idea.
> > > >>
> > > >>         [SM] L4S can be described as taking the ideas behind DCTCP and
> > > >> making them fit for use over the internet*. Yet the signaling
> > discussed
> > > >> here is to be used in e.g. data center contexts where DCTCP is
> > already used
> > > >> and found lacking compared to newer methods operating on richer
> > congestion
> > > >> information (HPCC, Swift, Poseidon, ...).
> > > >>         Given that L4S essentially uses a multi-packet signal(**) to
> > > >> report the "queue filling state" that is then stochastically
> > distributed
> > > >> over all concurrent flows, it seems obvious to me that reconstructing
> > a
> > > >> reliable estimate of on-path queueing will take some time and
> > averaging for
> > > >> each individual flow, I would guess that in some environments this
> > delay
> > > >> simply is too costly.
> > > >>         So L4S and CSIG seem complementary and in no way mutually
> > > >> exclusive.
> > > >>
> > > >>
> > > >> Regards
> > > >>         Sebastian
> > > >>
> > > >>
> > > >>
> > > >> *) I will not further discuss whether that is achieved or not as it
> > seems
> > > >> irrelevant here.
> > > >> **) In essence transmission of congestion state via a 1-bit serial
> > > >> channel, clocked at the (variable***) packet rate at the bottleneck.
> > > >> ***) as packets are not of uniform size
> > > >>
> > > >> >
> > > >> > CSIG sounds to me like something that belongs more in ICCRG or ?
> > > >> >
> > > >> > /Ingemar
> > > >> >
> > > >> > From: tsvwg <[e-mail]> On Behalf Of Vidhi Goel
> > > >> > Sent: Wednesday, 13 September 2023 00:59
> > > >> > To: Shihang(Vincent) <shihang9=[e-mail]>
> > > >> > Cc: Huangyihong (Rachel) <rachel.huang=[e-mail]
> > >;
> > > >> Tom Herbert <tom=[e-mail]>; Abhiram Ravi
> > > >> <abhiramr=[e-mail]>; IETF IPPM WG <[e-mail]>;
> > > >> tsvwg <[e-mail]>; [e-mail]; [e-mail]; Nandita
> > Dukkipati
> > > >> <[e-mail]>; Naoshad Mehta <[e-mail]>; Jai Kumar
> > <
> > > >> [e-mail]>
> > > >> > Subject: Re: [tsvwg] [iccrg] New Internet Draft: Congestion
> > Signaling
> > > >> (CSIG)
> > > >> >
> > > >> > Not sure why we are coming up with so many new techniques when ECN
> > just
> > > >> works fine.
> > > >> > ECN is a 2 bit field (not 1 bit) and seems to be sufficient to
> > indicate
> > > >> extent of congestion by marking it per packet. Adding more complexity
> > to
> > > >> any layer whether it is L2 or L3 doesn?t work well in deployments.
> > Our goal
> > > >> should be to simplify things and only add new headers if absolutely
> > > >> necessary.
> > > >> >
> > > >> > Vidhi
> > > >> >
> > > >> >
> > > >> > On Sep 12, 2023, at 3:12 AM, Shihang(Vincent) <shihang9=
> > > >> [e-mail]> wrote:
> > > >> >
> > > >> > Hi,
> > > >> > I agree L2 may not be the best choice to carry the congestion
> > signaling
> > > >> end-to-end and more bits are needed. We have submitted a draft to
> > carry the
> > > >> multi-bits congestion signaling in L3. We call it Advanced ECN. See
> > > >> https://datatracker.ietf.org/doc/draft-shi-ccwg-advanced-ecn/.
> > > >> >
> > > >> > Thanks,
> > > >> > Hang
> > > >> >
> > > >> > From: CCWG <[e-mail]> On Behalf Of Huangyihong
> > (Rachel)
> > > >> > Sent: Tuesday, September 12, 2023 5:41 PM
> > > >> > To: Tom Herbert <tom=[e-mail]>; Abhiram
> > Ravi
> > > >> <abhiramr=[e-mail]>
> > > >> > Cc: IETF IPPM WG <[e-mail]>; tsvwg <[e-mail]>;
> > [e-mail];
> > > >> [e-mail]; Nandita Dukkipati <[e-mail]>; Naoshad
> > Mehta <
> > > >> [e-mail]>; Jai Kumar <[e-mail]>
> > > >> > Subject: Re: [CCWG] [iccrg] [tsvwg] New Internet Draft: Congestion
> > > >> Signaling (CSIG)
> > > >> >
> > > >> > Hi,
> > > >> >
> > > >> > I also have the same feeling. Implementing in L2 may be difficult
> > to be
> > > >> used in e2e transport. Of course it can work well in limited domain,
> > like
> > > >> DC or HPC clusters. However, I also look for some solutions that
> > could be
> > > >> able to go through internet. We have submitted a draft to describe the
> > > >> transport challenges. See
> > > >>
> > https://datatracker.ietf.org/doc/html/draft-huang-tsvwg-transport-challenges
> > > >> .
> > > >> >
> > > >> > I share the same opinion that the congestion signal is useful and
> > > >> current 1-bit ECN solution is not fully sufficient. But I also feel
> > like
> > > >> the more straight way is to extend L3, or l4, like update IOAM, to
> > carry
> > > >> the information. For L2 solution, it should be developed together
> > with IEEE
> > > >> 802.1.
> > > >> >
> > > >> > BR?
> > > >> > Rachel
> > > >> >
> > > >> > ???: iccrg <[e-mail]> ?? Tom Herbert
> > > >> > ????: 2023?9?10? 0:10
> > > >> > ???: Abhiram Ravi <abhiramr=[e-mail]>
> > > >> > ??: IETF IPPM WG <[e-mail]>; tsvwg <[e-mail]>;
> > [e-mail];
> > > >> [e-mail]; Nandita Dukkipati <[e-mail]>; Naoshad
> > Mehta <
> > > >> [e-mail]>; Jai Kumar <[e-mail]>
> > > >> > ??: Re: [iccrg] [tsvwg] New Internet Draft: Congestion Signaling
> > (CSIG)
> > > >> >
> > > >> > Hi, thanks for draft!
> > > >> >
> > > >> > The first thing that stands out to me is the carrier of the new
> > packet
> > > >> headers. In the forward path it would be in L2 and in reflection it
> > would
> > > >> be L4. As the draft describes, this would entail having to support the
> > > >> protocol in multiple L2 and multiple L4 protocols-- that's going to
> > be a
> > > >> pretty big lift! Also, L2 is not really an end-to-end protocol (would
> > > >> legacy switches in the path also forward the header)l?).
> > > >> >
> > > >> > The signaling being described in the draft is network layer
> > > >> information, and hence IMO should be conveyed in network layer
> > headers.
> > > >> That's is L3 which conveniently is the average of L2+L4 :-)
> > > >> >
> > > >> > IMO, the proper carrier of the signal data is Hop-by-Hop Options.
> > This
> > > >> is end-to-end and allows modification of data in-flight. The typical
> > > >> concern with Hop-by-Hop Options is high drop rates on the Internet,
> > however
> > > >> in this case the protocol is explicitly confined to a limited domain
> > so I
> > > >> don't see that as a blocking issue for this use case.
> > > >> >
> > > >> > The information being carried seems very similar to that of IOAM
> > (IOAM
> > > >> uses Hop-by-Hop Options and supports reflection). I suppose the
> > differences
> > > >> are that this protocol is meant to be consumed by the transport Layer
> > and
> > > >> the data is a condensed summary of path characteristics. IOAM seems
> > pretty
> > > >> extensible, so maybe it could be adapted to carry the signals of this
> > draft?
> > > >> >
> > > >> > A related proposal might be FAST draft-herbert-fast. Where the CSIG
> > is
> > > >> network to host signaling, FAST is host to network signaling for the
> > > >> purposes of requesting network services. These might be complementary
> > and
> > > >> options for both may be in the same packet. FAST also uses
> > reflection, so
> > > >> we might be able to leverage some common implementation at a
> > destination.
> > > >> >
> > > >> > Tom
> > > >> >
> > > >> > On Fri, Sep 8, 2023, 7:43 PM Abhiram Ravi <abhiramr=
> > > >> [e-mail]> wrote:
> > > >> > Hi IPPM folks,
> > > >> >
> > > >> > I am pleased to announce the publication of a new internet draft,
> > > >> Congestion Signaling (CSIG):
> > > >> https://datatracker.ietf.org/doc/draft-ravi-ippm-csig/
> > > >> >
> > > >> > CSIG is a new end-to-end packet header mechanism for in-band
> > signaling
> > > >> that is simple, efficient, deployable, and grounded in concrete use
> > cases
> > > >> of congestion control, traffic management, and network debuggability.
> > We
> > > >> believe that CSIG is an important new protocol that builds on top of
> > > >> existing in-band network telemetry protocols.
> > > >> >
> > > >> > We encourage you to read the CSIG draft and provide your feedback
> > and
> > > >> comments. We have also cc'd the TSVWG, CCWG, and ICCRG mailing lists,
> > as we
> > > >> believe that this work may be of interest to their members as well.
> > > >> >
> > > >> > Thank you for your time and consideration.
> > > >> >
> > > >> > Sincerely,
> > > >> > Abhiram Ravi
> > > >> > On behalf of the CSIG authors
> > > >>
> > > >>
> > > >
> > > > --
> > > > Abhiram Ravi
> > > > --
> > > > CCWG mailing list
> > > > [e-mail]
> > > > https://www.ietf.org/mailman/listinfo/ccwg
> > > >
> >
> > --
> > Rod Grimes
> > [e-mail]
> >

-- 
Rod Grimes                                                 [e-mail]