Re: [iccrg] [ippm] [tsvwg] New Internet Draft: Congestion Signaling (CSIG)

[Greg Mirsky <[e-mail]>]

Hi, Abhiram et al.,
top-posting one thread of the discussion:

> Rachel: For L2 solution, it should be developed together with IEEE 802.1.


Acknowledging that IEEE will be involved as well.

It seems like the question of the SDO driving this work requires careful
consideration. It is not a mere coincidence that the applicability of IOAM
is defined at IETF WGs for data planes based on specifications developed at
IETF. AFAICS, which is not exactly the case for CSIG. Could the liaison
from IETF to IEEE, specifically to 802.3 and 802.1, be drafted before
deciding where in IETF this work is anchored?

Regards,
Greg

On Thu, Sep 14, 2023 at 1:01 AM Abhiram Ravi <abhiramr=
[e-mail]> wrote:

> Thank you for your comments. Responses inline.
>
> > Tom: The signaling being described in the draft is network layer
> information, and hence IMO should be conveyed in network layer headers.
> That's is L3 which conveniently is the average of L2+L4 :-)
> > Rachel:  Implementing in L2 may be difficult to be used in e2e transport.
> > Hang: I agree L2 may not be the best choice to carry the congestion
> signaling end-to-end and more bits are needed.
>
> The choice of using L2+L4 is an intentional and important design choice,
> one that is driven by need, for practicality and ease of deployment in data
> center networks. We acknowledge that previous approaches have attempted to
> extend L3 or L4, but these approaches do not meet the requirements for
> deployability in a heterogeneous network, including interoperability and
> backward compatibility. CSIG targets specific yet important use cases while
> meeting these requirements.
>
> Section 7.1
> <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-choice-of-layer-2>
> of the draft explicitly addresses why such a design was chosen.
>
> The key reasons are:
>
>    -
>
>    Simplicity of device implementations of CSIG (both at NICs and
>    switches), given the location of the CSIG-tag in the packet header is not
>    affected by any header at layer 3 and above. CSIG-tag being the last tag
>    before the ethertype also makes it simple for ASICs to implement it inside
>    the L2 header.
>    -
>
>    Support for delivering congestion signals even in the presence of
>    in-network tunneling and encryption, which are commonly used in data
>    centers, especially for Cloud traffic and in Traffic engineered domains.
>    CSIG tags are carried through Layer 3 tunnels e.g., IP-in-IP, VxLAN,
>    Geneve, at a fixed offset in the packet header. This avoids the need to
>    copy and relocate CSIG tags across inner / outer headers during
>    encapsulation and decapsulation of packets, which is a complex hardware
>    operation and would be necessary if implemented instead at layers 3 or
>    higher.
>
> > Tom: The signaling being described in the draft is network layer
> information, and hence IMO should be conveyed in network layer headers.
>
> The information carried in L2 CSIG-tags are device signals or port signals
> from the bottleneck, and should not be viewed as network-layer signals.
> Although the bottleneck signal is also a property of the packet path in
> aggregate, the signal carried on the tag has a scope of one hop i.e., the
> measurement on one specific link or device on the path. The tag gets
> conditionally swapped on each device in a compare-and-replace manner as
> described in Section 4.3
> <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-csig-operation-life-of-a-pa>
>
> Reflection is a property of the transport as described in Section 4.2, and
> hence CSIG reflection is implemented at layer 4 and is specific to the
> transport. This allows a clean separation of the mechanism used to collect
> signals from the network from the mechanism used for leveraging those
> signals at the transport.
>
> > Tom: Also, L2 is not really an end-to-end protocol (would legacy
> switches in the path also forward the header)
>
> Legacy devices can not only forward the packet, but also participate in
> the CSIG protocol via a software-assisted approach by treating CSIG-tags as
> VLAN-tags and leveraging VLAN resources to perform CSIG updates. This L2
> approach is the only solution we’ve found which allows immediate deployment
> for certain signals with legacy devices in data centers, while native
> support for CSIG is easily achievable in future devices. CSIG stripping can
> also be exercised for interop when needed - see Section 6
> <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-incremental-deployment-of-c>
> for interoperability.
>
> > Tom: As the draft describes, this would entail having to support the
> protocol in multiple L2 and multiple L4 protocols
>
> Section 4.1
> <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-csig-tag-header-format>
> lists how CSIG-tag operates with each relevant L2 encapsulation and Section
> 4.2
> <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-csig-reflection-header-form>
> addresses implementation of reflection in relevant transports.
>
> > Tom: IMO, the proper carrier of the signal data is Hop-by-Hop Options.
> IOAM seems pretty extensible, so maybe it could be adapted to carry the
> signals of this draft?
> > Rachel: Of course it can work well in limited domain, like DC or HPC
> clusters. However, I also look for some solutions that could be able to go
> through internet
>
> CSIG is targeted at specific use cases in limited domains, primarily data
> centers, for congestion control, traffic management and network
> debuggability. The requirements for such use cases vary significantly from
> those of internet deployments, which is not in scope for CSIG.
> IOAM and hop-by-hop options in general are tuned for a broader set of use
> cases and do not meet all the requirements for the specific use cases that
> CSIG addresses, including interop with legacy devices and in-network
> tunneling, due to the placement of the header.  Section 2
> <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-design-principles>
> speaks to these requirements.
>
> > Rachel: For L2 solution, it should be developed together with IEEE 802.1.
>
> Acknowledging that IEEE will be involved as well.
>
> > Sebastian: What happens if bottlenecks hand out immediate capacity/rate
> estimates and all greedy flows start increasing their cwin really fast?
>
> CSIG supports both min(ABW) and min(ABW/C). Using min(ABW/C), the each
> flow can increase its rate in proportion to the % available bandwidth, so
> that in aggregate the flows respect the available bottleneck bandwidth with
> reduced risk of overshoot. See Section 8.1.2
> <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-using-maximum-link-utilizat>
> .
>
> > Ingemar: CSIG sounds to me like something that belongs more in ICCRG
>
> CSIG is applicable to use cases beyond Congestion control (which is an
> important one), such as traffic management and network debugging. Section
> 8
> <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-use-cases-defined-by-bottle>
> describes the use cases in more detail.
>
> > Tom: A related proposal might be FAST draft-herbert-fast. These might be
> complementary and options for both may be in the same packet
> > Hang: We call it Advanced ECN. See draft-shi-ccwg-advanced-ecn
> > Ingemar: L4S is recently standardised and it is definitely gaining
> traction also in 3GPP
>
> Thank you for the pointers. We agree that these proposals are all
> complementary to CSIG.
>
>
> On Wed, Sep 13, 2023 at 5:20 AM <[e-mail]> wrote:
>
>> Hi Ingemar, hi Sebastian,
>>
>> Skimming over the draft only, isn't CSIG about Ethernet-domains, while
>> L4S is E2E?
>>
>> Regards,
>>
>> Ruediger
>>
>> -----Ursprüngliche Nachricht-----
>> Von: tsvwg <[e-mail]> Im Auftrag von Sebastian Moeller
>> Gesendet: Mittwoch, 13. September 2023 13:42
>> An: Ingemar Johansson S <ingemar.s.johansson=
>> [e-mail]>
>> Cc: Vidhi Goel <vidhi_goel=[e-mail]>; Shihang(Vincent)
>> <shihang9=[e-mail]>; tsvwg <[e-mail]>; Jai
>> Kumar <[e-mail]>; IETF IPPM WG <[e-mail]>; Tom
>> Herbert <tom=[e-mail]>; [e-mail]; Abhiram
>> Ravi <abhiramr=[e-mail]>; Nandita Dukkipati <
>> [e-mail]>; [e-mail]; Huangyihong (Rachel) <rachel.huang=
>> [e-mail]>; Naoshad Mehta <[e-mail]>
>> Betreff: Re: [tsvwg] [iccrg] New Internet Draft: Congestion Signaling
>> (CSIG)
>>
>> Hi Ingemar,
>>
>>
>> > On Sep 13, 2023, at 12:30, Ingemar Johansson S <ingemar.s.johansson=
>> [e-mail]> wrote:
>> >
>> > Hi
>> >
>> > I agree with Vihdi
>> >
>> > L4S is recently standardised
>>
>>         [SM] In experimental track, the goal is currently to test whether
>> it can/should be deployed at scale....
>>
>> > and it is definitely gaining traction also in 3GPP. We have an echo
>> system that is looking forward to having L4S widely deployed.
>> > Still, the congestion control aspects are not fully explored yet. One
>> interesting topic is if L4S allows to more safely deviate from additive
>> increase to make congestion control algorithms more quickly converge to
>> higher link capacity. There are a number of study topics around L4S
>> congestion control that are listed in e.g the TCP Prague draft.
>> >
>> > I cannot dictate what others should do with their time and money but
>> personally I'd prefer that the IETF explores L4S and its possibilities and
>> downsides before jumping on the next idea.
>>
>>         [SM] L4S can be described as taking the ideas behind DCTCP and
>> making them fit for use over the internet*. Yet the signaling discussed
>> here is to be used in e.g. data center contexts where DCTCP is already used
>> and found lacking compared to newer methods operating on richer congestion
>> information (HPCC, Swift, Poseidon, ...).
>>         Given that L4S essentially uses a multi-packet signal(**) to
>> report the "queue filling state" that is then stochastically distributed
>> over all concurrent flows, it seems obvious to me that reconstructing a
>> reliable estimate of on-path queueing will take some time and averaging for
>> each individual flow, I would guess that in some environments this delay
>> simply is too costly.
>>         So L4S and CSIG seem complementary and in no way mutually
>> exclusive.
>>
>>
>> Regards
>>         Sebastian
>>
>>
>>
>> *) I will not further discuss whether that is achieved or not as it seems
>> irrelevant here.
>> **) In essence transmission of congestion state via a 1-bit serial
>> channel, clocked at the (variable***) packet rate at the bottleneck.
>> ***) as packets are not of uniform size
>>
>> >
>> > CSIG sounds to me like something that belongs more in ICCRG or ?
>> >
>> > /Ingemar
>> >
>> > From: tsvwg <[e-mail]> On Behalf Of Vidhi Goel
>> > Sent: Wednesday, 13 September 2023 00:59
>> > To: Shihang(Vincent) <shihang9=[e-mail]>
>> > Cc: Huangyihong (Rachel) <rachel.huang=[e-mail]>;
>> Tom Herbert <tom=[e-mail]>; Abhiram Ravi
>> <abhiramr=[e-mail]>; IETF IPPM WG <[e-mail]>;
>> tsvwg <[e-mail]>; [e-mail]; [e-mail]; Nandita Dukkipati
>> <[e-mail]>; Naoshad Mehta <[e-mail]>; Jai Kumar <
>> [e-mail]>
>> > Subject: Re: [tsvwg] [iccrg] New Internet Draft: Congestion Signaling
>> (CSIG)
>> >
>> > Not sure why we are coming up with so many new techniques when ECN just
>> works fine.
>> > ECN is a 2 bit field (not 1 bit) and seems to be sufficient to indicate
>> extent of congestion by marking it per packet. Adding more complexity to
>> any layer whether it is L2 or L3 doesn’t work well in deployments. Our goal
>> should be to simplify things and only add new headers if absolutely
>> necessary.
>> >
>> > Vidhi
>> >
>> >
>> > On Sep 12, 2023, at 3:12 AM, Shihang(Vincent) <shihang9=
>> [e-mail]> wrote:
>> >
>> > Hi,
>> > I agree L2 may not be the best choice to carry the congestion signaling
>> end-to-end and more bits are needed. We have submitted a draft to carry the
>> multi-bits congestion signaling in L3. We call it Advanced ECN. See
>> https://datatracker.ietf.org/doc/draft-shi-ccwg-advanced-ecn/.
>> >
>> > Thanks,
>> > Hang
>> >
>> > From: CCWG <[e-mail]> On Behalf Of Huangyihong (Rachel)
>> > Sent: Tuesday, September 12, 2023 5:41 PM
>> > To: Tom Herbert <tom=[e-mail]>; Abhiram Ravi
>> <abhiramr=[e-mail]>
>> > Cc: IETF IPPM WG <[e-mail]>; tsvwg <[e-mail]>; [e-mail];
>> [e-mail]; Nandita Dukkipati <[e-mail]>; Naoshad Mehta <
>> [e-mail]>; Jai Kumar <[e-mail]>
>> > Subject: Re: [CCWG] [iccrg] [tsvwg] New Internet Draft: Congestion
>> Signaling (CSIG)
>> >
>> > Hi,
>> >
>> > I also have the same feeling. Implementing in L2 may be difficult to be
>> used in e2e transport. Of course it can work well in limited domain, like
>> DC or HPC clusters. However, I also look for some solutions that could be
>> able to go through internet. We have submitted a draft to describe the
>> transport challenges. See
>> https://datatracker.ietf.org/doc/html/draft-huang-tsvwg-transport-challenges
>> .
>> >
>> > I share the same opinion that the congestion signal is useful and
>> current 1-bit ECN solution is not fully sufficient. But I also feel like
>> the more straight way is to extend L3, or l4, like update IOAM, to carry
>> the information. For L2 solution, it should be developed together with IEEE
>> 802.1.
>> >
>> > BR，
>> > Rachel
>> >
>> > 发件人: iccrg <[e-mail]> 代表 Tom Herbert
>> > 发送时间: 2023年9月10日 0:10
>> > 收件人: Abhiram Ravi <abhiramr=[e-mail]>
>> > 抄送: IETF IPPM WG <[e-mail]>; tsvwg <[e-mail]>; [e-mail];
>> [e-mail]; Nandita Dukkipati <[e-mail]>; Naoshad Mehta <
>> [e-mail]>; Jai Kumar <[e-mail]>
>> > 主题: Re: [iccrg] [tsvwg] New Internet Draft: Congestion Signaling (CSIG)
>> >
>> > Hi, thanks for draft!
>> >
>> > The first thing that stands out to me is the carrier of the new packet
>> headers. In the forward path it would be in L2 and in reflection it would
>> be L4. As the draft describes, this would entail having to support the
>> protocol in multiple L2 and multiple L4 protocols-- that's going to be a
>> pretty big lift! Also, L2 is not really an end-to-end protocol (would
>> legacy switches in the path also forward the header)l?).
>> >
>> > The signaling being described in the draft is network layer
>> information, and hence IMO should be conveyed in network layer headers.
>> That's is L3 which conveniently is the average of L2+L4 :-)
>> >
>> > IMO, the proper carrier of the signal data is Hop-by-Hop Options. This
>> is end-to-end and allows modification of data in-flight. The typical
>> concern with Hop-by-Hop Options is high drop rates on the Internet, however
>> in this case the protocol is explicitly confined to a limited domain so I
>> don't see that as a blocking issue for this use case.
>> >
>> > The information being carried seems very similar to that of IOAM (IOAM
>> uses Hop-by-Hop Options and supports reflection). I suppose the differences
>> are that this protocol is meant to be consumed by the transport Layer and
>> the data is a condensed summary of path characteristics. IOAM seems pretty
>> extensible, so maybe it could be adapted to carry the signals of this draft?
>> >
>> > A related proposal might be FAST draft-herbert-fast. Where the CSIG is
>> network to host signaling, FAST is host to network signaling for the
>> purposes of requesting network services. These might be complementary and
>> options for both may be in the same packet. FAST also uses reflection, so
>> we might be able to leverage some common implementation at a destination.
>> >
>> > Tom
>> >
>> > On Fri, Sep 8, 2023, 7:43 PM Abhiram Ravi <abhiramr=
>> [e-mail]> wrote:
>> > Hi IPPM folks,
>> >
>> > I am pleased to announce the publication of a new internet draft,
>> Congestion Signaling (CSIG):
>> https://datatracker.ietf.org/doc/draft-ravi-ippm-csig/
>> >
>> > CSIG is a new end-to-end packet header mechanism for in-band signaling
>> that is simple, efficient, deployable, and grounded in concrete use cases
>> of congestion control, traffic management, and network debuggability. We
>> believe that CSIG is an important new protocol that builds on top of
>> existing in-band network telemetry protocols.
>> >
>> > We encourage you to read the CSIG draft and provide your feedback and
>> comments. We have also cc'd the TSVWG, CCWG, and ICCRG mailing lists, as we
>> believe that this work may be of interest to their members as well.
>> >
>> > Thank you for your time and consideration.
>> >
>> > Sincerely,
>> > Abhiram Ravi
>> > On behalf of the CSIG authors
>>
>>
>
> --
> Abhiram Ravi
> _______________________________________________
> ippm mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/ippm
>