Re: [ippm] [tsvwg] [iccrg] New Internet Draft: Congestion Signaling (CSIG)

[Abhiram Ravi <[e-mail]>]

Thank you for your comments. Responses inline.

> Tom: The signaling being described in the draft is network layer
information, and hence IMO should be conveyed in network layer headers.
That's is L3 which conveniently is the average of L2+L4 :-)
> Rachel:  Implementing in L2 may be difficult to be used in e2e transport.
> Hang: I agree L2 may not be the best choice to carry the congestion
signaling end-to-end and more bits are needed.

The choice of using L2+L4 is an intentional and important design choice,
one that is driven by need, for practicality and ease of deployment in data
center networks. We acknowledge that previous approaches have attempted to
extend L3 or L4, but these approaches do not meet the requirements for
deployability in a heterogeneous network, including interoperability and
backward compatibility. CSIG targets specific yet important use cases while
meeting these requirements.

Section 7.1
<https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-choice-of-layer-2>
of the draft explicitly addresses why such a design was chosen.

The key reasons are:

   -

   Simplicity of device implementations of CSIG (both at NICs and
   switches), given the location of the CSIG-tag in the packet header is not
   affected by any header at layer 3 and above. CSIG-tag being the last tag
   before the ethertype also makes it simple for ASICs to implement it inside
   the L2 header.
   -

   Support for delivering congestion signals even in the presence of
   in-network tunneling and encryption, which are commonly used in data
   centers, especially for Cloud traffic and in Traffic engineered domains.
   CSIG tags are carried through Layer 3 tunnels e.g., IP-in-IP, VxLAN,
   Geneve, at a fixed offset in the packet header. This avoids the need to
   copy and relocate CSIG tags across inner / outer headers during
   encapsulation and decapsulation of packets, which is a complex hardware
   operation and would be necessary if implemented instead at layers 3 or
   higher.

> Tom: The signaling being described in the draft is network layer
information, and hence IMO should be conveyed in network layer headers.

The information carried in L2 CSIG-tags are device signals or port signals
from the bottleneck, and should not be viewed as network-layer signals.
Although the bottleneck signal is also a property of the packet path in
aggregate, the signal carried on the tag has a scope of one hop i.e., the
measurement on one specific link or device on the path. The tag gets
conditionally swapped on each device in a compare-and-replace manner as
described in Section 4.3
<https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-csig-operation-life-of-a-pa>

Reflection is a property of the transport as described in Section 4.2, and
hence CSIG reflection is implemented at layer 4 and is specific to the
transport. This allows a clean separation of the mechanism used to collect
signals from the network from the mechanism used for leveraging those
signals at the transport.

> Tom: Also, L2 is not really an end-to-end protocol (would legacy switches
in the path also forward the header)

Legacy devices can not only forward the packet, but also participate in the
CSIG protocol via a software-assisted approach by treating CSIG-tags as
VLAN-tags and leveraging VLAN resources to perform CSIG updates. This L2
approach is the only solution we’ve found which allows immediate deployment
for certain signals with legacy devices in data centers, while native
support for CSIG is easily achievable in future devices. CSIG stripping can
also be exercised for interop when needed - see Section 6
<https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-incremental-deployment-of-c>
for interoperability.

> Tom: As the draft describes, this would entail having to support the
protocol in multiple L2 and multiple L4 protocols

Section 4.1
<https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-csig-tag-header-format>
lists how CSIG-tag operates with each relevant L2 encapsulation and Section
4.2
<https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-csig-reflection-header-form>
addresses implementation of reflection in relevant transports.

> Tom: IMO, the proper carrier of the signal data is Hop-by-Hop Options.
IOAM seems pretty extensible, so maybe it could be adapted to carry the
signals of this draft?
> Rachel: Of course it can work well in limited domain, like DC or HPC
clusters. However, I also look for some solutions that could be able to go
through internet

CSIG is targeted at specific use cases in limited domains, primarily data
centers, for congestion control, traffic management and network
debuggability. The requirements for such use cases vary significantly from
those of internet deployments, which is not in scope for CSIG.
IOAM and hop-by-hop options in general are tuned for a broader set of use
cases and do not meet all the requirements for the specific use cases that
CSIG addresses, including interop with legacy devices and in-network
tunneling, due to the placement of the header.  Section 2
<https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-design-principles>
speaks to these requirements.

> Rachel: For L2 solution, it should be developed together with IEEE 802.1.

Acknowledging that IEEE will be involved as well.

> Sebastian: What happens if bottlenecks hand out immediate capacity/rate
estimates and all greedy flows start increasing their cwin really fast?

CSIG supports both min(ABW) and min(ABW/C). Using min(ABW/C), the each flow
can increase its rate in proportion to the % available bandwidth, so that
in aggregate the flows respect the available bottleneck bandwidth with
reduced risk of overshoot. See Section 8.1.2
<https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-using-maximum-link-utilizat>
.

> Ingemar: CSIG sounds to me like something that belongs more in ICCRG

CSIG is applicable to use cases beyond Congestion control (which is an
important one), such as traffic management and network debugging. Section 8
<https://datatracker.ietf.org/doc/html/draft-ravi-ippm-csig-00#name-use-cases-defined-by-bottle>
describes the use cases in more detail.

> Tom: A related proposal might be FAST draft-herbert-fast. These might be
complementary and options for both may be in the same packet
> Hang: We call it Advanced ECN. See draft-shi-ccwg-advanced-ecn
> Ingemar: L4S is recently standardised and it is definitely gaining
traction also in 3GPP

Thank you for the pointers. We agree that these proposals are all
complementary to CSIG.


On Wed, Sep 13, 2023 at 5:20 AM <[e-mail]> wrote:

> Hi Ingemar, hi Sebastian,
>
> Skimming over the draft only, isn't CSIG about Ethernet-domains, while L4S
> is E2E?
>
> Regards,
>
> Ruediger
>
> -----Ursprüngliche Nachricht-----
> Von: tsvwg <[e-mail]> Im Auftrag von Sebastian Moeller
> Gesendet: Mittwoch, 13. September 2023 13:42
> An: Ingemar Johansson S <ingemar.s.johansson=[e-mail]
> >
> Cc: Vidhi Goel <vidhi_goel=[e-mail]>; Shihang(Vincent)
> <shihang9=[e-mail]>; tsvwg <[e-mail]>; Jai Kumar
> <[e-mail]>; IETF IPPM WG <[e-mail]>; Tom Herbert <tom=
> [e-mail]>; [e-mail]; Abhiram Ravi <abhiramr=
> [e-mail]>; Nandita Dukkipati <[e-mail]>;
> [e-mail]; Huangyihong (Rachel) <rachel.huang=
> [e-mail]>; Naoshad Mehta <[e-mail]>
> Betreff: Re: [tsvwg] [iccrg] New Internet Draft: Congestion Signaling
> (CSIG)
>
> Hi Ingemar,
>
>
> > On Sep 13, 2023, at 12:30, Ingemar Johansson S <ingemar.s.johansson=
> [e-mail]> wrote:
> >
> > Hi
> >
> > I agree with Vihdi
> >
> > L4S is recently standardised
>
>         [SM] In experimental track, the goal is currently to test whether
> it can/should be deployed at scale....
>
> > and it is definitely gaining traction also in 3GPP. We have an echo
> system that is looking forward to having L4S widely deployed.
> > Still, the congestion control aspects are not fully explored yet. One
> interesting topic is if L4S allows to more safely deviate from additive
> increase to make congestion control algorithms more quickly converge to
> higher link capacity. There are a number of study topics around L4S
> congestion control that are listed in e.g the TCP Prague draft.
> >
> > I cannot dictate what others should do with their time and money but
> personally I'd prefer that the IETF explores L4S and its possibilities and
> downsides before jumping on the next idea.
>
>         [SM] L4S can be described as taking the ideas behind DCTCP and
> making them fit for use over the internet*. Yet the signaling discussed
> here is to be used in e.g. data center contexts where DCTCP is already used
> and found lacking compared to newer methods operating on richer congestion
> information (HPCC, Swift, Poseidon, ...).
>         Given that L4S essentially uses a multi-packet signal(**) to
> report the "queue filling state" that is then stochastically distributed
> over all concurrent flows, it seems obvious to me that reconstructing a
> reliable estimate of on-path queueing will take some time and averaging for
> each individual flow, I would guess that in some environments this delay
> simply is too costly.
>         So L4S and CSIG seem complementary and in no way mutually
> exclusive.
>
>
> Regards
>         Sebastian
>
>
>
> *) I will not further discuss whether that is achieved or not as it seems
> irrelevant here.
> **) In essence transmission of congestion state via a 1-bit serial
> channel, clocked at the (variable***) packet rate at the bottleneck.
> ***) as packets are not of uniform size
>
> >
> > CSIG sounds to me like something that belongs more in ICCRG or ?
> >
> > /Ingemar
> >
> > From: tsvwg <[e-mail]> On Behalf Of Vidhi Goel
> > Sent: Wednesday, 13 September 2023 00:59
> > To: Shihang(Vincent) <shihang9=[e-mail]>
> > Cc: Huangyihong (Rachel) <rachel.huang=[e-mail]>;
> Tom Herbert <tom=[e-mail]>; Abhiram Ravi
> <abhiramr=[e-mail]>; IETF IPPM WG <[e-mail]>;
> tsvwg <[e-mail]>; [e-mail]; [e-mail]; Nandita Dukkipati <
> [e-mail]>; Naoshad Mehta <[e-mail]>; Jai Kumar <
> [e-mail]>
> > Subject: Re: [tsvwg] [iccrg] New Internet Draft: Congestion Signaling
> (CSIG)
> >
> > Not sure why we are coming up with so many new techniques when ECN just
> works fine.
> > ECN is a 2 bit field (not 1 bit) and seems to be sufficient to indicate
> extent of congestion by marking it per packet. Adding more complexity to
> any layer whether it is L2 or L3 doesn’t work well in deployments. Our goal
> should be to simplify things and only add new headers if absolutely
> necessary.
> >
> > Vidhi
> >
> >
> > On Sep 12, 2023, at 3:12 AM, Shihang(Vincent) <shihang9=
> [e-mail]> wrote:
> >
> > Hi,
> > I agree L2 may not be the best choice to carry the congestion signaling
> end-to-end and more bits are needed. We have submitted a draft to carry the
> multi-bits congestion signaling in L3. We call it Advanced ECN. See
> https://datatracker.ietf.org/doc/draft-shi-ccwg-advanced-ecn/.
> >
> > Thanks,
> > Hang
> >
> > From: CCWG <[e-mail]> On Behalf Of Huangyihong (Rachel)
> > Sent: Tuesday, September 12, 2023 5:41 PM
> > To: Tom Herbert <tom=[e-mail]>; Abhiram Ravi
> <abhiramr=[e-mail]>
> > Cc: IETF IPPM WG <[e-mail]>; tsvwg <[e-mail]>; [e-mail];
> [e-mail]; Nandita Dukkipati <[e-mail]>; Naoshad Mehta <
> [e-mail]>; Jai Kumar <[e-mail]>
> > Subject: Re: [CCWG] [iccrg] [tsvwg] New Internet Draft: Congestion
> Signaling (CSIG)
> >
> > Hi,
> >
> > I also have the same feeling. Implementing in L2 may be difficult to be
> used in e2e transport. Of course it can work well in limited domain, like
> DC or HPC clusters. However, I also look for some solutions that could be
> able to go through internet. We have submitted a draft to describe the
> transport challenges. See
> https://datatracker.ietf.org/doc/html/draft-huang-tsvwg-transport-challenges
> .
> >
> > I share the same opinion that the congestion signal is useful and
> current 1-bit ECN solution is not fully sufficient. But I also feel like
> the more straight way is to extend L3, or l4, like update IOAM, to carry
> the information. For L2 solution, it should be developed together with IEEE
> 802.1.
> >
> > BR，
> > Rachel
> >
> > 发件人: iccrg <[e-mail]> 代表 Tom Herbert
> > 发送时间: 2023年9月10日 0:10
> > 收件人: Abhiram Ravi <abhiramr=[e-mail]>
> > 抄送: IETF IPPM WG <[e-mail]>; tsvwg <[e-mail]>; [e-mail];
> [e-mail]; Nandita Dukkipati <[e-mail]>; Naoshad Mehta <
> [e-mail]>; Jai Kumar <[e-mail]>
> > 主题: Re: [iccrg] [tsvwg] New Internet Draft: Congestion Signaling (CSIG)
> >
> > Hi, thanks for draft!
> >
> > The first thing that stands out to me is the carrier of the new packet
> headers. In the forward path it would be in L2 and in reflection it would
> be L4. As the draft describes, this would entail having to support the
> protocol in multiple L2 and multiple L4 protocols-- that's going to be a
> pretty big lift! Also, L2 is not really an end-to-end protocol (would
> legacy switches in the path also forward the header)l?).
> >
> > The signaling being described in the draft is network layer information,
> and hence IMO should be conveyed in network layer headers. That's is L3
> which conveniently is the average of L2+L4 :-)
> >
> > IMO, the proper carrier of the signal data is Hop-by-Hop Options. This
> is end-to-end and allows modification of data in-flight. The typical
> concern with Hop-by-Hop Options is high drop rates on the Internet, however
> in this case the protocol is explicitly confined to a limited domain so I
> don't see that as a blocking issue for this use case.
> >
> > The information being carried seems very similar to that of IOAM (IOAM
> uses Hop-by-Hop Options and supports reflection). I suppose the differences
> are that this protocol is meant to be consumed by the transport Layer and
> the data is a condensed summary of path characteristics. IOAM seems pretty
> extensible, so maybe it could be adapted to carry the signals of this draft?
> >
> > A related proposal might be FAST draft-herbert-fast. Where the CSIG is
> network to host signaling, FAST is host to network signaling for the
> purposes of requesting network services. These might be complementary and
> options for both may be in the same packet. FAST also uses reflection, so
> we might be able to leverage some common implementation at a destination.
> >
> > Tom
> >
> > On Fri, Sep 8, 2023, 7:43 PM Abhiram Ravi <abhiramr=
> [e-mail]> wrote:
> > Hi IPPM folks,
> >
> > I am pleased to announce the publication of a new internet draft,
> Congestion Signaling (CSIG):
> https://datatracker.ietf.org/doc/draft-ravi-ippm-csig/
> >
> > CSIG is a new end-to-end packet header mechanism for in-band signaling
> that is simple, efficient, deployable, and grounded in concrete use cases
> of congestion control, traffic management, and network debuggability. We
> believe that CSIG is an important new protocol that builds on top of
> existing in-band network telemetry protocols.
> >
> > We encourage you to read the CSIG draft and provide your feedback and
> comments. We have also cc'd the TSVWG, CCWG, and ICCRG mailing lists, as we
> believe that this work may be of interest to their members as well.
> >
> > Thank you for your time and consideration.
> >
> > Sincerely,
> > Abhiram Ravi
> > On behalf of the CSIG authors
>
>

-- 
Abhiram Ravi