Re: [iccrg] [tsvwg] New Internet Draft: Congestion Signaling (CSIG)

[Sebastian Moeller <[e-mail]>]

Hi Rodney,

thanks for the information. I intended no harm when mentioning SCE, my clearly too terse reference was only meant to highlight that with-in the ECN bitfield there is potential for 2 explict congestion states as well as one no-congestion state (so 3 states informative of congestion). I think SCE is not the only draft/RFC that made better use of the 2bit ECN field, but it was the only one I looked at recently so it was fresher in my memory.


> On Sep 14, 2023, at 21:23, Rodney W. Grimes <[e-mail]> wrote:
> 
> Hello Sebastian, etc,
> 	I just want to insert a clarification on SCE, and perhaps a different
> view on the ECN ECT[0] and ECT[1] bits.  In line [RWG]
> 
> Regards,
> Rod Grimes
> 
>> Hi Ingemar
>> 
>>> On Sep 14, 2023, at 08:42, Ingemar Johansson S <[e-mail]> wrote:
>>> 
>>> Hi Ruediger
>>> 
>>> Even though CSIG is on ethernet, it appears to be e2e as the feedback is on L4. So I guess somehow the CSIG info needs to jump from domain to domain somewhow, and that sounds to me like L3, albeit perhaps brief jumps ? 
>> 
>> 	[SM] rfc3168 and L4S ECN signaling faces the same hurdle, forward path uses a different layer than feed-back path. And I do not think that it is conceptually all that cleaner to have fewer layers between forward and reverse signaling... "clean" would be to put both forward and reverse information into the same layer, but that is not on the table as far as I can see.
>> 	The other difference however, richness of signal, is a clear difference between 1 bit ECN and multibit CSIG (and alternatives). (Side-note, sure the ECN bitfield is two bits but it only carries 2 congestion states when ECN is in use, which makes it IMHO a 1-bit information channel, if we had followed the SCE proposal with its 3 congestion states we would be at 1.5 bits ;))
>> Regards
>> 	Sebastian
> 
> [RWG]
> First off, IMHO, we should always treat ECT[0] and ECT[1] togeather as
> the ECN field, which leads us to 4 possible states per packet, and we
> should not discuss these bits in any other way.  This applies to not
> only SCE, but to rfc3168 ECN, DCTCP and L4S.
> 
> Second SCE uses these 4 states as 2 input states to the network
> (Not-ECT and ECT) and 2 output states (SCE and CE):
> [From draft-morton-tsvwg-sce-03.txt secton 4]
> Binary  Keyword                                   References
> 00      Not-ECT   (Not ECN-Capable Transport)     [RFC3168]
> 01      SCE       (Some Congestion Experienced)   [This draft]
> 10      ECT       (ECN-Capable Transport)         [RFC3168]
> 11      CE        (Congestion Experienced)        [RFC3168]
> 
> This 2 states gives in effect double the signalling of Classic
> ECN or L4S with thier single output state "CE".

	[SM] The way I count SCE is:
10 ECT -> explicit information of no congestion
01 SCE -> explicit information of some congestion
11 CE  -> explicit information of "normal" congestion
=> total of 3 different states


while rfc3168/L4s only encode 2 congestion states
10 ECT -> explicit information of no congestion
01 ECT -> explicit information of no congestion
11 CE  -> explicit information of "normal" congestion
=>  total of 2 different states


both, for backward compatibility need to invest one code point to force dropping on packets...


> 
> For those that wish to play, the SCE drafts HAVE been updated
> by using DSCP's to allow safe experimental operation, even in
> a L4S network.

	[SM] Great, this is IMHO the best way to allow meaningful experiments, by requiring active opt-in (DSCPs are not perfect, but certainly better than no opt-in method at all).


> 
> For those that want to read a good paper on how many bits are
> needed I suggest "Switches Know the Exact Amount of Congestion"
> https://web.stanford.edu/~sarslan/files/Switches_Know_The_Exact_Amount_of_Congestion___BufferSizingWorkshop.pdf

	[SM] This is a great read! It also makes a decent point in showing the value of multibit signals over single bit (and serialization of multibit states over a single bit channel).

Regards
	Sebastian


> 
> Regards,
> Rod Grimes
> 
>> 
>>> 
>>> /Ingemar
>>> 
>>>> -----Original Message-----
>>>> From: [e-mail] <[e-mail]>
>>>> Sent: Wednesday, 13 September 2023 14:20
>>>> To: [e-mail]; Ingemar Johansson S
>>>> <[e-mail]>
>>>> Cc: [e-mail]; [e-mail]; [e-mail];
>>>> [e-mail]; [e-mail]; [e-mail];
>>>> [e-mail]; [e-mail]; [e-mail]; [e-mail];
>>>> [e-mail]; [e-mail]
>>>> Subject: AW: [tsvwg] [iccrg] New Internet Draft: Congestion Signaling
>>>> (CSIG)
>>>> 
>>>> Hi Ingemar, hi Sebastian,
>>>> 
>>>> Skimming over the draft only, isn't CSIG about Ethernet-domains, while
>>>> L4S is E2E?
>>>> 
>>>> Regards,
>>>> 
>>>> Ruediger
>>>> 
>>>> -----Urspr?ngliche Nachricht-----
>>>> Von: tsvwg <[e-mail]> Im Auftrag von Sebastian Moeller
>>>> Gesendet: Mittwoch, 13. September 2023 13:42
>>>> An: Ingemar Johansson S
>>>> <ingemar.s.johansson=[e-mail]>
>>>> Cc: Vidhi Goel <vidhi_goel=[e-mail]>; Shihang(Vincent)
>>>> <shihang9=[e-mail]>; tsvwg <[e-mail]>; Jai
>>>> Kumar <[e-mail]>; IETF IPPM WG <[e-mail]>; Tom
>>>> Herbert <tom=[e-mail]>; [e-mail]; Abhiram
>>>> Ravi <abhiramr=[e-mail]>; Nandita Dukkipati
>>>> <[e-mail]>; [e-mail]; Huangyihong (Rachel)
>>>> <rachel.huang=[e-mail]>; Naoshad Mehta
>>>> <[e-mail]>
>>>> Betreff: Re: [tsvwg] [iccrg] New Internet Draft: Congestion Signaling
>>>> (CSIG)
>>>> 
>>>> Hi Ingemar,
>>>> 
>>>> 
>>>>> On Sep 13, 2023, at 12:30, Ingemar Johansson S
>>>> <ingemar.s.johansson=[e-mail]> wrote:
>>>>> 
>>>>> Hi
>>>>> 
>>>>> I agree with Vihdi
>>>>> 
>>>>> L4S is recently standardised
>>>> 
>>>> 	[SM] In experimental track, the goal is currently to test whether
>>>> it can/should be deployed at scale....
>>>> 
>>>>> and it is definitely gaining traction also in 3GPP. We have an echo
>>>> system that is looking forward to having L4S widely deployed.
>>>>> Still, the congestion control aspects are not fully explored yet. One
>>>> interesting topic is if L4S allows to more safely deviate from additive
>>>> increase to make congestion control algorithms more quickly converge to
>>>> higher link capacity. There are a number of study topics around L4S
>>>> congestion control that are listed in e.g the TCP Prague draft.
>>>>> 
>>>>> I cannot dictate what others should do with their time and money but
>>>> personally I'd prefer that the IETF explores L4S and its possibilities
>>>> and downsides before jumping on the next idea.
>>>> 
>>>> 	[SM] L4S can be described as taking the ideas behind DCTCP and
>>>> making them fit for use over the internet*. Yet the signaling discussed
>>>> here is to be used in e.g. data center contexts where DCTCP is already
>>>> used and found lacking compared to newer methods operating on richer
>>>> congestion information (HPCC, Swift, Poseidon, ...).
>>>> 	Given that L4S essentially uses a multi-packet signal(**) to report
>>>> the "queue filling state" that is then stochastically distributed over
>>>> all concurrent flows, it seems obvious to me that reconstructing a
>>>> reliable estimate of on-path queueing will take some time and averaging
>>>> for each individual flow, I would guess that in some environments this
>>>> delay simply is too costly.
>>>> 	So L4S and CSIG seem complementary and in no way mutually
>>>> exclusive.
>>>> 
>>>> 
>>>> Regards
>>>> 	Sebastian
>>>> 
>>>> 
>>>> 
>>>> *) I will not further discuss whether that is achieved or not as it
>>>> seems irrelevant here.
>>>> **) In essence transmission of congestion state via a 1-bit serial
>>>> channel, clocked at the (variable***) packet rate at the bottleneck.
>>>> ***) as packets are not of uniform size
>>>> 
>>>>> 
>>>>> CSIG sounds to me like something that belongs more in ICCRG or ?
>>>>> 
>>>>> /Ingemar
>>>>> 
>>>>> From: tsvwg <[e-mail]> On Behalf Of Vidhi Goel
>>>>> Sent: Wednesday, 13 September 2023 00:59
>>>>> To: Shihang(Vincent) <shihang9=[e-mail]>
>>>>> Cc: Huangyihong (Rachel) <rachel.huang=[e-mail]>;
>>>> Tom Herbert <tom=[e-mail]>; Abhiram Ravi
>>>> <abhiramr=[e-mail]>; IETF IPPM WG <[e-mail]>;
>>>> tsvwg <[e-mail]>; [e-mail]; [e-mail]; Nandita Dukkipati
>>>> <[e-mail]>; Naoshad Mehta <[e-mail]>; Jai Kumar
>>>> <[e-mail]>
>>>>> Subject: Re: [tsvwg] [iccrg] New Internet Draft: Congestion Signaling
>>>> (CSIG)
>>>>> 
>>>>> Not sure why we are coming up with so many new techniques when ECN
>>>> just works fine.
>>>>> ECN is a 2 bit field (not 1 bit) and seems to be sufficient to
>>>> indicate extent of congestion by marking it per packet. Adding more
>>>> complexity to any layer whether it is L2 or L3 doesn?t work well in
>>>> deployments. Our goal should be to simplify things and only add new
>>>> headers if absolutely necessary.
>>>>> 
>>>>> Vidhi
>>>>> 
>>>>> 
>>>>> On Sep 12, 2023, at 3:12 AM, Shihang(Vincent)
>>>> <shihang9=[e-mail]> wrote:
>>>>> 
>>>>> Hi,
>>>>> I agree L2 may not be the best choice to carry the congestion
>>>> signaling end-to-end and more bits are needed. We have submitted a draft
>>>> to carry the multi-bits congestion signaling in L3. We call it Advanced
>>>> ECN. See https://datatracker.ietf.org/doc/draft-shi-ccwg-advanced-ecn/.
>>>>> 
>>>>> Thanks,
>>>>> Hang
>>>>> 
>>>>> From: CCWG <[e-mail]> On Behalf Of Huangyihong (Rachel)
>>>>> Sent: Tuesday, September 12, 2023 5:41 PM
>>>>> To: Tom Herbert <tom=[e-mail]>; Abhiram Ravi
>>>> <abhiramr=[e-mail]>
>>>>> Cc: IETF IPPM WG <[e-mail]>; tsvwg <[e-mail]>;
>>>> [e-mail]; [e-mail]; Nandita Dukkipati <[e-mail]>;
>>>> Naoshad Mehta <[e-mail]>; Jai Kumar <[e-mail]>
>>>>> Subject: Re: [CCWG] [iccrg] [tsvwg] New Internet Draft: Congestion
>>>> Signaling (CSIG)
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> I also have the same feeling. Implementing in L2 may be difficult to
>>>> be used in e2e transport. Of course it can work well in limited domain,
>>>> like DC or HPC clusters. However, I also look for some solutions that
>>>> could be able to go through internet. We have submitted a draft to
>>>> describe the transport challenges. See
>>>> https://datatracker.ietf.org/doc/html/draft-huang-tsvwg-transport-
>>>> challenges.
>>>>> 
>>>>> I share the same opinion that the congestion signal is useful and
>>>> current 1-bit ECN solution is not fully sufficient. But I also feel like
>>>> the more straight way is to extend L3, or l4, like update IOAM, to carry
>>>> the information. For L2 solution, it should be developed together with
>>>> IEEE 802.1.
>>>>> 
>>>>> BR?
>>>>> Rachel
>>>>> 
>>>>> ???: iccrg <[e-mail]> ?? Tom Herbert
>>>>> ????: 2023?9?10? 0:10
>>>>> ???: Abhiram Ravi <abhiramr=[e-mail]>
>>>>> ??: IETF IPPM WG <[e-mail]>; tsvwg <[e-mail]>;
>>>> [e-mail]; [e-mail]; Nandita Dukkipati <[e-mail]>;
>>>> Naoshad Mehta <[e-mail]>; Jai Kumar <[e-mail]>
>>>>> ??: Re: [iccrg] [tsvwg] New Internet Draft: Congestion Signaling
>>>> (CSIG)
>>>>> 
>>>>> Hi, thanks for draft!
>>>>> 
>>>>> The first thing that stands out to me is the carrier of the new packet
>>>> headers. In the forward path it would be in L2 and in reflection it
>>>> would be L4. As the draft describes, this would entail having to support
>>>> the protocol in multiple L2 and multiple L4 protocols-- that's going to
>>>> be a pretty big lift! Also, L2 is not really an end-to-end protocol
>>>> (would legacy switches in the path also forward the header)l?).
>>>>> 
>>>>> The signaling being described in the draft is network layer
>>>> information, and hence IMO should be conveyed in network layer headers.
>>>> That's is L3 which conveniently is the average of L2+L4 :-)
>>>>> 
>>>>> IMO, the proper carrier of the signal data is Hop-by-Hop Options. This
>>>> is end-to-end and allows modification of data in-flight. The typical
>>>> concern with Hop-by-Hop Options is high drop rates on the Internet,
>>>> however in this case the protocol is explicitly confined to a limited
>>>> domain so I don't see that as a blocking issue for this use case.
>>>>> 
>>>>> The information being carried seems very similar to that of IOAM (IOAM
>>>> uses Hop-by-Hop Options and supports reflection). I suppose the
>>>> differences are that this protocol is meant to be consumed by the
>>>> transport Layer and the data is a condensed summary of path
>>>> characteristics. IOAM seems pretty extensible, so maybe it could be
>>>> adapted to carry the signals of this draft?
>>>>> 
>>>>> A related proposal might be FAST draft-herbert-fast. Where the CSIG is
>>>> network to host signaling, FAST is host to network signaling for the
>>>> purposes of requesting network services. These might be complementary
>>>> and options for both may be in the same packet. FAST also uses
>>>> reflection, so we might be able to leverage some common implementation
>>>> at a destination.
>>>>> 
>>>>> Tom
>>>>> 
>>>>> On Fri, Sep 8, 2023, 7:43 PM Abhiram Ravi
>>>> <abhiramr=[e-mail]> wrote:
>>>>> Hi IPPM folks,
>>>>> 
>>>>> I am pleased to announce the publication of a new internet draft,
>>>> Congestion Signaling (CSIG): https://datatracker.ietf.org/doc/draft-
>>>> ravi-ippm-csig/
>>>>> 
>>>>> CSIG is a new end-to-end packet header mechanism for in-band signaling
>>>> that is simple, efficient, deployable, and grounded in concrete use
>>>> cases of congestion control, traffic management, and network
>>>> debuggability. We believe that CSIG is an important new protocol that
>>>> builds on top of existing in-band network telemetry protocols.
>>>>> 
>>>>> We encourage you to read the CSIG draft and provide your feedback and
>>>> comments. We have also cc'd the TSVWG, CCWG, and ICCRG mailing lists, as
>>>> we believe that this work may be of interest to their members as well.
>>>>> 
>>>>> Thank you for your time and consideration.
>>>>> 
>>>>> Sincerely,
>>>>> Abhiram Ravi
>>>>> On behalf of the CSIG authors
>>> 
>> 
>> _______________________________________________
>> iccrg mailing list
>> [e-mail]
>> https://www.irtf.org/mailman/listinfo/iccrg
>> 
>> 
> 
> -- 
> Rod Grimes                                                 [e-mail]
> 
> _______________________________________________
> iccrg mailing list
> [e-mail]
> https://www.irtf.org/mailman/listinfo/iccrg