Re: [Detnet] Role of the transport layer in the DetNet Architecture

["Black, David" <[e-mail]>]

Hi Pascal,

> Now it's getting really interesting (thanks Lou!, and thanks you too!).
Cool - add this discussion to the list of evidence for in-person IETF meetings being important & productive ...

Top posting to try to summarize ...

I think we have two related goals in this discussion that ought to be distinguished:
	- What's the minimal/necessary/ideal L4 transport functionality for DetNet?
	- What would it take to run an existing transport stack (L4 and above) over DetNet?
This discussion started around the first goal, where I agree with you that the ECN-based dynamic windowing functionality is not needed.  That functionality may be useful towards the second goal (e.g., run HTTP/TLS/QUIC over a DetNet data plane for device management).

I think we're in agreement that a DetNet-aware shaper is an important sender component - the shaper's primary responsibility is to send packets when they're supposed to be sent wrt the DetNet SLA - not early, not late.

On dealing with loss, I think this is fine:
> Not in plan. We cannot afford end to end ARQ. We want FEC and PREOF instead. My question to you is whether you call those things L3, because that would change my maybe naïve view of what L3 and L4 do.
When deployed in-network (how PREOF almost always appears in DetNet architecture diagrams), those things are L2 or L3 because as viewed from endpoint stacks, they're transparent to L4 functionality - TCP, UDP, etc.

Whatever is done in this area will want application of both DetNet and Transport expertise, as DetNet runs contrary to some typical Transport expectations (e.g., sender pacing is not helpful for at least a CQF-based DetNet data plane).

Thanks, --David

-----Original Message-----
From: Pascal Thubert (pthubert) <[e-mail]> 
Sent: Friday, July 29, 2022 12:37 PM
To: Black, David; Pascal Thubert (pthubert); DetNet WG; [e-mail]
Cc: Balázs Varga A; Black, David
Subject: RE: Role of the transport layer in the DetNet Architecture


[EXTERNAL EMAIL] 

Hello David,

Now it's getting really interesting (thanks Lou!, and thanks you too!).

Please note that I listed TCP not as the protocol I'd suggest to modify but to illustrate the do's and don'ts that DetNet would need. I have no idea if there is a bets match that would be changed or if a thin layer above UDP (e.g. to aggregate into DetNet expected block size), in combination with lower layers (e.g., to pull at DetNet expected times) could do the trick. Or else. IOW, I'm not in solution space yet. I'm trying to figure conceptually what belongs where since we have all those nice layers at IETF and those other nice layers at DetNet.

> So, starting from your transport layer goals:

I did not mean to be exhaustive but to steer the WG into writing those requirements. I do not mean to propose a solution either, or to work on one inside the DetNet group, but maybe to steer enough interest in TSV area to get something running.


> 
> > Bottom line, we'd want the application to have to support the minimum
> > and let the transport layer pack the data to fit the DetNet SLA and send
> it at the appropriate time, which could effectively be signaled over the
> UNI.
> > The stream behavior of TCP is appropriate, but the dynamic windowing and
> the retries and undesirable.
> > TCP has reordering but does not do PREOF along the path.
> 
> I see four pieces here that ought to be teased apart:
>    1) Transmission timing ("pack the data to fit the DetNet SLA")
>    2) TCP (and the like) dynamic windowing
>    3) Loss and retries
>    4) PREOF interaction
> 
> Quick summary of what could be done:
> 1) Transmission timing: Don't ask the transport protocol to do this.
>         - Disable sender pacing so the transport protocol (e.g., TCP) sends
> immediately (subject to its window, see item 2)

>From what I understand, a typical SLA will be a periodic block with a fixed period and a fixed block size. Compared with frame relay CIR (committed and excess rate, burst size), DetNet is a lot less "elastic" and no over-subscription. 

>         - At the sender, use a DetNet shaper below the transport protocol
> to align sent traffic to the DetNet SLA.

You mean packing multiple L4 packets into one? Doable, but segment size is usually L4 isn't it? A thin layer above UDP can pack the blocks and save header space.. 


> 2) Dynamic Windowing: Use a scalable congestion control with ECN congestion
> marking algorithms customized to DetNet (more below).

Yes, I can figure that having implemented FR switches and dynamic CIR based on FECN and BECN. But we do not have congestion issues in DetNet. The only thing we can get is bloat or discard at the detnet ingress where the first shaper is. So the signaling we want is between the DetNet shaper and the transport and or application. The problem I'm concerned with is when the application becomes too enthusiastic and transmits excessive amounts of data vs what the first shaper is willing to let I the DetNet network. We need either to tell the app to calm down if it can, or to control it by not pulling more from the socket than the shaper will accept just in time. 

> 3) Loss and retry behavior. Defer this, and target a lossless DetNet data
> plane for now to make initial progress on item 2.

Not in plan. We cannot afford end to end ARQ. We want FEC and PREOF instead. My question to you is whether you call those things L3, because that would change my maybe naïve view of what L3 and L4 do. Note that it's mostly an academic question anyway so we can name the boxes in the architecture; the function remains. I think Norm already blurred L2 and L3 is many of his speeches since many things could be implemented in either but the abstract behaviour remains.


>         - In initial work, treat any loss as a "cry for help" from the
> DetNet data plane, e.g., indicating a DetNet SLA violation.

That could be useful

>         - Deal with DetNet data planes that exhibit non-congestive losses
> later (e.g., starting from a transport protocol that doesn't do retries,
> such as DCCP [RFC 4340] or Datagram QUIC [RFC 9221]).

That's the idea, yes. 

> 4) PREOF in the network should "just work".

Someone has to do it. Then again do assign these functions to a layer? Traditional unicast L3 is one packet in, same packet out, either up or down the stack, or across the router. With IPv6 we even removed fragmentation from inside the network. PREOF is a very different beast.


> So, focusing on 2) Dynamic Windowing, there's an opportunity presented by
> changes in congestion control technology.

No desire for that; missing the bounded latency can be worse than loss. So we want simple: it's like a bus that will periodically leave the station. Fill it or lose the space. Excess passengers wait for the next, which probably means they'll walk or stays home. 

> 
> Transport protocols (e.g., TCP) that use classic congestion controls (e.g.,
> Reno, CUBIC) tend to fill buffers at bottleneck nodes when there's more
> traffic to send than the network can accommodate.  That's not desirable

No such thing in DetNet. There's no congestion inside the network. The only congestion would be at the ingress shaper, if we do not do right what's between the app and that shaper. That is my discussion.


> behavior for DetNet, particularly as there will be a sender buffer between
> the transport protocol and the DetNet shaper (item 1) that ought not to be
> overstuffed.  Starting with Datacenter TCP (DCTCP) a new class of scalable
> congestion controls has emerged that tends to empty buffers at bottleneck
> nodes under the same conditions.  These scalable congestion controls use
> ECN feedback to manage the sender's dynamic window, and do so in a very
> different fashion than classic drop-equivalent ECN.  The most visible
> current activities in this area are L4S, TCP Prague, and Prague for QUIC
> (all discussed briefly in yesterday's ICCRG meeting).  Details of what a
> "scalable congestion control" means are to be found in Section 4 and
> Appendix A of draft-ietf-tsvwg-ecn-l4s-id
> (https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/), but be
> aware that this is not a light read.
> 
> So, an opportunity to make a TCP-like transport protocol useful for DetNet
> would be to figure out what sort of active queue management (AQM) ought to
> be used to mark ECN congestion indications for the various possible DetNet
> queuing mechanisms (e.g., CQF, ATS) where the goal of the ECN congestion
> indications is to cause the "right thing" (whatever that is) to happen with
> the sender's transmission window so that the sender's DetNet shaper has
> "enough" traffic to do its "right thing."  Hacking together a running
> prototype is strongly suggested as a next step, because the interactions
> between transport protocols, DetNet's queuing mechanisms, and congestion
> marking algorithms are subtle, to put it mildly ... and there's a bunch of
> effort to be invested in figuring out what the two "right thing"s and one
> "enough" in this paragraph might mean in practice ...
> 

That's the thing, we do not need any of that. If the blocking function (e.g., in that shim above UDP) is not synchronized to send just in time, the simplest is that the shaper pulls the data just before that time. This is what my draft attempts to explain.

Thank you for all this, I hope we do more...

Pascal



> Thanks, --David
> 
> -----Original Message-----
> From: Balázs Varga A <[e-mail]>
> Sent: Tuesday, July 26, 2022 8:45 AM
> To: Black, David; Pascal Thubert (pthubert); DetNet WG; [e-mail]
> Cc: Black, David
> Subject: RE: Role of the transport layer in the DetNet Architecture
> 
> 
> [EXTERNAL EMAIL]
> 
> Hi,
> 
> Agree with David. Adding just one more aspect: DetNet MPLS never
> touch L4 during the MPLS based forwarding.
> 
> Cheers
> Bala'zs
> 
> -----Original Message-----
> From: detnet <[e-mail]> On Behalf Of Black, David
> Sent: Tuesday, July 19, 2022 3:20 PM
> To: Pascal Thubert (pthubert) <pthubert=[e-mail]>; DetNet
> WG <[e-mail]>; [e-mail]
> Cc: Black, David <[e-mail]>
> Subject: Re: [Detnet] Role of the transport layer in the DetNet
> Architecture
> 
> Hi Pascal,
> 
> > So my question is: should we spin off work in TSVWG?
> How long an explanation of the word "no" would you like ... how many years
> ... :-)?
> 
> Seriously, a new L4 transport protocol for DetNet seems rather unlikely to
> be adopted, and the mention of TCP in the email below looks like an
> exercise in "strawman demolition," e.g., as RFC 8655 does not mention TCP.
> 
> I would suggest looking at RTP and perhaps the multi-stream support in QUIC
> (uses UDP) or SCTP (via SCTP/UDP)  to understand what sort of adaptations
> to DetNet would be both plausible and useful.
> 
> Thanks, --David
> 
> -----Original Message-----
> From: detnet <[e-mail]> On Behalf Of Pascal Thubert
> (pthubert)
> Sent: Tuesday, July 19, 2022 3:18 AM
> To: DetNet WG; [e-mail]
> Subject: [Detnet] Role of the transport layer in the DetNet Architecture
> 
> 
> [EXTERNAL EMAIL]
> 
> Dear all,
> 
> At the RAW interim we discussed again the role of UNI and Transport layer,
> see in the architecture
> https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=31323334-
> 501d5122-313273af-454445555731-2b0c3973f1cab171&q=1&e=14c922c0-2015-4d1d-
> b68f-cb029ff5417a&u=https*3A*2F*2Fwww.rfc-
> editor.org*2Frfc*2Frfc8655.html*2Afig_DetNetservice__;JSUlJSUl!!LpKI!mLX9vm
> 8aqPY8KibxgO1XK4yIyPC-FgoEzzgUzLqKOw5oFZGMHkk5WEV3h_PArs-
> rK5G5ZqsQwObAEeQFVKsOz7VR2S8$ [protect2[.]fireeye[.]com].
> RAW typically operates within the DetNet Service subLayer.
> 
> The question is whether the DetNet Service sublayer is fully a L3 operation
> or if some of it is really L4. E.g., L3 unicast processing in a router is
> typically one packet in, same packet out. PREOF is certainly beyond that.
> 
> Also, a DetNet End System may not have TSN and/or Time Sync handy all the
> way to the app layer for the app to send the right volume of data at the
> right time. A transport that is not adapted may delay the transmission and
> interfere with the application process desires. Bottom line, we'd want the
> application to have to support the minimum and let the transport layer pack
> the data to fit the DetNet SLA and send it at the appropriate time, which
> could effectively be signaled over the UNI. The stream behavior of TCP is
> appropriate, but the dynamic windowing and the retries and undesirable. TCP
> has reordering but does not do PREOF along the path.
> 
> So we see the ideal transport is not available off the shelf. We could make
> it look like UDP to the network, but there's still a need for a particular
> functionality to suit DetNet. And if we agree to place PREOF at L4, then
> the packet should emerge to L4 at the relay nodes (see
> https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=31323334-
> 501d5122-313273af-454445555731-78e6c37e3d3533b3&q=1&e=14c922c0-2015-4d1d-
> b68f-cb029ff5417a&u=https*3A*2F*2Fwww.rfc-
> editor.org*2Frfc*2Frfc8655.html*2Afig_detnet__;JSUlJSUl!!LpKI!mLX9vm8aqPY8K
> ibxgO1XK4yIyPC-FgoEzzgUzLqKOw5oFZGMHkk5WEV3h_PArs-
> rK5G5ZqsQwObAEeQFVKsO8e_qJV8$ [protect2[.]fireeye[.]com]) and plunge again
> in L3. Which is fine, that's what L4 proxies do, remember DLSW anyone?
> 
> I wrote some basic of this in
> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft-
> thubert-tsvwg-detnet-transport-01__;!!LpKI!mLX9vm8aqPY8KibxgO1XK4yIyPC-
> FgoEzzgUzLqKOw5oFZGMHkk5WEV3h_PArs-rK5G5ZqsQwObAEeQFVKsOOh98TzQ$
> [datatracker[.]ietf[.]org]. The document was intended to introduce the need
> for next steps as DetNet becomes real, which is now on us. There was great
> feedback (e.g., Xuesong and Mach) but the focus was elsewhere.
> 
> So my question is: should we spin off work in TSVWG? If so, could we
> prepare a problem statement, e.g., with the doc above as starting point?
> 
> I will not be in person at IETF 114, sorry for that. But it would be nice
> that the topic shows on the agendas for open discussions.
> 
> Keep safe;
> 
> Pascal
> 
> _______________________________________________
> detnet mailing list
> [e-mail]
> https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=31323334-
> 501d5122-313273af-454445555731-6a55b9dd3f9aedbf&q=1&e=14c922c0-2015-4d1d-
> b68f-
> cb029ff5417a&u=https*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Fwww.ietf
> .org*2Fmailman*2Flistinfo*2Fdetnet__*3B*21*21LpKI*21hG_2Ga7PoRVYUsVHjqhyX3p
> HQWG3nvwEu2xPrtdF95Yv_aq_c7VE43Y747tcgpwb9kjLhxTeRRwP1GJnog_3mYZqXxC9GR2D*2
> 4__;JSUlJSUlJSUlJSUlJSUlJQ!!LpKI!mLX9vm8aqPY8KibxgO1XK4yIyPC-
> FgoEzzgUzLqKOw5oFZGMHkk5WEV3h_PArs-rK5G5ZqsQwObAEeQFVKsOiKC_Z8M$
> [protect2[.]fireeye[.]com] [ietf[.]org]
> 
> _______________________________________________
> detnet mailing list
> [e-mail]
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/detnet__;
> !!LpKI!mLX9vm8aqPY8KibxgO1XK4yIyPC-FgoEzzgUzLqKOw5oFZGMHkk5WEV3h_PArs-
> rK5G5ZqsQwObAEeQFVKsO-PzkXeY$ [ietf[.]org]
> 
> --
> RAW mailing list
> [e-mail]
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/raw__;!!LpKI!jhsW7TcMxLJKEJFecxRk0SA3Vevo86u6QoIv7mUQgKCtZOq7Rk2Vpg_1N2a3ui_fOOvR59ATebbBG76A$ [ietf[.]org]