Re: [mpls] [Detnet] DetNet: draft-eckert-detnet-mpls-tc-tcqf-03 request for feedback

[Toerless Eckert <[e-mail]>]

Thanks, Pascal

And i am not claiming that this application model is the only we should wonder about,
instead really maybe write such a hopefully not too long informational draft capturing
how we think different applications can utilize DetNet, and their requirements as
DetNet hosts. Or as non-DetNet hosts, which would typically introduce a large
latency/queue on the first-hop DetNet node, which is absorbing and gating/shaping
traffic from the non-DetNet host.

Wrt to latency of poll: Most of the control loop algoritms are really synchronous and the
maximum end-to-end latency between PLC and any involved sensor/actor will likely
determine the maximum control loop performance. The fact that a sensor data
has to be polled does not reduce that performance, because the PLC can trigger it
periodically upfront.

Cheers
    Toerless

On Sat, Aug 06, 2022 at 10:04:54AM +0000, Pascal Thubert (pthubert) wrote:
> Hello Toerless 
> 
> We’re in sync. 
> 
> From the standpoint of the sensor your model fits the case where DetNet pulls the data from the transport in the sensor and TCQF absorbs the jitter introduced by the sensor processing, the variation being that the pull is really a poll by the remote PLC as opposed to a go-nogo like PFC provides from the DetNet ingress Edge. If we can afford the latency of the poll that certainly works.
> 
> From the standpoint of the PLC the model used is the time-based pull, as opposed to go-no go. The need of a rough syntonisation remains, to avoid drift between the transport doing TCQF and the DetNet network. So ideally the very good timing you’re talking about in the PLC is ticked by the network. Which fits exactly what I was describing.
> 
> All the best,
> 
> Pascal
> 
> > Le 6 août 2022 à 02:57, Toerless Eckert <[e-mail]> a écrit :
> > 
> > Thanks, Pascal
> > 
> > One answer to your concern is the application scenario i described as a
> > specific benefit of TCQF's small jitter (equall of course TSN CQF, for "smaller"/
> > "slower" networks):
> > 
> > I do have actors, sensors and a controller (PLC). The PLC is a host with
> > really good timing. The control loop starts at the PLC sending poll packets
> > to sensors. It can time those polls and because of TCQF low jitter, the PLC
> > will know accurately enough when the poll arrives at the sensor. Likewise
> > let's assume PLC knows the maximum latency between a poll arriving at a
> > sensor and the sensor using it, e.g.: reading some sensor data element(s). Likewise
> > the latency between that and the sensor sending a reply packet. For both these
> > in-sensor delays, there is no complex timing or anything like this, just
> > the worst case speed of the device (e.g.: if received packet triggers interrupt,
> > worst case I/O delay for sensor data - all easy in good sensor specs). Likewise
> > with actors except here we do not evenhave poll/reply, but just action data
> > from PLC to actor, which is often calculated values and the PLC knows when the
> > packet arrives at the actor and how long the actor will take to execute on it.
> > (in practive actors likely will send back some also well predictable data update,
> > so it's pretty much like sensorin that case).
> > 
> > Now, when the PLC sends packets it is conforming to the DetNet flows
> > requirements (max bitrate, burst-size). Likewise, because sensors will only
> > send packets back in reply to those poll messages (or actors in reply to action
> > packets), the PLC is equally in control of packets from sensors/actors
> > to conform to their DetNet flows limits. As long as the reply-size of packets
> > from sensors/actors is within size limits derived from the PLC packet (e.g.:
> > PLS sends one packet, reply will always be one packet of some max size that
> > fits into the defined limit of the sensor/actor DetNet flow.
> > 
> > So, this is think is the most fundamental model to promote bounded-latency
> > low-jitter DetNet service - and it does not have the ingres congestion issue
> > you mention, and its not only attractive for all the upcoming 2-wire ethernet
> > industrial replacement of pre-packet networks - with TCQF allowing to expand
> > such control loops across metro-spaces, where needed, but i would hope that
> > this model can also expand across a sensor/actor to network radio interfaces
> > from IoT space (low power). But i have not tried to analyze the different
> > radios to know which ones fit.
> > 
> > Now, this is not to invalidate what you said, but i think we need to understand
> > which target DetNet scenario needs which type of "ingres throttling" of
> > senders, and what then the likely most easily adoptable technology is to do it.
> > 
> > Non-congestable media streams with higher bitrate would IMHO be the next
> > more complex case to analyze - e.g.: could i claim that lets say all those
> > imaging devices that are now part of industrialand city control loops
> > (video streams) can perfectly time their packet bursts with existing worst-case
> > CPU/interrupt timing, but no special L2 support like IEEE PFC PAUSE. I am
> > not sure. I hope though.
> > 
> > In other words: I would  like to identify the most important nearest case use
> > case where we would safely say we need something like PFC PAUSE or anything
> > beyond pure CPU in devices that need to be low-cost and using only
> > commodity-hardware, but connecting and benefitting from DetNet service.
> > 
> > And i am for example not sure if Eth PAUSE or better yet PFC is commodity.
> > At one point in time i had hoped that PTP was becoming commodity in Eth chips,
> > but it looks more like a specialty for high-end ethernet chips in servers
> > and PLC and the like.
> > 
> > To avoid gigantic buffering in the network, we certainly want burst sizes
> > in usec. And i am not sure what usec accuracy for packet sending we should
> > expect from the worst-case large-scale entpoints that could benefit from
> > DetNet.
> > 
> > I think in TSN this is typically solved with the "Gates" on the first-hop switches,
> > especially also because client devices are by default highly untrusted, I don't
> > know if TSN does have an established model to integrate those gates with PAUSE/PFC.
> > That would be lovely to learn. Maybe we need to hang out at a TSN meeting once
> > to inherit these details ;-))
> > 
> > I certainly would like to avoid bare Eth PAUSE because i feel all devices
> > should NOT ONLY have DetNet traffic but also other traffic, so PFC would IMHO
> > be minimum (to be able to throttle DetNet but not other traffic from the device)
> > if we then can scope the minimum use case where we can show we
> > must have it.
> > 
> > Cheers
> >    Toerless
> > 
> >> On Fri, Aug 05, 2022 at 10:09:30AM +0000, Pascal Thubert (pthubert) wrote:
> >> Hello Toerless 
> >> 
> >> I agree that the current definition of DetNet makes ECN useless inside the DetNet network which from the outside emulates an ingress to egress serial wireline. It’s not only that we do not need ECN, there’s no bit grim reaper either because loss, even organized loss, is not acceptable.
> >> 
> >> DetNet inside does not eliminate congestion, though. It places it between the application and the DetNet network ingress, in the stack in the source end system and across the UNI, be it the bus to the NIC or a first Ethernet hop.
> >> 
> >> We can either a) make all that deterministic but that will be hard for a common OS or 2) provide congestion control, like ECN or whatever else. Ideally in the form of traffic backwards from the ingress to the application. Think Qbb PFC relayed all the way up the stack in the source end system.
> >> 
> >> An alternative is to make the transport layer aware of DetNet and terminating the congestion signaling so the app would just see a more traditional flow control. The benefit of that approach is that the application is agnostic of whether we do a) or b) below. 
> >> 
> >> If we do nothing to pace the application then the excess data will be squeezed at the source. Note that if the pacing could be either in the from of a tick to the application or in the form of the pull of a burst by the transport. The same goes over the UNI.
> >> 
> >> I see your TCQF as one cool way to pull the burst and absorb the jitter the jitter in the application, the stack, and the access network to the DetNet ingress. Using 3 (or even more) cycles like we do for CQF over long lines, we could absorb the UNI jitter. The transport doing CQF would just need a rough syntonization to avoid drifting.
> >> 
> >> I hope I make sense to you,
> >> 
> >> Pascal
> >> 
> >>>> Le 5 août 2022 à 09:46, Toerless Eckert <[e-mail]> a écrit :
> >>> 
> >>> On Thu, Aug 04, 2022 at 08:21:59PM +0000, Black, David wrote:
> >>>>> I think the initial customers for DetNet are those who have the no-loss, no-jitter, no-congestion
> >>>>> requirements. Hence i want the first round that we can do without new packet headers
> >>>> 
> >>>> Sorry, that's not an answer to the question that was asked.
> >>>> 
> >>>> An architectural decision now that DetNet/MPLS will *never* support ECN could be rather short-sighted (e.g., if the management stack use case is reasonable).  OTOH, the approach of locating TCQF in the Service layer might be extensible to also locating ECN marking in that layer.
> >>> 
> >>> I am not saying we should never have ECN in DetNet. I am just saying that we don't have it now:
> >>> For any traffic with the DetNet guaranteed bandwidth or bounded latency services,
> >>> there is no DetNet definition what ECN would mean, because neither of these services is
> >>> defined to have any congestion. Instead, both of them are defined to have no congestion loss
> >>> or ECN.
> >>> 
> >>> If you have a good definition of a proposed DetNet guaranteed bandwidth or DetNet bounded latency
> >>> service with ECN, please elaborate. I did have ideas of my own,  but given how those ideas even
> >>> before DetNet where considered by IETF - and you - to be too complex on non-compliant with standard
> >>> PHB, i'll let others step forward first now (if you remember, it was the AF41/AF42 discuss).
> >>> 
> >>> And just because DetNet IP has an ECN field in the IP header does not mean we have a DetNet definition
> >>> as to what it should mean with DetNet guaranteed throughput / bounded latency traffic.
> >>> 
> >>> I am happy to see us "abuse" ECN bits as a form of OAM: I calculate by DetNet traffic queue
> >>> size based on my calculus for my admitted traffic (whether its guaranteed bandwidth or
> >>> bounded latency), and then i make my queues somewhat longer and trigger ECN when the queue gets into that
> >>> range.
> >>> 
> >>> Technically i call this an "abuse", because i do not expect the sender/receiver app to reduce their
> >>> bitrate, instead i am signalling that i (the network service) have screwed up. Its more like an
> >>> "Apology Bit" then. But i would very much like that.
> >>> 
> >>> And finally, if i just use DetNet for no-bandwidth guarantee traffic, for example because
> >>> i just want to do PREOF, then i definitely want to support ECN, but for that traffic i can not
> >>> use TCQF or any other bounded latency option, because they all depend on reservation of
> >>> path resources (bitrate, buffers).
> >>> 
> >>> Cheers
> >>>   Toerless
> >>> 
> >>>> Thanks, --David
> >>>> 
> >>>> -----Original Message-----
> >>>> From: Toerless Eckert <[e-mail]> 
> >>>> Sent: Thursday, August 4, 2022 2:14 PM
> >>>> To: Black, David
> >>>> Cc: [e-mail]; DetNet WG; [e-mail]; draft-eckert-detnet-mpls-tc-tcqf
> >>>> Subject: Re: [Detnet] DetNet: draft-eckert-detnet-mpls-tc-tcqf-03 request for feedback
> >>>> 
> >>>> 
> >>>> [EXTERNAL EMAIL] 
> >>>> 
> >>>>> On Wed, Aug 03, 2022 at 01:27:33PM +0000, Black, David wrote:
> >>>>> Hi Toerless,
> >>>>> 
> >>>>> I'll observe that the rather small MPLS TC field is being used to serve a lot of purposes.  A question to consider (and one that likely reflects my limited MPLS expertise) is whether cycle tag information is always needed in the forwarding layer F-label vs. an alternative where the cycle tag is in the service layer S-label, resulting in an approach where the service layer handles tagged cycles, relying on the unmodified forwarding layer of the MPLS data plane between service nodes.  That could result in a hierarchy where service nodes are deployed to bound the level of cycle skew exposed to the unmodified forwarding layer that does not see the cycle tags.  Does that make sense?
> >>>> 
> >>>> The goal definitely is to come up with the most easily first-round deployable
> >>>> solution, so i am happy to put the bounded latency function into whatever layer
> >>>> (Service or Forwarding) where it is most easily implemented. But: a) not sure
> >>>> what the other detnet folks think about that, or b) we probably would need
> >>>> to ask the MPLS WG what they think aout looking up QoS from the second label
> >>>> in the stack - complexity wise.
> >>>> 
> >>>>> On ECN ...
> >>>>> 
> >>>>>> Wrt. ECN: For TCQF, we do not need ECN: There CANNOT be any TCQF/DetNet traffic
> >>>>>> with bounded latency that would need ECN markings. That would simply be a failure
> >>>>>> of the bounded latency service.
> >>>>> 
> >>>>> There may be a "never say never" reality check lurking in there ;-).
> >>>>> 
> >>>>> While use of ECN to signal actual congestion is almost certainly indicative of a DetNet failure, I'll repeat my comments to Pascal on this topic:
> >>>>> 
> >>>>>>> I think we have two related goals in this discussion that ought to be distinguished:
> >>>>>>>   - What's the minimal/necessary/ideal L4 transport functionality for DetNet?
> >>>>>>>   - What would it take to run an existing transport stack (L4 and above) over DetNet?
> >>>>>>> This discussion started around the first goal, where I agree with you that the ECN-based dynamic windowing
> >>>>>>> functionality is not needed.  That functionality may be useful towards the second goal (e.g., run HTTP/TLS/QUIC
> >>>>>>> over a DetNet data plane for device management).
> >>>>> 
> >>>>> I suggest more consideration of that second goal.
> >>>>> 
> >>>>> Are you taking the position that it is *never* appropriate to run an existing web-based management stack (e.g., based on HTTP/TLS/TCP or HTTP/TLS/QUIC) over a DetNet MPLS data plane?
> >>>> 
> >>>> I think the initial customers for DetNet are those who have the no-loss, no-jitter, no-congestion
> >>>> requirements. Hence i want the first round that we can do without new packet headers to be
> >>>> focussing on that, so we (DetNet) gets a good foot in the market. And especially that we
> >>>> can establish simpler DetNet-Only solutions, instead of creating an unfortunately even
> >>>> more complex solution we have now: layering DetNet on top of TSN to get bounded latency.
> >>>> 
> >>>> So, i primarily would not want to spare bit for more loftiger goals in this first-round
> >>>> encoding options (reuse TC, DSCP), unless we know we have them available without  cutting
> >>>> into that first goal.
> >>>> 
> >>>> Once we know we have the packet header bits to do more:
> >>>> 
> >>>> First of all, i would like us to think about OAM bits to recognize when there are errors,
> >>>> e.g.: when there is packet drop even though there should be non, because something
> >>>> malfunctioned. How would we do this ? Aka: Whats the best OAM for failure of
> >>>> bounded latency that we can give those first-round applications.
> >>>> 
> >>>> If that OAM turns out to also be usable for classical congestion controlled applications
> >>>> (which i don't think), then we're done. Otherwise we'd have to look for encoding
> >>>> of bits for both those OAM goals and the ECN goals.
> >>>> 
> >>>> If we go with SR-MPLS, our LSE would turn into SIDs, so we could have SIDs for just
> >>>> DetNet, not shared by other traffic. Meaning wew have the full 3 bit free. And we
> >>>> could then also easier (IMHO) hack around even more, redesignating e.g.: some more
> >>>> bits from TTL (if Cisco gives license to IETF on its patent for that...).
> >>>> 
> >>>> Aka: Lots of options, which is why i was trying to figure out whats hopefully most important
> >>>> first so we can get more deployment experience from a first round RFC.
> >>>> 
> >>>> Cheers
> >>>>   Toerless
> >>>> 
> >>>>> Thanks, --David
> >>>>> 
> >>>>> -----Original Message-----
> >>>>> From: detnet <[e-mail]> On Behalf Of Toerless Eckert
> >>>>> Sent: Tuesday, August 2, 2022 10:41 PM
> >>>>> To: [e-mail]
> >>>>> Cc: DetNet WG; [e-mail]; draft-eckert-detnet-mpls-tc-tcqf
> >>>>> Subject: Re: [Detnet] DetNet: draft-eckert-detnet-mpls-tc-tcqf-03 request for feedback
> >>>>> 
> >>>>> 
> >>>>> [EXTERNAL EMAIL] 
> >>>>> 
> >>>>> Thanks for the interest Zongpeng,
> >>>>> 
> >>>>> Given how the TC-TCQF solution is now within scope of DetNet charter, i think this
> >>>>> is also a good time to also get more input/review from the MPLS WG, hence Cc'ing it
> >>>>> (the chair where already suggesting to engage with th MPLS WG on this in the past).
> >>>>> 
> >>>>> For background: The draft currently covers what i think is the most easy deployable option to
> >>>>> enable tagged-cyclic-queuing-and-forwarding (TCQF) with the MPLS forwarding plane -
> >>>>> without changes on the wire, by reusing the existing MPLS header field TC.
> >>>>> 
> >>>>> If we choose to adopt this work in DetNet it would be good work to document not
> >>>>> only this most simple TC encoding, but also any other encoding that do not require
> >>>>> on-the-wire-changes, but that may be necessary when this most simple method is
> >>>>> considered to be not sufficient for some MPLS networks.
> >>>>> 
> >>>>> TCQF encoding that would encode the cycle in a new heder field (change on the wire)
> >>>>> should IMHO be work reserved for another draft. Primarily because i think we can
> >>>>> much faster finish specifying and deploying the mechanism with the options that do
> >>>>> not require changes on the wire - and collect experience with it (this is true for
> >>>>> MPLS as well as IP/IPv6, where we can use DSCP).
> >>>>> 
> >>>>> The draft therefore proposed allocation of MPLS TC values for use with TC-TCQF is
> >>>>> described in section 3.3 of the draft TC-TCQF draft. We would need 3 or 4 TC values.
> >>>>> 
> >>>>> Here is a bit more detail:
> >>>>> 
> >>>>> RFC3270 defines how to use the TC field
> >>>>> (actually, it calls it EXP, the field was later renamed to TC in RFC5462)
> >>>>> 
> >>>>> What our TCQF draft proposes is to propose to use the so-called 
> >>>>> "TC- Inferred-PSC LSP (E-LSP) behavior". This is described in section 1.2 of
> >>>>> RFC3270.
> >>>>> 
> >>>>> This is IMHO the most widely deployed method for using DiffServ with MPLS
> >>>>> and also the most simple to deploy, because it can get away without additional
> >>>>> control plane / labels. What this effectively means is that the network operator
> >>>>> configures in a DiffServ style 
> >>>>> 
> >>>>> When a deployment has already used more than 4/5 TC and therefore
> >>>>> does not have 3 or 4 TC free, then (i think!) the operator could allocate additional
> >>>>> labels space where the TC values are free. Depending on how MPLS experts read
> >>>>> RFC3270, this might require resorting to calling those LSPs SIDs and call that
> >>>>> encoding then an SR-MPLS mechanism.
> >>>>> 
> >>>>> We could also look into allocating 3 or 4 additional L-LSP, but that looks like
> >>>>> the most complicated option to me.
> >>>>> 
> >>>>> IMHO, a first RFC does not have to be complete with all possible encoding
> >>>>> options. I'd rather see an RFC finished that can be deployed, and when that
> >>>>> raises interest in the industry, but the vendors recognize some customers who
> >>>>> already used up all TC, then this would be good justification to do a -bis of
> >>>>> the RFC with the more complicated encodings.
> >>>>> 
> >>>>> Wrt. ECN: For TCQF, we do not need ECN: There CANNOT be any TCQF/DetNet traffic
> >>>>> with bounded latency that would need ECN markings. That would simply be a failure
> >>>>> of the bounded latency service. The non-TCQF TC values may indicate ECN
> >>>>> for non-TCQF traffic, but to the best of my knowledge ECN with TC is practically
> >>>>> not deployed.
> >>>>> 
> >>>>> Hope this help, please comment!
> >>>>> 
> >>>>> Cheers
> >>>>>   Toerless
> >>>>> 
> >>>>> On Tue, Aug 02, 2022 at 09:14:24AM +0800, [e-mail] wrote:
> >>>>>> Hi, Toerless   
> >>>>>> 
> >>>>>>   I am Zongpeng Du from China Mobile.
> >>>>>> 
> >>>>>>   We are interested in the draft. However, we have a question about the method to convey the tag in the TOS of an MPLS label.
> >>>>>> 
> >>>>>>   MPLS label: 32bits
> >>>>>> 
> >>>>>>   00-19     20-21                                                 23                                    24-31
> >>>>>>   label       TC: Traffic Class(QoS and ECN)        S: Bottom-of-Stack          TTL: Time-to-live
> >>>>>> 
> >>>>>> As shown in the Figure, the three bits for TC have been defined as the format of "QoS and ECN". 
> >>>>>>   IMO, unless the IETF agrees to disuse it, we can not use the bits that have been used.
> >>>>>>   Do I have any misunderstanding here?
> >>>>>> 
> >>>>>>  Or, the label is special defined, such as an S-Label, so that we can define new use of the TC field ? 
> >>>>>> 
> >>>>>> Best Regards
> >>>>>> Zongpeng Du
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> [e-mail] & [e-mail]
> >>>>>> 
> >>>>>> From: Toerless Eckert
> >>>>>> Date: 2022-07-12 03:20
> >>>>>> To: detnet
> >>>>>> CC: draft-eckert-detnet-mpls-tc-tcqf
> >>>>>> Subject: [Detnet] DetNet: draft-eckert-detnet-mpls-tc-tcqf-03 request for feedback
> >>>>>> Dear DetNet WG
> >>>>>> 
> >>>>>> I just posted rev 3 of https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-eckert-detnet-mpls-tc-tcqf/__;!!LpKI!mdHkRkynsjbKwb-lZDUF4l-5wA4KD2nDtRSdgiUzwt_OR1Ia9AOMNJyskNQCYsQzZKh4xiORb915kQ$ [datatracker[.]ietf[.]org]
> >>>>>> 
> >>>>>> Diff over previous version 2:
> >>>>>> https://urldefense.com/v3/__http://www.ietf.org/*rfcdiff?url1=https:**Awww.ietf.org*archive*id*draft-eckert-detnet-mpls-tc-tcqf-02.txt&url2=https:**Awww.ietf.org*archive*id*draft-eckert-detnet-mpls-tc-tcqf-03.txt__;Ly8vLy8vLy8vLy8!!LpKI!mdHkRkynsjbKwb-lZDUF4l-5wA4KD2nDtRSdgiUzwt_OR1Ia9AOMNJyskNQCYsQzZKh4xiNYE2hEcw$ [ietf[.]org]
> >>>>>> 
> >>>>>> This version adds new text about the per-flow ingres enqueueing into the
> >>>>>> cycles, informational text about high-speed HW implementation considerations
> >>>>>> and a pointer to a research paper also describing such HW validation.
> >>>>>> 
> >>>>>> I think this makes the proposed work complete, and as authors, we would
> >>>>>> like to ask the WG for adoption - and the participants for feedback.
> >>>>>> 
> >>>>>> Btw: The authors are also happy to discuss any restructuring, for example
> >>>>>> Eshan told me it might be better to separaate the genreic mechanism from
> >>>>>> the MPLS encoding (separate drafts). 
> >>>>>> 
> >>>>>> Quick summary, if you have not kept track:
> >>>>>> 
> >>>>>> TCQF stands for Tagged Cyclic Queuing and Forwarding, and is a proposal
> >>>>>> how to adopt the IEEE TSN "Cyclic Queuing and Forwarding" Queuing for
> >>>>>> 'native' use in DetNet, specifically with MPLS forwarding plane.
> >>>>>> 
> >>>>>> CQF is a great queuing mechanism for DetNet because it does not only offer
> >>>>>> bounded latency, but also tight-bounded jitter, important for all
> >>>>>> industrial "synchronous" applications. It also is most easily scalable
> >>>>>> in HW and operations for metropolitan or larger size WAN DetNet deployments
> >>>>>> because it does not have per-hop state on every router.
> >>>>>> 
> >>>>>> By tagging packets with the cycle number, TCQF overcomes also the issues
> >>>>>> that keeps CQF limited to small-scale (few Km) Ethernet: it can support
> >>>>>> large link delays and link-delay variation, and it reduces relatively
> >>>>>> the required clock accuracy, therefore making it feasible to use this
> >>>>>> on 100, 400 Gbs or faster networks.
> >>>>>> 
> >>>>>> There is an expired problem description draft i wrote some time ago,
> >>>>>> if you need more background: https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-eckert-detnet-bounded-latency-problems/__;!!LpKI!mdHkRkynsjbKwb-lZDUF4l-5wA4KD2nDtRSdgiUzwt_OR1Ia9AOMNJyskNQCYsQzZKh4xiOUluv-oQ$ [datatracker[.]ietf[.]org]
> >>>>>> 
> >>>>>> Cheers
> >>>>>>   Toerless
> >>>>>> 
> >>>>>> _______________________________________________
> >>>>>> detnet mailing list
> >>>>>> [e-mail]
> >>>>>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/detnet__;!!LpKI!mdHkRkynsjbKwb-lZDUF4l-5wA4KD2nDtRSdgiUzwt_OR1Ia9AOMNJyskNQCYsQzZKh4xiMZSWyvmw$ [ietf[.]org]
> >>>>> 
> >>>>> -- 
> >>>>> ---
> >>>>> [e-mail]
> >>>>> 
> >>>>> _______________________________________________
> >>>>> detnet mailing list
> >>>>> [e-mail]
> >>>>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/detnet__;!!LpKI!mdHkRkynsjbKwb-lZDUF4l-5wA4KD2nDtRSdgiUzwt_OR1Ia9AOMNJyskNQCYsQzZKh4xiMZSWyvmw$ [ietf[.]org]
> >>>>> 
> >>>> 
> >>>> -- 
> >>>> ---
> >>>> [e-mail]
> >>>> 
> >>> 
> >>> -- 
> >>> ---
> >>> [e-mail]
> >>> 
> >>> _______________________________________________
> >>> detnet mailing list
> >>> [e-mail]
> >>> https://www.ietf.org/mailman/listinfo/detnet
> > 
> > -- 
> > ---
> > [e-mail]
> > 
> > _______________________________________________
> > detnet mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/detnet

-- 
---
[e-mail]