Re: [Detnet] comments on draft-ietf-raw-architecture-16 (Janos Farkas)

[Ike Alisson <[e-mail]>]

Mr. Farkas et al, I would like hereby to express my support for your
proposal for a feature rich RAW API, that provides interaction
possibilities between the Reliability Features, so, the Reliability
Features, in essence, remain the same in the different Layers, but they can
take input form the other Layer.

To substantiate the grounds/Technology evolvements for my support to Mr.
Farkas Proposal, please find below attached:

1. Link to a presentation on Intent-based Management with enhanced Network
Analytics (including the snapshots below):
https://www.linkedin.com/posts/ike-alisson-21173_beyond-5g-b5g-6g-networks-capabilities-activity-7115271926347919362-MgxO?utm_source=share&utm_medium=member_desktop
[image: bild.png]
[image: bild.png]
[image: bild.png]
[image: bild.png]
[image: bild.png]

2. Shift from (2G, 3G, 4G) Design offering "Best-effort Services" to Design
(5G SA Rel-18 aka 1st "5G Advanced" Release) that offers Services with
"Performance & User Exerience Guarantees".

[image: 3GPP 5G Adv Rel-18 use of AI ML key aspects.png]


3. 5G PMF (Performance Measutrement Functionality) currently supporting E2E
from the UE through RAN to the CN (with "Echo" & "Responce") with RTT, PLR,
"Suspend"/"Resume" Traffic Duplication.
[image: 3GPP 5G PMF Rel 18 4 0 Dec 2023.png]
[image: 3GPP 5G SA Perform and User Experience Guarantees.png]

4. 5G APIs evolvement in the 5G Common API Framework from SNA
(Subscriber-aware Northbound APIs Access) to RNAA (Resource-owner aware
Northbound APIs Access) attached below
[image: 3GPP 5G Adv APIs from SNA to RNAA 2023.png]

3. 5G evolved "Identifiers" from the "traditional" User being "Subscriber"
to no adding "Application" as well as an UE behind a GW and/or UEs
off-premises (Mobility type 3 & 4) attached below.
[image: 3GPP 5G Adv Rel 19 User Identifiers with Appl Layer E Agreement Oct
2023.png]
This is just FYI to share with you evolvements that would have an impact on
the Networks Capabilities enabling End-User Services Releiability and
Availability assuring Performance & User Experience Guarantees.
//Ike A.
P.S. If the attached is of no use/interest, please just press the "del"
button.
In case of interest on the use of the specified use of AI ML in 5G Networks
(Telecom oriented, not AI ML oriented), you may look at:
1. Presentation 1 on the specified use of AI ML in 5G Networks:
https://www.linkedin.com/posts/ike-alisson-21173_5g-enhancements-for-aiml-on-network-application-activity-7138761933222055936-SInx?utm_source=share&utm_medium=member_desktop

2.Presentation 2 on the use of AI ML in 5G both, Network and Application
Layers with specified Model Training and Model Transfer with specified AI
ML 5G Service KPIs:
https://www.linkedin.com/posts/ike-alisson-21173_5g-advanced-use-of-aiml-with-aiml-models-activity-7125775255331090432-nQvD?utm_source=share&utm_medium=member_desktop

<https://www.linkedin.com/feed/update/urn:li:activity:7168826038691598336/#>
<https://www.linkedin.com/feed/update/urn:li:activity:7168826038691598336/#>
Ike Alisson on LinkedIn: 5G Advanced use of AI/ML with AI/ML Models &
Service KPIs

<https://www.linkedin.com/feed/update/urn:li:activity:7125775255331090432>
<https://www.linkedin.com/feed/update/urn:li:activity:7125775255331090432>
<https://www.linkedin.com/feed/update/urn:li:activity:7125775255331090432>
3. Presentation 3 on AI ML Device selected Standard specified Functional
Requirements:
<https://www.linkedin.com/feed/update/urn:li:activity:7125775255331090432>
https://www.linkedin.com/posts/ike-alisson-21173_ai-ml-mobile-device-requirements-with-the-activity-7131864690543968256-L8yM?utm_source=share&utm_medium=member_desktop
<https://www.linkedin.com/feed/update/urn:li:activity:7168826038691598336/#>
<https://www.linkedin.com/feed/update/urn:li:activity:7168826038691598336/#>
<https://www.linkedin.com/feed/update/urn:li:activity:7131864690543968256>

Den tis 27 feb. 2024 kl 01:28 skrev <[e-mail]>:

> Send detnet mailing list submissions to
>         [e-mail]
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://www.ietf.org/mailman/listinfo/detnet
> or, via email, send a message with subject or body 'help' to
>         [e-mail]
>
> You can reach the person managing the list at
>         [e-mail]
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of detnet digest..."
> Today's Topics:
>
>    1. Re: comments on draft-ietf-raw-architecture-16 (Janos Farkas)
>
> ---------- Forwarded message ----------
> From: Janos Farkas <[e-mail]>
> To: "[e-mail]" <[e-mail]>
> Cc:
> Bcc:
> Date: Tue, 27 Feb 2024 00:28:13 +0000
> Subject: Re: [Detnet] comments on draft-ietf-raw-architecture-16
>
> Hi,
>
>
>
> Picking this up again.
>
>
>
> I am happy to propose/make changes to add the figures to the draft.
>
>
>
> I still have concerns about PAREO. After some off line discussions, my
> understanding is that the idea behind the term PAREO really is to have a
> term for a kind of superset of reliability functions provided by the DetNet
> layer and the lower wireless/radio layer. However, I’m afraid it is not
> that useful  to talk about the superset, esp. given the controversy it
> brings. Thinking it further, each layer operates its reliability function
> in its own layer. If the RAW API is not feature rich, e.g., not so much
> information exchanged / exposed, then superset is not really meaningful. If
> the RAW API is feature rich, then it actually provides interaction
> possibilities between the reliability features. So, the reliability
> features are in essence remain the same in the different layers, but they
> can take input form the other layer. (In particular, thinking of PLR.) This
> allows cross-layer optimization if one wants to go that way.
>
>
>
> I’ve been thinking and trying to come up with some potential way forward
> with the draft along the lines above.
>
>
>
> For instance, a new paragraph could be added to Section 1 Introduction,
> e.g., as penultimate paragraph along the lines:
>
> “In addition, RAW introduces the RAW API, which is an interface between
> the lower layer wireless technology and the DetNet layers. The RAW API is
> RAW technology [RAW-TECHNOS] dependent as it can vary what the different
> RAW technologies expose towards the DetNet layers. Furthermore, the
> different RAW technologies are equipped with different reliability
> features, e.g., short range broadcast, MUMIMO, PHY rate and other
> Modulation Coding Scheme (MCS) adaptation, (H)ARQ, constructive
> interference and overhearing. The RAW API enables interactions between the
> reliability functions provided by the wireless technology and the
> reliability functions provided by DetNet. That is, the RAW API makes
> cross-layer optimization possible for the reliability functions of
> different layers depending on the actual exposure provided via the RAW API
> by the give RAW technology.”
>
>
>
> (Note that I tried to make this text in-line with the other definitions,
> e.g., 2.6.7 Lower Layer Information.)
>
>
>
> Along the proposed direction, the term PAREO could be actually removed
> from the draft.
>
>
>
> What do you think?
>
>
>
> If the WG is ok with it, as a contribution, I can make suggested changes
> to the rest of the draft along the lines proposed above.
>
>
>
> Regards,
>
> Janos
>
>
>
> *From:* Janos Farkas
> *Sent:* Wednesday, November 8, 2023 1:14 PM
> *To:* [e-mail]; [e-mail]
> *Subject:* RE: [Detnet] comments on draft-ietf-raw-architecture-16
>
>
>
> Hi Pascal, all,
>
>
>
> I did not mention this in my mail, but I guess it is easy to see that the
> figure(s) I had in my mail have been inspired by Figure 4 in RFC 8655:
> https://www.rfc-editor.org/rfc/rfc8655.html#figure-4. I was trying to
> gasp the essence of RAW for myself to  have some guideline for my comments.
> If the group sees value in it, some version might be added to the draft.
>
>
>
> Regards,
>
> Janos
>
>
>
> *From:* Pascal Thubert <[e-mail]>
> *Sent:* Monday, November 6, 2023 2:49 PM
> *To:* Lou Berger <[e-mail]>; Janos Farkas <
> [e-mail]>; [e-mail]
> *Subject:* Re: [Detnet] comments on draft-ietf-raw-architecture-16
>
>
>
> Hello Lou and Janos
>
> Parsing through Janos ' mail cannot be done before the meeting today, so I
> guess we can only discuss highlights today like the PAREO term/concept. I
> agree with Janos's drawing in his mail below. We already discussed that the
> DetNet layers Request but do not actuate (via APIs) the PAREO service that
> are operated at the lower layer.
>
> "
>
> PAREO functions that are actuated at the lower layers may be controlled
> through abstract interfaces by the RAW extensions within the DetNet Service
> sub-layer.
>
> "
>
> I do not see that this constitutes a layer violation. There are shaper
> services in DetNet that can be operated at various layers as well.
> Certainly there should be something between the layers to control who's
> doing what.
>
> About the lane stuff, the term came in the discussion with Don Fedyk after
> the London meeting that you guys asked us to have. The term supersedes
> 'legs'. It is one specific sequential path within the recovery domain, we
> can expect several parallel ones for redundancy. I'm unclear if your
> protection path is one such lane or the redundant collection of parallel
> lanes. See also
> https://mailarchive.ietf.org/arch/browse/roll/?q=%20%22leg%22%20vs%20%22lane%22
>
> For recovery graph, it is my understanding that we came up with this
> agreement in the corridor meeting with Greg and Adrian. The story is that
> the centralized CPF may not provide a set of protection paths to be
> selected from, but the union of them, in which case the PLR will build the
> temporary paths dynamically based on elements in the graph.
>
>
>
>  all the best
>
>
>
> Pascal
>
>
>
>
>
>
>
> Le 06/11/2023 à 11:52, Lou Berger a écrit :
>
> Pascal, (Janos)
>
> Thank you for the much improved versions.  I think (as contributor) the
> document will be in good shape once the address the comments below.
>
> I think Janos' comments are very helpful and  together with the drawings
> will answer one of my major open questions - how does a raw domain work
> with a non-raw detnet domain to support end to end detnet services.
>
> A few terminology items to add to the comments below:
>
> WRT Recovery graphs, I thought we were going to reuse one of the existing
> terms (i.e., protection group or recovery group) see,
> https://mailarchive.ietf.org/arch/msg/detnet/xj6bBURD66eB0ANhjqMd3anZ5vA/
>
> WRT Lane - how does a lane differ from a 'protection path', i.e. do we
> really need this new term? I know I've asked this before, but I  do not see
> the difference based on the latest text.
>
> At the nit level uplink and downlink can be dropped as terms as they are
> only used in their definition sections.
>
> Thanks again,
>
> Lou
>
> On 11/6/2023 2:21 AM, Janos Farkas wrote:
>
> Hi Pascal, all,
>
>
>
> I am very sorry for not being able to respond earlier.
>
>
>
> First of all, many thanks for the updates and significant changes in the
> draft!
>
> The draft has been improved a lot!
>
>
>
> I’ve read v16. I think it is better if I place my thoughts on v16 directly
> here than embedding in-line in our dialogue. These thoughts actually
> include responses too. In addition, I’ve added responses below too.
>
>
>
> text from the draft are marked with <v16>
>
> my thoughts are marked with <JF>
>
>
>
> <JF>:
>
> A couple of things are still confusing in the draft imo, e.g., some
> self-inconsistencies. I’ve been trying to wrap my head around how to
> comment actually. Perhaps it might be good to make sure that there is
> consensus on high level principles before diving into details.
>
> Perhaps first what we are doing, what is RAW actually. I think there are
> very good statements on this already in the draft, e.g.:
>
>
>
> <v16>:
>
> RAW improves the reliability of transmissions and the availability
>
> of the communication resources, but does not provide scheduling and
>
> shaping, so RAW itself does not provide guarantees such as latency
>
> for the application payload. Rather, it should be seen as a dynamic
>
> optimization of the use of redundancy to maintain it within certain
>
> boundaries. For instance, ARQ
>
> is operated by the lower layers and
>
> RAW will only abstract the concept and hint the lower layers on the
>
> desired outcome, as opposed to performing the retries at Layer-3.
>
>
>
> <JF>: note that I’ve intentionally removed the part I do not like and will
> come back to it.
>
>
>
> <v16>:
>
> using a common radio abstraction and
>
> APIs
>
>  the capability
>
> to push reliability and timing hints like suggest X retries (min,
>
> max) within a time window, or send unicast (one next hop) or
>
> multicast (for overhearing). The other way around RAW needs hints
>
> about the radio conditions like L2 triggers
>
>
>
> <JF>: again, a few words left out intentionally for a sec.
>
>
>
> I like the text above. I think they reflect what I think I heard we want:
> a clen layered model (not an integrated one)
>
> I guess I would illustrate it something like:
>
>                    .
>
>                    .
>
>      +-----------------------------+
>
>      |                             |
>
>      |  DetNet Service sub-layer   | (e.g., (PREOF)
>
>      |                             |
>
>      +-----------------------------+
>
>      |                             |
>
>      | DetNet Forwarding sub-layer |
>
>      |         _________           |
>
>      +--------< RAW API >----------+
>
>      |         ---------           |
>
>      |    Wireless/Radio Layer     | (e.g., (H)ARQ)
>
>      |                             |
>
>      +-----------------------------+
>
>                    .
>
>
>
> However, the draft still contains the term PAREO. I’m afraid I need to
> keep repeating that the term/concept PAREO is a layer violation itself. I
> guess PAREO comes from an integrated model. The figure above clearly
> illustrates why PAREO is a layer violation. PAREO combines (H)ARQ and
> PREOF. However, (H)ARQ resides in the Wireless/Radio Layer whereas, PREOF
> is part of the DetNet Service sub-layer. They are very different layers,
> and there is the DetNet Forwarding sub-layer in between them. Therefore,
> the term and concept PAREO should be removed completely from the document.
>
>
>
> So, to the question whether I suggest changes to Figure 4: Yes, remove
> PARO at the minimum. Nonetheless, there may be further potential changes
> falling out of the train of thoughts below.
>
> I do not have a replacement term yet, but perhaps we might get somewhat
> closer via the thoughts below.
>
>
>
> So, I think one essence of RAW is the API.
>
> I think the draft contains further good text about the API, e.g.:
>
>
>
> <v16>:
>
> Conceptually, RAW is
>
> agnostic to the radio layer underneath
>
> <v16>:
>
> hint on
>
> the desired outcome, and the lower layer acts on those hints to
>
> provide the best approximation of that outcome,
>
> <v16:>:
>
> abstract
>
> interface, while they are really operated at the lower layers
>
>
>
> <JF>: Nonetheless, the draft sometimes uses “controls”, “manipulates” and
> alike from what is done from the DetNet layer towards the Wireless/Radio
> Layer.
>
> The draft also (imo correctly) includes statements like:
>
>
>
> <v16>:
>
> it is not expected that all lower layers
>
> support all those capabilities
>
>
>
> <JF>:
>
> I think that the RAW architecture should be crafted such that it covers at
> least the wireless technologies in draft-ietf-raw-technologies. They are
> quire different; provide different ways / levels of exposure. So, I rather
> like the language around “hint” than “control”.
>
> For instance,
>
> “RAW manipulates abstractions of the lower layer services to hint on
>
> the desired outcome,”
>
> could be replaced with
>
> “RAW provides hints to the lower layer services on the desired outcome,”
>
>
>
>
>
> <JF>:
>
> Back to what is RAW:
>
> In my understanding an essence of RAW is the capability of fast reaction
> to rapidly changing wireless conditions on order to maintain the desired
> QoS for the end to end service, esp., reliability.
>
> (I know I’m simplifying a couple of aspects a bit, but this is with the
> attempt to make it easier to come to some similar page.)
>
> To this extent, the concept of recovery graph has been introduced. The
> term protection path is widely used too. Perhaps, at a high level, it is
> fair to say that the recovery graph is the superset of all possible
> protection paths. (again, there may be some simplifications here to be able
> to express some thoughts below)
>
> I found very good statements in the draft:
>
>
>
> <v16>:
>
> RAW
>
> itself operates in the DetNet Network Plane at a faster time scale
>
> <v16>:
>
> take the routing decision early
>
> enough along the possible paths to route around the failure. RAW
>
> defines a end-to-end control loop that dynamically controls the
>
> activation and deactivation of the feasible Protection Paths
>
> <v16>:
>
> An Operational Plane PLR that hosts the Decision function of
>
> which DetNet Paths to use for the future packets that will be
>
> routed within the recovery graph
>
> <v16>:
>
> The main action by the PLR is the swapping of the DetNet Path within
>
> the recovery graph for the future packets. The candidate DetNet
>
> Paths represent different energy and spectrum profiles, and provide
>
> protection against different failures.
>
> <v16>:
>
> RAW defines the Path Selection Engine (PLR) as a
>
> synchronous CPF that performs rapid local adjustments of the
>
> forwarding tables within the diversity that the asynchronous CPF has
>
> in store for the recovery graph
>
> <v16>:
>
> the system observed by the RAW OAM is the
>
> recovery graph
>
>
>
> <JF>:
>
> In my understanding, the key component being added by RAW to the DetNet
> architecture is the PLR to perform the fast reactions to the fast changes
> in wireless/radio conditions.
>
> In my understanding, the establishment of the forwarding paths, i.e., the
> Protection Paths and ultimately the recovery graph belongs to the DetNet
> Forwarding sub-layer. In my read, this is illustrated, e.g., in Figure 2 in
> RFC 8655. The explicit routes (source routing whatsoever) belong to the
> DetNet Forwarding sub-layer.
>
> In my understanding, PLR reacts based on the information received from
> lower layers via the RAW API and based on OAM observing the recovery graph.
>
> Which DetNet sub-layer the PLR then belongs to?
>
> Maybe not that easy to answer.
>
> Placing it in the DetNet Service sub-layer results in an additional layer
> in between the PLR and the API the PLR operates upon. So, it may not be the
> best approach.
>
> Placing it in the DetNet Forwarding sub-layer has the benefit that PLR is
> adjacent / directly connected to the API it operates upon.
>
>
>
> With that, perhaps a simplified view on what RAW is, i.e., what are the
> RAW add-ons / extensions to the DetNet architecture might be roughly viewed
> as:
>
>
>
>                    .
>
>                    .
>
>      +-----------------------------+
>
>      |                             |
>
>      |  DetNet Service sub-layer   | (e.g., (PREOF)
>
>      |                             |
>
>      +-----------------------------+
>
>      |                             |
>
>      | DetNet Forwarding sub-layer |
>
>      |                             |
>
>      |           _____    _____    |
>
>      |          < PLR >--< OAM >   |
>
>      |           -+--     -----    |
>
>      |            | ^              |
>
>      |            | |              |
>
>      |         ___v_+_____         |
>
>      +--------< RAW API >----------+
>
>      |         ---------           |
>
>      |    Wireless/Radio Layer     | (e.g., (H)ARQ)
>
>      |                             |
>
>      +-----------------------------+
>
>                    .
>
>
>
> So, I’d suggest more changes to Figure 4.
>
> As for where the OAM is based upon which the PLR acts: Well, I guess this
> part of the OAM really monitors the forwarding paths, i.e., the protection
> paths, the protection graph. So, I do not see it alien to be located at the
> DetNet Forwarding sub-layer.
>
> But, it all depends on the consensus of the WG.
>
>
>
>
>
> Some further comments:
>
> I’m not sure on the use of some terminology, e.g.,
> synchronous/asynchronous CPF, distributed (e.g., PLR).
>
> Synchronous/asynchronous often refer to time synchronization. In my
> understanding, what the PLR does is actually a local decision. So, it is
> rather some local CPF making decision on Protection Path compared to the
> main CPF crafting the protection graph.
>
> As for distributed, my understanding is that in case of distributed,
> multiple entities have to act together to reach some goal, e.g., RSVP and
> OSPF are distributed and multiple nodes have to play together.
>
> I’m not sure I get, e.g., what “the PLR decision may be distributed”
> means. As I understand the point is that it is a local decision based on
> the local situation, independent of what’s going on at some remote location.
>
>
>
> I am not sure about the “loose” approach. My concern is that we do not
> know the guarantees of the loose segments, if any. Perhaps it would be
> better not going into “loose”.
>
>
>
> I’d suggest to use “extend” instead of “enhance”.
>
>
>
>
>
> Some detailed  comments on v16:
>
>
>
> Section 1:
>
> <v16>:
>
> Deterministic Networking is an attempt to emulate the properties of
>
> a serial link over a switched fabric, by providing a bounded latency
>
> and eliminating congestion loss, even when co-existing with best effort
>
> traffic.
>
> <JF>:
>
> I’d suggest:
>
> Deterministic Networking is to provide bounded latency and eliminate
>
> congestion loss, even when co-existing with best effort traffic.
>
>
>
> as I never had the emulation of serial link in mind.
>
>
>
> <v16>:
>
> Bringing determinism in a packet network means eliminating the
>
> statistical effects
>
> <JF>:
>
> I’m afraid complete elimination is not possible, so I’d suggest something
> like:
>
> Bringing determinism in a packet network means minimizing the
>
> statistical effects
>
>
>
>
>
> <v16>:
>
> This innovation was initially introduced on wired networks, with
>
> IEEE 802.1 Time Sensitive networking (TSN) - for Ethernet LANs - and
>
> IETF DetNet.
>
> <JF>:
>
> RFC 8578 DetNet Use Cases includes wireless.
>
>
>
> <v16>:
>
> With new scheduled
>
> radios such as TSCH and OFDMA
>
> <JF>
>
> Suggest to remove “new” as they are not new.
>
>
>
> <v16>:
>
> To achieve this, RAW
>
> leverages
>
> <JF>
>
> As it depends on the actual wireless technology etc., suggest to replace
> with:
>
> To achieve this, RAW
>
> can leverage
>
>
>
> <v16>:
>
> As opposed to routing trees, Distance-Vector protocols can enable
>
> more than one feasible successors along non-equal-cost multipath
>
> forwarding graphs. This provide redundancy and allow to dynamically
>
> adapt the forwarding operation to the state of the links. But this
>
> protection is limited since only a subset of the nodes along the
>
> path will have an alternate feasible successor
>
> <JF>
>
> I’m afraid this is confusing. A distance vector protocol may also build
> up a tree, see, e.g., RSTP. Sticking to the RSTP example, it is true that
> RSTP has the Alternate Port concept for fast reaction to a change, however,
> the concept has been then introduced to routing protocols as well, see LFA.
>
>
>
>
>
>
>
> Section 2:
>
> <v16>:
>
> In an quantic analogy, a recovery graph is to a path what an atomic
>
> orbital is to a planetary orbit, in that the electron has a
>
> probability of presence within a known shape as opposed to a
>
> deterministic trajectory.
>
>
>
> <JF>:
>
> Consider removing the paragraph as it may be unsure how much it is helpful
> to the reader in the given context.
>
> 2.1.7:
>
> Remove PAREO completely from the document for the reasons explained above.
>
>
>
> 2.2.1:
>
> Is this really flapping? Isn’t what the text explain is rather just
> degradation? Isn’t flapping rather a link going down and coming back?
>
>
>
> 2.2.4:
>
> Should "in the direction from the source towards the destination" be added?
>
> 2.3.1:
>
> Replace
>
> a packet from A to B
>
> experiences 2 paths,
>
> with
>
> B, a packet from A to B
>
> can experience 2 paths,
>
> as one particular packet only experiences one path
>
>
>
> 2.3.2
>
> I’m not sure about the need of “usage metadata;”. Perhaps remove?
>
> Figure 1 is without any mention in the text. Perhaps add explanation.
>
> <v16>:
>
> The vertices of that graph are DetNet Relay nodes that operate at
>
> the DetNet Service sub-layer and provide the PAREO functions.
>
>
>
> The topological edges of the graph are strict sequences of DetNet
>
> Transit nodes that operate at the DetNet Forwarding sub-layer.
>
>
>
> These bullet may need some updates depending on how the discussion goes.
>
>
>
> 2.5.1
>
> <v16>:
>
> an SLA (service level agreement) is a
>
> contract between a provider, the network, and a client, the
>
> application flow, about measurable metrics such as latency
>
> boundaries, consecutive losses, and packet delivery ratio (PDR).
>
> <JF>:
>
> In my understanding, an SLA is rather a contract between two parties (not
> more).
>
> Consider updating to:
>
> an SLA (service level agreement) is a
>
> contract between a network provider and a client about measurable metrics
> of a flow such as latency boundaries, consecutive losses, and packet
> delivery ratio (PDR).
>
>
>
> 2.5.6:
>
> “ Because a wireless lane may not be good enough to
>
> provide the required reliability, and even 2 parallel lanes may not
>
> be over a longer period of time, the RAW availability implies a
>
> journey that is a lot more complex than following a serial path.”
>
> Could be removed as not needed for the definition.
>
>
>
> 3.1.1:
>
> <v16>:
>
> elimination of single points of failure
>
> <JF>:
>
> Perhaps rather: elimination of each single point of failure
>
> <v16>:
>
> prompt detection of failures as they occur.
>
> <JF>:
>
> Perhaps rather: prompt detection and reaction to failures as they occur.
>
>
>
> 3.1.1.1:
>
> <v16>:
>
> IP Routers leverage routing protocols to compute alternate routes
>
> <JF>:
>
> replace “compute” with "reroute to" as it is not just compute
>
>
>
> 3.1.1.2:
>
>
>
> <v16>:
>
> Having a backup equipment has a limited value unless it can be
>
> reliably switched into use within the down-time parameters.
>
>
>
> <JF>:
>
> Some schemes, e.g., PREOF, do not require switchover.
>
>
>
> <v16>:
>
> delayed reestablishment of the second path
>
> <JF>:
>
> There is no reestablishment of 2nd path in case of 1+1 protection; esp. in
> case of PREOF, where the basic idea is to send multiple copies of the same
> packet over pre-established maximally disjoint paths all the time.
>
>
>
> 3.1.2:
>
> <v16>:
>
> In data
>
> networks, this is rarely the case. Packet loss cannot never be fully
>
> avoided and the systems are built to resist to one loss, e.g., using
>
> redundancy with Retries (HARQ) or Packet Replication and Elimination
>
> (PRE), or, in a typical control loop, by linear interpolation from
>
> the previous measurements.
>
> <JF>
>
> Packet loss cannot be fully avoided and the systems are built to resist
> only a limited number of losses e.g., using retransmissions (HARQ), or,
> in a typical control loop, by linear interpolation from the previous
> measurements.
>
> Note that the RLC mechanism exists too.
>
>
>
> 3.2:
>
> <v16>:
>
> Conceptually, RAW is
>
> agnostic to the radio layer underneath though the capability to
>
> schedule transmissions is assumed.
>
> <JF>:
>
> I suggest to remove “though the capability to
>
> schedule transmissions is assumed”
>
> or replace with
>
> though it is assumed that the radio layer underneath is capable of
> schedule transmissions
>
>
>
> <v16>:
>
> Nevertheless,
>
> cross-layer optimizations may take place to ensure proper
>
> <JF>
>
> As we follow a layered approach (not an integrated one), it may be clearer
> to have :
>
> cross-layer optimization may take place via the APIs provided between RAW
> and the radio layer.
>
>
>
> 4.1:
>
> Suggest to remove “such as a Wi-Fi extended service set (ESS) or a 5G
> Core;”
>
> For instance ,the 5G core is part of the 5G System. When it comes to 5G, a
> better example would be a 5G System connected to a wireline domain.
>
>
>
> 4.3
>
> The RAW Management sub-layer
>
> functionality includes the PLR
>
> RAW aims to be faster than usual control plane. Management plane is slower
> than control plane. Management plane does not seem to be a good place for
> PLR, which intends to be very fast.
>
>
>
> 5.1
>
> Suggest to remove “In the wired world,”
>
> There could be a wireless subnet/segment along some of these paths.
>
>
>
>
>
> *From:* Pascal Thubert (pthubert) <pthubert=[e-mail]>
> <pthubert=[e-mail]>
> *Sent:* Friday, August 11, 2023 3:03 PM
> *To:* Janos Farkas <[e-mail]> <[e-mail]>
> *Cc:* [e-mail]; [e-mail]; Adrian Farrel ([e-mail])
> <[e-mail]> <[e-mail]>; Greg Mirsky
> <[e-mail]> <[e-mail]>; Lou Berger (
> [e-mail]) <[e-mail]> <[e-mail]>
> *Subject:* RE: Terminology: draft-ietf-raw-architecture-14.txt and what's
> still nuclear
>
>
>
> Hello Janos
>
>
>
> Many thanks for your review!
>
>
>
> >  I think RAW, i.e., wireless aspects actually rather reside in the
> Forwarding sub-layer (not in the Service sub-layer).
>
> > The Service sub-layer is pretty much IP in this case. Ideally, it is
> hidden from the Service sub-layer as much as possible what kind of
> forwarding is used in the Forwarding > sub-layer, e.g., whether it is
> wireline or wireless. (Well, the Forwarding sub-layer may provide some
> certain APIs towards the Service sub-layer if we want to.)
>
>
>
>
>
> We defined the sublayers by listing functions as hints as opposed to
> mapping to IP or L2. Our intention (Norm and I) at the start was that the
> DetNet operated at L3 but deterministic networking at large could be widely
> done at either L2 or L3 (or a mix) and the abstractions and service, from a
> distance, would be the same.
>
>
>
> RAW being DetNet it is L3, so neither wireless nor wired, though it is
> meant to address gaps that are more common in wireless like quick flaps,
> reliability degradation, and PHY speed variations. Wireless, like wire, is
> only observed and requested through APIs. As Lou said, some aspects like
> promiscuous and ARQ are more commonly observed on wireless, but are not
> exclusive to wireless. We provide a L3 architecture that is sensitive to
> those elements but does not include them since they are operated in another
> layer altogether.
>
>
>
> The RAW architecture is focused on a fast protection/recovery loop. This
> is why the main operation (PAREO actuation and OAM processing/compiling )
> is in the DetNet (=> L3) service sublayer. But not only:
>
>
>
> “
>
>     +------------------------------+ +--------------------------------+
>
>     |                              | |                                |
>
>    .....................................................................
>
>     |                              | |                                |
>
>     | +--------------------------+ | | +----------------------------+ |
>
>     | |     aCPF                 | | | |         aCPF               | |
>
>     | +--------------------------+ | | +----------------------------+ |
>
>     | +----------+  +------------+ | | .-.-.-.-.-.--.  .-.-.-.-.-.--. |
>
>     | | PLR      |  | OAM        | | | | Distr. PLR |  | Distr. OAM | |
>
>     | |          |  | Supervisor | | | |            |  | Supervisor | |
>
>     | +----------+  +------------+ | | .-.-.-.-.-.--.  .-.-.-.-.-.--. |
>
>     |                              | |    optional         optional   |
>
>        RAW Management sub-layer
>
>    .....................................................................
>
>        DetNet Service sub-layer
>
>     |                              | |                                |
>
>     | +----------+  +------------+ | | +------------+  +------------+ |
>
>     | | PAREO    |  |  OAM       | | | |  PAREO     |  |  OAM       | |
>
>     | | Actuator |  |  Observer  | | | |  Actuator  |  |  Observer  | |
>
>     | +----------+  +------------+ | | +------------+  +------------+ |
>
>     |                              | |                                |
>
>        DetNet Service sub-layer
>
>    .....................................................................
>
>        DetNet Forwarding sub-layer
>
>     |                              | |                                |
>
>     |               +------------+ | |                 +------------+ |
>
>     |               |In-Situ OAM | | |                 |In-Situ OAM | |
>
>     |               +------------+ | |                 +------------+ |
>
>     |                              | |                                |
>
>     +------------------------------+ +--------------------------------+
>
>
>
>             End System or                       Relay
>
>           Ingress Edge Node                     Node
>
>
>
>           Figure 4: RAW functional posture within DetNet sublayers
>
>
>
> “
>
>
>
> Is there a change you would suggest to this picture?
>
>
>
> <JF>
>
> Yes, see above.
>
>
>
>
>
>
>
>
>
>
>
> > As I read, the 1st paragraph in section 4.3 is actually along these lines
>
>
> https://www.ietf.org/archive/id/draft-ietf-raw-architecture-14.html#name-raw-and-detnet
>
> > “RAW leverages the DetNet Forwarding sub-layer”
>
> > “RAW enhances DetNet to improve the protection against link errors such
> as transient flapping that are far more common in wireless links.” – Well,
> this reads to me to be in the Forwarding sub-layer.
>
>
>
> Well, protection is service sub-layer actually (section 2.1 of RFC 8655).
> Forwarding is packet handling, like find which buffer and which protection
> path for this packet.
>
>
>
> <JF>
>
> This is my point: “Forwarding is packet handling, like find which
> protection path for this packet.”
>
> So, you suggest Forwarding sub-layer too. 😉
>
>
>
>
>
> The PAREO actuator impacts the forwarding operation for future packets but
> does not necessarily run on every packet, see the discussion with David.
>
>
>
> <JF>
>
> Imo, PAREO should be completely removed from the document because it is a
> layer violation, as I keep saying for a while. See more above.
>
>
>
>
>
> > However, I’m quire unsure of the beginning of the 2nd paragraph in
> section 4.3:
>
> > “RAW extends the DetNet Stack (see fig 4 of [
> https://www.ietf.org/archive/id/draft-ietf-raw-architecture-14.html#RFC8655])
> with additional functionality at the DetNet Service sub-layer for the PLR
> operation.”
>
> >  and the location of PLR in Figure 4, where the PLR is not in the data
> plane.
>
>
>
> You’re correct about the inconsistency. The PLR is a piece of the
> controller that was exported to the forwarding devices. The discussion with
> Greg placed that component in the management sublayer. The text that you
> point out is a victim of the renaming and the component it discusses is now
> called PAREO actuator. Proposed beginning of that section:
>
> “
>
>    RAW leverages the DetNet Forwarding sub-layer and requires the
>
>    support of in-situ OAM in DetNet Transit Nodes (see fig 3 of
>
>    [RFC8655] for the dynamic acquisition of link capacity and state to
>
>    maintain a strict RAW service, end-to-end, over a DetNet Network.
>
>    RAW enhances DetNet to improve the protection against link errors
>
>    such as transient flapping that are far more common in wireless
>
>    links.  Nevertheless, the RAW methods are for the most part
>
>    applicable to wired links as well, e.g., when energy savings are
>
>    desirable and the available path diversity exceeds 1+1 linear
>
>    redundancy.
>
>
>
>    RAW adds a Management sub-layer that operates in the DetNet
>
>    Operational Plane.  The RAW Management sub-layer typically runs only
>
>    in the DetNet Ingress Edge Node or End System, though it may also run
>
>    in DetNet Relay Nodes when the RAW Control sub-layer is distributed
>
>    along the recovery graph.  The RAW Management sub-layer functionality
>
>    includes the PLR that decides the DetNet Path for the future packets
>
>    of a flows and controls the PAREO Actuators along the DetNet Path
>
>    through specific signaling, and the OAM Supervisor that triggers, and
>
>    learns from, OAM observations, and feeds the PLR for its next
>
>    decision.
>
>
>
>    RAW extends the DetNet Stack (see fig 4 of [RFC8655]) with additional
>
>    functionality at the DetNet Service sub-layer for the actuation of
>
>    the PLR decision by the PAREO Actuator.  Layer-3 in general and
>
>    DetNet in particular operates on abstractions of the lower layers and
>
>    through APIs to control those abstractions.  For instance, DetNet
>
>    already leverages lower layers for time-sensitive operations such as
>
>    time synchronization and traffic shapers.  Because the performances
>
>    of the radio layers are subject to rapid changes, so RAW needs more
>
>    dynamic gauges and knobs.  To that effect, the DetNet PREOF is
>
>    extended with the PAREO capabilities (see Section 5.6) and the RAW
>
>    PAREO Actuator manages dynamically the PAREO operations, which may be
>
>    performed either within the DetNet sublayers or at a lower layer,
>
>    using a common radio abstraction and APIs in the latter case.  In
>
>    particular, PAREO needs the capability to push reliability and timing
>
>    hints like suggest X retries (min, max) within a time window, or send
>
>    unicast (one next hop) or multicast (for overhearing).  The other way
>
>    around RAW needs hints about the radio conditions like L2 triggers
>
>    (RSSI, LQI, ETX...) over all the wireless hops.  This information is
>
>    useful to both the aCPF and the PLR.
>
>
>
> “
>
>
>
> > I think PLR should be located in the Forwarding sub-layer, but I may
> have misunderstood the intention.
>
>
>
> I think the operation you have in mind is the realization of the PLR
> decision. We did not create a box for it. Should we? What name? Is there
> really something new there vs. DetNet?
>
>
>
> > As far as I understood, the intention with the PLR is fast reaction to
> actual wireless situation, i.e., where to send a given packet actually.
>
>
>
> True, and it reacts to signals that arrive asynchronously to the packets,
> for the most part. They are either coming from L2 or from OAM.
>
>
>
>
>
>
>
>                |
>
>         packet | going
>
>       down the | stack
>
>     +==========v==========+=====================+=====================+
>
>     |   (iOAM + iCTRL)    | (L2 Triggers, DLEP) |       (oOAM)        |
>
>     +==========v==========+=====================+=====================+
>
>     |     Learn from      |                     |    Learn from       |
>
>     |    packet tagging   >       Maintain      <    end-to-end       |
>
>     +----------v----------+      Forwarding     |    OAM packets      |
>
>     | Forwarding decision <        State        +---------^-----------|
>
>     +----------v----------+                     |      Enrich or      |
>
>     +    Retag Packet     |  Learn abstracted   >     Regenerate      |
>
>     |    and Forward      | metrics about Links |     OAM packets     |
>
>     +..........v..........+..........^..........+.........^.v.........+
>
>     |                          Lower layers                           |
>
>     +..........v.....................^....................^.v.........+
>
>          frame | sent          Frame | L2 Ack        oOAM | | packet
>
>           over | wireless        In  |                 In | | and out
>
>                v                     |                    | v
>
>
>
>                          Figure 10: PLR Interfaces
>
>
>
>
>
>
>
> > Imo, this is very similar to FRR, see, e.g.:
> https://datatracker.ietf.org/doc/html/rfc5286. A key aspect is that the
> alternate next hops are precalculated. That is, it resides in the data
> plane, no control plane reaction is needed.
>
>
>
> Hum FRR is still reroute. It installs new path in the forwarding plane
> (think software operation downloading a new FIB) and then the forwarding
> plane (think silicon now) uses the new paths possibly with no idea that it
> is a FRR path. Same here.
>
>
>
> > I guess a more recent (segment routing) document on the subject with PLR
> in it is:
> https://datatracker.ietf.org/doc/html/draft-ietf-rtgwg-segment-routing-ti-lfa,
> see, e.g.: “no need for any co-ordination or message exchange between the
> PLR and any other router in the network.”
>
> Note that segment routing is part of the DetNet Forwarding sub-layer (not
> the Service sub-layer).
>
>
>
> Like RAW, SR is a lot of things; the SRH operation (execution) is indeed
> forwarding plane. The decision of which SRH to insert upon which condition
> is control plane. Now we have this interesting 3D problem of how our planes
> intersect those planes with similar names, e.g., DetNet Controller plane
> and usual control plane.
>
>
>
> > Perhaps I’m the only one with these kind of thoughts.
>
> > If not, then perhaps it might be good to converge on PLR (e.g., its
> location in the architecture) before figuring out what actual changes are
> needed in the draft.
>
>
>
> Let’s see how others react. For now, you debunked at least one bug and
> that already very valuable. Committed as af8771a
> <https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-03fb2e6dd7e27dec&q=1&e=5ab1955b-c117-42bb-bcee-c4e507e491b9&u=https%3A%2F%2Fgithub.com%2Fraw-wg%2Fraw-architecture%2Fcommit%2Faf8771aeb4bc7dad5d2ae012cd53a4581e63f9df>
>
>
>
>
> <JF>
>
> Yes, see what the group thinks.
>
>
>
> Thank you,
>
> Cheers,
>
> Janos
>
>
>
>
>
> Many thanks!
>
>
>
> Pascal
>
> *From:* Janos Farkas <Janos.Farkas=[e-mail]>
> *Sent:* Monday, July 31, 2023 4:02 PM
> *To:* Pascal Thubert (pthubert) <[e-mail]>
> *Cc:* [e-mail]; [e-mail]; Adrian Farrel ([e-mail]) <
> [e-mail]>; Greg Mirsky <[e-mail]>; Lou Berger (
> [e-mail]) <[e-mail]>
> *Subject:* RE: Terminology: draft-ietf-raw-architecture-14.txt and what's
> still nuclear
>
>
>
> Dear Pascal, all,
>
>
>
> Many thanks for the updates!
>
> Really great improvements towards architectural cleanliness.
>
>
>
> Otoh, distillation may reveal further questions and further distillation.
> 😉
>
>
>
> For instance, now that it is called PLR, it triggers a couple of questions
> on my part. (Well, with the assumption that I have some clue on PLR.)
>
>
>
> Perhaps one step back first:
>
> We actually introduced the division of DetNet data plane to Forwarding
> sub-layer and Service sub-layer for architectural cleanliness during DetNet
> data plane design team discussions. Furthermore, in order to divide (and
> conquer) in the sense to be able to identify the functions and to place
> them to some order, i.e., for functional decomposition.
>
>
>
> I suppose, the RAW architecture would follow DetNet architectural
> principles and place the RAW related functions accordingly.
>
>
>
> I think RAW, i.e., wireless aspects actually rather reside in the
> Forwarding sub-layer (not in the Service sub-layer).
>
> The Service sub-layer is pretty much IP in this case. Ideally, it is
> hidden from the Service sub-layer as much as possible what kind of
> forwarding is used in the Forwarding sub-layer, e.g., whether it is
> wireline or wireless. (Well, the Forwarding sub-layer may provide some
> certain APIs towards the Service sub-layer if we want to.)
>
>
>
> As I read, the 1st paragraph in section 4.3 is actually along these lines
>
>
> https://www.ietf.org/archive/id/draft-ietf-raw-architecture-14.html#name-raw-and-detnet
>
> “RAW leverages the DetNet Forwarding sub-layer”
>
> “RAW enhances DetNet to improve the protection against link errors such as
> transient flapping that are far more common in wireless links.” – Well,
> this reads to me to be in the Forwarding sub-layer.
>
>
>
> However, I’m quire unsure of the beginning of the 2nd paragraph in
> section 4.3:
>
> “RAW extends the DetNet Stack (see fig 4 of [RFC8655
> <https://www.ietf.org/archive/id/draft-ietf-raw-architecture-14.html#RFC8655>])
> with additional functionality at the DetNet Service sub-layer for the PLR
> operation.”
>
> and the location of PLR in Figure 4, where the PLR is not in the data
> plane.
>
>
>
> I think PLR should be located in the Forwarding sub-layer, but I may have
> misunderstood the intention.
>
>
>
> As far as I understood, the intention with the PLR is fast reaction to
> actual wireless situation, i.e., where to send a given packet actually.
>
> Imo, this is very similar to FRR, see, e.g.:
> https://datatracker.ietf.org/doc/html/rfc5286. A key aspect is that the
> alternate next hops are precalculated. That is, it resides in the data
> plane, no control plane reaction is needed.
>
> I guess a more recent (segment routing) document on the subject with PLR
> in it is:
> https://datatracker.ietf.org/doc/html/draft-ietf-rtgwg-segment-routing-ti-lfa,
> see, e.g.: “no need for any co-ordination or message exchange between the
> PLR and any other router in the network.”
>
> Note that segment routing is part of the DetNet Forwarding sub-layer (not
> the Service sub-layer).
>
>
>
> Perhaps I’m the only one with these kind of thoughts.
>
> If not, then perhaps it might be good to converge on PLR (e.g., its
> location in the architecture) before figuring out what actual changes are
> needed in the draft.
>
>
>
> My 2 cents,
>
> János
>
>
>
> ps:
>
> Sorry for late commenting. It was not the plan. And now we are hit by
> August = vacation. I’m not sure how much I can chime in the discussion from
> vacation. I just thought sharing the comment instead of waiting till
> September.
>
>
>
>
>
> *From:* detnet <[e-mail]> *On Behalf Of *Pascal Thubert
> (pthubert)
> *Sent:* Sunday, July 30, 2023 10:39 PM
> *To:* [e-mail]; Adrian Farrel ([e-mail]) <
> [e-mail]>; Greg Mirsky <[e-mail]>; Lou Berger (
> [e-mail]) <[e-mail]>
> *Cc:* [e-mail]
> *Subject:* [Detnet] Terminology: draft-ietf-raw-architecture-14.txt and
> what's still nuclear
>
>
>
>
>
> Dear all:
>
>
>
> the publication of 14 adds terminologies and typos. The goal is to serve
> as the new reference for the WGLC so we can use the new terms in our
> discussions. If someone still uses PSE and Track, well, I guess we'll still
> understand for a little while, but he will be harshly reprimanded.
>
>
>
> What I did not do yet though I started is work out the positioning of the
> aCPF (the component that talks asynchronously to the rCPF == PCE to report
> performance and get route updates), the Point of Local Repair (PLR is the
> term that replaces the PSE) and the OAM supervisor that triggers OAM and
> aggregates results for the PLR.
>
>
>
> These are 3 new architectural blocks, and we want to position them well in
> the DetNet architecture.
>
>
>
> The DetNet architecture (section 4.4) has 3 planes that are mapped to
> classical SDN, with the controller plane in the middle, a southbound
> interface to the network plane (in the case of RAW used between rCPF and
> aCPF) and a northbound interface to the Application Plane.
>
>
>
> The Controller plane has the typical route servers like PCEs, and network
> management entities. In the SDN model they are "far away" and monitor the
> whole network. Which is what causing the RAW issue of lack of reactivity
> and pushed us to port functionality in the network plane. In networking
> planes parlance, the PCE is control plane and the NMEs are management plane.
>
>
>
> As we see, though the term controller plane looks like control plane, they
> are different beasts. A classical IGP is a control plane function that
> operates in the DetNet network plane. The controller plane hosts
> controllers, which physically may embed routing and management entities. In
> the DetNet architecture, "The Controller Plane corresponds to the
> aggregation of the Control and Management Planes in [RFC7426
> <https://datatracker.ietf.org/doc/html/rfc7426>], though Common Control
> and Measurement Plane (CCAMP) (as defined by the CCAMP Working Group [
> CCAMP <https://datatracker.ietf.org/wg/ccamp/charter/>]) makes an
> additional distinction between management and measurement."
>
>
>
> In my book, the OAM supervisor, the aCPF and the PLR operate in the
> control plane. The OAM supervisor talks to the OAM handlers in the
> management plane. And all of the above being distributed in the network,
> they operate in the DetNet Network plane.  So 1) we extend the DetNet
> architecture to place functional blocks in the Network Plane and 2) one
> could say we need 3D pictures to represent the intersection of the DetNet
> planes and the traditional control and management planes. While the data
> plane remains firmly in the Network plane.
>
>
>
> Now this is my view and the way I intend to update the text to publish 15,
> hopefully quite soon. But I need confirmation that we are on the same line,
> or else debates to reach a consensus.
>
>
>
> What do you all think?
>
>
>
> Pascal
> ------------------------------
>
> *De :* [e-mail] <[e-mail]>
> *Envoyé :* samedi 29 juillet 2023 15:40
> *À :* Pascal Thubert (pthubert) <[e-mail]>
> *Objet :* New Version Notification for draft-ietf-raw-architecture-14.txt
>
>
>
>
> A new version of I-D, draft-ietf-raw-architecture-14.txt
> has been successfully submitted by Pascal Thubert and posted to the
> IETF repository.
>
> Name:           draft-ietf-raw-architecture
> Revision:       14
> Title:          Reliable and Available Wireless Architecture
> Document date:  2023-07-29
> Group:          raw
> Pages:          43
> URL:
> https://www.ietf.org/archive/id/draft-ietf-raw-architecture-14.txt
> Status:
> https://datatracker.ietf.org/doc/draft-ietf-raw-architecture/
> Html:
> https://www.ietf.org/archive/id/draft-ietf-raw-architecture-14.html
> Htmlized:
> https://datatracker.ietf.org/doc/html/draft-ietf-raw-architecture
> Diff:
> https://author-tools.ietf.org/iddiff?url2=draft-ietf-raw-architecture-14
>
> Abstract:
>    Reliable and Available Wireless (RAW) provides for high reliability
>    and availability for IP connectivity across any combination of wired
>    and wireless network segments.  The RAW Architecture extends the
>    DetNet Architecture and other standard IETF concepts and mechanisms
>    to adapt to the specific challenges of the wireless medium, in
>    particular intermittently lossy connectivity.  This document defines
>    a network control loop that optimizes the use of constrained spectrum
>    and energy while maintaining the expected connectivity properties,
>    typically reliability and latency.  The loop involves OAM, DetNet
>    Controller Plane, and PREOF extensions, and specifically a new
>    recovery Function called PAREO and a new Point of Local Repair
>    operation, that dynamically selects the DetNet path(s) for the future
>    packets to route around local degradations and failures.
>
>
>
>
>
> The IETF Secretariat
>
>
>
> _______________________________________________
>
> detnet mailing list
>
> [e-mail]
>
> https://www.ietf.org/mailman/listinfo/detnet
>
> _______________________________________________
> detnet mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/detnet
>