Re: [Detnet] comments on draft-ietf-raw-architecture-16

[Pascal Thubert <[e-mail]>]

Hello Lou and Janos

Parsing through Janos ' mail cannot be done before the meeting today, so 
I guess we can only discuss highlights today like the PAREO 
term/concept. I agree with Janos's drawing in his mail below. We already 
discussed that the DetNet layers Request but do not actuate (via APIs) 
the PAREO service that are operated at the lower layer.

"

PAREO functions that are actuated at the lower layers may be controlled 
through abstract interfaces by the RAW extensions within the DetNet 
Service sub-layer.

"

I do not see that this constitutes a layer violation. There are shaper 
services in DetNet that can be operated at various layers as well. 
Certainly there should be something between the layers to control who's 
doing what.

About the lane stuff, the term came in the discussion with Don Fedyk 
after the London meeting that you guys asked us to have. The term 
supersedes 'legs'. It is one specific sequential path within the 
recovery domain, we can expect several parallel ones for redundancy. I'm 
unclear if your protection path is one such lane or the redundant 
collection of parallel lanes. See also 
https://mailarchive.ietf.org/arch/browse/roll/?q=%20%22leg%22%20vs%20%22lane%22

For recovery graph, it is my understanding that we came up with this 
agreement in the corridor meeting with Greg and Adrian. The story is 
that the centralized CPF may not provide a set of protection paths to be 
selected from, but the union of them, in which case the PLR will build 
the temporary paths dynamically based on elements in the graph.


  all the best


Pascal



Le 06/11/2023 à 11:52, Lou Berger a écrit :
>
> Pascal, (Janos)
>
> Thank you for the much improved versions.  I think (as contributor) 
> the document will be in good shape once the address the comments below.
>
> I think Janos' comments are very helpful and  together with the 
> drawings will answer one of my major open questions - how does a raw 
> domain work with a non-raw detnet domain to support end to end detnet 
> services.
>
> A few terminology items to add to the comments below:
>
> WRT Recovery graphs, I thought we were going to reuse one of the 
> existing terms (i.e., protection group or recovery group) see, 
> https://mailarchive.ietf.org/arch/msg/detnet/xj6bBURD66eB0ANhjqMd3anZ5vA/
>
> WRT Lane - how does a lane differ from a 'protection path', i.e. do we 
> really need this new term? I know I've asked this before, but I  do 
> not see the difference based on the latest text.
>
> At the nit level uplink and downlink can be dropped as terms as they 
> are only used in their definition sections.
>
> Thanks again,
>
> Lou
>
> On 11/6/2023 2:21 AM, Janos Farkas wrote:
>
>> Hi Pascal, all,
>>
>> I am very sorry for not being able to respond earlier.
>>
>> First of all, many thanks for the updates and significant changes in 
>> the draft!
>>
>> The draft has been improved a lot!
>>
>> I’ve read v16. I think it is better if I place my thoughts on v16 
>> directly here than embedding in-line in our dialogue. These thoughts 
>> actually include responses too. In addition, I’ve added responses 
>> below too.
>>
>> text from the draft are marked with <v16>
>>
>> my thoughts are marked with <JF>
>>
>> <JF>:
>>
>> A couple of things are still confusing in the draft imo, e.g., some 
>> self-inconsistencies. I’ve been trying to wrap my head around how to 
>> comment actually. Perhaps it might be good to make sure that there is 
>> consensus on high level principles before diving into details.
>>
>> Perhaps first what we are doing, what is RAW actually. I think there 
>> are very good statements on this already in the draft, e.g.:
>>
>> <v16>:
>>
>> RAW improves the reliability of transmissions and the availability
>>
>> of the communication resources, but does not provide scheduling and
>>
>> shaping, so RAW itself does not provide guarantees such as latency
>>
>> for the application payload. Rather, it should be seen as a dynamic
>>
>> optimization of the use of redundancy to maintain it within certain
>>
>> boundaries. For instance, ARQ
>>
>> is operated by the lower layers and
>>
>> RAW will only abstract the concept and hint the lower layers on the
>>
>> desired outcome, as opposed to performing the retries at Layer-3.
>>
>> <JF>: note that I’ve intentionally removed the part I do not like and 
>> will come back to it.
>>
>> <v16>:
>>
>> using a common radio abstraction and
>>
>> APIs
>>
>>  the capability
>>
>> to push reliability and timing hints like suggest X retries (min,
>>
>> max) within a time window, or send unicast (one next hop) or
>>
>> multicast (for overhearing). The other way around RAW needs hints
>>
>> about the radio conditions like L2 triggers
>>
>> <JF>: again, a few words left out intentionally for a sec.
>>
>> I like the text above. I think they reflect what I think I heard we 
>> want: a clen layered model (not an integrated one)
>>
>> I guess I would illustrate it something like:
>>
>>                    .
>>
>>                    .
>>
>>      +-----------------------------+
>>
>>      |                             |
>>
>>      |  DetNet Service sub-layer   | (e.g., (PREOF)
>>
>>      |                             |
>>
>>      +-----------------------------+
>>
>>      |                             |
>>
>>      | DetNet Forwarding sub-layer |
>>
>>      |         _________           |
>>
>>      +--------< RAW API >----------+
>>
>>      |         ---------           |
>>
>>      |    Wireless/Radio Layer     | (e.g., (H)ARQ)
>>
>>      |                             |
>>
>>      +-----------------------------+
>>
>>                    .
>>
>> However, the draft still contains the term PAREO. I’m afraid I need 
>> to keep repeating that the term/concept PAREO is a layer violation 
>> itself. I guess PAREO comes from an integrated model. The figure 
>> above clearly illustrates why PAREO is a layer violation. PAREO 
>> combines (H)ARQ and PREOF. However, (H)ARQ resides in the 
>> Wireless/Radio Layer whereas, PREOF is part of the DetNet Service 
>> sub-layer. They are very different layers, and there is the DetNet 
>> Forwarding sub-layer in between them. Therefore, the term and concept 
>> PAREO should be removed completely from the document.
>>
>> So, to the question whether I suggest changes to Figure 4: Yes, 
>> remove PARO at the minimum. Nonetheless, there may be further 
>> potential changes falling out of the train of thoughts below.
>>
>> I do not have a replacement term yet, but perhaps we might get 
>> somewhat closer via the thoughts below.
>>
>> So, I think one essence of RAW is the API.
>>
>> I think the draft contains further good text about the API, e.g.:
>>
>> <v16>:
>>
>> Conceptually, RAW is
>>
>> agnostic to the radio layer underneath
>>
>> <v16>:
>>
>> hint on
>>
>> the desired outcome, and the lower layer acts on those hints to
>>
>> provide the best approximation of that outcome,
>>
>> <v16:>:
>>
>> abstract
>>
>> interface, while they are really operated at the lower layers
>>
>> <JF>: Nonetheless, the draft sometimes uses “controls”, “manipulates” 
>> and alike from what is done from the DetNet layer towards the 
>> Wireless/Radio Layer.
>>
>> The draft also (imo correctly) includes statements like:
>>
>> <v16>:
>>
>> it is not expected that all lower layers
>>
>> support all those capabilities
>>
>> <JF>:
>>
>> I think that the RAW architecture should be crafted such that it 
>> covers at least the wireless technologies in 
>> draft-ietf-raw-technologies. They are quire different; provide 
>> different ways / levels of exposure. So, I rather like the language 
>> around “hint” than “control”.
>>
>> For instance,
>>
>> “RAW manipulates abstractions of the lower layer services to hint on
>>
>> the desired outcome,”
>>
>> could be replaced with
>>
>> “RAW provides hints to the lower layer services on the desired outcome,”
>>
>> <JF>:
>>
>> Back to what is RAW:
>>
>> In my understanding an essence of RAW is the capability of fast 
>> reaction to rapidly changing wireless conditions on order to maintain 
>> the desired QoS for the end to end service, esp., reliability.
>>
>> (I know I’m simplifying a couple of aspects a bit, but this is with 
>> the attempt to make it easier to come to some similar page.)
>>
>> To this extent, the concept of recovery graph has been introduced. 
>> The term protection path is widely used too. Perhaps, at a high 
>> level, it is fair to say that the recovery graph is the superset of 
>> all possible protection paths. (again, there may be some 
>> simplifications here to be able to express some thoughts below)
>>
>> I found very good statements in the draft:
>>
>> <v16>:
>>
>> RAW
>>
>> itself operates in the DetNet Network Plane at a faster time scale
>>
>> <v16>:
>>
>> take the routing decision early
>>
>> enough along the possible paths to route around the failure. RAW
>>
>> defines a end-to-end control loop that dynamically controls the
>>
>> activation and deactivation of the feasible Protection Paths
>>
>> <v16>:
>>
>> An Operational Plane PLR that hosts the Decision function of
>>
>> which DetNet Paths to use for the future packets that will be
>>
>> routed within the recovery graph
>>
>> <v16>:
>>
>> The main action by the PLR is the swapping of the DetNet Path within
>>
>> the recovery graph for the future packets. The candidate DetNet
>>
>> Paths represent different energy and spectrum profiles, and provide
>>
>> protection against different failures.
>>
>> <v16>:
>>
>> RAW defines the Path Selection Engine (PLR) as a
>>
>> synchronous CPF that performs rapid local adjustments of the
>>
>> forwarding tables within the diversity that the asynchronous CPF has
>>
>> in store for the recovery graph
>>
>> <v16>:
>>
>> the system observed by the RAW OAM is the
>>
>> recovery graph
>>
>> <JF>:
>>
>> In my understanding, the key component being added by RAW to the 
>> DetNet architecture is the PLR to perform the fast reactions to the 
>> fast changes in wireless/radio conditions.
>>
>> In my understanding, the establishment of the forwarding paths, i.e., 
>> the Protection Paths and ultimately the recovery graph belongs to the 
>> DetNet Forwarding sub-layer. In my read, this is illustrated, e.g., 
>> in Figure 2 in RFC 8655. The explicit routes (source routing 
>> whatsoever) belong to the DetNet Forwarding sub-layer.
>>
>> In my understanding, PLR reacts based on the information received 
>> from lower layers via the RAW API and based on OAM observing the 
>> recovery graph.
>>
>> Which DetNet sub-layer the PLR then belongs to?
>>
>> Maybe not that easy to answer.
>>
>> Placing it in the DetNet Service sub-layer results in an additional 
>> layer in between the PLR and the API the PLR operates upon. So, it 
>> may not be the best approach.
>>
>> Placing it in the DetNet Forwarding sub-layer has the benefit that 
>> PLR is adjacent / directly connected to the API it operates upon.
>>
>> With that, perhaps a simplified view on what RAW is, i.e., what are 
>> the RAW add-ons / extensions to the DetNet architecture might be 
>> roughly viewed as:
>>
>>                    .
>>
>>                    .
>>
>>      +-----------------------------+
>>
>>      |                             |
>>
>>      |  DetNet Service sub-layer   | (e.g., (PREOF)
>>
>>      |                             |
>>
>>      +-----------------------------+
>>
>>      |                             |
>>
>>      | DetNet Forwarding sub-layer |
>>
>>      |                             |
>>
>>      |           _____    _____    |
>>
>>      |          < PLR >--< OAM >   |
>>
>>      |           -+--     -----    |
>>
>>      |            | ^              |
>>
>>      |            | |              |
>>
>>      |         ___v_+_____         |
>>
>>      +--------< RAW API >----------+
>>
>>      |         ---------           |
>>
>>      |    Wireless/Radio Layer     | (e.g., (H)ARQ)
>>
>>      |                             |
>>
>>      +-----------------------------+
>>
>>                    .
>>
>> So, I’d suggest more changes to Figure 4.
>>
>> As for where the OAM is based upon which the PLR acts: Well, I guess 
>> this part of the OAM really monitors the forwarding paths, i.e., the 
>> protection paths, the protection graph. So, I do not see it alien to 
>> be located at the DetNet Forwarding sub-layer.
>>
>> But, it all depends on the consensus of the WG.
>>
>> Some further comments:
>>
>> I’m not sure on the use of some terminology, e.g., 
>> synchronous/asynchronous CPF, distributed (e.g., PLR).
>>
>> Synchronous/asynchronous often refer to time synchronization. In my 
>> understanding, what the PLR does is actually a local decision. So, it 
>> is rather some local CPF making decision on Protection Path compared 
>> to the main CPF crafting the protection graph.
>>
>> As for distributed, my understanding is that in case of distributed, 
>> multiple entities have to act together to reach some goal, e.g., RSVP 
>> and OSPF are distributed and multiple nodes have to play together.
>>
>> I’m not sure I get, e.g., what “the PLR decision may be distributed” 
>> means. As I understand the point is that it is a local decision based 
>> on the local situation, independent of what’s going on at some remote 
>> location.
>>
>> I am not sure about the “loose” approach. My concern is that we do 
>> not know the guarantees of the loose segments, if any. Perhaps it 
>> would be better not going into “loose”.
>>
>> I’d suggest to use “extend” instead of “enhance”.
>>
>> Some detailed  comments on v16:
>>
>> Section 1:
>>
>> <v16>:
>>
>> Deterministic Networking is an attempt to emulate the properties of
>>
>> a serial link over a switched fabric, by providing a bounded latency
>>
>> and eliminating congestion loss, even when co-existing with best effort
>>
>> traffic.
>>
>> <JF>:
>>
>> I’d suggest:
>>
>> Deterministic Networking is to provide bounded latency and eliminate
>>
>> congestion loss, even when co-existing with best effort traffic.
>>
>> as I never had the emulation of serial link in mind.
>>
>> <v16>:
>>
>> Bringing determinism in a packet network means eliminating the
>>
>> statistical effects
>>
>> <JF>:
>>
>> I’m afraid complete elimination is not possible, so I’d suggest 
>> something like:
>>
>> Bringing determinism in a packet network means minimizing the
>>
>> statistical effects
>>
>> <v16>:
>>
>> This innovation was initially introduced on wired networks, with
>>
>> IEEE 802.1 Time Sensitive networking (TSN) - for Ethernet LANs - and
>>
>> IETF DetNet.
>>
>> <JF>:
>>
>> RFC 8578 DetNet Use Cases includes wireless.
>>
>> <v16>:
>>
>> With new scheduled
>>
>> radios such as TSCH and OFDMA
>>
>> <JF>
>>
>> Suggest to remove “new” as they are not new.
>>
>> <v16>:
>>
>> To achieve this, RAW
>>
>> leverages
>>
>> <JF>
>>
>> As it depends on the actual wireless technology etc., suggest to 
>> replace with:
>>
>> To achieve this, RAW
>>
>> can leverage
>>
>> <v16>:
>>
>> As opposed to routing trees, Distance-Vector protocols can enable
>>
>> more than one feasible successors along non-equal-cost multipath
>>
>> forwarding graphs. This provide redundancy and allow to dynamically
>>
>> adapt the forwarding operation to the state of the links. But this
>>
>> protection is limited since only a subset of the nodes along the
>>
>> path will have an alternate feasible successor
>>
>> <JF>
>>
>> I’m afraid this is confusing. A distance vector protocol may also 
>> build up a tree, see, e.g., RSTP. Sticking to the RSTP example, it is 
>> true that RSTP has the Alternate Port concept for fast reaction to a 
>> change, however, the concept has been then introduced to routing 
>> protocols as well, see LFA.
>>
>> Section 2:
>>
>> <v16>:
>>
>> In an quantic analogy, a recovery graph is to a path what an atomic
>>
>> orbital is to a planetary orbit, in that the electron has a
>>
>> probability of presence within a known shape as opposed to a
>>
>> deterministic trajectory.
>>
>> <JF>:
>>
>> Consider removing the paragraph as it may be unsure how much it is 
>> helpful to the reader in the given context.
>>
>> 2.1.7:
>>
>> Remove PAREO completely from the document for the reasons explained 
>> above.
>>
>> 2.2.1:
>>
>> Is this really flapping? Isn’t what the text explain is rather just 
>> degradation? Isn’t flapping rather a link going down and coming back?
>>
>> 2.2.4:
>>
>> Should "in the direction from the source towards the destination" be 
>> added?
>>
>> 2.3.1:
>>
>> Replace
>>
>> a packet from A to B
>>
>> experiences 2 paths,
>>
>> with
>>
>> B, a packet from A to B
>>
>> can experience 2 paths,
>>
>> as one particular packet only experiences one path
>>
>> 2.3.2
>>
>> I’m not sure about the need of “usage metadata;”. Perhaps remove?
>>
>> Figure 1 is without any mention in the text. Perhaps add explanation.
>>
>> <v16>:
>>
>> The vertices of that graph are DetNet Relay nodes that operate at
>>
>> the DetNet Service sub-layer and provide the PAREO functions.
>>
>> The topological edges of the graph are strict sequences of DetNet
>>
>> Transit nodes that operate at the DetNet Forwarding sub-layer.
>>
>> These bullet may need some updates depending on how the discussion goes.
>>
>> 2.5.1
>>
>> <v16>:
>>
>> an SLA (service level agreement) is a
>>
>> contract between a provider, the network, and a client, the
>>
>> application flow, about measurable metrics such as latency
>>
>> boundaries, consecutive losses, and packet delivery ratio (PDR).
>>
>> <JF>:
>>
>> In my understanding, an SLA is rather a contract between two parties 
>> (not more).
>>
>> Consider updating to:
>>
>> an SLA (service level agreement) is a
>>
>> contract between a network provider and a client about measurable 
>> metrics of a flow such as latency boundaries, consecutive losses, and 
>> packet delivery ratio (PDR).
>>
>> 2.5.6:
>>
>> “Because a wireless lane may not be good enough to
>>
>> provide the required reliability, and even 2 parallel lanes may not
>>
>> be over a longer period of time, the RAW availability implies a
>>
>> journey that is a lot more complex than following a serial path.”
>>
>> Could be removed as not needed for the definition.
>>
>> 3.1.1:
>>
>> <v16>:
>>
>> elimination of single points of failure
>>
>> <JF>:
>>
>> Perhaps rather: elimination of each single point of failure
>>
>> <v16>:
>>
>> prompt detection of failures as they occur.
>>
>> <JF>:
>>
>> Perhaps rather: prompt detection and reaction to failures as they occur.
>>
>> 3.1.1.1:
>>
>> <v16>:
>>
>> IP Routers leverage routing protocols to compute alternate routes
>>
>> <JF>:
>>
>> replace “compute” with "reroute to" as it is not just compute
>>
>> 3.1.1.2:
>>
>> <v16>:
>>
>> Having a backup equipment has a limited value unless it can be
>>
>> reliably switched into use within the down-time parameters.
>>
>> <JF>:
>>
>> Some schemes, e.g., PREOF, do not require switchover.
>>
>> <v16>:
>>
>> delayed reestablishment of the second path
>>
>> <JF>:
>>
>> There is no reestablishment of 2nd path in case of 1+1 protection; 
>> esp. in case of PREOF, where the basic idea is to send multiple 
>> copies of the same packet over pre-established maximally disjoint 
>> paths all the time.
>>
>> 3.1.2:
>>
>> <v16>:
>>
>> In data
>>
>> networks, this is rarely the case. Packet loss cannot never be fully
>>
>> avoided and the systems are built to resist to one loss, e.g., using
>>
>> redundancy with Retries (HARQ) or Packet Replication and Elimination
>>
>> (PRE), or, in a typical control loop, by linear interpolation from
>>
>> the previous measurements.
>>
>> <JF>
>>
>> Packet loss cannot be fully avoided and the systems are built to 
>> resist only a limited number of losses e.g., using retransmissions 
>> (HARQ), or, in a typical control loop, by linear interpolation from 
>> the previous measurements.
>>
>> Note that the RLC mechanism exists too.
>>
>> 3.2:
>>
>> <v16>:
>>
>> Conceptually, RAW is
>>
>> agnostic to the radio layer underneath though the capability to
>>
>> schedule transmissions is assumed.
>>
>> <JF>:
>>
>> I suggest to remove “though the capability to
>>
>> schedule transmissions is assumed”
>>
>> or replace with
>>
>> though it is assumed that the radio layer underneath is capable of 
>> schedule transmissions
>>
>> <v16>:
>>
>> Nevertheless,
>>
>> cross-layer optimizations may take place to ensure proper
>>
>> <JF>
>>
>> As we follow a layered approach (not an integrated one), it may be 
>> clearer to have :
>>
>> cross-layer optimization may take place via the APIs provided between 
>> RAW and the radio layer.
>>
>> 4.1:
>>
>> Suggest to remove “such as a Wi-Fi extended service set (ESS) or a 5G 
>> Core;”
>>
>> For instance ,the 5G core is part of the 5G System. When it comes to 
>> 5G, a better example would be a 5G System connected to a wireline 
>> domain.
>>
>> 4.3
>>
>> The RAW Management sub-layer
>>
>> functionality includes the PLR
>>
>> RAW aims to be faster than usual control plane. Management plane is 
>> slower than control plane. Management plane does not seem to be a 
>> good place for PLR, which intends to be very fast.
>>
>> 5.1
>>
>> Suggest to remove “In the wired world,”
>>
>> There could be a wireless subnet/segment along some of these paths.
>>
>> *From:* Pascal Thubert (pthubert) <pthubert=[e-mail]>
>> *Sent:* Friday, August 11, 2023 3:03 PM
>> *To:* Janos Farkas <[e-mail]>
>> *Cc:* [e-mail]; [e-mail]; Adrian Farrel 
>> ([e-mail]) <[e-mail]>; Greg Mirsky 
>> <[e-mail]>; Lou Berger ([e-mail]) <[e-mail]>
>> *Subject:* RE: Terminology: draft-ietf-raw-architecture-14.txt and 
>> what's still nuclear
>>
>> Hello Janos
>>
>> Many thanks for your review!
>>
>> >  I think RAW, i.e., wireless aspects actually rather reside in the 
>> Forwarding sub-layer (not in the Service sub-layer).
>>
>> > The Service sub-layer is pretty much IP in this case. Ideally, it 
>> is hidden from the Service sub-layer as much as possible what kind of 
>> forwarding is used in the Forwarding > sub-layer, e.g., whether it is 
>> wireline or wireless. (Well, the Forwarding sub-layer may provide 
>> some certain APIs towards the Service sub-layer if we want to.)
>>
>> We defined the sublayers by listing functions as hints as opposed to 
>> mapping to IP or L2. Our intention (Norm and I) at the start was that 
>> the DetNet operated at L3 but deterministic networking at large could 
>> be widely done at either L2 or L3 (or a mix) and the abstractions and 
>> service, from a distance, would be the same.
>>
>> RAW being DetNet it is L3, so neither wireless nor wired, though it 
>> is meant to address gaps that are more common in wireless like quick 
>> flaps, reliability degradation, and PHY speed variations. Wireless, 
>> like wire, is only observed and requested through APIs. As Lou said, 
>> some aspects like promiscuous and ARQ are more commonly observed on 
>> wireless, but are not exclusive to wireless. We provide a L3 
>> architecture that is sensitive to those elements but does not include 
>> them since they are operated in another layer altogether.
>>
>> The RAW architecture is focused on a fast protection/recovery loop. 
>> This is why the main operation (PAREO actuation and OAM 
>> processing/compiling ) is in the DetNet (=> L3) service sublayer. But 
>> not only:
>>
>> “
>>
>>     +------------------------------+ +--------------------------------+
>>
>>     |                              | |                                |
>>
>> .....................................................................
>>
>>     |                              | |                                |
>>
>>     | +--------------------------+ | | +----------------------------+ |
>>
>>     | |     aCPF                 | | | |         aCPF               | |
>>
>>     | +--------------------------+ | | +----------------------------+ |
>>
>>     | +----------+  +------------+ | | .-.-.-.-.-.--.  .-.-.-.-.-.--. |
>>
>>     | | PLR      |  | OAM        | | | | Distr. PLR |  | Distr. OAM | |
>>
>>     | |          |  | Supervisor | | | |            |  | Supervisor | |
>>
>>     | +----------+  +------------+ | | .-.-.-.-.-.--.  .-.-.-.-.-.--. |
>>
>>     |                              | |    optional         optional   |
>>
>>        RAW Management sub-layer
>>
>> .....................................................................
>>
>>        DetNet Service sub-layer
>>
>>     |                              | |                                |
>>
>>     | +----------+  +------------+ | | +------------+  +------------+ |
>>
>>     | | PAREO    |  |  OAM       | | | |  PAREO     |  |  OAM       | |
>>
>>     | | Actuator |  |  Observer  | | | |  Actuator  |  |  Observer  | |
>>
>>     | +----------+  +------------+ | | +------------+  +------------+ |
>>
>>     |                              | |                                |
>>
>>        DetNet Service sub-layer
>>
>> .....................................................................
>>
>>        DetNet Forwarding sub-layer
>>
>>     |                              | |                                |
>>
>>     |               +------------+ | |                 +------------+ |
>>
>>     |               |In-Situ OAM | | |                 |In-Situ OAM | |
>>
>>     |               +------------+ | |                 +------------+ |
>>
>>     |                              | |                                |
>>
>>     +------------------------------+ +--------------------------------+
>>
>>             End System or                       Relay
>>
>>           Ingress Edge Node                     Node
>>
>>           Figure 4: RAW functional posture within DetNet sublayers
>>
>> “
>>
>> Is there a change you would suggest to this picture?
>>
>> <JF>
>>
>> Yes, see above.
>>
>> > As I read, the 1st paragraph in section 4.3 is actually along these 
>> lines
>>
>> https://www.ietf.org/archive/id/draft-ietf-raw-architecture-14.html#name-raw-and-detnet
>>
>> > “RAW leverages the DetNet Forwarding sub-layer”
>>
>> > “RAW enhances DetNet to improve the protection against link errors 
>> such as transient flapping that are far more common in wireless 
>> links.” – Well, this reads to me to be in the Forwarding sub-layer.
>>
>> Well, protection is service sub-layer actually (section 2.1 of RFC 
>> 8655). Forwarding is packet handling, like find which buffer and 
>> which protection path for this packet.
>>
>> <JF>
>>
>> This is my point: “Forwarding is packet handling, like find which 
>> protection path for this packet.”
>>
>> So, you suggest Forwarding sub-layer too. 😉
>>
>> The PAREO actuator impacts the forwarding operation for future 
>> packets but does not necessarily run on every packet, see the 
>> discussion with David.
>>
>> <JF>
>>
>> Imo, PAREO should be completely removed from the document because it 
>> is a layer violation, as I keep saying for a while. See more above.
>>
>> > However, I’m quire unsure of the beginning of the 2nd paragraph in 
>> section 4.3:
>>
>> > “RAW extends the DetNet Stack (see fig 4 of 
>> [https://www.ietf.org/archive/id/draft-ietf-raw-architecture-14.html#RFC8655]) 
>> with additional functionality at the DetNet Service sub-layer for the 
>> PLR operation.”
>>
>> >  and the location of PLR in Figure 4, where the PLR is not in the 
>> data plane.
>>
>> You’re correct about the inconsistency. The PLR is a piece of the 
>> controller that was exported to the forwarding devices. The 
>> discussion with Greg placed that component in the management 
>> sublayer. The text that you point out is a victim of the renaming and 
>> the component it discusses is now called PAREO actuator. Proposed 
>> beginning of that section:
>>
>> “
>>
>>    RAW leverages the DetNet Forwarding sub-layer and requires the
>>
>>    support of in-situ OAM in DetNet Transit Nodes (see fig 3 of
>>
>>    [RFC8655] for the dynamic acquisition of link capacity and state to
>>
>>    maintain a strict RAW service, end-to-end, over a DetNet Network.
>>
>>    RAW enhances DetNet to improve the protection against link errors
>>
>>    such as transient flapping that are far more common in wireless
>>
>>    links.  Nevertheless, the RAW methods are for the most part
>>
>>    applicable to wired links as well, e.g., when energy savings are
>>
>>    desirable and the available path diversity exceeds 1+1 linear
>>
>>    redundancy.
>>
>>    RAW adds a Management sub-layer that operates in the DetNet
>>
>>    Operational Plane.  The RAW Management sub-layer typically runs only
>>
>>    in the DetNet Ingress Edge Node or End System, though it may also run
>>
>>    in DetNet Relay Nodes when the RAW Control sub-layer is distributed
>>
>>    along the recovery graph.  The RAW Management sub-layer functionality
>>
>>    includes the PLR that decides the DetNet Path for the future packets
>>
>>    of a flows and controls the PAREO Actuators along the DetNet Path
>>
>>    through specific signaling, and the OAM Supervisor that triggers, and
>>
>>    learns from, OAM observations, and feeds the PLR for its next
>>
>>    decision.
>>
>> RAW extends the DetNet Stack (see fig 4 of [RFC8655]) with additional
>>
>> functionality at the DetNet Service sub-layer for the actuation of
>>
>>    the PLR decision by the PAREO Actuator.Layer-3 in general and
>>
>>    DetNet in particular operates on abstractions of the lower layers and
>>
>>    through APIs to control those abstractions.  For instance, DetNet
>>
>>    already leverages lower layers for time-sensitive operations such as
>>
>>    time synchronization and traffic shapers.  Because the performances
>>
>>    of the radio layers are subject to rapid changes, so RAW needs more
>>
>>    dynamic gauges and knobs.  To that effect, the DetNet PREOF is
>>
>>    extended with the PAREO capabilities (see Section 5.6) and the RAW
>>
>>    PAREO Actuator manages dynamically the PAREO operations, which may be
>>
>>    performed either within the DetNet sublayers or at a lower layer,
>>
>>    using a common radio abstraction and APIs in the latter case.  In
>>
>>    particular, PAREO needs the capability to push reliability and timing
>>
>>    hints like suggest X retries (min, max) within a time window, or send
>>
>>    unicast (one next hop) or multicast (for overhearing).  The other way
>>
>>    around RAW needs hints about the radio conditions like L2 triggers
>>
>>    (RSSI, LQI, ETX...) over all the wireless hops.  This information is
>>
>>    useful to both the aCPF and the PLR.
>>
>> “
>>
>> > I think PLR should be located in the Forwarding sub-layer, but I 
>> may have misunderstood the intention.
>>
>> I think the operation you have in mind is the realization of the PLR 
>> decision. We did not create a box for it. Should we? What name? Is 
>> there really something new there vs. DetNet?
>>
>> > As far as I understood, the intention with the PLR is fast reaction 
>> to actual wireless situation, i.e., where to send a given packet 
>> actually.
>>
>> True, and it reacts to signals that arrive asynchronously to the 
>> packets, for the most part. They are either coming from L2 or from OAM.
>>
>>                |
>>
>>         packet | going
>>
>>       down the | stack
>>
>> +==========v==========+=====================+=====================+
>>
>>     |   (iOAM + iCTRL)    | (L2 Triggers, DLEP) |       (oOAM)        |
>>
>> +==========v==========+=====================+=====================+
>>
>>     |     Learn from |                     |    Learn from       |
>>
>>     |    packet tagging   > Maintain      <    end-to-end       |
>>
>>     +----------v----------+ Forwarding     |    OAM packets      |
>>
>>     | Forwarding decision < State        +---------^-----------|
>>
>> +----------v----------+                     |      Enrich or      |
>>
>>     +    Retag Packet     |  Learn abstracted   >     Regenerate      |
>>
>>     |    and Forward      | metrics about Links |     OAM packets     |
>>
>> +..........v..........+..........^..........+.........^.v.........+
>>
>>     |                          Lower layers                           |
>>
>> +..........v.....................^....................^.v.........+
>>
>>          frame | sent          Frame | L2 Ack        oOAM | | packet
>>
>>           over | wireless        In |                 In | | and out
>>
>>                v |                    | v
>>
>>                          Figure 10: PLR Interfaces
>>
>> > Imo, this is very similar to FRR, see, e.g.: 
>> https://datatracker.ietf.org/doc/html/rfc5286. A key aspect is that 
>> the alternate next hops are precalculated. That is, it resides in the 
>> data plane, no control plane reaction is needed.
>>
>> Hum FRR is still reroute. It installs new path in the forwarding 
>> plane (think software operation downloading a new FIB) and then the 
>> forwarding plane (think silicon now) uses the new paths possibly with 
>> no idea that it is a FRR path. Same here.
>>
>> > I guess a more recent (segment routing) document on the subject 
>> with PLR in it is: 
>> https://datatracker.ietf.org/doc/html/draft-ietf-rtgwg-segment-routing-ti-lfa, 
>> see, e.g.: “no need for any co-ordination or message exchange between 
>> the PLR and any other router in the network.”
>>
>> Note that segment routing is part of the DetNet Forwarding sub-layer 
>> (not the Service sub-layer).
>>
>> Like RAW, SR is a lot of things; the SRH operation (execution) is 
>> indeed forwarding plane. The decision of which SRH to insert upon 
>> which condition is control plane. Now we have this interesting 3D 
>> problem of how our planes intersect those planes with similar names, 
>> e.g., DetNet Controller plane and usual control plane.
>>
>> > Perhaps I’m the only one with these kind of thoughts.
>>
>> > If not, then perhaps it might be good to converge on PLR (e.g., its 
>> location in the architecture) before figuring out what actual changes 
>> are needed in the draft.
>>
>> Let’s see how others react. For now, you debunked at least one bug 
>> and that already very valuable. Committed as af8771a 
>> <https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-03fb2e6dd7e27dec&q=1&e=5ab1955b-c117-42bb-bcee-c4e507e491b9&u=https%3A%2F%2Fgithub.com%2Fraw-wg%2Fraw-architecture%2Fcommit%2Faf8771aeb4bc7dad5d2ae012cd53a4581e63f9df>
>>
>>
>> <JF>
>>
>> Yes, see what the group thinks.
>>
>> Thank you,
>>
>> Cheers,
>>
>> Janos
>>
>> Many thanks!
>>
>> Pascal
>>
>> *From:* Janos Farkas <Janos.Farkas=[e-mail]>
>> *Sent:* Monday, July 31, 2023 4:02 PM
>> *To:* Pascal Thubert (pthubert) <[e-mail]>
>> *Cc:* [e-mail]; [e-mail]; Adrian Farrel 
>> ([e-mail]) <[e-mail]>; Greg Mirsky 
>> <[e-mail]>; Lou Berger ([e-mail]) <[e-mail]>
>> *Subject:* RE: Terminology: draft-ietf-raw-architecture-14.txt and 
>> what's still nuclear
>>
>> Dear Pascal, all,
>>
>> Many thanks for the updates!
>>
>> Really great improvements towards architectural cleanliness.
>>
>> Otoh, distillation may reveal further questions and further 
>> distillation. 😉
>>
>> For instance, now that it is called PLR, it triggers a couple of 
>> questions on my part. (Well, with the assumption that I have some 
>> clue on PLR.)
>>
>> Perhaps one step back first:
>>
>> We actually introduced the division of DetNet data plane to 
>> Forwarding sub-layer and Service sub-layer for architectural 
>> cleanliness during DetNet data plane design team discussions. 
>> Furthermore, in order to divide (and conquer) in the sense to be able 
>> to identify the functions and to place them to some order, i.e., for 
>> functional decomposition.
>>
>> I suppose, the RAW architecture would follow DetNet architectural 
>> principles and place the RAW related functions accordingly.
>>
>> I think RAW, i.e., wireless aspects actually rather reside in the 
>> Forwarding sub-layer (not in the Service sub-layer).
>>
>> The Service sub-layer is pretty much IP in this case. Ideally, it is 
>> hidden from the Service sub-layer as much as possible what kind of 
>> forwarding is used in the Forwarding sub-layer, e.g., whether it is 
>> wireline or wireless. (Well, the Forwarding sub-layer may provide 
>> some certain APIs towards the Service sub-layer if we want to.)
>>
>> As I read, the 1^st paragraph in section 4.3 is actually along these 
>> lines
>>
>> https://www.ietf.org/archive/id/draft-ietf-raw-architecture-14.html#name-raw-and-detnet 
>>
>>
>> “RAW leverages the DetNet Forwarding sub-layer”
>>
>> “RAW enhances DetNet to improve the protection against link errors 
>> such as transient flapping that are far more common in wireless 
>> links.” – Well, this reads to me to be in the Forwarding sub-layer.
>>
>> However, I’m quire unsure of the beginning of the 2^nd paragraph in 
>> section 4.3:
>>
>> “RAW extends the DetNet Stack (see fig 4 of [RFC8655 
>> <https://www.ietf.org/archive/id/draft-ietf-raw-architecture-14.html#RFC8655>]) 
>> with additional functionality at the DetNet Service sub-layer for the 
>> PLR operation.”
>>
>> and the location of PLR in Figure 4, where the PLR is not in the data 
>> plane.
>>
>> I think PLR should be located in the Forwarding sub-layer, but I may 
>> have misunderstood the intention.
>>
>> As far as I understood, the intention with the PLR is fast reaction 
>> to actual wireless situation, i.e., where to send a given packet 
>> actually.
>>
>> Imo, this is very similar to FRR, see, e.g.: 
>> https://datatracker.ietf.org/doc/html/rfc5286. A key aspect is that 
>> the alternate next hops are precalculated. That is, it resides in the 
>> data plane, no control plane reaction is needed.
>>
>> I guess a more recent (segment routing) document on the subject with 
>> PLR in it is: 
>> https://datatracker.ietf.org/doc/html/draft-ietf-rtgwg-segment-routing-ti-lfa, 
>> see, e.g.: “no need for any co-ordination or message exchange between 
>> the PLR and any other router in the network.”
>>
>> Note that segment routing is part of the DetNet Forwarding sub-layer 
>> (not the Service sub-layer).
>>
>> Perhaps I’m the only one with these kind of thoughts.
>>
>> If not, then perhaps it might be good to converge on PLR (e.g., its 
>> location in the architecture) before figuring out what actual changes 
>> are needed in the draft.
>>
>> My 2 cents,
>>
>> János
>>
>> ps:
>>
>> Sorry for late commenting. It was not the plan. And now we are hit by 
>> August = vacation. I’m not sure how much I can chime in the 
>> discussion from vacation. I just thought sharing the comment instead 
>> of waiting till September.
>>
>> *From:* detnet <[e-mail]> *On Behalf Of *Pascal 
>> Thubert (pthubert)
>> *Sent:* Sunday, July 30, 2023 10:39 PM
>> *To:* [e-mail]; Adrian Farrel ([e-mail]) 
>> <[e-mail]>; Greg Mirsky <[e-mail]>; Lou 
>> Berger ([e-mail]) <[e-mail]>
>> *Cc:* [e-mail]
>> *Subject:* [Detnet] Terminology: draft-ietf-raw-architecture-14.txt 
>> and what's still nuclear
>>
>> Dear all:
>>
>> the publication of 14 adds terminologies and typos. The goal is to 
>> serve as the new reference for the WGLC so we can use the new terms 
>> in our discussions. If someone still uses PSE and Track, well, I 
>> guess we'll still understand for a little while, but he will be 
>> harshly reprimanded.
>>
>> What I did not do yet though I started is work out the positioning of 
>> the aCPF (the component that talks asynchronously to the rCPF == PCE 
>> to report performance and get route updates), the Point of Local 
>> Repair (PLR is the term that replaces the PSE) and the OAM supervisor 
>> that triggers OAM and aggregates results for the PLR.
>>
>> These are 3 new architectural blocks, and we want to position them 
>> well in the DetNet architecture.
>>
>> The DetNet architecture (section 4.4) has 3 planes that are mapped to 
>> classical SDN, with the controller plane in the middle, a southbound 
>> interface to the network plane (in the case of RAW used between rCPF 
>> and aCPF) and a northbound interface to the Application Plane.
>>
>> The Controller plane has the typical route servers like PCEs, and 
>> network management entities. In the SDN model they are "far away" and 
>> monitor the whole network. Which is what causing the RAW issue of 
>> lack of reactivity and pushed us to port functionality in the network 
>> plane. In networking planes parlance, the PCE is control plane and 
>> the NMEs are management plane.
>>
>> As we see, though the term controller plane looks like control plane, 
>> they are different beasts. A classical IGP is a control plane 
>> function that operates in the DetNet network plane. The controller 
>> plane hosts controllers, which physically may embed routing and 
>> management entities. In the DetNet architecture, "The Controller 
>> Plane corresponds to the aggregation of the Control and Management 
>> Planes in [RFC7426 <https://datatracker.ietf.org/doc/html/rfc7426>], 
>> though Common Control and Measurement Plane (CCAMP) (as defined by 
>> the CCAMP Working Group [CCAMP 
>> <https://datatracker.ietf.org/wg/ccamp/charter/>]) makes an 
>> additional distinction between management and measurement."
>>
>> In my book, the OAM supervisor, the aCPF and the PLR operate in the 
>> control plane. The OAM supervisor talks to the OAM handlers in the 
>> management plane. And all of the above being distributed in the 
>> network, they operate in the DetNet Network plane.  So 1) we extend 
>> the DetNet architecture to place functional blocks in the Network 
>> Plane and 2) one could say we need 3D pictures to represent the 
>> intersection of the DetNet planes and the traditional control and 
>> management planes. While the data plane remains firmly in the Network 
>> plane.
>>
>> Now this is my view and the way I intend to update the text to 
>> publish 15, hopefully quite soon. But I need confirmation that we are 
>> on the same line, or else debates to reach a consensus.
>>
>> What do you all think?
>>
>> Pascal
>>
>> ------------------------------------------------------------------------
>>
>> *De :*[e-mail] <[e-mail]>
>> *Envoyé :* samedi 29 juillet 2023 15:40
>> *À :* Pascal Thubert (pthubert) <[e-mail]>
>> *Objet :* New Version Notification for 
>> draft-ietf-raw-architecture-14.txt
>>
>>
>> A new version of I-D, draft-ietf-raw-architecture-14.txt
>> has been successfully submitted by Pascal Thubert and posted to the
>> IETF repository.
>>
>> Name:           draft-ietf-raw-architecture
>> Revision:       14
>> Title:          Reliable and Available Wireless Architecture
>> Document date:  2023-07-29
>> Group:          raw
>> Pages:          43
>> URL: https://www.ietf.org/archive/id/draft-ietf-raw-architecture-14.txt
>> Status: https://datatracker.ietf.org/doc/draft-ietf-raw-architecture/
>> Html: https://www.ietf.org/archive/id/draft-ietf-raw-architecture-14.html
>> Htmlized: 
>> https://datatracker.ietf.org/doc/html/draft-ietf-raw-architecture
>> Diff: 
>> https://author-tools.ietf.org/iddiff?url2=draft-ietf-raw-architecture-14
>>
>> Abstract:
>>    Reliable and Available Wireless (RAW) provides for high reliability
>>    and availability for IP connectivity across any combination of wired
>>    and wireless network segments.  The RAW Architecture extends the
>>    DetNet Architecture and other standard IETF concepts and mechanisms
>>    to adapt to the specific challenges of the wireless medium, in
>>    particular intermittently lossy connectivity. This document defines
>>    a network control loop that optimizes the use of constrained spectrum
>>    and energy while maintaining the expected connectivity properties,
>>    typically reliability and latency.  The loop involves OAM, DetNet
>>    Controller Plane, and PREOF extensions, and specifically a new
>>    recovery Function called PAREO and a new Point of Local Repair
>>    operation, that dynamically selects the DetNet path(s) for the future
>>    packets to route around local degradations and failures.
>>
>>
>>
>>
>> The IETF Secretariat
>>
>>
>> _______________________________________________
>> detnet mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/detnet