Re: [Detnet] comments on draft-ietf-raw-architecture-16

[Don Fedyk <[e-mail]>]

Hi Janos

Your wording is good.

As I read it there are multiple factors for ARQ and delay:

  *   Per hop reduces delay vs end-to-end
  *   HARC combines FEC and ARC
  *   Controlling the number of retransmissions to meet end-to end delay constraints.  Applies to ARQ and HARQ and important for RAW and DetNet.

Your wording is in line with that,
Cheers
Don


From: Janos Farkas <[e-mail]>
Sent: Thursday, February 29, 2024 4:28 AM
To: Don Fedyk <[e-mail]>; [e-mail]
Subject: RE: [Detnet] comments on draft-ietf-raw-architecture-16

Hi Don,

Thank you for checking and for the suggestions on ARQ. I overall like it; however, not sure about:
“On lossy wireless links this improves delivery and minimizes the impact on end-to-end latency. ”
My understanding is that (H)ARQ improves delivery at the price of latency. I mean a repeat introduces latency.

What about?:
Automatic Repeat Request, a well-known mechanism, enabling an acknowledged transmission with retries to mitigate errors and loss.  ARQ may be implemented at various layers in a network. ARQ is typically implemented at Layer-2, per hop and not end-to-end in wireless networks. ARQ improves delivery on lossy wireless. Additionally, ARQ retransmission may be further limited by a bounded time to meet end-to-end packet latency constraints.  Additional details and considerations for ARC are detailed in RFC 3366.

Thanks,
Janos

From: detnet <[e-mail]<mailto:[e-mail]>> On Behalf Of Don Fedyk
Sent: Wednesday, February 28, 2024 8:38 PM
To: Janos Farkas <Janos.Farkas=[e-mail]<mailto:Janos.Farkas=[e-mail]>>; [e-mail]<mailto:[e-mail]>
Subject: Re: [Detnet] comments on draft-ietf-raw-architecture-16

Hi Janos

I think the changes are good.  PREOF and ARQ is a good way to describe it.

Additionally, this text made me read the current definition of ARQ. I think it could be improved.

Current RAW Architecture Definition.
Automatic Repeat Request, enabling an acknowledged transmission and retries. ARQ is a typical model at Layer-2 on a wireless medium. ARQ is typically implemented hop-by-hop and not end-to-end in wireless networks. Else, it introduces excessive indetermination in latency, but a limited number of retries within a bounded time may be used within end-to-end constraints.
(I had trouble parsing the last sentence).
Suggest :
Automatic Repeat Request, a well-known mechanism, enabling an acknowledged transmission with retries to mitigate errors and loss.  ARQ may be implemented at various layers in a network. ARQ is typically implemented at Layer-2, per hop and not end-to-end in wireless networks. On lossy wireless links this improves delivery and minimizes the impact on end-to-end latency.  Additionally, ARQ retransmission may be further limited by a bounded time to meet end-to-end packet latency constraints.  Additional details and considerations for ARC are detailed in RFC 3366.
(Or something like that)
Cheers
Don


From: detnet <[e-mail]<mailto:[e-mail]>> On Behalf Of Janos Farkas
Sent: Wednesday, February 28, 2024 11:47 AM
To: [e-mail]<mailto:[e-mail]>
Subject: Re: [Detnet] comments on draft-ietf-raw-architecture-16

Hi,

Please find attached my individual contribution on suggested changes to text related to PAREO.
This is a diff to https://datatracker.ietf.org/doc/draft-ietf-raw-architecture/16/

Please let me know what do you think.

Regards,
Janos

From: detnet <[e-mail]<mailto:[e-mail]>> On Behalf Of Janos Farkas
Sent: Tuesday, February 27, 2024 1:28 AM
To: [e-mail]<mailto:[e-mail]>
Subject: Re: [Detnet] comments on draft-ietf-raw-architecture-16

Hi,

Picking this up again.

I am happy to propose/make changes to add the figures to the draft.

I still have concerns about PAREO. After some off line discussions, my understanding is that the idea behind the term PAREO really is to have a term for a kind of superset of reliability functions provided by the DetNet layer and the lower wireless/radio layer. However, I’m afraid it is not that useful  to talk about the superset, esp. given the controversy it brings. Thinking it further, each layer operates its reliability function in its own layer. If the RAW API is not feature rich, e.g., not so much information exchanged / exposed, then superset is not really meaningful. If the RAW API is feature rich, then it actually provides interaction possibilities between the reliability features. So, the reliability features are in essence remain the same in the different layers, but they can take input form the other layer. (In particular, thinking of PLR.) This allows cross-layer optimization if one wants to go that way.

I’ve been thinking and trying to come up with some potential way forward with the draft along the lines above.

For instance, a new paragraph could be added to Section 1 Introduction, e.g., as penultimate paragraph along the lines:
“In addition, RAW introduces the RAW API, which is an interface between the lower layer wireless technology and the DetNet layers. The RAW API is RAW technology [RAW-TECHNOS] dependent as it can vary what the different RAW technologies expose towards the DetNet layers. Furthermore, the different RAW technologies are equipped with different reliability features, e.g., short range broadcast, MUMIMO, PHY rate and other Modulation Coding Scheme (MCS) adaptation, (H)ARQ, constructive interference and overhearing. The RAW API enables interactions between the reliability functions provided by the wireless technology and the reliability functions provided by DetNet. That is, the RAW API makes cross-layer optimization possible for the reliability functions of different layers depending on the actual exposure provided via the RAW API by the give RAW technology.”

(Note that I tried to make this text in-line with the other definitions, e.g., 2.6.7 Lower Layer Information.)

Along the proposed direction, the term PAREO could be actually removed from the draft.

What do you think?

If the WG is ok with it, as a contribution, I can make suggested changes to the rest of the draft along the lines proposed above.

Regards,
Janos

From: Janos Farkas
Sent: Wednesday, November 8, 2023 1:14 PM
To: [e-mail]<mailto:[e-mail]>; [e-mail]<mailto:[e-mail]>
Subject: RE: [Detnet] comments on draft-ietf-raw-architecture-16

Hi Pascal, all,

I did not mention this in my mail, but I guess it is easy to see that the figure(s) I had in my mail have been inspired by Figure 4 in RFC 8655: https://www.rfc-editor.org/rfc/rfc8655.html#figure-4. I was trying to gasp the essence of RAW for myself to  have some guideline for my comments. If the group sees value in it, some version might be added to the draft.

Regards,
Janos

From: Pascal Thubert <[e-mail]<mailto:[e-mail]>>
Sent: Monday, November 6, 2023 2:49 PM
To: Lou Berger <[e-mail]<mailto:[e-mail]>>; Janos Farkas <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]>
Subject: Re: [Detnet] comments on draft-ietf-raw-architecture-16


Hello Lou and Janos

Parsing through Janos ' mail cannot be done before the meeting today, so I guess we can only discuss highlights today like the PAREO term/concept. I agree with Janos's drawing in his mail below. We already discussed that the DetNet layers Request but do not actuate (via APIs) the PAREO service that are operated at the lower layer.

"

PAREO functions that are actuated at the lower layers may be controlled through abstract interfaces by the RAW extensions within the DetNet Service sub-layer.

"

I do not see that this constitutes a layer violation. There are shaper services in DetNet that can be operated at various layers as well. Certainly there should be something between the layers to control who's doing what.

About the lane stuff, the term came in the discussion with Don Fedyk after the London meeting that you guys asked us to have. The term supersedes 'legs'. It is one specific sequential path within the recovery domain, we can expect several parallel ones for redundancy. I'm unclear if your protection path is one such lane or the redundant collection of parallel lanes. See also https://mailarchive.ietf.org/arch/browse/roll/?q=%20%22leg%22%20vs%20%22lane%22

For recovery graph, it is my understanding that we came up with this agreement in the corridor meeting with Greg and Adrian. The story is that the centralized CPF may not provide a set of protection paths to be selected from, but the union of them, in which case the PLR will build the temporary paths dynamically based on elements in the graph.



 all the best



Pascal






Le 06/11/2023 à 11:52, Lou Berger a écrit :

Pascal, (Janos)

Thank you for the much improved versions.  I think (as contributor) the document will be in good shape once the address the comments below.

I think Janos' comments are very helpful and  together with the drawings will answer one of my major open questions - how does a raw domain work with a non-raw detnet domain to support end to end detnet services.

A few terminology items to add to the comments below:

WRT Recovery graphs, I thought we were going to reuse one of the existing terms (i.e., protection group or recovery group) see, https://mailarchive.ietf.org/arch/msg/detnet/xj6bBURD66eB0ANhjqMd3anZ5vA/

WRT Lane - how does a lane differ from a 'protection path', i.e. do we really need this new term? I know I've asked this before, but I  do not see the difference based on the latest text.

At the nit level uplink and downlink can be dropped as terms as they are only used in their definition sections.

Thanks again,

Lou

On 11/6/2023 2:21 AM, Janos Farkas wrote:
Hi Pascal, all,

I am very sorry for not being able to respond earlier.

First of all, many thanks for the updates and significant changes in the draft!
The draft has been improved a lot!

I’ve read v16. I think it is better if I place my thoughts on v16 directly here than embedding in-line in our dialogue. These thoughts actually include responses too. In addition, I’ve added responses below too.

text from the draft are marked with <v16>
my thoughts are marked with <JF>

<JF>:
A couple of things are still confusing in the draft imo, e.g., some self-inconsistencies. I’ve been trying to wrap my head around how to comment actually. Perhaps it might be good to make sure that there is consensus on high level principles before diving into details.
Perhaps first what we are doing, what is RAW actually. I think there are very good statements on this already in the draft, e.g.:

<v16>:
RAW improves the reliability of transmissions and the availability
of the communication resources, but does not provide scheduling and
shaping, so RAW itself does not provide guarantees such as latency
for the application payload. Rather, it should be seen as a dynamic
optimization of the use of redundancy to maintain it within certain
boundaries. For instance, ARQ
is operated by the lower layers and
RAW will only abstract the concept and hint the lower layers on the
desired outcome, as opposed to performing the retries at Layer-3.

<JF>: note that I’ve intentionally removed the part I do not like and will come back to it.

<v16>:
using a common radio abstraction and
APIs
 the capability
to push reliability and timing hints like suggest X retries (min,
max) within a time window, or send unicast (one next hop) or
multicast (for overhearing). The other way around RAW needs hints
about the radio conditions like L2 triggers

<JF>: again, a few words left out intentionally for a sec.

I like the text above. I think they reflect what I think I heard we want: a clen layered model (not an integrated one)
I guess I would illustrate it something like:
                   .
                   .
     +-----------------------------+
     |                             |
     |  DetNet Service sub-layer   | (e.g., (PREOF)
     |                             |
     +-----------------------------+
     |                             |
     | DetNet Forwarding sub-layer |
     |         _________           |
     +--------< RAW API >----------+
     |         ---------           |
     |    Wireless/Radio Layer     | (e.g., (H)ARQ)
     |                             |
     +-----------------------------+
                   .

However, the draft still contains the term PAREO. I’m afraid I need to keep repeating that the term/concept PAREO is a layer violation itself. I guess PAREO comes from an integrated model. The figure above clearly illustrates why PAREO is a layer violation. PAREO combines (H)ARQ and PREOF. However, (H)ARQ resides in the Wireless/Radio Layer whereas, PREOF is part of the DetNet Service sub-layer. They are very different layers, and there is the DetNet Forwarding sub-layer in between them. Therefore, the term and concept PAREO should be removed completely from the document.

So, to the question whether I suggest changes to Figure 4: Yes, remove PARO at the minimum. Nonetheless, there may be further potential changes falling out of the train of thoughts below.
I do not have a replacement term yet, but perhaps we might get somewhat closer via the thoughts below.

So, I think one essence of RAW is the API.
I think the draft contains further good text about the API, e.g.:

<v16>:
Conceptually, RAW is
agnostic to the radio layer underneath
<v16>:
hint on
the desired outcome, and the lower layer acts on those hints to
provide the best approximation of that outcome,
<v16:>:
abstract
interface, while they are really operated at the lower layers

<JF>: Nonetheless, the draft sometimes uses “controls”, “manipulates” and alike from what is done from the DetNet layer towards the Wireless/Radio Layer.
The draft also (imo correctly) includes statements like:

<v16>:
it is not expected that all lower layers
support all those capabilities

<JF>:
I think that the RAW architecture should be crafted such that it covers at least the wireless technologies in draft-ietf-raw-technologies. They are quire different; provide different ways / levels of exposure. So, I rather like the language around “hint” than “control”.
For instance,
“RAW manipulates abstractions of the lower layer services to hint on
the desired outcome,”
could be replaced with
“RAW provides hints to the lower layer services on the desired outcome,”


<JF>:
Back to what is RAW:
In my understanding an essence of RAW is the capability of fast reaction to rapidly changing wireless conditions on order to maintain the desired QoS for the end to end service, esp., reliability.
(I know I’m simplifying a couple of aspects a bit, but this is with the attempt to make it easier to come to some similar page.)
To this extent, the concept of recovery graph has been introduced. The term protection path is widely used too. Perhaps, at a high level, it is fair to say that the recovery graph is the superset of all possible protection paths. (again, there may be some simplifications here to be able to express some thoughts below)
I found very good statements in the draft:

<v16>:
RAW
itself operates in the DetNet Network Plane at a faster time scale
<v16>:
take the routing decision early
enough along the possible paths to route around the failure. RAW
defines a end-to-end control loop that dynamically controls the
activation and deactivation of the feasible Protection Paths
<v16>:
An Operational Plane PLR that hosts the Decision function of
which DetNet Paths to use for the future packets that will be
routed within the recovery graph
<v16>:
The main action by the PLR is the swapping of the DetNet Path within
the recovery graph for the future packets. The candidate DetNet
Paths represent different energy and spectrum profiles, and provide
protection against different failures.
<v16>:
RAW defines the Path Selection Engine (PLR) as a
synchronous CPF that performs rapid local adjustments of the
forwarding tables within the diversity that the asynchronous CPF has
in store for the recovery graph
<v16>:
the system observed by the RAW OAM is the
recovery graph

<JF>:
In my understanding, the key component being added by RAW to the DetNet architecture is the PLR to perform the fast reactions to the fast changes in wireless/radio conditions.
In my understanding, the establishment of the forwarding paths, i.e., the Protection Paths and ultimately the recovery graph belongs to the DetNet Forwarding sub-layer. In my read, this is illustrated, e.g., in Figure 2 in RFC 8655. The explicit routes (source routing whatsoever) belong to the DetNet Forwarding sub-layer.
In my understanding, PLR reacts based on the information received from lower layers via the RAW API and based on OAM observing the recovery graph.
Which DetNet sub-layer the PLR then belongs to?
Maybe not that easy to answer.
Placing it in the DetNet Service sub-layer results in an additional layer in between the PLR and the API the PLR operates upon. So, it may not be the best approach.
Placing it in the DetNet Forwarding sub-layer has the benefit that PLR is adjacent / directly connected to the API it operates upon.

With that, perhaps a simplified view on what RAW is, i.e., what are the RAW add-ons / extensions to the DetNet architecture might be roughly viewed as:

                   .
                   .
     +-----------------------------+
     |                             |
     |  DetNet Service sub-layer   | (e.g., (PREOF)
     |                             |
     +-----------------------------+
     |                             |
     | DetNet Forwarding sub-layer |
     |                             |
     |           _____    _____    |
     |          < PLR >--< OAM >   |
     |           -+--     -----    |
     |            | ^              |
     |            | |              |
     |         ___v_+_____         |
     +--------< RAW API >----------+
     |         ---------           |
     |    Wireless/Radio Layer     | (e.g., (H)ARQ)
     |                             |
     +-----------------------------+
                   .

So, I’d suggest more changes to Figure 4.
As for where the OAM is based upon which the PLR acts: Well, I guess this part of the OAM really monitors the forwarding paths, i.e., the protection paths, the protection graph. So, I do not see it alien to be located at the DetNet Forwarding sub-layer.
But, it all depends on the consensus of the WG.


Some further comments:
I’m not sure on the use of some terminology, e.g., synchronous/asynchronous CPF, distributed (e.g., PLR).
Synchronous/asynchronous often refer to time synchronization. In my understanding, what the PLR does is actually a local decision. So, it is rather some local CPF making decision on Protection Path compared to the main CPF crafting the protection graph.
As for distributed, my understanding is that in case of distributed, multiple entities have to act together to reach some goal, e.g., RSVP and OSPF are distributed and multiple nodes have to play together.
I’m not sure I get, e.g., what “the PLR decision may be distributed” means. As I understand the point is that it is a local decision based on the local situation, independent of what’s going on at some remote location.

I am not sure about the “loose” approach. My concern is that we do not know the guarantees of the loose segments, if any. Perhaps it would be better not going into “loose”.

I’d suggest to use “extend” instead of “enhance”.


Some detailed  comments on v16:

Section 1:
<v16>:
Deterministic Networking is an attempt to emulate the properties of
a serial link over a switched fabric, by providing a bounded latency
and eliminating congestion loss, even when co-existing with best effort
traffic.
<JF>:
I’d suggest:
Deterministic Networking is to provide bounded latency and eliminate
congestion loss, even when co-existing with best effort traffic.

as I never had the emulation of serial link in mind.

<v16>:
Bringing determinism in a packet network means eliminating the
statistical effects
<JF>:
I’m afraid complete elimination is not possible, so I’d suggest something like:
Bringing determinism in a packet network means minimizing the
statistical effects


<v16>:
This innovation was initially introduced on wired networks, with
IEEE 802.1 Time Sensitive networking (TSN) - for Ethernet LANs - and
IETF DetNet.
<JF>:
RFC 8578 DetNet Use Cases includes wireless.

<v16>:
With new scheduled
radios such as TSCH and OFDMA
<JF>
Suggest to remove “new” as they are not new.

<v16>:
To achieve this, RAW
leverages
<JF>
As it depends on the actual wireless technology etc., suggest to replace with:
To achieve this, RAW
can leverage

<v16>:
As opposed to routing trees, Distance-Vector protocols can enable
more than one feasible successors along non-equal-cost multipath
forwarding graphs. This provide redundancy and allow to dynamically
adapt the forwarding operation to the state of the links. But this
protection is limited since only a subset of the nodes along the
path will have an alternate feasible successor
<JF>
I’m afraid this is confusing. A distance vector protocol may also build up a tree, see, e.g., RSTP. Sticking to the RSTP example, it is true that RSTP has the Alternate Port concept for fast reaction to a change, however, the concept has been then introduced to routing protocols as well, see LFA.



Section 2:
<v16>:
In an quantic analogy, a recovery graph is to a path what an atomic
orbital is to a planetary orbit, in that the electron has a
probability of presence within a known shape as opposed to a
deterministic trajectory.

<JF>:
Consider removing the paragraph as it may be unsure how much it is helpful to the reader in the given context.
2.1.7:
Remove PAREO completely from the document for the reasons explained above.

2.2.1:
Is this really flapping? Isn’t what the text explain is rather just degradation? Isn’t flapping rather a link going down and coming back?

2.2.4:
Should "in the direction from the source towards the destination" be added?
2.3.1:
Replace
a packet from A to B
experiences 2 paths,
with
B, a packet from A to B
can experience 2 paths,
as one particular packet only experiences one path

2.3.2
I’m not sure about the need of “usage metadata;”. Perhaps remove?
Figure 1 is without any mention in the text. Perhaps add explanation.
<v16>:
The vertices of that graph are DetNet Relay nodes that operate at
the DetNet Service sub-layer and provide the PAREO functions.

The topological edges of the graph are strict sequences of DetNet
Transit nodes that operate at the DetNet Forwarding sub-layer.

These bullet may need some updates depending on how the discussion goes.

2.5.1
<v16>:
an SLA (service level agreement) is a
contract between a provider, the network, and a client, the
application flow, about measurable metrics such as latency
boundaries, consecutive losses, and packet delivery ratio (PDR).
<JF>:
In my understanding, an SLA is rather a contract between two parties (not more).
Consider updating to:
an SLA (service level agreement) is a
contract between a network provider and a client about measurable metrics of a flow such as latency boundaries, consecutive losses, and packet delivery ratio (PDR).

2.5.6:
“ Because a wireless lane may not be good enough to
provide the required reliability, and even 2 parallel lanes may not
be over a longer period of time, the RAW availability implies a
journey that is a lot more complex than following a serial path.”
Could be removed as not needed for the definition.

3.1.1:
<v16>:
elimination of single points of failure
<JF>:
Perhaps rather: elimination of each single point of failure
<v16>:
prompt detection of failures as they occur.
<JF>:
Perhaps rather: prompt detection and reaction to failures as they occur.

3.1.1.1:
<v16>:
IP Routers leverage routing protocols to compute alternate routes
<JF>:
replace “compute” with "reroute to" as it is not just compute

3.1.1.2:

<v16>:
Having a backup equipment has a limited value unless it can be
reliably switched into use within the down-time parameters.

<JF>:
Some schemes, e.g., PREOF, do not require switchover.

<v16>:
delayed reestablishment of the second path
<JF>:
There is no reestablishment of 2nd path in case of 1+1 protection; esp. in case of PREOF, where the basic idea is to send multiple copies of the same packet over pre-established maximally disjoint paths all the time.

3.1.2:
<v16>:
In data
networks, this is rarely the case. Packet loss cannot never be fully
avoided and the systems are built to resist to one loss, e.g., using
redundancy with Retries (HARQ) or Packet Replication and Elimination
(PRE), or, in a typical control loop, by linear interpolation from
the previous measurements.
<JF>
Packet loss cannot be fully avoided and the systems are built to resist only a limited number of losses e.g., using retransmissions (HARQ), or, in a typical control loop, by linear interpolation from the previous measurements.
Note that the RLC mechanism exists too.

3.2:
<v16>:
Conceptually, RAW is
agnostic to the radio layer underneath though the capability to
schedule transmissions is assumed.
<JF>:
I suggest to remove “though the capability to
schedule transmissions is assumed”
or replace with
though it is assumed that the radio layer underneath is capable of schedule transmissions

<v16>:
Nevertheless,
cross-layer optimizations may take place to ensure proper
<JF>
As we follow a layered approach (not an integrated one), it may be clearer to have :
cross-layer optimization may take place via the APIs provided between RAW and the radio layer.

4.1:
Suggest to remove “such as a Wi-Fi extended service set (ESS) or a 5G Core;”
For instance ,the 5G core is part of the 5G System. When it comes to 5G, a better example would be a 5G System connected to a wireline domain.

4.3
The RAW Management sub-layer
functionality includes the PLR
RAW aims to be faster than usual control plane. Management plane is slower than control plane. Management plane does not seem to be a good place for PLR, which intends to be very fast.

5.1
Suggest to remove “In the wired world,”
There could be a wireless subnet/segment along some of these paths.


From: Pascal Thubert (pthubert) <pthubert=[e-mail]><mailto:pthubert=[e-mail]>
Sent: Friday, August 11, 2023 3:03 PM
To: Janos Farkas <[e-mail]><mailto:[e-mail]>
Cc: [e-mail]<mailto:[e-mail]>; [e-mail]<mailto:[e-mail]>; Adrian Farrel ([e-mail]<mailto:[e-mail]>) <[e-mail]><mailto:[e-mail]>; Greg Mirsky <[e-mail]><mailto:[e-mail]>; Lou Berger ([e-mail]<mailto:[e-mail]>) <[e-mail]><mailto:[e-mail]>
Subject: RE: Terminology: draft-ietf-raw-architecture-14.txt and what's still nuclear


Hello Janos



Many thanks for your review!



>  I think RAW, i.e., wireless aspects actually rather reside in the Forwarding sub-layer (not in the Service sub-layer).

> The Service sub-layer is pretty much IP in this case. Ideally, it is hidden from the Service sub-layer as much as possible what kind of forwarding is used in the Forwarding > sub-layer, e.g., whether it is wireline or wireless. (Well, the Forwarding sub-layer may provide some certain APIs towards the Service sub-layer if we want to.)





We defined the sublayers by listing functions as hints as opposed to mapping to IP or L2. Our intention (Norm and I) at the start was that the DetNet operated at L3 but deterministic networking at large could be widely done at either L2 or L3 (or a mix) and the abstractions and service, from a distance, would be the same.



RAW being DetNet it is L3, so neither wireless nor wired, though it is meant to address gaps that are more common in wireless like quick flaps, reliability degradation, and PHY speed variations. Wireless, like wire, is only observed and requested through APIs. As Lou said, some aspects like promiscuous and ARQ are more commonly observed on wireless, but are not exclusive to wireless. We provide a L3 architecture that is sensitive to those elements but does not include them since they are operated in another layer altogether.



The RAW architecture is focused on a fast protection/recovery loop. This is why the main operation (PAREO actuation and OAM processing/compiling ) is in the DetNet (=> L3) service sublayer. But not only:



“

    +------------------------------+ +--------------------------------+

    |                              | |                                |

   .....................................................................

    |                              | |                                |

    | +--------------------------+ | | +----------------------------+ |

    | |     aCPF                 | | | |         aCPF               | |

    | +--------------------------+ | | +----------------------------+ |

    | +----------+  +------------+ | | .-.-.-.-.-.--.  .-.-.-.-.-.--. |

    | | PLR      |  | OAM        | | | | Distr. PLR |  | Distr. OAM | |

    | |          |  | Supervisor | | | |            |  | Supervisor | |

    | +----------+  +------------+ | | .-.-.-.-.-.--.  .-.-.-.-.-.--. |

    |                              | |    optional         optional   |

       RAW Management sub-layer

   .....................................................................

       DetNet Service sub-layer

    |                              | |                                |

    | +----------+  +------------+ | | +------------+  +------------+ |

    | | PAREO    |  |  OAM       | | | |  PAREO     |  |  OAM       | |

    | | Actuator |  |  Observer  | | | |  Actuator  |  |  Observer  | |

    | +----------+  +------------+ | | +------------+  +------------+ |

    |                              | |                                |

       DetNet Service sub-layer

   .....................................................................

       DetNet Forwarding sub-layer

    |                              | |                                |

    |               +------------+ | |                 +------------+ |

    |               |In-Situ OAM | | |                 |In-Situ OAM | |

    |               +------------+ | |                 +------------+ |

    |                              | |                                |

    +------------------------------+ +--------------------------------+



            End System or                       Relay

          Ingress Edge Node                     Node



          Figure 4: RAW functional posture within DetNet sublayers



“



Is there a change you would suggest to this picture?



<JF>

Yes, see above.











> As I read, the 1st paragraph in section 4.3 is actually along these lines

https://www.ietf.org/archive/id/draft-ietf-raw-architecture-14.html#name-raw-and-detnet

> “RAW leverages the DetNet Forwarding sub-layer”

> “RAW enhances DetNet to improve the protection against link errors such as transient flapping that are far more common in wireless links.” – Well, this reads to me to be in the Forwarding sub-layer.



Well, protection is service sub-layer actually (section 2.1 of RFC 8655). Forwarding is packet handling, like find which buffer and which protection path for this packet.



<JF>

This is my point: “Forwarding is packet handling, like find which protection path for this packet.”

So, you suggest Forwarding sub-layer too. 😉





The PAREO actuator impacts the forwarding operation for future packets but does not necessarily run on every packet, see the discussion with David.



<JF>

Imo, PAREO should be completely removed from the document because it is a layer violation, as I keep saying for a while. See more above.





> However, I’m quire unsure of the beginning of the 2nd paragraph in section 4.3:

> “RAW extends the DetNet Stack (see fig 4 of [https://www.ietf.org/archive/id/draft-ietf-raw-architecture-14.html#RFC8655]) with additional functionality at the DetNet Service sub-layer for the PLR operation.”

>  and the location of PLR in Figure 4, where the PLR is not in the data plane.



You’re correct about the inconsistency. The PLR is a piece of the controller that was exported to the forwarding devices. The discussion with Greg placed that component in the management sublayer. The text that you point out is a victim of the renaming and the component it discusses is now called PAREO actuator. Proposed beginning of that section:

“

   RAW leverages the DetNet Forwarding sub-layer and requires the

   support of in-situ OAM in DetNet Transit Nodes (see fig 3 of

   [RFC8655] for the dynamic acquisition of link capacity and state to

   maintain a strict RAW service, end-to-end, over a DetNet Network.

   RAW enhances DetNet to improve the protection against link errors

   such as transient flapping that are far more common in wireless

   links.  Nevertheless, the RAW methods are for the most part

   applicable to wired links as well, e.g., when energy savings are

   desirable and the available path diversity exceeds 1+1 linear

   redundancy.



   RAW adds a Management sub-layer that operates in the DetNet

   Operational Plane.  The RAW Management sub-layer typically runs only

   in the DetNet Ingress Edge Node or End System, though it may also run

   in DetNet Relay Nodes when the RAW Control sub-layer is distributed

   along the recovery graph.  The RAW Management sub-layer functionality

   includes the PLR that decides the DetNet Path for the future packets

   of a flows and controls the PAREO Actuators along the DetNet Path

   through specific signaling, and the OAM Supervisor that triggers, and

   learns from, OAM observations, and feeds the PLR for its next

   decision.



   RAW extends the DetNet Stack (see fig 4 of [RFC8655]) with additional

   functionality at the DetNet Service sub-layer for the actuation of

   the PLR decision by the PAREO Actuator.  Layer-3 in general and

   DetNet in particular operates on abstractions of the lower layers and

   through APIs to control those abstractions.  For instance, DetNet

   already leverages lower layers for time-sensitive operations such as

   time synchronization and traffic shapers.  Because the performances

   of the radio layers are subject to rapid changes, so RAW needs more

   dynamic gauges and knobs.  To that effect, the DetNet PREOF is

   extended with the PAREO capabilities (see Section 5.6) and the RAW

   PAREO Actuator manages dynamically the PAREO operations, which may be

   performed either within the DetNet sublayers or at a lower layer,

   using a common radio abstraction and APIs in the latter case.  In

   particular, PAREO needs the capability to push reliability and timing

   hints like suggest X retries (min, max) within a time window, or send

   unicast (one next hop) or multicast (for overhearing).  The other way

   around RAW needs hints about the radio conditions like L2 triggers

   (RSSI, LQI, ETX...) over all the wireless hops.  This information is

   useful to both the aCPF and the PLR.



“



> I think PLR should be located in the Forwarding sub-layer, but I may have misunderstood the intention.



I think the operation you have in mind is the realization of the PLR decision. We did not create a box for it. Should we? What name? Is there really something new there vs. DetNet?



> As far as I understood, the intention with the PLR is fast reaction to actual wireless situation, i.e., where to send a given packet actually.



True, and it reacts to signals that arrive asynchronously to the packets, for the most part. They are either coming from L2 or from OAM.







               |

        packet | going

      down the | stack

    +==========v==========+=====================+=====================+

    |   (iOAM + iCTRL)    | (L2 Triggers, DLEP) |       (oOAM)        |

    +==========v==========+=====================+=====================+

    |     Learn from      |                     |    Learn from       |

    |    packet tagging   >       Maintain      <    end-to-end       |

    +----------v----------+      Forwarding     |    OAM packets      |

    | Forwarding decision <        State        +---------^-----------|

    +----------v----------+                     |      Enrich or      |

    +    Retag Packet     |  Learn abstracted   >     Regenerate      |

    |    and Forward      | metrics about Links |     OAM packets     |

    +..........v..........+..........^..........+.........^.v.........+

    |                          Lower layers                           |

    +..........v.....................^....................^.v.........+

         frame | sent          Frame | L2 Ack        oOAM | | packet

          over | wireless        In  |                 In | | and out

               v                     |                    | v



                         Figure 10: PLR Interfaces







> Imo, this is very similar to FRR, see, e.g.: https://datatracker.ietf.org/doc/html/rfc5286. A key aspect is that the alternate next hops are precalculated. That is, it resides in the data plane, no control plane reaction is needed.



Hum FRR is still reroute. It installs new path in the forwarding plane (think software operation downloading a new FIB) and then the forwarding plane (think silicon now) uses the new paths possibly with no idea that it is a FRR path. Same here.



> I guess a more recent (segment routing) document on the subject with PLR in it is: https://datatracker.ietf.org/doc/html/draft-ietf-rtgwg-segment-routing-ti-lfa, see, e.g.: “no need for any co-ordination or message exchange between the PLR and any other router in the network.”

Note that segment routing is part of the DetNet Forwarding sub-layer (not the Service sub-layer).



Like RAW, SR is a lot of things; the SRH operation (execution) is indeed forwarding plane. The decision of which SRH to insert upon which condition is control plane. Now we have this interesting 3D problem of how our planes intersect those planes with similar names, e.g., DetNet Controller plane and usual control plane.



> Perhaps I’m the only one with these kind of thoughts.

> If not, then perhaps it might be good to converge on PLR (e.g., its location in the architecture) before figuring out what actual changes are needed in the draft.



Let’s see how others react. For now, you debunked at least one bug and that already very valuable. Committed as af8771a<https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-03fb2e6dd7e27dec&q=1&e=5ab1955b-c117-42bb-bcee-c4e507e491b9&u=https%3A%2F%2Fgithub.com%2Fraw-wg%2Fraw-architecture%2Fcommit%2Faf8771aeb4bc7dad5d2ae012cd53a4581e63f9df>


<JF>
Yes, see what the group thinks.

Thank you,
Cheers,
Janos


Many thanks!

Pascal
From: Janos Farkas <Janos.Farkas=[e-mail]<mailto:Janos.Farkas=[e-mail]>>
Sent: Monday, July 31, 2023 4:02 PM
To: Pascal Thubert (pthubert) <[e-mail]<mailto:[e-mail]>>
Cc: [e-mail]<mailto:[e-mail]>; [e-mail]<mailto:[e-mail]>; Adrian Farrel ([e-mail]<mailto:[e-mail]>) <[e-mail]<mailto:[e-mail]>>; Greg Mirsky <[e-mail]<mailto:[e-mail]>>; Lou Berger ([e-mail]<mailto:[e-mail]>) <[e-mail]<mailto:[e-mail]>>
Subject: RE: Terminology: draft-ietf-raw-architecture-14.txt and what's still nuclear

Dear Pascal, all,

Many thanks for the updates!
Really great improvements towards architectural cleanliness.

Otoh, distillation may reveal further questions and further distillation. 😉

For instance, now that it is called PLR, it triggers a couple of questions on my part. (Well, with the assumption that I have some clue on PLR.)

Perhaps one step back first:
We actually introduced the division of DetNet data plane to Forwarding sub-layer and Service sub-layer for architectural cleanliness during DetNet data plane design team discussions. Furthermore, in order to divide (and conquer) in the sense to be able to identify the functions and to place them to some order, i.e., for functional decomposition.

I suppose, the RAW architecture would follow DetNet architectural principles and place the RAW related functions accordingly.

I think RAW, i.e., wireless aspects actually rather reside in the Forwarding sub-layer (not in the Service sub-layer).
The Service sub-layer is pretty much IP in this case. Ideally, it is hidden from the Service sub-layer as much as possible what kind of forwarding is used in the Forwarding sub-layer, e.g., whether it is wireline or wireless. (Well, the Forwarding sub-layer may provide some certain APIs towards the Service sub-layer if we want to.)

As I read, the 1st paragraph in section 4.3 is actually along these lines
https://www.ietf.org/archive/id/draft-ietf-raw-architecture-14.html#name-raw-and-detnet
“RAW leverages the DetNet Forwarding sub-layer”
“RAW enhances DetNet to improve the protection against link errors such as transient flapping that are far more common in wireless links.” – Well, this reads to me to be in the Forwarding sub-layer.

However, I’m quire unsure of the beginning of the 2nd paragraph in section 4.3:
“RAW extends the DetNet Stack (see fig 4 of [RFC8655<https://www.ietf.org/archive/id/draft-ietf-raw-architecture-14.html#RFC8655>]) with additional functionality at the DetNet Service sub-layer for the PLR operation.”
and the location of PLR in Figure 4, where the PLR is not in the data plane.

I think PLR should be located in the Forwarding sub-layer, but I may have misunderstood the intention.

As far as I understood, the intention with the PLR is fast reaction to actual wireless situation, i.e., where to send a given packet actually.
Imo, this is very similar to FRR, see, e.g.: https://datatracker.ietf.org/doc/html/rfc5286. A key aspect is that the alternate next hops are precalculated. That is, it resides in the data plane, no control plane reaction is needed.
I guess a more recent (segment routing) document on the subject with PLR in it is: https://datatracker.ietf.org/doc/html/draft-ietf-rtgwg-segment-routing-ti-lfa, see, e.g.: “no need for any co-ordination or message exchange between the PLR and any other router in the network.”
Note that segment routing is part of the DetNet Forwarding sub-layer (not the Service sub-layer).

Perhaps I’m the only one with these kind of thoughts.
If not, then perhaps it might be good to converge on PLR (e.g., its location in the architecture) before figuring out what actual changes are needed in the draft.

My 2 cents,
János

ps:
Sorry for late commenting. It was not the plan. And now we are hit by August = vacation. I’m not sure how much I can chime in the discussion from vacation. I just thought sharing the comment instead of waiting till September.


From: detnet <[e-mail]<mailto:[e-mail]>> On Behalf Of Pascal Thubert (pthubert)
Sent: Sunday, July 30, 2023 10:39 PM
To: [e-mail]<mailto:[e-mail]>; Adrian Farrel ([e-mail]<mailto:[e-mail]>) <[e-mail]<mailto:[e-mail]>>; Greg Mirsky <[e-mail]<mailto:[e-mail]>>; Lou Berger ([e-mail]<mailto:[e-mail]>) <[e-mail]<mailto:[e-mail]>>
Cc: [e-mail]<mailto:[e-mail]>
Subject: [Detnet] Terminology: draft-ietf-raw-architecture-14.txt and what's still nuclear


Dear all:

the publication of 14 adds terminologies and typos. The goal is to serve as the new reference for the WGLC so we can use the new terms in our discussions. If someone still uses PSE and Track, well, I guess we'll still understand for a little while, but he will be harshly reprimanded.

What I did not do yet though I started is work out the positioning of the aCPF (the component that talks asynchronously to the rCPF == PCE to report performance and get route updates), the Point of Local Repair (PLR is the term that replaces the PSE) and the OAM supervisor that triggers OAM and aggregates results for the PLR.

These are 3 new architectural blocks, and we want to position them well in the DetNet architecture.

The DetNet architecture (section 4.4) has 3 planes that are mapped to classical SDN, with the controller plane in the middle, a southbound interface to the network plane (in the case of RAW used between rCPF and aCPF) and a northbound interface to the Application Plane.

The Controller plane has the typical route servers like PCEs, and network management entities. In the SDN model they are "far away" and monitor the whole network. Which is what causing the RAW issue of lack of reactivity and pushed us to port functionality in the network plane. In networking planes parlance, the PCE is control plane and the NMEs are management plane.

As we see, though the term controller plane looks like control plane, they are different beasts. A classical IGP is a control plane function that operates in the DetNet network plane. The controller plane hosts controllers, which physically may embed routing and management entities. In the DetNet architecture, "The Controller Plane corresponds to the aggregation of the Control and Management Planes in [RFC7426<https://datatracker.ietf.org/doc/html/rfc7426>], though Common Control and Measurement Plane (CCAMP) (as defined by the CCAMP Working Group [CCAMP<https://datatracker.ietf.org/wg/ccamp/charter/>]) makes an additional distinction between management and measurement."

In my book, the OAM supervisor, the aCPF and the PLR operate in the control plane. The OAM supervisor talks to the OAM handlers in the management plane. And all of the above being distributed in the network, they operate in the DetNet Network plane.  So 1) we extend the DetNet architecture to place functional blocks in the Network Plane and 2) one could say we need 3D pictures to represent the intersection of the DetNet planes and the traditional control and management planes. While the data plane remains firmly in the Network plane.

Now this is my view and the way I intend to update the text to publish 15, hopefully quite soon. But I need confirmation that we are on the same line, or else debates to reach a consensus.

What do you all think?

Pascal
________________________________
De : [e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>
Envoyé : samedi 29 juillet 2023 15:40
À : Pascal Thubert (pthubert) <[e-mail]<mailto:[e-mail]>>
Objet : New Version Notification for draft-ietf-raw-architecture-14.txt


A new version of I-D, draft-ietf-raw-architecture-14.txt
has been successfully submitted by Pascal Thubert and posted to the
IETF repository.

Name:           draft-ietf-raw-architecture
Revision:       14
Title:          Reliable and Available Wireless Architecture
Document date:  2023-07-29
Group:          raw
Pages:          43
URL:            https://www.ietf.org/archive/id/draft-ietf-raw-architecture-14.txt
Status:         https://datatracker.ietf.org/doc/draft-ietf-raw-architecture/
Html:           https://www.ietf.org/archive/id/draft-ietf-raw-architecture-14.html
Htmlized:       https://datatracker.ietf.org/doc/html/draft-ietf-raw-architecture
Diff:           https://author-tools.ietf.org/iddiff?url2=draft-ietf-raw-architecture-14

Abstract:
   Reliable and Available Wireless (RAW) provides for high reliability
   and availability for IP connectivity across any combination of wired
   and wireless network segments.  The RAW Architecture extends the
   DetNet Architecture and other standard IETF concepts and mechanisms
   to adapt to the specific challenges of the wireless medium, in
   particular intermittently lossy connectivity.  This document defines
   a network control loop that optimizes the use of constrained spectrum
   and energy while maintaining the expected connectivity properties,
   typically reliability and latency.  The loop involves OAM, DetNet
   Controller Plane, and PREOF extensions, and specifically a new
   recovery Function called PAREO and a new Point of Local Repair
   operation, that dynamically selects the DetNet path(s) for the future
   packets to route around local degradations and failures.




The IETF Secretariat


_______________________________________________

detnet mailing list

[e-mail]<mailto:[e-mail]>

https://www.ietf.org/mailman/listinfo/detnet