Re: [Raw] WG Last Call: draft-ietf-raw-architecture-11 - David Black's comments

["Pascal Thubert (pthubert)" <[e-mail]>]

Great! many thanks again and again David 🙂


>  (1) Decide step in 5.2: “Decides which DetNet Path to use for the next packet(s)” sounds like packet-by-packet forwarding decisions.  s/next/future/ might be most of what’s needed.

done

>  (2) The packet going down the stack path in Figure 10 (PSE) reads the packet tag, makes a forwarding decision and retags the packet before forwarding.

Renames fig 10 as "PSE interfaces. The PSE is really the central space in the picture.
I changed the text above to clarify that:
"
    The PSE sits in the DetNet Service sub-Layer of Edge and Relay Nodes. On the
    one hand, it operates on the packet flow, learning the Track and path
    selection information from the packet, possibly making local decision and
    retagging the packet to indicate so. On the other hand, the PSE interacts
    with the lower layers (through triggers and DLEP) and with its peers
    (through iOAM and IOAM) to obtain up-to-date information about its links and
    the quality of the overall Track, respectively, as illustrated in figure 10.
"

> s/the next/future/

I looked up throughout the document to make the changes.

>   This sentence that’s just above that OODA Loop list will likely also need some attention:

RAW Network Plane operations happen at the forwarding time scale on one DetNet flow over a complex path delineated by a Track (see Section 2.3.2<https://datatracker.ietf.org/doc/html/draft-ietf-raw-architecture-13#trk>).




 I see; what about:

" RAW distinguishes the longer time scale at which routes are computed from the
   the shorter time scale where forwarding decisions are made for a limited time
   RAW Network Plane operations happen at a time scale that sits between the
   routing and the forwarding time scales, on one DetNet flow, to select a DetNet
   path within the resources delineated by a Track
"

I have to make the changes in the repo (https://github.com/raw-wg/raw-architecture/commit/123c26d6b059aef09008c1f9a4e5059aef8e17ea) but will push when we're ready.

regards,

Pascal

________________________________
De : Black, David <[e-mail]>
Envoyé : mercredi 12 juillet 2023 16:19
À : Pascal Thubert (pthubert) <[e-mail]>; Black, David <David.Black=[e-mail]>
Cc : Eve Schooler <[e-mail]>; [e-mail] <[e-mail]>; [e-mail] <[e-mail]>; [e-mail] <[e-mail]>; [e-mail] <[e-mail]>; John Scudder <[e-mail]>; Black, David <[e-mail]>
Objet : RE: [Raw] WG Last Call: draft-ietf-raw-architecture-11 - David Black's comments


> > I’d suggest more discussion.  At the very least, the PSE’s ability to select path (including next hop) on a packet-by-packet basis suggests a tight coupling with packet forwarding.

> That's our mismatch. it's not on a "packet-by-packet basis".



Then an editing pass is needed to remove requirements for packet-by-packet functionality.  The two most important items are:

(1) Decide step in 5.2: “Decides which DetNet Path to use for the next packet(s)” sounds like packet-by-packet forwarding decisions.  s/next/future/ might be most of what’s needed.

(2) The packet going down the stack path in Figure 10 (PSE) reads the packet tag, makes a forwarding decision and retags the packet before forwarding.



Additional minor edits will be needed in other places, e.g., in item 3 in OODA Loop list in Section 3.2:

3. A Runtime distributed Path Selection Engine (PSE) that Decides which DetNet Path to use for the next packet(s) that are routed along the Track

s/the next/future/



This sentence that’s just above that OODA Loop list will likely also need some attention:

RAW Network Plane operations happen at the forwarding time scale on one DetNet flow over a complex path delineated by a Track (see Section 2.3.2<https://datatracker.ietf.org/doc/html/draft-ietf-raw-architecture-13#trk>).



>> I’d think that the PSE would be capable of obtaining L2 information directly without an OAM protocol as an intermediary.  I agree that more discussion would be good.

> We're in sync there. The L2 triggers are not part of OAM. They are local upcalls (.ind) from the lower layer and calls (.req/.conf) from the PSE to the lower layer and back.



Good – all uses of OAM should be checked to ensure that L2 info is not required to be routed through OAM.  The good news is that Figure 10 appears to get this right, in contrast to the packet-by-packet concern.



Thanks, --David



From: Pascal Thubert (pthubert) <[e-mail]>
Sent: Wednesday, July 12, 2023 8:04 AM
To: Black, David; Black, David
Cc: Eve Schooler; [e-mail]; [e-mail]; [e-mail]; [e-mail]; John Scudder
Subject: RE: [Raw] WG Last Call: draft-ietf-raw-architecture-11 - David Black's comments



[EXTERNAL EMAIL]

>> I see it this way: when the conditions are stable (no hint from the OAM) the PSE can sleep for a while. So it is not involved in the data path, does not "forward" the packets.

>> When the OAM indicates a change, the PSE tweaks the forwarding rules and an alternate DetNet path (within the track) is now used.



> I’d suggest more discussion.  At the very least, the PSE’s ability to select path (including next hop) on a packet-by-packet basis suggests a tight coupling with packet forwarding.



That's our mismatch. it's not on a "packet-by-packet basis".



There's little point to that since the cost of AOM gathering is much higher than that of forwarding; I see it this way:

RAW is an optimization, it has to perform better than using the maximal DetNet path, that spans the entire Track. => the saving of not flooding the Track with each packet must overwhelm that the cost of the signaling including OAM, and whatever IPPM and what-else. => OAM is order(s) of magnitude rarer than packets.



Since the OODA loop is triggered by the OAM, it is also orders of magnitude rarer. The loop when the PSE acts. After that the packet forwarding operation lives with the most recent PSE conclusion till the PSE changes its mind at the soonest at the next iteration of the control loop.



Net-net: the time granularity of the PCE is much longer than that of the PSE, which in turn is probably orders of magnitude longer than that of per-packet operations.



>> I have a sense that I'm missing something, I hope we can use WG time to dig at IETF 117. I added text in the Observe piece of the OODA definition.



> I’d think that the PSE would be capable of obtaining L2 information directly without an OAM protocol as an intermediary.  I agree that more discussion would be good.



We're in sync there. The L2 triggers are not part of OAM. They are local upcalls (.ind) from the lower layer and calls (.req/.conf) from the PSE to the lower layer and back.



What do you think?



Pascal

________________________________

De : Black, David <[e-mail]<mailto:[e-mail]>>
Envoyé : mardi 11 juillet 2023 21:23
À : Pascal Thubert (pthubert) <[e-mail]<mailto:[e-mail]>>; Black, David <David.Black=[e-mail]<mailto:David.Black=[e-mail]>>
Cc : Eve Schooler <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>; John Scudder <[e-mail]<mailto:[e-mail]>>; Black, David <[e-mail]<mailto:[e-mail]>>
Objet : RE: [Raw] WG Last Call: draft-ietf-raw-architecture-11 - David Black's comments



Hi Pascal,



Thanks for doing what was possible in the time available – the result is a visible improvement.  Summarizing here …



>> Sounds good - the crucial change is to get the PCE (as well as the entire detnet controller plane) out of the RAW OODA loop, starting with item 2 at the end of Section 3.2 and a revision of Figure 8.

[…]

> what about  (removing PCE, replaced by a DetNet Routing CPF + a local asynchronous CPF that can be queried inside the loop).

> Suggestion to introduce an acronym for the in-loop deported piece, e.g., aCPF, to be used in fig 8.



Result looks good to me – significant improvement, thank you.



>> In support of this, it might help to swap the order of section 4 and section 5, so that the PSE-centric control loop is explained (revision of current section 5)

>> before discussion of how it fits into the layers of the detnet architecture (section 4).

> Top - down vs bottom-up again. I guess there are people in both camps.



Ok, I can live with the current structure.



>> Section 4 also needs work, as it places the PSE in the detnet service sub-layer with the premise that it uses the detnet forwarding sub-layer with minor changes.

>> That doesn’t make sense – to phrase this as a question: Given that the PSE is selecting Path within Track on a packet-by-packet basis, how and why is that not packet forwarding functionality

> I see it this way: when the conditions are stable (no hint from the OAM) the PSE can sleep for a while. So it is not involved in the data path, does not "forward" the packets.

> When the OAM indicates a change, the PSE tweaks the forwarding rules and an alternate DetNet path (within the track) is now used.



I’d suggest more discussion.  At the very least, the PSE’s ability to select path (including next hop) on a packet-by-packet basis suggests a tight coupling with packet forwarding.



>> Sections 2.6 and 5.3 along with item 1 at the end of section 3.2 discuss OAM in terms of measurement protocols and packets.

>> In contrast, Figure 10 in section 5.5 shows an additional PSE interface to L2 which appears to be outside the scope of OAM as used elsewhere in this draft.

>> Perhaps broadening “Observe” to be more than OAM (which implies protocols and packets) would help.

> I'm used to the term L2-triggers and then BFD/ DLEP. Is that your intent? I added text in the OAM section to indicate that the information transported by RAW OAM may be obtained that way.

> I have a sense that I'm missing something, I hope we can use WG time to dig at IETF 117. I added text in the Observe piece of the OODA definition.



I’d think that the PSE would be capable of obtaining L2 information directly without an OAM protocol as an intermediary.  I agree that more discussion would be good.



Thanks, --David



From: Pascal Thubert (pthubert) <[e-mail]<mailto:[e-mail]>>
Sent: Monday, July 10, 2023 11:44 AM
To: Black, David
Cc: Eve Schooler; [e-mail]<mailto:[e-mail]>; [e-mail]<mailto:[e-mail]>; [e-mail]<mailto:[e-mail]>; [e-mail]<mailto:[e-mail]>; John Scudder; Black, David
Subject: RE: [Raw] WG Last Call: draft-ietf-raw-architecture-11 - David Black's comments



[EXTERNAL EMAIL]

Hello David:



With the cutoff today, I handled what I could starting with the removal of the term PCE and the clarification on the orient piece in the loop.

The HTML version of the chanegs available at:

https://www.ietf.org/archive/id/draft-ietf-raw-architecture-13.html [ietf.org]<https://urldefense.com/v3/__https:/www.ietf.org/archive/id/draft-ietf-raw-architecture-13.html__;!!LpKI!hy8jPLVfQhdT9bHWMaJHnHF5Gyzk2TJg_nfOBWav8vNVecXfLVOC2wM86frmO1THCbVfQ0h23wMisgTz$>

A diff from the previous version is available at:
https://author-tools.ietf.org/iddiff?url2=draft-ietf-raw-architecture-13 [author-tools.ietf.org]<https://urldefense.com/v3/__https:/author-tools.ietf.org/iddiff?url2=draft-ietf-raw-architecture-13__;!!LpKI!hy8jPLVfQhdT9bHWMaJHnHF5Gyzk2TJg_nfOBWav8vNVecXfLVOC2wM86frmO1THCbVfQ0h232mqntMb$>



In more details:





Sounds good - the crucial change is to get the PCE (as well as the entire detnet controller plane) out of the RAW OODA loop, starting with item 2 at the end of Section 3.2 and a revision of Figure 8.



I guess that our mismatch comes from the fact that I failed to describe the Orient thingy as something inloop that consumes wisdom that is generated offloop. to express that, I need to separate the local entity that talks to the DetNet Routing CPF (ex PCE) from the DetNet Routing CPF itself.





BEFORE Item 2 says

   2.  Optional Controller plane elements that report the links

       statistics to be used to compute and install the Tracks, and

       provides meta data to Orient the routing decision, e.g., by a PCE

       in a centralized controller



what about  (removing PCE, replaced by a DetNet Routing CPF + a local asynchronous CPF that can be queried inside the loop). Suggestion to introduce an acronym for the in-loop deported piece, e.g., aCPF, to be used in fig 8.



AFTER (proposed)

  2.  An optional asynchronous Controller Plane Function (aCPF) that reports

      data and information such as link statistics to be used asynchronously

      by the DetNet Routing CPF to compute, install, and maintain the Tracks,

      e.g., by generating knowledge and wisdom such as a trained model for

      link quality prediction, which in turn can be used by the aCPF to

      Orient the Path selection by the PSE within the RAW OODA loop.



Fig 8 becomes:



     +-------> Orient (aCPF) -------+

     |        reflex actions        |

     |       pre-trained model      |

     |             ...              |

     |                              v

 Observe (OAM)                Decide (PSE)

     ^                              |

     |                              |

     |                              |

     +-------- Act (PAREO) <--------+

                At DetNet

             Service sub-layer







In support of this, it might help to swap the order of section 4 and section 5, so that the PSE-centric control loop is explained (revision of current section 5) before discussion of how it fits into the layers of the detnet architecture (section 4).



Top - down vs bottom-up again. I guess there are prople in both camps.





  As noted below, I also think that the PSE (or at least its path selection functionality) belongs in the detnet forwarding sub-layer, not just the service sub-layer (although I think I see some rationale for a portion of the PSE being in the service sub-layer).



+



Section 4 also needs work, as it places the PSE in the detnet service sub-layer with the premise that it uses the detnet forwarding sub-layer with minor changes.  That doesn’t make sense – to phrase this as a question: Given that the PSE is selecting Path within Track on a packet-by-packet basis, how and why is that not packet forwarding functionality





I see it this way: when the conditions are stable (no hint from the OAM) the PSE can sleep for a while. So it is not involved in the data path, does not "forward" the packets.

When the OAM indicates a change, the PSE tweaks the forwarding rules and an alternate DetNet path (within the track) is now used.



My mind image:

  *   The asynchronous CPF provides and maintains a RIB (with some RAW metadata) to represent the Track, and that RIB is pushed to FIB as a starting point.
  *   The RIB is associated to the DetNet flow(s) that are routed through the Track
  *   The PSE controls the FIB at a highly reactive pace, e.g., by a variation of the ususal load balancing hashes you find in current fwd silicon.
  *   FIB tweaking is not done upon each packet, but hopefully more at the speed of a BFD than at that of an IGP.
  *   The snapshot of the tweaks in the FIBs by the PSEs inside the Track define the DetNet path in use at that time, ever fluctuating.





As noted below, I’d suggest an initial rewrite that never mentions the PCE.  A plausible approach would be to make a raw-enhanced DetNet controller plane responsible for producing and updating Tracks.



done



Sections 2.6 and 5.3 along with item 1 at the end of section 3.2 discuss OAM in terms of measurement protocols and packets.  In contrast, Figure 10 in section 5.5 shows an additional PSE interface to L2 which appears to be outside the scope of OAM as used elsewhere in this draft.   Perhaps broadening “Observe” to be more than OAM (which implies protocols and packets) would help.



I'm used to the term L2-triggers and then BFD/ DLEP. Is that your intent? I added text in the OAM section to indicate that the information transported by RAW OAM may be obtained that way. I have a sense that I'm missing something, I hope we can use WG time to dig at IETF 117. I added text in the Observe piece of teh OODA defintion





regards,



Pascal



________________________________

De : Black, David <David.Black=[e-mail]<mailto:David.Black=[e-mail]>>
Envoyé : vendredi 7 juillet 2023 18:59
À : Pascal Thubert (pthubert) <[e-mail]<mailto:[e-mail]>>
Cc : Eve Schooler <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>; John Scudder <[e-mail]<mailto:[e-mail]>>; Black, David <[e-mail]<mailto:[e-mail]>>
Objet : RE: [Raw] WG Last Call: draft-ietf-raw-architecture-11 - David Black's comments



Hi Pascal,



> Many thanks ! I’ll need to clarify that the ooda loop of effectively the one you describe. There is no other.

> It certainly never goes through the PCE, the whole point of RAW is that the communication with the PCE is too slow and expensive so we export functionality in the routers.



Sounds good - the crucial change is to get the PCE (as well as the entire detnet controller plane) out of the RAW OODA loop, starting with item 2 at the end of Section 3.2 and a revision of Figure 8.



In support of this, it might help to swap the order of section 4 and section 5, so that the PSE-centric control loop is explained (revision of current section 5) before discussion of how it fits into the layers of the detnet architecture (section 4).   As noted below, I also think that the PSE (or at least its path selection functionality) belongs in the detnet forwarding sub-layer, not just the service sub-layer (although I think I see some rationale for a portion of the PSE being in the service sub-layer).



> The PCE is expected to build some hints based on its history and predictions of the links state.

> This is exported to the network with the initial Track structure and possibly updated overtime as a background task.



As noted below, I’d suggest an initial rewrite that never mentions the PCE.  A plausible approach would be to make a raw-enhanced DetNet controller plane responsible for producing and updating Tracks.



> This knowledge/wisdom level set of hints constitutes the Orient piece which is the base for the reroute decision. I’m game for suggestions on which text should be reworded that caused this confusion.

Sections 2.6 and 5.3 along with item 1 at the end of section 3.2 discuss OAM in terms of measurement protocols and packets.  In contrast, Figure 10 in section 5.5 shows an additional PSE interface to L2 which appears to be outside the scope of OAM as used elsewhere in this draft.   Perhaps broadening “Observe” to be more than OAM (which implies protocols and packets) would help.



Thanks, --David



From: Pascal Thubert (pthubert) <pthubert=[e-mail]<mailto:pthubert=[e-mail]>>
Sent: Friday, July 7, 2023 5:54 AM
To: Black, David
Cc: Eve Schooler; [e-mail]<mailto:[e-mail]>; [e-mail]<mailto:[e-mail]>; [e-mail]<mailto:[e-mail]>; [e-mail]<mailto:[e-mail]>; John Scudder; Black, David
Subject: Re: [Raw] WG Last Call: draft-ietf-raw-architecture-11 - David Black's comments



[EXTERNAL EMAIL]

Hello David



Many thanks ! I’ll need to clarify that the ooda loop of effectively the one you describe. There is no other.



It certainly never goes through the PCE, the whole point of RAW is that the communication with the PCE is too slow and expensive so we export functionality in the routers.



The PCE is expected to build some hints based on its history and predictions of the links state. This is exported to the network with the initial Track structure and possibly updated overtime as a background task.



This knowledge/wisdom level set of hints constitutes the Orient piece which is the base for the reroute decision. I’m game for suggestions on which text should be reworded that caused this confusion.

Again many thanks !



Pascal



Le 7 juil. 2023 à 01:32, Black, David <David.Black=[e-mail]<mailto:David.Black=[e-mail]>> a écrit :



Lou and Janos (detnet chairs) asked me to take a look at this draft – these comments are posted as an individual, not in my role as a Technical Advisor to the detnet WG.  I actually looked at -12, which has some minor changes from -11.



Overall, there are valuable and important concepts in the architecture.  The primary functional additions of raw over detnet that I noticed are the concepts of multi-Path Tracks and the PSE (Path Selection Engine) that can select Path within Track on a packet-by-packet basis, with the control path revising Tracks on a longer timescale.  That’s more dynamic than the static path provisioning that is envisioned by detnet, e.g., in the discussion of explicit routes in section 3.2.2 of RFC 8655 (Deterministic Networking Architecture).  In addition, the ability of the PSE to monitor, react to, and possibly modify wireless link characteristics (e.g., monitor level of interference and signal loss, modify ARQ & FEC parameters) results in a significantly more dynamic version of detnet PREOF, although I’m not convinced that introducing the new PAREO term and acronym is a productive way of describing that.



Unfortunately, the OODA Raw Control Loop in Section 5 doesn’t work well with this, as exemplified by Figure 8 showing only one control loop that runs through a centralized PCE (Path Computation Engine).  There is a far more important control loop that runs between the PSE and the network, based on far more information than RAW OAM (in Figure 8) provides.  While the draft does characterize the PCE as an example of a control plane structure, it doesn’t have much to say about any others.  That’s too easy to read as reinventing the detnet control plane, which would not be a good idea.



This draft would be considerably stronger if it jettisoned the PCE and its Figure 8 coarse control plane feedback loop to focus on the PSE and its fine-grain path selection feedback loop.  In particular, there’s a lot of good content in Figure 10 that deserves much more explanation than currently provided in sections 5.5 and 5.6.   This sort of approach would involve extending the detnet architecture to introduce Tracks as the entities via which traffic is explicitly “routed” by the detnet/raw control plane.  Section 4 also needs work, as it places the PSE in the detnet service sub-layer with the premise that it uses the detnet forwarding sub-layer with minor changes.  That doesn’t make sense – to phrase this as a question: Given that the PSE is selecting Path within Track on a packet-by-packet basis, how and why is that not packet forwarding functionality?



This is intended as constructive criticism for discussion – as I’ve not been following raw closely, some portion of this is likely to be incorrect in some fashion.



Thanks, --David



From: detnet <[e-mail]<mailto:[e-mail]>> On Behalf Of Eve Schooler
Sent: Friday, June 23, 2023 1:11 PM
To: [e-mail]<mailto:[e-mail]>; [e-mail]<mailto:[e-mail]>
Cc: [e-mail]<mailto:[e-mail]>; [e-mail]<mailto:[e-mail]>; John Scudder
Subject: [Detnet] WG Last Call: draft-ietf-raw-architecture-11



[EXTERNAL EMAIL]

All,



This starts working group last call on draft-ietf-raw-architecture-11

https://datatracker.ietf.org/doc/draft-ietf-raw-architecture/ [datatracker.ietf.org]<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-ietf-detnet-pof/__;!!LpKI!gwAT9SygbUUZ8R5o9vGdxPNzKSMI44j13KrTFoVFxilR3iN3ZAcjdFYU1uqKEJwOtjWHwje5_7am6fHX4_EpuxQ$>



The working group last call ends on Friday, July 21st, the Friday before IETF 117.

Please send your comments to the working group mailing list.



Two IPR disclosures have been made for this document thus far (in tandem with WGLC, in a separate e-mail, we will solicit all co-authors/contributors for any additional disclosures).

All comments, e.g., "I've reviewed this document and believe it is ready for publication", are welcome.

This is useful and important, even from authors.

In preparation for the transition of the RAW WG to roll back into the DetNet WG, we are issuing this as a joint WGLC to both WGs.

Thank you,

Eve (RAW Co-Chair & doc Shepherd)

--
RAW mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/raw [ietf.org]<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/raw__;!!LpKI!neGbePuwOEQnPaz0Blsdxcf55SE0K9zSdcQlv_ao9x35SDrFjI-PZqODRmM_f48qGdZncuOvti7M3F-Ju1jXd-MrETxGwjdT$>