Re: [Detnet] Last Call: <draft-ietf-detnet-yang-18.txt> (Deterministic Networking (DetNet) YANG Model) to Proposed Standard

[Florian Kauer <[e-mail]>]

Hi Don,

See [FK2] inline.

Thanks,
Florian

On 23.12.23 00:09, Don Fedyk wrote:
> On 19.12.23 22:19, Don Fedyk wrote:
>>       typedef ipsec-spi {
>> @@ -1113,6 +1113,79 @@
>>           description
>>             "The Application flow type choices.";
>>           container tsn-app-flow {
>> +---
>> +fk: In case of a tsn-app-flow, how does the actual encapsulation take place?
>>
>> [Don] This is a YANG specification that satisfies the operations in 
>> DetNet RFCs Listed in the references.  You can think of it as a 
>> configuration for an SDN controller that specifies 
>> encapsulation/decapsulation on various nodes/devices along the data 
>> path. It is not specifying "how" it is specifying "what" aggregation, disaggregation functions are applied along the data path and which technology is used.
>>
>> Also reading through your comments I think there is mis understanding.  
>>
>> DetNet is not an any to any encapsulation. It is specifically what is covered in the DetNet architecture to support DetNet encapsulation flows for aggregation or replication.  We have specific type of encapsulations and behaviors that must be specified on ingress. The reverse operations don’t need as much specification because you are removing an outer encapsulation and exposing another header. I will try to see if this point is spelled out in the ID. 
> 
> [Don2] There is an egress specification operation for every disaggregation but sometimes is just remove the layer and link to the next layer. 
> 
> [FK]: Yes, I agree that the YANG specification document is not the correct place to specify how to implement the encapsulations in detail (for that we have for example RFC 9024 and others). The reason why I raised this in the context of the YANG model is that it makes it obvious how many different options of combining TSN/IP/MPLS/... there are. And there are options that are both reasonable and permitted by the YANG model, but in my opinion not unambiguously covered by any DetNet specification (including TSN over IP over TSN without MPLS).
> 
> Again, that is not a weakness of the YANG model, but I am just worried that if we come to an agreement about all combinations we might find something that needs to be added to the YANG model to properly cover that case.
> 
> 
>>   
>> +
>> +For reference, lets look at a few cases and how I have understood them:
>> +
>> +1. ip-app-flow, nothing MPLS related in service or forwarding sub-layers:
>> +   -> Just transmit unmodified packet with new L2 header.
>> ++---------------+      +----------+
>> +| Payload       |      | Payload  |
>> ++---------------+      +----------+
>> +|   UDP         |      |   UDP    |
>> ++---------------+  ->  +----------+
>> +|   IP          |      |   IP     |
>> ++---------------+      +----------+
>> +| L2 (original) |      | L2 (new) |
>> ++---------------+      +----------+
>>
>> [Don]  Is this is a DetNet Appflow?  Please see DetNet Architecture RFC 8655 Figure 3. Also Data Plane Framework RFC8938.
>> We are only concerned with DetNet.
>>
>> The drawing is a bit ambiguous for DetNet.  
>> The Payload could be the TSN flow then the IP is a DetNet sub-network and you have a DetNet packet. The L2 is irrelevant.  
>> If the whole stack Payload/UDP/IP is a TSN payload then TSN is the subnetwork layer. It would be a TSN application frame in ingress and the SAME frame on egress.    
>>
>> The YANG model aggregates (encapsulates) and disaggregates (decapsulates) DetNet packet but it does not change/translate a TSN L2.  The encapsulation layers for DetNet are IP/MPLS not Ethernet L2.   (we don’t specify egress L2 headers we can match L2 ingress headers at the Application interface).  
> 
> [FK] Yes, sorry for the confusion! Of course, explicitly changing the L2 header not in the scope of DetNet (and thus the YANG model). I just added the L2 to indicate that the packet can move from one TSN domain to another, while everything L3 related in the packet stays constant.
> 
> In this example, the payload was meant to be the actual application payload (like MQTT) and not an encapsulated TSN L2 (that is covered by case 4).
> 
>>       
>>
>> +
>> +2. ip-app-flow, but MPLS label is added:
>> +   -> Squeeze in MPLS header.
>> ++---------------+      +----------+
>> +| Payload       |      | Payload  |
>> ++---------------+      +----------+
>> +|   UDP         |      |   UDP    |
>> ++---------------+  ->  +----------+
>> +|   IP          |      |   IP     |
>> ++---------------+      +----------+
>> +| L2 (original) |      |   MPLS   |
>> ++---------------+      +----------+
>> +                       | L2 (new) |
>> +                       +----------+
>> [Don] Again if this is an IP App flow you have this. L2 Not relevant.
>>
>> ++---------------+      +----------+
>> +| Payload       |      | Payload  |
>> ++---------------+      +----------+
>> +|   UDP         |      |   UDP    |
>> ++---------------+  ->  +----------+
>> +|   IP          |      |   IP     |
>> ++---------------+      +----------+
>>                         |   MPLS   |
>>                         +----------+
> 
> [FK] Yes, see my comment above regarding L2.
>>
>> [Don] Or if it is TSN app flow you have this: 
>>
>> ++---------------+      +----------+
>> +| Payload       |      | Original |
>> ++---------------+      | L2 Frame |
>> +|   UDP         |      |          |
>> ++---------------+  ->  |          |
>> +|   IP          |      |          |
>> ++---------------+      |          |
>> +| L2 (original) |      |          |
>> ++---------------+      +---------------------------+
>> +                       | DetNet Service Sublayer|  |
>> +                       +---------------------------+
>> +                       | DetNet Forwarding Sublayer|  |
>> +                       +---------------------------+
> 
> 
> [FK] In my classification, TSN app flow is case 4. But anyway, the question I am having for case 4 is how what you denote as "DetNet Service Sublayer" and "DetNet Forwarding Sublayer" specifically looks like in the TSN app flow case (see below).
>                         
>>  
>>
>> +
>> +3. mpls-app-flow:
>> +   -> Only optionally modify MPLS header
>> ++-----------------+      +------------+
>> +| Payload         |      | Payload    |
>> ++-----------------+      +------------+
>> +|   UDP           |      |   UDP      |
>> ++-----------------+  ->  +------------+
>> +|   IP            |      |   IP       |
>> ++-----------------+      +------------+
>> +| MPLS (original) |      | MPLS (new) |
>> ++-----------------+      +------------+
>> +| L2 (original)   |      | L2 (new)   |
>> ++-----------------+      +------------+
>>
>>
>> [Don] If it is an APP flow it gets encapsulated. We can encapsulate service sub-layer as application layer for example.  This header format could also be a service sub layer and we show examples of changing the labels as a serve sub layer and forwarding layer.   
> 
> [FK] Sorry that was depicted ambiguously since I merged all MPLS headers into one. I actually meant:
> 
> +-----------------+      +-----------------+
> | Payload         |      | Payload         |
> +-----------------+      +-----------------+
> |   UDP           |      |   UDP           |
> +-----------------+  ->  +-----------------+
> |   IP            |      |   IP            |
> +-----------------+      +-----------------+
> | MPLS (original) |      | MPLS (original) |
> +-----------------+      +-----------------+
>                          | MPLS (new)      |
>                          +-----------------+
> 
> Is that better this way or did I misinterpret your comment?
> [Don2] This is supported by the model.  The New MPLS can be a service sub-layer or a forwarding sub-layer. 


[FK2] Yes, agreed.



> 
>> +
>> +4. tsn-app-flow:
>> +   -> In order to be able to reconstruct the full original L2 packet, we need
>> +      to retain the L2 header. So we need something like:
>> +
>> ++---------------+      +-----------------+
>> +| Payload       |      |    Payload      |
>> ++---------------+  ->  +-----------------+
>> +| L2 (Original) |      |  L2 (Original)  |
>> ++---------------+      +-----------------+
>> +                       |      ???        |
>> +                       +-----------------+
>> +                       |   L2 (new)      |
>> +                       +-----------------+
>>
>> [Don] This is specified in IEEE TSN.    DetNet uses IP/MPLS as the service sub layer. 
> 
> [FK] ??? here is a placeholder for UDP/IP/MPLS headers, sorry that this was not clear.
> In RFC 9024 Figure 3 the following two options are depicted (translated into the notation of my mail):
> 
> +---------------+      +-----------------+
> | Payload       |      |    Payload      |
> +---------------+  ->  +-----------------+
> | L2 (Original) |      |  L2 (Original)  |
> +---------------+      +-----------------+
>                        |      MPLS       |
>                        +-----------------+
>                        |  L2 (new)       |
>                        +-----------------+
> [Don2] The L2 (new) is not in the YANG but you have a L2 new if you look on the wire.  You are using MPLS as the service sub-layer and forwarding sub-layer. L2 is hop by hop. That’s on purpose we do not specify TSN Ethernet encapsulation as a service/forwarding sub-layer in DetNet YANG. There are TSN config parameters that are specified in the IEEE. A box that supported TSN configuration could specify those parameters. See RFC 9023.  If that box supported that our YANG then you would have a box that did both.  So as far as RFC 9024 figure 3 the L2 case and the TSN case are the same for us.   
> See management section of https://datatracker.ietf.org/doc/rfc9023/ Section 5. 


[FK2] Yes, I agree that "L2 (new)" is not (and should not) be covered by the DetNet YANG. My current approach for that (and I hope that is how it is supposed to be) is to link the DetNet YANG and the TSN YANG just via the forwarding sub-layer's outgoing-interface (see for example https://mailarchive.ietf.org/arch/msg/detnet/paairBoBOB2ETRP-yKadT5Lvqn0/ ).


> 
> +---------------+      +-----------------+
> | Payload       |      |    Payload      |
> +---------------+  ->  +-----------------+
> | L2 (Original) |      |  L2 (Original)  |
> +---------------+      +-----------------+
>                        |      MPLS       |
>                        +-----------------+
>                        |      UDP        |
>                        +-----------------+
>                        |      IP         |
>                        +-----------------+
>                        |   L2 (new)      |
>                        +-----------------+
> [Don2] This stack is supported. (same comment about L2 though)
> 
> (L2 can either be TSN or something else, therefore there are three options in RFC 9024 Figure 3.)  
> 
> I would expect that at least these two cases could be differentiated via YANG somehow (either in the draft-ietf-detnet-yang or in another model). What do you think?
> 
> [Don2] As I recall the service sub-layer and forwarding sub-layer were heavily discussed in the DetNet WG as being IP and/or MPLS. The DetNet architecture shows the layer clearly - we support IP and MPLS - IP tunneling was not in scope - there were no DetNet ID that specified tunneling at least when the effort was undertaken.  DetNet works with TSN both as an application and as a TSN under lay application/service/forwarding layer. In DetNet we only specify the IP/and MPLS for service and forwarding sub-layers because that is DetNet scope. We detailed many combinations with the cases so that people could see the combinations and the WG wanted it captured in the ID.  Our PowerPoint became the SVG and config examples in the appendix.    


[FK2] There is at least GRE mentioned as IP tunneling protocol in Figure 4 of RFC 8655, but I am fine with that if configuring GRE, VXLAN, L2TP3 or similar is not in scope of the YANG (or even DetNet in general) at the moment.


> But there are certainly some more possibilities like:
> 
> +---------------+      +-----------------+
> | Payload       |      |    Payload      |
> +---------------+  ->  +-----------------+
> | L2 (Original) |      |  L2 (Original)  |
> +---------------+      +-----------------+
>                        |      UDP        |
>                        +-----------------+
>                        |      IP         |
>                        +-----------------+
>                        |   L2 (new)      |
>                        +-----------------+
> 
> But I have not seen that being specified anywhere yet, but it would be one option to realize the "TSN over IP over TSN without MPLS" case.
> 
> [Don2] Again nothing in our model to prevent this but the if the L2 is TSN then same comments apply. 


[FK2] I am sorry, but that is what I still don't understand. If you say that all three stack options
- L2 over MPLS
- L2 over MPLS over UDP over IP
- L2 over UDP over IP
are supported by the model, how does the node know which one of these three options (or any potential further ones) it should apply?


>> +I know there are several different ways to realize this, including 
>> +just transmitting the L2 packet over MPLS (apparently not IETF
>> +standardized?) or several others like L2TPv3 (RFC3931) or VXLAN (RFC
>> +7348) if no MPLS labels are to be added (like in case 1 above).
>>
>> +
>> +I would expect the YANG model (either this or a linked one) to 
>> +somehow specify which of these options shall actually be applied. 
>> +Otherwise, two DetNet routers might not be able to communicate with 
>> +each other, because one might use L2TPv3 and the other one VXLAN. And 
>> +also for example in the case of VXLAN, we would need the option to specify a VXLAN Network Identifier.
>>
>> [Don] The scope of the YANG does what DetNet documents outline. There is no VXLAN or L2TP3 specified in the DetNet when this applied. 
>> The DetNet YANG works by specified the whole path and applying the appropriate encapsulation format at the appropriate places. Yes the encapsulation specified on each device must match just the way SDN controllers specify packet forwarding in several technologies.
> 
> [FK] You have convinced me that it does not make sense to hold back the YANG model until someday someone might come along who finds a use case for VXLAN or L2TP3 in DetNet. And there is still the option to specify an extension to the YANG model in that particular spec if needed. 
> 
> 
>>               uses data-flow-spec;
>>               choice application-type {
>>                 description
>>                   "This is the application type choices.";
>> +---
>> +fk: This part is quite confusing to me, especially since it does not directly mirror the ingress since.
>> +I guess one reason for that is there could be multiple interfaces that could be selected based on the packet headers?
>> +But for the same reason you could also have multiple ingress interfaces.
>> [Don]  For Egress the options are different because we get here by decapsulating the service sub-layer. We have either some type of TSN frame (Ethernet) or IP/MPLS but we are basically only associating the flow with an interface at the end of a service sub-layer.  
> 
> [FK] So the TSN case is clear to me because in both cases you have zero or one interface (app_flow->ingress->interface and app_flow->egress->ethernet->interface).
> [Don2] You can have many interfaces. (see below).


[FK2] But only if you specify multiple app flows, not for a single app flow, right?


> 
> But in the case of IP or MPLS we have still zero or one interface for ingress (app_flow->ingress->interface), but since there can be multiple different next hops for A SINGLE app-flow, we can have multiple interfaces for egress (app_flow->egress->ip-mpls->next_hop*->outgoing_interface).
> 
> That means the current model allows to split up A SINGLE app-flow like this:
> 
>                                                  +---------------+
>                                             ---> | Destination 1 |
> +----------+    +---------+    +--------+  /     +---------------+
> | Source 1 | -> | Ingress | -> | Egress |--
> +----------+    +---------+    +--------+  \     +---------------+
>                                             ---> | Destination 2 |
>                                                  +---------------+
> 
> But not to merge one like this:
> 
> +----------+
> | Source 1 | --
> +----------+   \    +---------+    +--------+    +---------------+
>                 --> | Ingress | -> | Egress | -> | Destination 1 |
> +----------+   /    +---------+    +--------+    +---------------+
> | Source 2 | --
> +----------+
> 
> [Don2]  You can aggregate incoming stream at all layers app/service/forwarding we have examples of that in the appendixes. 
> 
> So either we have exactly one source and one destination for an app flow. 
> [Don2] I may be missing your point several examples show multiple interfaces. Case A1 in the appendix shows aggregation on multiple interfaces.


[FK2] Do you really meant A1? It has eth0 as ingress for both app-flows.

  
> Then why do we have multiple next hops for egress?
> [Don2]  You are talking IP?  If the application is IP it is forwarded as IP natively, so we support IP forwarding if the layer is IP.


[FK2] It is the same for both IP and MPLS in the YANG model. For both there is the option to have a next-hop-list with multiple entries:

     |        +--rw (application-type)?
     |           +--:(ethernet)
     |           |  +--rw ethernet
     |           |     +--rw interface?   if:interface-ref
     |           +--:(ip-mpls)
     |              +--rw ip-mpls
     |                 +--rw (next-hop-options)?
     |                    +--:(simple-next-hop)
     |                    |  +--rw outgoing-interface?
     |                    |  |       if:interface-ref
     |                    |  +--rw (flow-type)?
     |                    |     +--:(ip)
     |                    |     |  +--rw next-hop-address?
     |                    |     |          inet:ip-address-no-zone
     |                    |     +--:(mpls)
     |                    |        +--rw mpls-label-stack
     |                    |           +--rw entry* [id]
     |                    |              +--rw id             uint8
     |                    |              +--rw label?
     |                    |              |       rt-types:mpls-label
     |                    |              +--rw ttl?           uint8
     |                    |              +--rw traffic-class? uint8
     |                    +--:(next-hop-list)
     |                       +--rw next-hop* [hop-index]
     |                          +--rw hop-index               uint8
     |                          +--rw outgoing-interface?
     |                          |       if:interface-ref
     |                          +--rw (flow-type)?
     |                             +--:(ip)
     |                             |  +--rw next-hop-address?
     |                             |          inet:ip-address-no-
     |                             |            zone
     |                             +--:(mpls)
     |                                +--rw mpls-label-stack
     |                                   +--rw entry* [id]
     |                                      +--rw id
     |                                      |       uint8
     |                                      +--rw label?
     |                                      |       rt-types:mpls-
     |                                      |         label
     |                                      +--rw ttl?
     |                                      |       uint8
     |                                      +--rw traffic-class?
     |                                              uint8



> Or we allow multiple sources and destinations per app flow, then we need to support multiple interfaces at ingress, too. Or do I overlook something?
> [Don2] You can have App1 on interface eth0 and App 2 on interface eth1 etc - there are examples e.g. Figure 4. For IP you have wild card IP prefixes as aggregation or multiple Application definitions.  There is an ingress config and egress config per flow direction. 


[FK2] Yes, there are several examples how to aggregate MULTIPLE app flows into one flow in the service or forwarding sub-layer. And by using wild card IP prefixes I can also have a single app flow for multiple source, but only if they come in via the same interface, not if they come in via different interfaces. In the latter case, I only have the fallback option to use multiple app flow definitions each with a different interface. That would be fine, but for egress it is sufficient to use a single app flow definition and still specify multiple interfaces via the next-hop-list. And that is what is confusing me.


>  
> 
>>     /detnet/service/sub-layer/incoming/app-flow: This links applications
>>     to services.
>> @@ -2608,6 +2713,14 @@
>>          |     +--rw operation?            operation
>>          |     +--rw incoming
>>          |     |  +--rw (incoming)?
>> +---
>> +fk: One thing I raised before, but it either my comment got lost or i missed the response:
>> +Why is there no incoming forwarding sub-layer for the service sub-layer?
>> +An app-flow has outgoing service-sub-layer and incoming 
>> +service-sub-layer, a forwarding sub-layer has incoming 
>> +service-sub-layer and outgoing service-sub-layer, a service sub-layer 
>> +has incoming app-flow and outgoing app-flow and it has outgoing forwarding sub-layer, but NO incoming forwarding sub-layer.
>> [Don] Sorry I missed your comment before.   
>> The operation of Service-sub-layer to forwarding sub-layer is an association of an encapsulation operation. The reverse direction is simply a removal of the forwarding layer and a match in the now exposed service Sub-layer.  Therefore, you do not need the same level of description.
> 
> [FK] Yes, but what do I specify as service->sub-layer->incoming for an egress node? Just keep it empty? Especially if no MPLS is involved.
> 
> [Don2] Please look at some of examples of aggregation/ disaggregation. Note at egress we call it outgoing service sub-layer service-disaggregation.   Remember there are 2 configs for one each end for a unidirectional flow. Also depending on the example, you may have app->service->forwarding or just app-service or just service->forwarding or just forwarding etc.  It depends on what operations you need.  
> 
> For a disaggregation example with 2 interfaces see Figure 23.  There is an aggregated MPLS forwarding sub-layer afl-1 when the label is matched it pops the label and trys matching fsl-1 and fsl-2 --  if fsl-1 is the matched label swap and forward the next label out one interface.  If fsl- 2 matches it swap and forwards out a different interface.  And this logic is used for other disaggregation at other layers. This case is Forwarding sub-layer outgoing forwarding-disaggregation.  


[FK2] I meant an example for a final egress node (like the ones labeled Egress 1 or 2 in the pictures) and if I do not overlook something, there is none provided. But anyway, I would expect derived from asl-1 in example C-3 that the service sub-layer for Egress 1 in example A-1 should look like this:

"service": {
  "sub-layer": [
    {
      "name": "ssl-1",
      "service-rank": 10,
      "traffic-profile": "pf-2",
      "service-protection": {
        "protection": "none",
        "sequence-number-length": "long-sn"
      },
      "operation": "termination",
      "incoming": {
        "service-id": {
          "mpls-label-stack": {
            "entry": [
              {
                "id": 0,
                "label": "102"
              }
            ]
          }
        }
      },
      "outgoing": {
        "app-flow": {
          "flow": [
            "app-0",
            "app-1"
          ]
        }
      }
    }
  ]
}

So in a case where there is no MPLS S-label to be removed (like in the L2 over UDP over IP case mentioned above), it would just look like this:


"service": {
  "sub-layer": [
    {
      "name": "ssl-1",
      "service-rank": 10,
      "traffic-profile": "pf-2",
      "service-protection": {
        "protection": "none",
        "sequence-number-length": "long-sn"
      },
      "operation": "termination",
      "incoming": {
      },
      "outgoing": {
        "app-flow": {
          "flow": [
            "app-0",
            "app-1"
          ]
        }
      }
    }
  ]
}

So the incoming container would be empty. That might be as it is supposed to be, but it just looks odd to have an empty incoming container, especially since - as mentioned before - all other pairs of sub-layers have the option to specify the link between the layers on both sides (incoming and outgoing), but not this specific case.