Re: [Detnet] IPv6 encapsulation in dataplane doc

Lou Berger <lberger@labn.net> Mon, 12 February 2018 20:46 UTC

Return-Path: <lberger@labn.net>
X-Original-To: detnet@ietfa.amsl.com
Delivered-To: detnet@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 38BD6127010 for <detnet@ietfa.amsl.com>; Mon, 12 Feb 2018 12:46:04 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.901
X-Spam-Level:
X-Spam-Status: No, score=-1.901 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (768-bit key) header.d=labn.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HgDhnNm3qNlz for <detnet@ietfa.amsl.com>; Mon, 12 Feb 2018 12:46:00 -0800 (PST)
Received: from gproxy2-pub.mail.unifiedlayer.com (gproxy2-pub.mail.unifiedlayer.com [69.89.18.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 53C63126FDC for <detnet@ietf.org>; Mon, 12 Feb 2018 12:46:00 -0800 (PST)
Received: from cmgw2 (unknown [10.0.90.83]) by gproxy2.mail.unifiedlayer.com (Postfix) with ESMTP id C5B9C1E0B80 for <detnet@ietf.org>; Mon, 12 Feb 2018 13:45:59 -0700 (MST)
Received: from box313.bluehost.com ([69.89.31.113]) by cmgw2 with id 9wlw1x0092SSUrH01wlz7P; Mon, 12 Feb 2018 13:45:59 -0700
X-Authority-Analysis: v=2.2 cv=TIA1cxta c=1 sm=1 tr=0 a=h1BC+oY+fLhyFmnTBx92Jg==:117 a=h1BC+oY+fLhyFmnTBx92Jg==:17 a=xqWC_Br6kY4A:10 a=Op4juWPpsa0A:10 a=r77TgQKjGQsHNAKrUKIA:9 a=pGLkceISAAAA:8 a=wU2YTnxGAAAA:8 a=AUd_NHdVAAAA:8 a=48vgC7mUAAAA:8 a=wld5ZzBma5Q97v1U5I4A:9 a=mmbURYpjFNgws_GB:21 a=DELfdGRLsoxJTCqo:21 a=QEXdDO2ut3YA:10 a=WP0VnmNyAAAA:8 a=rrKNdljagIRSFoFf32cA:9 a=EbN3sSXJiRvKkX-j:21 a=-XpjyNt1bGG9-3eG:21 a=cx6ir3Cv7ypSt4u-:21 a=_W_S_7VecoQA:10 a=Yz9wTY_ffGCQnEDHKrcv:22 a=w1C3t2QeGrPiZgrLijVG:22 a=GAxSCXi1NP3-_krMBJ48:22
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=labn.net; s=default; h=Content-Type:MIME-Version:Subject:References:In-Reply-To: Message-ID:Date:CC:To:From:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=YpqfpTHOZUwpjSghymiHXWxKBZrenK7GFzmIizeq2HA=; b=iCqX9t1+1CUaQBEVqAKHt9jU0y eL+t/ldbnR7XAua8/MQTkxPsbo0bI9MS1Ldt9kqj8I58WDhmAbtIWPXHyEFMhfeAIp2vretfFwKyS yYxnlmIbqildN7NDTC4JCtzfa;
Received: from [172.58.233.69] (port=39313 helo=[IPV6:2607:fb90:711f:b11f:0:45:1769:9301]) by box313.bluehost.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.89_1) (envelope-from <lberger@labn.net>) id 1elKz5-003KUp-Gj; Mon, 12 Feb 2018 13:45:55 -0700
From: Lou Berger <lberger@labn.net>
To: "Andrew G. Malis" <agmalis@gmail.com>
CC: "Pascal Thubert (pthubert)" <pthubert@cisco.com>, Jouni <jouni.nospam@gmail.com>, detnet@ietf.org
Date: Mon, 12 Feb 2018 15:45:52 -0500
Message-ID: <1618bc44418.27d3.9b4188e636579690ba6c69f2c8a0f1fd@labn.net>
In-Reply-To: <CAA=duU2zsqp4WeNHjh1_EMLtVRRWj7gsigC4z=+uCSxx4FQoQg@mail.gmail.com>
References: <cff50c52d2f945cfa8eff149f5242fb0@XCH-RCD-001.cisco.com> <16170c78f30.27d3.9b4188e636579690ba6c69f2c8a0f1fd@labn.net> <22c101d3a0bc$5795b080$06c11180$@gmail.com> <b17a1c79-010d-b4a0-aa07-8182b72f2577@labn.net> <00A9AB24-830A-46C7-B3DB-5C17D2AC91EC@gmail.com> <b9c9fe85-fbd9-9859-51ca-27c9bda061a5@labn.net> <7CF3359D-204E-46E9-B7C6-B6134120502E@gmail.com> <d879c877a1a04760920c9c9c3edbe6e0@XCH-RCD-001.cisco.com> <1ecb9fc9-de35-c543-f53a-6640fddf2079@labn.net> <CAA=duU0fgVx+eqE-YRePz-QCfU107Jcu-2iSn8=+YvC+XmYKBQ@mail.gmail.com> <CAA=duU2zsqp4WeNHjh1_EMLtVRRWj7gsigC4z=+uCSxx4FQoQg@mail.gmail.com>
User-Agent: AquaMail/1.13.2-730 (build: 101300200)
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="----------1618bc448e048be27d3cdf1470"
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - box313.bluehost.com
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - labn.net
X-BWhitelist: no
X-Source-IP: 172.58.233.69
X-Exim-ID: 1elKz5-003KUp-Gj
X-Source:
X-Source-Args:
X-Source-Dir:
X-Source-Sender: ([IPV6:2607:fb90:711f:b11f:0:45:1769:9301]) [172.58.233.69]:39313
X-Source-Auth: lberger@labn.net
X-Email-Count: 1
X-Source-Cap: bGFibm1vYmk7bGFibm1vYmk7Ym94MzEzLmJsdWVob3N0LmNvbQ==
X-Local-Domain: yes
Archived-At: <https://mailarchive.ietf.org/arch/msg/detnet/kfDscc2xgxqvguoCxUgpUg0bu-4>
Subject: Re: [Detnet] IPv6 encapsulation in dataplane doc
X-BeenThere: detnet@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Discussions on Deterministic Networking BoF and Proposed WG <detnet.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/detnet>, <mailto:detnet-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/detnet/>
List-Post: <mailto:detnet@ietf.org>
List-Help: <mailto:detnet-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/detnet>, <mailto:detnet-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 12 Feb 2018 20:46:04 -0000

I thought I heard IP over mpls not mpls over IP, although this is certainly 
possible but my bet is that this a third document or at least a new section.

I expect that this will come up on Wednesdays interim...

Lou


On February 12, 2018 2:16:05 PM "Andrew G. Malis" <agmalis@gmail.com> wrote:

> Whoops, that was a typo, should be RFC 7510, not 7520.
>
> Cheers,
> Andy
>
>
> On Mon, Feb 12, 2018 at 2:14 PM, Andrew G. Malis <agmalis@gmail.com> wrote:
>
>> Lou and Pascal,
>>
>> On the most recent interim call, we discussed an encapsulation for IPv4
>> and IPv6 that reused the MPLS encapsulation inside UDP inside IP (ala RFC
>> 7520). This works for both v4 and v6, obviously.  Or at least that’s how I
>> interpreted the conversation at the time and the following notes from the
>> Etherpad:
>>
>> Jouni: Good point. Prob with IP is have been narrow focused on carrying
>> DetNet info. But if can assume that in a DetNet domain for routed traffic,
>> transport below is TSN or equivalent, e.g. MPLS, then what we do at IP
>> layer is just routing or mapping flows into MPLS tunnels. An underlay and
>> then just do routing.
>> Lou: Is that assuming that PREF always done by e.g. TSN layer?
>> Jouni: Yes.
>> Jouni: Then starts looking like over PWs as we discussed earlier.
>> Lou: Others find this approach acceptable? This proposal fits within our
>> charter and process, and is more like what some hoped to see from our
>> process. Do we, the people building the solution, like this approach or
>> find it objectionable?
>> Jouni: Starting to like this idea.
>> Lou: So next rev of draft will propose largely common approach on v4 and
>> v6, i.e. 5-tuple, depends on TSN and MPLS to provide transport and PREF
>> functions?
>>
>> So is this still the case, and what we should expect to see in the next
>> revision of the draft?
>>
>> Thanks,
>> Andy
>>
>>
>> On Mon, Feb 12, 2018 at 11:25 AM, Lou Berger <lberger@labn.net> wrote:
>>
>>> Hi Pascal,
>>>
>>> On 2/12/2018 5:30 AM, Pascal Thubert (pthubert) wrote:
>>>
>>>> Hello Jouni and Lou:
>>>>
>>>> Just to make sure we are on the same line:
>>>>
>>>> - IPvX (==v4/v6) would signal the flow implicitly - derived from the
>>>> tuples- but not the source sequencing. No need to change IP.
>>>>
>>> if you substitute "encode" with "signal", I agree.
>>>
>>> - DetNet may provide PREF/IOD from the edge ingress. DetNet would provide
>>>> its own sequencing based on reception order at ingress edge.
>>>>
>>> I'm not sure edge ingress is right here.  an IP host may still be a
>>> DetNet edge, but PREF may on start at the DetNet/MPLS ingress.
>>>
>>> - The  ingress edge may be a NIC card in the originating node. It would
>>>> be operating below IPvX and possibly equipped with 2 Ethernet ports and
>>>> capable of PREF.
>>>>
>>> are you talking 802.1cb or detnet PREF? if the former, yes, if the latter
>>> this is FFS for IP only detnet.
>>>
>>> - Anything above IP is beyond charter. A upper layer sequence may be
>>>> visible in the L4-transport or at the application layer but DetNet will not
>>>> look for it.
>>>>
>>> Yes (from my personal view)
>>>
>>>> A snooping edge device doing DPI may dig in the packet and find
>>>> information for packet identification and ordering, for a particular
>>>> application, and that is beyond the interests of the WG.
>>>>
>>> I don't think there has any proposal for this discussed in the WG either
>>> way.
>>>
>>> - A L4 transport may be designed to provide PREF, IOD, FEC (network
>>>> coding) and whatever else needed. This would be done in TSV Area with
>>>> DetNet's blessing.
>>>>
>>> agreed, as would be any changes in TCP congestion control modes to take
>>> advantage of DetNet allocations.
>>>
>>>
>>> Do we agree?
>>>>
>>> I hope so!
>>>
>>>
>>> Pascal
>>>>
>>>> -----Original Message-----
>>>> From: Jouni [mailto:jouni.nospam@gmail.com]
>>>> Sent: dimanche 11 février 2018 22:43
>>>> To: Lou Berger <lberger@labn.net>
>>>> Cc: Pascal Thubert (pthubert) <pthubert@cisco.com>; detnet@ietf.org
>>>> Subject: Re: [Detnet] IPv6 encapsulation in dataplane doc
>>>>
>>>>
>>>>
>>>> * End systems cannot participate to DetNet service layer packet
>>>>>>>> elimination & duplication business. If they still do e.g. using
>>>>>>>> DetNet DstOpt that is separate from subnet/transport layer provided
>>>>>>>> service.
>>>>>>>>
>>>>>>> True, end station participation in PREF wouldn't be possible in this
>>>>>>> simplified approach for IP, but the end stations *could* still
>>>>>>> participate in the equivalent function provided by a sub-net layer
>>>>>>> such as TSN. So end to end detnet traffic protection is not
>>>>>>> possible, but it doesn't mean it can't be provided by lower network
>>>>>>> layers.  The gain here is a simplified and unified IP solution and
>>>>>>> simplified host processing/support, albeit with a loss of one of the
>>>>>>> three end to end detnet service attributes.
>>>>>>>
>>>>>> Yes, that's true (as I already clarified in my latest mail to Pascal).
>>>>>>
>>>>>> yes.  It's good for all to remember that DetNet is just as much (if
>>>>> not
>>>>> more) about supporting explicit routes with congestion protection and
>>>>> latency control....
>>>>>
>>>> Indeed..
>>>>
>>>> * Different subnet/transport layer segments in the DetNet domain
>>>>>>>> are likely to use their own sequencing and duplication &
>>>>>>>> elimination solutions. I am not sure how independent those will be
>>>>>>>> e.g., is the sequence numbering unified across or only per segment.
>>>>>>>> The former is not IMO easy and the latter easily reduces to single
>>>>>>>> points between segments to handle number book keeping properly.
>>>>>>>>
>>>>>>> Agreed.  The question for the WG is this a reasonable compromise in
>>>>>>> order to in our *initial* solution defined by the WG.  -- Nothing
>>>>>>> says we can't come back and improve on this in the future, but it
>>>>>>> will allow us to get *something* useful defined with less complexity.
>>>>>>>
>>>>>> I would first specify the required steps that make IP as-is work
>>>>>> within one subnet/transport layer segment. That should be an achievable
>>>>>> goal with a limitation of e.g. a single interconnection point between
>>>>>> subnet/transport layer segments of different types/solutions.
>>>>>>
>>>>>> sure. as long as one of the 'attached stations' can be a router.
>>>>>
>>>> Yes. That's the plan at least from my side.
>>>>
>>>> - Juoni
>>>>
>>>>
>>>> After that look into solving multiple interconnection points.. and there
>>>>>> the L4 path that Pascal mentioned or that IPv6 DetNet DstOpt of mine could
>>>>>> be way forward.
>>>>>>
>>>>>> yes, this would be good follow on work...
>>>>>
>>>>> Cheers,
>>>>> Lou
>>>>>
>>>>>> - Jouni
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Regardless the above I am keen to explore this alternative approach..
>>>>>>>>
>>>>>>>> Great - look forward to hear the results!
>>>>>>> Lou
>>>>>>>
>>>>>>> -          Jouni
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *From:* detnet [mailto:detnet-bounces@ietf.org] *On Behalf Of *Lou
>>>>>>>> Berger
>>>>>>>> *Sent:* Wednesday, February 7, 2018 17:00 PM
>>>>>>>> *To:* Pascal Thubert (pthubert) <pthubert@cisco.com>;
>>>>>>>> detnet@ietf.org
>>>>>>>> *Subject:* Re: [Detnet] IPv6 encapsulation in dataplane doc
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Pascal/all,
>>>>>>>>
>>>>>>>> At the last interim a proposal was made to simplify IP processing,
>>>>>>>> at least for the initial detnet solution, by leaving PREF to the
>>>>>>>> subnet/transport layers (i.e., TSN and MPLS) and providing DetNet
>>>>>>>> flow identification based on typical IP 5-tuple perhaps + dscp.
>>>>>>>> This approach has several benefits beyond simplification, notably
>>>>>>>> it will work for both ipv4 and IPv6, and doesn't require any
>>>>>>>> modification to encapsulation / formats.
>>>>>>>>
>>>>>>>> It would be really valuable to get feedback from the whole working
>>>>>>>> group if this simplification is acceptable or has unacceptable
>>>>>>>> limitations.
>>>>>>>>
>>>>>>>> Lou
>>>>>>>>
>>>>>>>> On February 7, 2018 8:28:02 AM "Pascal Thubert (pthubert)"
>>>>>>>> <pthubert@cisco.com <mailto:pthubert@cisco.com>> wrote:
>>>>>>>>
>>>>>>>>    Dear all
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>    This is about the IPv6 encapsulation and more precisely
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                            Therefore, if a DetNet-aware end system
>>>>>>>> only
>>>>>>>>
>>>>>>>>       inserted the DetNet Destination Option into the IPv6 but e.g.,
>>>>>>>> a
>>>>>>>>
>>>>>>>>       DetNet Edge node is configured to enforce an explicit route
>>>>>>>> for the
>>>>>>>>
>>>>>>>>       IPv6 packet using a source routing header, then it has no
>>>>>>>> other
>>>>>>>>
>>>>>>>>       possibility than add an outer tunneling IPv6 header with
>>>>>>>> required
>>>>>>>>
>>>>>>>>       extension headers in it.  The processing of IPv6 packets in a
>>>>>>>> DetNet
>>>>>>>>
>>>>>>>>       Edge node is discussed further in Section 6.4.1
>>>>>>>>    <https://tools.ietf.org/html/draft-ietf-detnet-dp-sol-01#se
>>>>>>>> ction-6.4.1>.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>    With the current spec, a source sends a DetNet packet as
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                        +---------------------------------+
>>>>>>>>
>>>>>>>>                        |                                 |
>>>>>>>>
>>>>>>>>                        |           DetNet Flow           |
>>>>>>>>
>>>>>>>>                        |             Payload             |
>>>>>>>>
>>>>>>>>                        |                                 |
>>>>>>>>
>>>>>>>>                        /---------------------------------\
>>>>>>>>
>>>>>>>>                        H   Optional DetNet DstOpt Hdr    H
>>>>>>>>
>>>>>>>>                        \---------------------------------/
>>>>>>>>
>>>>>>>>                        |          IPv6 header            |
>>>>>>>>
>>>>>>>>                        |     (with set Flow label)       |
>>>>>>>>
>>>>>>>>                        +---------------------------------+
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>    And then the ingress node needs to re-encapsulate as
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                        +---------------------------------+
>>>>>>>>
>>>>>>>>                        |                                 |
>>>>>>>>
>>>>>>>>                        |           DetNet Flow           |
>>>>>>>>
>>>>>>>>                        |             Payload             |
>>>>>>>>
>>>>>>>>                       |                                 |
>>>>>>>>
>>>>>>>>                        /---------------------------------\
>>>>>>>>
>>>>>>>>                        H        DetNet DstOpt Hdr        H
>>>>>>>>
>>>>>>>>                        \---------------------------------/
>>>>>>>>
>>>>>>>>                        |          IPv6 header            |
>>>>>>>>
>>>>>>>>                        |     (with set Flow label)       |
>>>>>>>>
>>>>>>>>                        +=================================+
>>>>>>>>
>>>>>>>>                        |          Routing header         |
>>>>>>>>
>>>>>>>>                        /---------------------------------\
>>>>>>>>
>>>>>>>>                        H        DetNet DstOpt Hdr        H
>>>>>>>>
>>>>>>>>                        \---------------------------------/
>>>>>>>>
>>>>>>>>                        |          IPv6 header            |
>>>>>>>>
>>>>>>>>                        |     (with set Flow label)       |
>>>>>>>>
>>>>>>>>                        +---------------------------------+
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>    This creates a duplication of the DetNet Destination Option.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>    There are alternatives
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>    a)  whereby the packet is tunneled from the source to the detnet
>>>>>>>>    ingress, and based on its state the DetNet ingress accepts the
>>>>>>>>    packet, processes it and then resends it. The tunneled version of
>>>>>>>>    this could be:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                        +---------------------------------+
>>>>>>>>
>>>>>>>>                        |                                 |
>>>>>>>>
>>>>>>>>                        |           DetNet Flow           |
>>>>>>>>
>>>>>>>>                        |             Payload             |
>>>>>>>>
>>>>>>>>                        |                                 |
>>>>>>>>
>>>>>>>>                        +---------------------------------+
>>>>>>>>
>>>>>>>>                        |          IPv6 header            |
>>>>>>>>
>>>>>>>>                        |   (dest = final destination)    |
>>>>>>>>
>>>>>>>>                        /=================================\
>>>>>>>>
>>>>>>>>                        H   Optional DetNet DstOpt Hdr    H
>>>>>>>>
>>>>>>>>                        \---------------------------------/
>>>>>>>>
>>>>>>>>                        |          IPv6 header            |
>>>>>>>>
>>>>>>>>                        |   (dest = DetNet ingress edge)  |
>>>>>>>>
>>>>>>>>                        |     (with set Flow label)       |
>>>>>>>>
>>>>>>>>                        +---------------------------------+
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>    Which allows the ingress to tunnel to the egress as follows:
>>>>>>>>
>>>>>>>>                        +---------------------------------+
>>>>>>>>
>>>>>>>>                        |                                 |
>>>>>>>>
>>>>>>>>                        |           DetNet Flow           |
>>>>>>>>
>>>>>>>>                        |             Payload             |
>>>>>>>>
>>>>>>>>                        |                                 |
>>>>>>>>
>>>>>>>>                        +---------------------------------+
>>>>>>>>
>>>>>>>>                        |          IPv6 header            |
>>>>>>>>
>>>>>>>>                        |      (to final destination)     |
>>>>>>>>
>>>>>>>>                        +=================================+
>>>>>>>>
>>>>>>>>                        |          Routing header         |
>>>>>>>>
>>>>>>>>                        /---------------------------------\
>>>>>>>>
>>>>>>>>                        H   Optional DetNet DstOpt Hdr    H
>>>>>>>>
>>>>>>>>                        \---------------------------------/
>>>>>>>>
>>>>>>>>                        |          IPv6 header            |
>>>>>>>>
>>>>>>>>                        |   (dest = DetNet egress edge)   |
>>>>>>>>
>>>>>>>>                        |     (with set Flow label)       |
>>>>>>>>
>>>>>>>>                        +---------------------------------+
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>    b)  whereby the PREF is done by the end nodes and the tunnel is
>>>>>>>>    transport only, meaning that there are 2 tunnels A and B and that
>>>>>>>>    the source sends twice a packet like this:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                        +---------------------------------+
>>>>>>>>
>>>>>>>>                        |                                 |
>>>>>>>>
>>>>>>>>                        |           DetNet Flow           |
>>>>>>>>
>>>>>>>>                        |             Payload             |
>>>>>>>>
>>>>>>>>                        |                                 |
>>>>>>>>
>>>>>>>>                        /---------------------------------\
>>>>>>>>
>>>>>>>>                        H   Optional DetNet DstOpt Hdr    H
>>>>>>>>
>>>>>>>>                        \---------------------------------/
>>>>>>>>
>>>>>>>>                        |          IPv6 header            |
>>>>>>>>
>>>>>>>>                        |   (dest = DetNet ingress edge X)|
>>>>>>>>
>>>>>>>>                        |     (with set Flow label)       |
>>>>>>>>
>>>>>>>>                        +---------------------------------+
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>    And then the ingress node needs to re-encapsulate as
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                        +---------------------------------+
>>>>>>>>
>>>>>>>>                        |                                 |
>>>>>>>>
>>>>>>>>                        |           DetNet Flow           |
>>>>>>>>
>>>>>>>>                        |             Payload             |
>>>>>>>>
>>>>>>>>                       |                                 |
>>>>>>>>
>>>>>>>>                        /---------------------------------\
>>>>>>>>
>>>>>>>>                        H        DetNet DstOpt Hdr        H
>>>>>>>>
>>>>>>>>                        \---------------------------------/
>>>>>>>>
>>>>>>>>                        |          IPv6 header            |
>>>>>>>>
>>>>>>>>                        |   (dest = final destination)    |
>>>>>>>>
>>>>>>>>                        |     (with set Flow label)       |
>>>>>>>>
>>>>>>>>                        +=================================+
>>>>>>>>
>>>>>>>>                        |          Routing header         |
>>>>>>>>
>>>>>>>>                        +---------------------------------+
>>>>>>>>
>>>>>>>>                        |          IPv6 header            |
>>>>>>>>
>>>>>>>>                        |   (dest = DetNet egress edge X) |
>>>>>>>>
>>>>>>>>                        |     (with set Flow label)       |
>>>>>>>>
>>>>>>>>                        +---------------------------------+
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>    Cheers,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>    Pascal
>>>>>>>>
>>>>>>>>    _______________________________________________
>>>>>>>>    detnet mailing list
>>>>>>>>    detnet@ietf.org <mailto:detnet%40ietf.org>
>>>>>>>>    https://www.ietf.org/mailman/listinfo/detnet
>>>>>>>>
>>>>>>>>
>>>>>> _______________________________________________
>>>> detnet mailing list
>>>> detnet@ietf.org
>>>> https://www.ietf.org/mailman/listinfo/detnet
>>>>
>>>>
>>> _______________________________________________
>>> detnet mailing list
>>> detnet@ietf.org
>>> https://www.ietf.org/mailman/listinfo/detnet
>>>
>>
>>