From nobody Tue Mar 30 06:46:24 2021 Return-Path: X-Original-To: ipv6@ietfa.amsl.com Delivered-To: ipv6@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E08B23A12E6 for ; Tue, 30 Mar 2021 06:46:23 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.003 X-Spam-Level: X-Spam-Status: No, score=0.003 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, FROM_LOCAL_NOVOWEL=0.5, HK_RANDOM_ENVFROM=0.001, HK_RANDOM_FROM=0.999, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, URI_HEX=0.1, URI_NOVOWEL=0.5] autolearn=no autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id b-87_VfbK2pA for ; Tue, 30 Mar 2021 06:46:19 -0700 (PDT) Received: from mail-io1-xd2c.google.com (mail-io1-xd2c.google.com [IPv6:2607:f8b0:4864:20::d2c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id B9BDB3A12DA for ; Tue, 30 Mar 2021 06:46:18 -0700 (PDT) Received: by mail-io1-xd2c.google.com with SMTP id z136so16440335iof.10 for ; Tue, 30 Mar 2021 06:46:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=fHnFkIW2ooUm9funHeqA5GE4d767tya291cnUiQ4Zs0=; b=mjkU5sceZ5Nl66QEj1GaIXK8vVeY/FY/CyAaFaCBodgpZqFj0VEjUkSlqe3AzDxIkp AqwAQ/k4DhwkfjzI/LaMugnyiJCdYXWcZsZeq7hxmzk5fTLf3FpY18POzj9HVJwp19+R t8skls7ay29A1QTbkXZb00axekU9ZmcW1cKdPrLCBYy04rFC8XqeGy42Olk7Ig+ltt6/ //0srTdV2o563pbuk/Z5v6ooLop0aQwdfOt5pz41526rWbF2TQvH0zX26aCk987VbvPC nKPoHAw1SIXeCyQMrTeWPjoBX/v/VbuD/83tgAX8oS47Jaq295wmSMd/vZDR0CeX1inS mGtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=fHnFkIW2ooUm9funHeqA5GE4d767tya291cnUiQ4Zs0=; b=OYb1OSt3rPxc8/jNKyJOfm52ouTw/HNeoyVbfn8ugA4werEf99bSo0QreVfYgTa+Nb 4gpPfBfavALdy2zP/F7u7UYjQPB0yTvICKLJUagNYURmVoRy3pnV1QC6mfwFH+XdTl/U eA23Oiy51t6PxYTz6PbGQIhaG9fbuNc8tHCYWtGOBMtdxPOvUfULZ0L8Q/Io18qoyEaf 7cE7X+ulys+DZr317+na2NLRVw5HXJ0P/oC8qKRF78QAI3UQXiisjDvZuC0y4keN4wCQ o6lh/lTxZKfUiJObTyefXJq+/roq23THTEbcT6gOSYZ37xMkwUcwMCYAhbJwcEfYmbda PHbg== X-Gm-Message-State: AOAM530tHq0cZmvpYkMMsXFA26Vc9AF1b25vpbUSEqPCGDkFK6jiDeQt DhiTgV4Y2B56+4OSTyGmLoV88qii4RjgsCIUx84= X-Google-Smtp-Source: ABdhPJz+H2Z+3+JEwzt13EDxibO1MaGDhjvo3ibcs+opSCue7xE7IA9M2VDUEj8obJibcvrgCPfOSTg+IRKbjUIKaEI= X-Received: by 2002:a5d:9e09:: with SMTP id h9mr25691756ioh.178.1617111976915; Tue, 30 Mar 2021 06:46:16 -0700 (PDT) MIME-Version: 1.0 References: <712706464d4048c9840c4e62151dec5e@huawei.com> In-Reply-To: From: Mark Smith Date: Wed, 31 Mar 2021 00:45:50 +1100 Message-ID: Subject: Re: Questions/comments about draft-dunbar-6man-5g-edge-compute-sticky-service To: "Jeffrey (Zhaohui) Zhang" Cc: Linda Dunbar , "Joel M. Halpern" , IPv6 List , Kaippallimalil John Content-Type: multipart/alternative; boundary="000000000000f23a2105bec13882" Archived-At: X-BeenThere: ipv6@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "IPv6 Maintenance Working Group \(6man\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Mar 2021 13:46:24 -0000 --000000000000f23a2105bec13882 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, 31 Mar 2021, 00:05 Jeffrey (Zhaohui) Zhang, wrote: > Hi Linda, > > You're not "leverage network control plane" - you're adding complications > to the network forwarding plane (and the control plane because you need t= o > coordinate among the ingress routers) with that sticky-service-table with > (Sticky Service ID, Flow Label. Sticky Egress address, Timer) entries. Th= e > 5G/MEC control plane solution I mentioned does not need a UE to query for= a > new address when its location changes. > > The variation of that sticky-service-table based on (Sticky Service ID, U= E > address, Sticky Egress address, Timer) is slightly better, because it cou= ld > be generalized to forwarding based on (source, destination), which is not > different from IP multicast (except that there is no replication). While > that reduces the complication in the forwarding plane, it still has the > scaling issue - you will still have a large table and you will still have > to coordinate among different ingress routers. > > Granted, with the trendy AI, the coordination can be more precisely - the > 5G system can predict where a UE is moving to and push the corresponding > entry only to the new ingress router and ahead of time. If the UE address > does not change after its anchoring UPF changes, the coordination can be > simpler (otherwise the new ingress router needs to be able to associate t= he > old and new addresses). > > Retaining the same UE address after relocation can be done, and it is > documented in " Solution #26: Persistent address allocation for mobile UE= s > that need MEC access" of 3GPP SA2 TR 23.748. I did not get to defend it > during the final stage of study but I still think it is a viable solution > (and preferred for application servers that do not deal well with client > address changes) - would love to discuss that further with John on 3GPP S= A2 > mailing list. > > Having said all the above, I still believe it's the easiest and most > scalable solution if the UEs are told by 5G/MEC control plane what true > unicast address of the server or load balancer to use. > Anycast to initially set things up, switch to unicast once that is done so there is no ongoing "anycast stickiness" required, and therefore no corresponding ongoing "anycast stickiness" state required in the network forwarding plane. I think the existing and coming multipath transport layer protocols can facilitate that. Initial connection setup via anycast, once that is established, then establish 1 or more unicast sub-connections (subflows in MPTCP) for ongoing communication, and abandon the anycast connection once the first unicast sub-connection is established. Anycast is used for setup, but not for ongoing communication. See section 5.7.7 of this ID. https://tools.ietf.org/id/draft-smith-6man-form-func-anycast-addresses-01.h= tml#rfc.section.5.7.7 Regards, Mark. > Jeffrey > > -----Original Message----- > From: Linda Dunbar > Sent: Monday, March 29, 2021 5:16 PM > To: Joel M. Halpern ; Jeffrey (Zhaohui) Zhang < > zzhang@juniper.net>; 'IPv6 List' > Subject: RE: Questions/comments about > draft-dunbar-6man-5g-edge-compute-sticky-service > > [External Email. Be cautious of content] > > > Joel, > > I am confused of your statement. > > The problem I described is: > An end node (e.g. a drone) moves constantly. There are multiple > servers hosted in Edge DCs close to Cell towers to process requests from > the end node to achieve ultra-low latency. It is not realistic for the e= nd > node to send DNS query every time it anchors to a new Cell Tower. > > The suggested solution is to leverage network control plane to find the > optimal server to serve the end node. > > You stated that "it is not at all clear that is the best answer or even a > good one". Why? > > Linda > > > -----Original Message----- > From: Joel M. Halpern > Sent: Monday, March 29, 2021 1:56 PM > To: Linda Dunbar ; Jeffrey (Zhaohui) Zhang < > zzhang@juniper.net>; 'IPv6 List' > Subject: Re: Questions/comments about > draft-dunbar-6man-5g-edge-compute-sticky-service > > Linda, you are assserting that the answer is "to leverage the network > control plane". > As Jeffrey has pointed out, given the problem you have described, it is > not at all clear that is the best answer, or even a good one. > > Yours, > Joel > > On 3/29/2021 1:30 PM, Linda Dunbar wrote: > > Jeffrey, > > > > The draft is to leverage the network control plane to determine which > server is the most appropriate one for the device. > > > > You can definitely use the Controller to choose the optimal server. To > > simplify the IP address allocation, the draft suggest using the Egress > > router addresses as the proxy for reaching the desired server. We can > > add a section to describe this scenario > > > > Linda > > > > -----Original Message----- > > From: Jeffrey (Zhaohui) Zhang > > Sent: Monday, March 29, 2021 12:20 PM > > To: Linda Dunbar ; Vasilenko Eduard > > ; Kaippallimalil John > > ; 'IPv6 List' > > Subject: RE: Questions/comments about > > draft-dunbar-6man-5g-edge-compute-sticky-service > > > > Since it is for "sticky service", you don't want to get a new server > address every time you move - unless the previous one is no longer > appropriate. That means it is best for a controller to determine which on= e > to use both initially and later when situation changes (when a UE relocat= es > or server load situation changes), and that does not necessarily mean it = is > always through DNS. > > > > Jeffrey > > > > -----Original Message----- > > From: Linda Dunbar > > Sent: Monday, March 29, 2021 1:15 PM > > To: Jeffrey (Zhaohui) Zhang ; Vasilenko Eduard > > ; Kaippallimalil John > > ; 'IPv6 List' > > Subject: RE: Questions/comments about > > draft-dunbar-6man-5g-edge-compute-sticky-service > > > > [External Email. Be cautious of content] > > > > > > Jeffrey, > > > > The Devices are moving consistently, it is not reasonable to require > them to consistently query DNS for the "correct" non-ANYcast address . > > > > Linda > > > > -----Original Message----- > > From: Jeffrey (Zhaohui) Zhang > > Sent: Monday, March 29, 2021 12:04 PM > > To: Linda Dunbar ; Vasilenko Eduard > > ; Kaippallimalil John > > ; 'IPv6 List' > > Subject: RE: Questions/comments about > > draft-dunbar-6man-5g-edge-compute-sticky-service > > > > Even if you could get over the security/trust hurdle, using a controlle= r > to let the UEs know which unicast non-anycast address to use is a much > simpler/better solution. > > > > Jeffrey > > > > -----Original Message----- > > From: Linda Dunbar > > Sent: Monday, March 29, 2021 12:50 PM > > To: Vasilenko Eduard ; Jeffrey (Zhaohui) > > Zhang ; Kaippallimalil John > > ; 'IPv6 List' > > Subject: RE: Questions/comments about > > draft-dunbar-6man-5g-edge-compute-sticky-service > > > > [External Email. Be cautious of content] > > > > > > Ed, > > > > Yes, they are in one domain. Here is one example: > > > > 5G Connected devices, such as drones for fighting fires or natural > disasters or robots in Industry 4.0 environments, need ultra-low latenc= y > responses from their analytic servers hosted in the Edge data centers. To > reach ultra-low latency, there are multiple servers hosting the analytic > functions in the Edge DCs. > > All the functions (including networking and analytics) and devices are > administrated by one operator. Those functions might be provided by > different vendors, therefore needing interoperable solutions. > > > > Linda > > > > -----Original Message----- > > From: Vasilenko Eduard > > Sent: Monday, March 29, 2021 11:41 AM > > To: Linda Dunbar ; Jeffrey (Zhaohui) Zhang > > ; Kaippallimalil John > > ; 'IPv6 List' > > Subject: RE: Questions/comments about > > draft-dunbar-6man-5g-edge-compute-sticky-service > > > > It could be the problem. > > Because all SR RFCs and drafts clearly say: only inside the domain. > > Else could be a huge security risk. UE could not be trusted. > > Cross-domain security is the principal question that should be discusse= d > in SPRING first. > > Current SR architecture does not try to resolve it yet. > > > > > > Segment Routing in general and SRv6 in particular are claimed to be > designed for Trusted environments only: > > - Segment routing architecture (RFC 8402) section 8 > > - SRH - Segment Routing Header (RFC 8754) section 5 > > - SRv6 Network Programming > (draft-ietf-spring-srv6-network-programming-25) section 9 SRH RFC is > especially verbal how to filter-out any SR-related information on the > border of "SR domain". > > Ed/ > > -----Original Message----- > > From: ipv6 [mailto:ipv6-bounces@ietf.org] On Behalf Of Linda Dunbar > > Sent: Monday, March 29, 2021 7:09 PM > > To: Jeffrey (Zhaohui) Zhang ; Kaippallimalil John > > ; 'IPv6 List' > > Subject: RE: Questions/comments about > > draft-dunbar-6man-5g-edge-compute-sticky-service > > > > Jeffrey, > > > > We can definitely add the option of UE inserting SRH. I am just not sur= e > how many UEs or end devices will do those actions. If very few UEs can do > this action, the solution itself is not useful. However, it doesn't hurt > for IETF to specify such a solution so that future IoT or 5G devices can > have a reference to do the actions. > > > > Another point, the number of Sticky Service is not large. The Ingress > routers are configured with the policies ( ACLs) to filter those flows. > > > > Linda > > > > -----Original Message----- > > From: Jeffrey (Zhaohui) Zhang > > Sent: Monday, March 29, 2021 9:18 AM > > To: Linda Dunbar ; Kaippallimalil John > > ; 'IPv6 List' > > Subject: RE: Questions/comments about > > draft-dunbar-6man-5g-edge-compute-sticky-service > > > > Hi Linda, > > > > You proposed two ways of providing "sticky services" - when a UE moves > to a new location, the ingress router at that new location will still rou= te > the packets of the same flow to the previous egress router. That flow > cannot be identified by the destination address alone, since it is an > anycast address that are shared by servers behind different egress router= s. > > > > Essentially, you're trying to turn the ingress router into a load > balancer, especially with your option #2 (section 5, "tunnel based" > solution). I don't think we want the routers to do that - while routers c= an > make use of 5-tuple for ECMP hashing, we don't want to make routers more > complicated and do forwarding based on a sticky-service-table with (Stick= y > Service ID, Flow Label. Sticky Egress address, Timer) entries. It's not > only complicated but also does not scale (we can discuss the scaling aspe= ct > wrt the flow labels separately). > > > > The variation of option #1 that I suggested would be better, if the > following were true: > > > > 1. The UE can insert an SRH > > 2. The ingress router can trust the SRH from the UEs > > > > In that case, it would be better for the UE to learn the egress router > via 5G/MEC control plane, instead of relying on the egress router to put > that into the DOH of every server->UE packets for sticky services and for > the UE to retrieve that information from each incoming sticky service > packets. One thing I learned is that the entire 5G system is very much > heavy with control/management plane and I would think it is a much better > option to provide that information to the UEs. > > > > On the other hand, once you go that way, the control plane can simply > provide the regular, non-anycast addresses of the servers instead of the > egress router address. Then, all the problems disappear and corresponding > proposals are no longer needed, including the ones in > draft-dunbar-idr-5g-edge-compute-app-meta-data, and we only need existing > simple routing functions. > > > > Thanks. > > > > Jeffrey > > > > -----Original Message----- > > From: Linda Dunbar > > Sent: Saturday, March 27, 2021 10:45 PM > > To: Jeffrey (Zhaohui) Zhang ; Kaippallimalil John > > ; 'IPv6 List' > > Subject: RE: Questions/comments about > > draft-dunbar-6man-5g-edge-compute-sticky-service > > > > [External Email. Be cautious of content] > > > > > > Jeffrey, > > > > Thank you very much for the constructive comments. > > Replies are inserted below: > > > > -----Original Message----- > > From: Jeffrey (Zhaohui) Zhang > > Sent: Friday, March 26, 2021 3:59 PM > > To: Linda Dunbar ; Kaippallimalil John > > ; 'IPv6 List' > > Subject: Questions/comments about > > draft-dunbar-6man-5g-edge-compute-sticky-service > > > > Hi Linda, John, > > > > When a UE (User Equipment) initiates application packets using the > > destination address from a DNS reply or from its own cache, the > > packets from the UE are carried in a PDU session through 5G Core > > [5GC] to the 5G UPF-PSA (User Plan Function - PDU Session Anchor). > > The UPF-PSA decapsulate the 5G GTP outer header and forwards the > > packets from the UEs to the Ingress router of the Edge Computing (E= C) > > Local Data Network (LDN). The LDN for 5G EC, which is the IP Networ= ks > > from 5GC perspective, is responsible for forwarding the packets to > > the intended destinations. > > > > A nit comment about "5G Core" above. When I first started learning 4G/5= G > It took me a while to realize the 3GPP "core network" concept in vastly > different from what IETF people are used to. It's not about topology and > now the "core network" functions are being more and more distributed into > edges. Therefore, in this context it may be better to simply strike the > "through 5G Core [5GC]" wording to reduce the confusion to some readers. > > > > [Linda] That is very true. Removed the term per your suggestion. 5G Cor= e > refers to all the functions from Radio to UPF. > > > > 1.3. Problem #1: ANYCAST in 5G EC Environment > > > > Increasingly, ANYCAST is used extensively by various application > > providers and CDNs because it is possible to dynamically load balan= ce > > across multiple locations of the same address based on network > > conditions. BGP is an integral part in the way IP anycast usually > > functions. Within BGP routing there are multiple routes for the sam= e > > IP address which are pointing to different locations. > > > > Not only BGP - but all IP routing protocols should work well with > anycast. My understanding is that BGP being integral part here is really > that the data network here is likely realized by VPNs over the same > transport network. Is that a correct understanding? > > > > [Linda] ANYCAST has traditionally been used for servers or loader > balancers that are placed in geographically diverse locations, so that BG= P > alone is enough for the traffic in one region to be forwarded to one > server. But for the 5G Edge Computing where multiple Servers/load > Balancers with the same ANYCAST addresses are placed close proximity, IGP > is needed. > > > > Of course, BGP does have flexibility in providing better/more control o= f > route selection than IGP does in the context of the companion > draft-dunbar-idr-5g-edge-compute-app-meta-data. > > [Linda] Correct. > > > > But, having multiple locations for the same ANYCAST address in 5G > > Edge Computing environment can be problematic because all those edg= e > > computing Data Centers can be close in proximity. There might not = be > > any difference in the routing cost to reach the Application Servers > > in different Edge DCs. Same routing cost to multiple ANYCAST > > locations can cause packets from one flow to be forwarded to > > different locations, which can cause service glitches. > > > > As pointed out later in this same document, modern routers support "Flo= w > Affinity" and should not cause packets of a flow on a specific router to = be > forwarded to different locations. The real problem is when a UE moves to = a > different location, the new router at that location may send it to a > different egress router. However, that is the "sticky service" problem > described in 1.4. > > [Linda] Correct. > > > >>From draft-dunbar-idr-5g-edge-compute-app-meta-data, I understand that > on a specific router it needs to choose a location that can best serve an > application based on some non-routing factors. If 1.3 is really for that > purpose, it should be reworded accordingly. As I mentioned in an earlier > email, the two documents should better align on the problem descriptions. > > > > Here is the overview of the End-Node based Sticky Service solution: > > - Each ANYCAST Edge Computing server either learns or is informed > > of the unicast Sticky Egress address (Section 3). The goal of > > the network is to deliver packets belonging to one flow to the > > same Sticky Egress address for the ANYCAST address. Section 4.= 1 > > describes how Edge Computing Servers discover their > > corresponding Sticky Egress unicast addresses. > > - When an Edge Computing server sends data packets to a UE (or > > client), it inserts the Sticky-Dst-SubTLV (described in Sectio= n > > 4.2) into the packets' Destination Option Header. > > - UE (or client) needs to copy the Destination Option Header from > > the received packet to the next packet's Destination Header if > > the next packet belongs to the same flow as the previous packe= t. > > > > I was really confused by "next packet". I finally realized you may be > referring to response packets from the UE to the server, and the "same > flow" should be "same service". Better wording is needed here. > > > > - If the following conditions are true, the ingress router > > encapsulates the packet from the UE in a tunnel whose outer > > destination address is set to the Sticky Egress Address > > extracted from the packet's Sticky-Dst-SubTLV: > > o The destination of the packet from the UE side matches > > with one of the Sticky Service ACLs configured on the > > ingress router of the LDN, > > o the packet header has the Destination Option present with > > Sticky-Dst-SubTLV. > > > > Wouldn't it be better for the UE to put in an SRH with one SID for the > server address and set the DA to be the egress router address? That way y= ou > don't need the ACL or the DOH (the Sticky-Dst-SubTLV information in the > DOH is not for consumption by the server anyway), and you don't even need > tunneling or BGP (unless VPN is used - but that's orthogonal to this). > Existing SRv6 function takes care of it. > > > > [Linda] 3GPP has rejected using SRH in the 5G Core. We can think about > using them in the N6 interface. > > > > Also, the Sticky-Dst-SubTLV in DOH of the server->UE traffic would be > better renamed as "return waypoint" for more generic purpose. > > [Linda] that is interesting suggestion. > > > > 4.1. Sticky Egress Address Discovery > > > > To an App server with ANYCAST address, the Sticky Egress address is > > same as its default Gateway address. > > > > To prevent malicious UEs (or clients) sending DDOS attacks to route= rs > > within 5G EC LDN, e.g. the Sticky Egress address that is encoded in > > the Destination option header in the packets sent back to the UEs (= or > > clients), a proxy Sticky Egress address can be encoded in the > > Destination option header. The proxy Sticky Egress address is only > > recognizable by the 5G EC LDN ingress nodes, i.e. the Ra and Rb in > > the Figure 1, but not routable in other networks. The LDN ingress > > routers can translate the proxy Sticky Egress to a routable address > > for the Sticky Egress node after the source addresses of the packet= s > > are authenticated. > > > > Why is the 4.1 title called "... discovery"? Does not seem to be about > "discovery". > > [Linda] it is about remembering which Egress router was used for the > flow. Should it be "Sticky Egress Memory"? > > > > 4.3. Expected behavior at the UE > > ... > > Section 4 describes the network layer processing if UEs do not > > perform the steps described here. > > > > Should be "Section 5". > > > > [Linda] Thank you. > > > > 5. Tunnel based Sticky Service Solutions 5.1. Ingress and Egress > > Routers Processing Behavior > > > > The solution assumes that both ingress routers and egress routers > > support at least one type of tunnel and are configured with ACLs to > > filter out packets whose destination or source addresses match with > > the Sticky Service Identifier. The solution also assumes there are > > only limited number of Sticky Services to be supported. > > An ingress router needs to build a Sticky-Service-Table, with the > > minimum following attributes. The Sticky-Service-Table is initializ= ed > > to be empty. > > - Sticky Service ID > > - Flow Label > > - Sticky Egress address > > - Timer > > > > Editor's Note: > > When a UE moves from one 5G Site to another, the same UE will hav= e > > a new IP address. "Flow Label + Sticky Service ID" stays the same > > when a UE is anchored to a new PSA. Therefore, this solution use > > "Flow Label + Sticky Service ID" to identify a sticky flow. Since > > the chance of different UEs sending packets to the same ANYCAST > > address using the same Flow Label is very low, it is with high > > probability that "Flow Label + Sticky Service ID" can uniquely > > identify a flow. When multiple UEs using the same Flow Label > > sending packets to the same ANYCAST address, the solution describ= ed > > in this section will stick the flows to the same ANYCAST server > > attached to the Sticky Egress router. This behavior doesn't cause > > any harm. > > > > It seems that the same flow label is used for traffic of the same > service in both directions. So who will assign the flow label? > > [Linda] The "flow label" from the IPv6 header should be managed by the > hosts & servers. > > > > If two UEs of the same service happen to use the same flow label, then > sticky service is not guaranteed. For example, initially they're anchored > at different UPFs, and UE1 traffic is sent to egress router 1 while UE2 > traffic is sent to egress router 2. When UE 1 relocates to the same UPF a= s > UE 2's, its traffic will be sent to egress node 2 because the same flow > label is used. > > > > Therefore, there should be a central controller to assign flow labels > based on UE id, and the UE id is not based on IP address (since it could > change). > > [Linda] Since the "Flow Label" is randomly generated (by Host OS), the > chance of two UEs reaching the same service having the same Flow Label is > very small. We can explore the option of getting the Control Plane > involved. > > > > Note: since there are only small number of Sticky services, the > > Sticky-Service-Table is not very large. > > > > With the above understanding, the table could get large? > > [Linda]? > > > > When an ingress router receives a packet from a UE matching with on= e > > of the Sticky Service ACLs and there is no entry in the Sticky- > > Service-Table matching the Flow Label and the Sticky Service ID, th= e > > ingress router considers the packet to be the first packet of the > > flow. There is no need to sticking the packet to any location. The > > ingress router uses its own algorithm to select the optimal egress > > node as the Sticky Egress address for the ANYCAST address, > > encapsulates the packet with a tunnel that is supported by the egre= ss > > node. The tunnel's destination address is set to the egress node > > address. > > > > If a UE was using egress router 1 and it relocates to a new UPF, the ne= w > ingress router will likely have no corresponding entry for it? What if th= e > new ingress router pick egress router 2? > > It seems that the ingress routers need to pre-exchange entries in the > table? > > I see it's discussed later that the routers do exchange the information= . > It should be mentioned up front when the table is introduced. > > [Linda] Would Adding a reference be enough? > > > > When an ingress router receives a packet in a tunnel from any egres= s > > router and the packet's source address matches with a Sticky Servic= e > > ID, the egress router address is set as the Sticky Egress address f= or > > the Sticky Service ID. The ingress router adds the entry of "Sticky= - > > Service-ID + Flow Label + the associated Sticky Egress address + > > Timer" to the Sticky-Service-Table if the entry doesn't exist yet i= n > > the table. If the entry exists, the ingress router refreshes the > > Timer of the entry in the table. > > > > When the ingress router receives the subsequent packets of a flow > > from the 5G side matching with an Sticky Service ID and the Sticky- > > Service ID exists in the Sticky-Service-Table, the ingress router > > uses the Sticky Egress address found in the Sticky-Service-Table to > > encapsulate the packet and refresh the Timer of the entry. If the > > Sticky-Service ID doesn't exist in the table, the ingress router > > considers the packet as the first packet of a flow. > > > > The above is what leads me to believe that the flow label is the same i= n > both directions. > > [Linda] they don't have to be the same, do they? > > > > 5.3. Scenario 2: With communication with 5G system > > ... > > The ingress and egress router processing are the same as described = in > > Section 5.1 except a flow is now uniquely identified by the "Sticky > > Service ID" + "UE address" instead of "Sticky Service ID" + "Flow > > Label". > > > > This confirms my earlier understanding for scenario 1 that "there shoul= d > be a central controller to assign flow labels based on UE id, and the UE = id > is not based on IP address (since it could change)" and that the table > could get large. > > > > Of course now for scenario 2, you're not using the flow label any more. > While the table only contains entries that this ingress router actually > need, the following are still true: > > - The table could still get large (if the number of attached UEs for > > the sticky services is large) > > - On demand fetching of the table entry may not be fast enough > > > > Additionally, instead of "scenario", "option" or "solution" would be a > better wording. > > [Linda] Good suggestion! > > > > More importantly, this stateful flow steering based on the additional > table is just too heavy and complicated. Why not simply have the UEs > support SRH so that traffic will be routed via the desired egress router > using standard SRv6 mechanism? > > [Linda] It is not realistic for UEs (your smart phone) to support SRH. > > > > Jeffrey > > > > > > -----Original Message----- > > From: Jeffrey (Zhaohui) Zhang > > Sent: Thursday, March 25, 2021 3:46 PM > > To: Linda Dunbar ; Kaippallimalil John > > ; IPv6 List ; > > idr@ietf. org > > Subject: questions about > > draft-dunbar-idr-5g-edge-compute-app-meta-data and > > draft-dunbar-6man-5g-edge-compute-sticky-service > > > > Hi Linda, John, > > > > I have the following questions. > > > > The two related drafts listed the following three problems respectively= : > > > > 1.3. Problem#1: ANYCAST in 5G EC Environment.............. 6 > > 1.4. Problem #2: Unbalanced Anycast Distribution due to UE > Mobility.................................................. 7 > > 1.5. Problem 3: Application Server Relocation............. 7 > > > > 1.2. Problem #1: ANYCAST in 5G EC Environment.............. 4 > > 1.3. Problem #2: sticking to original App Server........... 5 > > 1.4. Problem #3: Application Server Relocation............. 5 > > > > Why is problem #2 different in the two drafts? Is it true that none of > the two drafts address problem #3? > > The idr draft talk about "soft anchoring" problem and solution - how is > that different from the "sticky service"? > > > > Thanks. > > Jeffrey > > > > Juniper Business Use Only > > > > Juniper Business Use Only > > > > Juniper Business Use Only > > -------------------------------------------------------------------- > > IETF IPv6 working group mailing list > > ipv6@ietf.org > > Administrative Requests: > https://urldefense.com/v3/__https://nam11.safelinks.protection.outlook.co= m/?url=3Dhttps*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Fnam11.safelink= s.protection.outlook.com*2F*3Furl*3Dhttps*3A*2F*2Furldefense.com*2Fv3*2F__h= ttps*3A*2F*2Fnam11.safelinks.protection.outlook.com*2F*3Furl*3Dhttps*3A*2F*= 2Fwww.ietf.org*2Fmailman*2Flistinfo*2Fipv6*26amp*3Bdata*3D04*7C01*7Clinda.d= unbar*40futurewei.com*7C4209e8a9acae47b96d1808d8f2d16b8d*7C0fee8ff2a3b24018= 9c753a1d5591fedc*7C1*7C0*7C637526328578769822*7CUnknown*7CTWFpbGZsb3d8eyJWI= joiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000*26amp*= 3Bsdata*3DA39pqHVUBFsssO3DLSqrTUtPpcXAr*2F8pi*2Bmw*2BtIJNME*3D*26amp*3Brese= rved*3D0__*3BJSUlJSUlJSUlJSUlJSUlJSUlJSUlJQ!!NEt6yMaO-gk!QWI34EOzIdgzRLkkdD= 3rdv_fn4CLHXnnMvDpOOeQB4ELlElbfawu6WXv0nbjgi-z*24*26amp*3Bdata*3D04*7C01*7C= linda.dunbar*40futurewei.com*7C2b126cc5be8541bd43cc08d8f2d4a7ab*7C0fee8ff2a= 3b240189c753a1d5591fedc*7C1*7C0*7C637526342463012256*7CUnknown*7CTWFpbGZsb3 > > d8eyJWIjoiMC4wLjAwMDAiLCJQIjoi__;JSUlJSUlJSUlJSUqKioqKioqKioqKioqKioqKio= qKioqKioqKioqKioqKioqKioqKioqKioqKiUlJSoqKioqKioqKio!!NEt6yMaO-gk!VemUe-2PH= PVnuVFBu0NeOKMAx3tulW_X0zDToehm7avqQhBqbE3FDykd8nYVAqHW$ > > > > V2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000%26amp%3Bsdata%3DaOw1DkDc > > Qu0*2FmMi6RfWlUyRcLB2jRcsbBAhcpoaX5yE*3D%26amp%3Breserved%3D0__%3BJSUl > > JSUlJSUlJSUqKioqKiolJSUqKioqKioqKioqKiolJSUqKioqJSUlJSUlJSUlJSUlJSUlJS > > UlJQ!!NEt6yMaO-gk!Q-hLtDzPuot4CQsvyUhfrEcNgHIIBEdRDT4RgyHgVCE1f5Vt6Dlv > > zYC-o7693kZ1%24&data=3D04%7C01%7Clinda.dunbar%40futurewei.com%7C67c= 9 > > 07b1d8e643cc5a5108d8f2d6f104%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C > > 0%7C637526352290503133%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJ > > QIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3DBe%2FU8M= h > > JT%2BA5b83e8ZLFCeamlBcSoit3SJ6Xk3X9%2Bz8%3D&reserved=3D0 > > -------------------------------------------------------------------- > > > > Juniper Business Use Only > > > > Juniper Business Use Only > > > > Juniper Business Use Only > > > > -------------------------------------------------------------------- > > IETF IPv6 working group mailing list > > ipv6@ietf.org > > Administrative Requests: > > > https://urldefense.com/v3/__https://nam11.safelinks.protection.outlook.co= m/?url=3Dhttps*3A*2F*2Fwww__;JSUl!!NEt6yMaO-gk!VemUe-2PHPVnuVFBu0NeOKMAx3tu= lW_X0zDToehm7avqQhBqbE3FDykd8ujmRF-e$ > . > > ietf.org%2Fmailman%2Flistinfo%2Fipv6&data=3D04%7C01%7Clinda.dunbar%= 4 > > 0futurewei.com%7Ca73842e8d0e7473e591d08d8f2e43b9f%7C0fee8ff2a3b240189c > > 753a1d5591fedc%7C1%7C0%7C637526409383434704%7CUnknown%7CTWFpbGZsb3d8ey > > JWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C100 > > 0&sdata=3DqkqfI%2B3ZLo76yZFXDIxUxjfwyhA5MNJIqUwzUyTGXqc%3D&rese= r > > ved=3D0 > > -------------------------------------------------------------------- > > > > Juniper Business Use Only > > -------------------------------------------------------------------- > IETF IPv6 working group mailing list > ipv6@ietf.org > Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6 > -------------------------------------------------------------------- > --000000000000f23a2105bec13882 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Wed, 31 Mar 2021, 00:05 Jeffrey (Z= haohui) Zhang, <zzhang=3D40juniper.net@dmarc.ietf.org> = wrote:
Hi Linda,

You're not "leverage network control plane" - you're addi= ng complications to the network forwarding plane (and the control plane bec= ause you need to coordinate among the ingress routers) with that sticky-ser= vice-table with (Sticky Service ID, Flow Label. Sticky Egress address, Time= r) entries. The 5G/MEC control plane solution I mentioned does not need a U= E to query for a new address when its location changes.

The variation of that sticky-service-table based on (Sticky Service ID, UE = address, Sticky Egress address, Timer) is slightly better, because it could= be generalized to forwarding based on (source, destination), which is not = different from IP multicast (except that there is no replication). While th= at reduces the complication in the forwarding plane, it still has the scali= ng issue - you will still have a large table and you will still have to coo= rdinate among different ingress routers.

Granted, with the trendy AI, the coordination can be more precisely - the 5= G system can predict where a UE is moving to and push the corresponding ent= ry only to the new ingress router and ahead of time. If the UE address does= not change after its anchoring UPF changes, the coordination can be simple= r (otherwise the new ingress router needs to be able to associate the old a= nd new addresses).

Retaining the same UE address after relocation can be done, and it is docum= ented in " Solution #26: Persistent address allocation for mobile UEs = that need MEC access" of 3GPP SA2 TR 23.748. I did not get to defend i= t during the final stage of study but I still think it is a viable solution= (and preferred for application servers that do not deal well with client a= ddress changes) - would love to discuss that further with John on 3GPP SA2 = mailing list.

Having said all the above, I still believe it's the easiest and most sc= alable solution if the UEs are told by 5G/MEC control plane what true unica= st address of the server or load balancer to use.


Anycast to initially set things up, switch to unicast once that is done s= o there is no ongoing "anycast stickiness" required, and therefor= e no corresponding ongoing "anycast stickiness" state required in= the network forwarding plane.

I think the existing and coming multipath transport layer protocols = can facilitate that. Initial connection setup via anycast, once that is est= ablished, then establish 1 or more unicast sub-connections (subflows in MPT= CP) for ongoing communication, and abandon the anycast connection once the = first unicast sub-connection is established.

Anycast is used for setup, but not for ongoing communi= cation.


Regards,
Mark.





Jeffrey

-----Original Message-----
From: Linda Dunbar <linda.dunbar@futurewei.com>= ;
Sent: Monday, March 29, 2021 5:16 PM
To: Joel M. Halpern <jmh@joelhalpern.com>; Jeffrey (Z= haohui) Zhang <zzhang@juniper.net>; 'IPv6 List'= ; <ipv6@ietf.org>
Subject: RE: Questions/comments about draft-dunbar-6man-5g-edge-compute-sti= cky-service

[External Email. Be cautious of content]


Joel,

I am confused of your statement.

The problem I described is:
=C2=A0 =C2=A0 An end node (e.g. a drone) moves constantly. There are multip= le servers hosted in Edge DCs close to Cell towers to process requests from= the end node to achieve ultra-low latency.=C2=A0 It is not realistic for t= he end node to send DNS query every time it anchors to a new Cell Tower.
The suggested solution is to leverage network control plane to find the opt= imal server to serve the end node.

You stated that "it is not at all clear that is the best answer or eve= n a good one".=C2=A0 Why?

Linda


-----Original Message-----
From: Joel M. Halpern <jmh@joelhalpern.com>
Sent: Monday, March 29, 2021 1:56 PM
To: Linda Dunbar <linda.dunbar@futurewei.com>;= Jeffrey (Zhaohui) Zhang <zzhang@juniper.net>; 'IP= v6 List' <ipv6@ietf.org>
Subject: Re: Questions/comments about draft-dunbar-6man-5g-edge-compute-sti= cky-service

Linda, you are assserting that the answer is "to leverage the network = control plane".
As Jeffrey has pointed out, given the problem you have described, it is not= at all clear that is the best answer, or even a good one.

Yours,
Joel

On 3/29/2021 1:30 PM, Linda Dunbar wrote:
> Jeffrey,
>
> The draft is to leverage the network control plane to determine which = server is the most appropriate one for the device.
>
> You can definitely use the Controller to choose the optimal server. To=
> simplify the IP address allocation, the draft suggest using the Egress=
> router addresses as the proxy for reaching the desired server. We can<= br> > add a section to describe this scenario
>
> Linda
>
> -----Original Message-----
> From: Jeffrey (Zhaohui) Zhang <zzhang@juniper.net>= ;
> Sent: Monday, March 29, 2021 12:20 PM
> To: Linda Dunbar <linda.dunbar@futurewei.com= >; Vasilenko Eduard
> <vasilenko.eduard@huawei.com>; Kaippalli= malil John
> <john.kaippallimalil@futurewei.com>= ;; 'IPv6 List' <ipv6@ietf.org>
> Subject: RE: Questions/comments about
> draft-dunbar-6man-5g-edge-compute-sticky-service
>
> Since it is for "sticky service", you don't want to get = a new server address every time you move - unless the previous one is no lo= nger appropriate. That means it is best for a controller to determine which= one to use both initially and later when situation changes (when a UE relo= cates or server load situation changes), and that does not necessarily mean= it is always through DNS.
>
> Jeffrey
>
> -----Original Message-----
> From: Linda Dunbar <linda.dunbar@futurewei.com>
> Sent: Monday, March 29, 2021 1:15 PM
> To: Jeffrey (Zhaohui) Zhang <zzhang@juniper.net>;= Vasilenko Eduard
> <vasilenko.eduard@huawei.com>; Kaippalli= malil John
> <john.kaippallimalil@futurewei.com>= ;; 'IPv6 List' <ipv6@ietf.org>
> Subject: RE: Questions/comments about
> draft-dunbar-6man-5g-edge-compute-sticky-service
>
> [External Email. Be cautious of content]
>
>
> Jeffrey,
>
> The Devices are moving consistently, it is not reasonable to require t= hem to consistently query DNS for the "correct" non-ANYcast addre= ss .
>
> Linda
>
> -----Original Message-----
> From: Jeffrey (Zhaohui) Zhang <zzhang@juniper.net>= ;
> Sent: Monday, March 29, 2021 12:04 PM
> To: Linda Dunbar <linda.dunbar@futurewei.com= >; Vasilenko Eduard
> <vasilenko.eduard@huawei.com>; Kaippalli= malil John
> <john.kaippallimalil@futurewei.com>= ;; 'IPv6 List' <ipv6@ietf.org>
> Subject: RE: Questions/comments about
> draft-dunbar-6man-5g-edge-compute-sticky-service
>
> Even if you could get over the security/trust hurdle, using a controll= er to let the UEs know which unicast non-anycast address to use is a much s= impler/better solution.
>
> Jeffrey
>
> -----Original Message-----
> From: Linda Dunbar <linda.dunbar@futurewei.com>
> Sent: Monday, March 29, 2021 12:50 PM
> To: Vasilenko Eduard <vasilenko.eduard@huawei.c= om>; Jeffrey (Zhaohui)
> Zhang <zzhang@juniper.net>; Kaippallimalil John > <john.kaippallimalil@futurewei.com>= ;; 'IPv6 List' <ipv6@ietf.org>
> Subject: RE: Questions/comments about
> draft-dunbar-6man-5g-edge-compute-sticky-service
>
> [External Email. Be cautious of content]
>
>
> Ed,
>
> Yes, they are in one domain. Here is one example:
>
> 5G Connected devices, such as drones for fighting fires or natural dis= asters or robots in Industry 4.0=C2=A0 environments,=C2=A0 need ultra-low l= atency=C2=A0 responses from their analytic servers hosted in the Edge data = centers. To reach ultra-low latency, there are multiple servers hosting the= analytic functions in the Edge DCs.
> All the functions (including networking and analytics) and devices are= administrated by one operator.=C2=A0 Those functions might be provided by = different vendors, therefore needing interoperable solutions.
>
> Linda
>
> -----Original Message-----
> From: Vasilenko Eduard <vasilenko.eduard@huawei= .com>
> Sent: Monday, March 29, 2021 11:41 AM
> To: Linda Dunbar <linda.dunbar@futurewei.com= >; Jeffrey (Zhaohui) Zhang
> <zzhang@juniper.net>; Kaippallimalil John
> <john.kaippallimalil@futurewei.com>= ;; 'IPv6 List' <ipv6@ietf.org>
> Subject: RE: Questions/comments about
> draft-dunbar-6man-5g-edge-compute-sticky-service
>
> It could be the problem.
> Because all SR RFCs and drafts clearly say: only inside the domain. > Else could be a huge security risk. UE could not be trusted.
> Cross-domain security is the principal question that should be discuss= ed in SPRING first.
> Current SR architecture does not try to resolve it yet.
>
>
> Segment Routing in general and SRv6 in particular are claimed to be de= signed for Trusted environments only:
> - Segment routing architecture (RFC 8402) section 8
> - SRH - Segment Routing Header (RFC 8754) section 5
> - SRv6 Network Programming (draft-ietf-spring-srv6-network-programming= -25) section 9 SRH RFC is especially verbal how to filter-out any SR-relate= d information on the border of "SR domain".
> Ed/
> -----Original Message-----
> From: ipv6 [mailto:ipv6-bounces@ietf.org] On Behalf = Of Linda Dunbar
> Sent: Monday, March 29, 2021 7:09 PM
> To: Jeffrey (Zhaohui) Zhang <zzhang@juniper.net>;= Kaippallimalil John
> <john.kaippallimalil@futurewei.com>= ;; 'IPv6 List' <ipv6@ietf.org>
> Subject: RE: Questions/comments about
> draft-dunbar-6man-5g-edge-compute-sticky-service
>
> Jeffrey,
>
> We can definitely add the option of UE inserting SRH. I am just not su= re how many UEs or end devices will do those actions. If very few UEs can d= o this action, the solution itself is not useful. However, it doesn't h= urt for IETF to specify such a solution so that future IoT or 5G devices ca= n have a reference to do the actions.
>
> Another point, the number of Sticky Service is not large. The Ingress = routers are configured with the policies ( ACLs) to filter those flows.
>
> Linda
>
> -----Original Message-----
> From: Jeffrey (Zhaohui) Zhang <zzhang@juniper.net>= ;
> Sent: Monday, March 29, 2021 9:18 AM
> To: Linda Dunbar <linda.dunbar@futurewei.com= >; Kaippallimalil John
> <john.kaippallimalil@futurewei.com>= ;; 'IPv6 List' <ipv6@ietf.org>
> Subject: RE: Questions/comments about
> draft-dunbar-6man-5g-edge-compute-sticky-service
>
> Hi Linda,
>
> You proposed two ways of providing "sticky services" - when = a UE moves to a new location, the ingress router at that new location will = still route the packets of the same flow to the previous egress router. Tha= t flow cannot be identified by the destination address alone, since it is a= n anycast address that are shared by servers behind different egress router= s.
>
> Essentially, you're trying to turn the ingress router into a load = balancer, especially with your option #2 (section 5, "tunnel based&quo= t; solution). I don't think we want the routers to do that - while rout= ers can make use of 5-tuple for ECMP hashing, we don't want to make rou= ters more complicated and do forwarding based on a sticky-service-table wit= h (Sticky Service ID, Flow Label. Sticky Egress address, Timer) entries. It= 's not only complicated but also does not scale (we can discuss the sca= ling aspect wrt the flow labels separately).
>
> The variation of option #1 that I suggested would be better, if the fo= llowing were true:
>
> 1. The UE can insert an SRH
> 2. The ingress router can trust the SRH from the UEs
>
> In that case, it would be better for the UE to learn the egress router= via 5G/MEC control plane, instead of relying on the egress router to put t= hat into the DOH of every server->UE packets for sticky services and for= the UE to retrieve that information from each incoming sticky service pack= ets. One thing I learned is that the entire 5G system is very much heavy wi= th control/management plane and I would think it is a much better option to= provide that information to the UEs.
>
> On the other hand, once you go that way, the control plane can simply = provide the regular, non-anycast addresses of the servers instead of the eg= ress router address. Then, all the problems disappear and corresponding pro= posals are no longer needed, including the ones in draft-dunbar-idr-5g-edge= -compute-app-meta-data, and we only need existing simple routing functions.=
>
> Thanks.
>
> Jeffrey
>
> -----Original Message-----
> From: Linda Dunbar <linda.dunbar@futurewei.com>
> Sent: Saturday, March 27, 2021 10:45 PM
> To: Jeffrey (Zhaohui) Zhang <zzhang@juniper.net>;= Kaippallimalil John
> <john.kaippallimalil@futurewei.com>= ;; 'IPv6 List' <ipv6@ietf.org>
> Subject: RE: Questions/comments about
> draft-dunbar-6man-5g-edge-compute-sticky-service
>
> [External Email. Be cautious of content]
>
>
> Jeffrey,
>
> Thank you very much for the constructive comments.
> Replies are inserted below:
>
> -----Original Message-----
> From: Jeffrey (Zhaohui) Zhang <zzhang@juniper.net>= ;
> Sent: Friday, March 26, 2021 3:59 PM
> To: Linda Dunbar <linda.dunbar@futurewei.com= >; Kaippallimalil John
> <john.kaippallimalil@futurewei.com>= ;; 'IPv6 List' <ipv6@ietf.org>
> Subject: Questions/comments about
> draft-dunbar-6man-5g-edge-compute-sticky-service
>
> Hi Linda, John,
>
>=C2=A0 =C2=A0 =C2=A0When a UE (User Equipment) initiates application pa= ckets using the
>=C2=A0 =C2=A0 =C2=A0destination address from a DNS reply or from its ow= n cache, the
>=C2=A0 =C2=A0 =C2=A0packets from the UE are carried in a PDU session th= rough 5G Core
>=C2=A0 =C2=A0 =C2=A0[5GC] to the 5G UPF-PSA (User Plan Function - PDU S= ession Anchor).
>=C2=A0 =C2=A0 =C2=A0The UPF-PSA decapsulate the 5G GTP outer header and= forwards the
>=C2=A0 =C2=A0 =C2=A0packets from the UEs to the Ingress router of the E= dge Computing (EC)
>=C2=A0 =C2=A0 =C2=A0Local Data Network (LDN). The LDN for 5G EC, which = is the IP Networks
>=C2=A0 =C2=A0 =C2=A0from 5GC perspective, is responsible for forwarding= the packets to
>=C2=A0 =C2=A0 =C2=A0the intended destinations.
>
> A nit comment about "5G Core" above. When I first started le= arning 4G/5G It took me a while to realize the 3GPP "core network"= ; concept in vastly different from what IETF people are used to. It's n= ot about topology and now the "core network" functions are being = more and more distributed into edges. Therefore, in this context it may be = better to simply strike the "through 5G Core [5GC]" wording to re= duce the confusion to some readers.
>
> [Linda] That is very true. Removed the term per your suggestion. 5G Co= re refers to all the functions from Radio to UPF.
>
>=C2=A0 =C2=A0 1.3. Problem #1: ANYCAST in 5G EC Environment
>
>=C2=A0 =C2=A0 =C2=A0Increasingly, ANYCAST is used extensively by variou= s application
>=C2=A0 =C2=A0 =C2=A0providers and CDNs because it is possible to dynami= cally load balance
>=C2=A0 =C2=A0 =C2=A0across multiple locations of the same address based= on network
>=C2=A0 =C2=A0 =C2=A0conditions. BGP is an integral part in the way IP a= nycast usually
>=C2=A0 =C2=A0 =C2=A0functions. Within BGP routing there are multiple ro= utes for the same
>=C2=A0 =C2=A0 =C2=A0IP address which are pointing to different location= s.
>
> Not only BGP - but all IP routing protocols should work well with anyc= ast. My understanding is that BGP being integral part here is really that t= he data network here is likely realized by VPNs over the same transport net= work. Is that a correct understanding?
>
> [Linda] ANYCAST has traditionally been used for servers or loader bala= ncers that are placed in geographically diverse locations, so that BGP alon= e is enough for the traffic in one region to be forwarded to one server.=C2= =A0 But for the 5G Edge Computing where multiple Servers/load Balancers wit= h the same ANYCAST addresses are placed close proximity, IGP is needed.
>
> Of course, BGP does have flexibility in providing better/more control = of route selection than IGP does in the context of the companion draft-dunb= ar-idr-5g-edge-compute-app-meta-data.
> [Linda] Correct.
>
>=C2=A0 =C2=A0 =C2=A0But, having multiple locations for the same ANYCAST= address in 5G
>=C2=A0 =C2=A0 =C2=A0Edge Computing environment can be problematic becau= se all those edge
>=C2=A0 =C2=A0 =C2=A0computing Data Centers can be close in proximity.= =C2=A0 There might not be
>=C2=A0 =C2=A0 =C2=A0any difference in the routing cost to reach the App= lication Servers
>=C2=A0 =C2=A0 =C2=A0in different Edge DCs.=C2=A0 =C2=A0Same routing cos= t to multiple ANYCAST
>=C2=A0 =C2=A0 =C2=A0locations can cause packets from one flow to be for= warded to
>=C2=A0 =C2=A0 =C2=A0different locations, which can cause service glitch= es.
>
> As pointed out later in this same document, modern routers support &qu= ot;Flow Affinity" and should not cause packets of a flow on a specific= router to be forwarded to different locations. The real problem is when a = UE moves to a different location, the new router at that location may send = it to a different egress router. However, that is the "sticky service&= quot; problem described in 1.4.
> [Linda] Correct.
>
>>From draft-dunbar-idr-5g-edge-compute-app-meta-data, I understand t= hat on a specific router it needs to choose a location that can best serve = an application based on some non-routing factors. If 1.3 is really for that= purpose, it should be reworded accordingly. As I mentioned in an earlier e= mail, the two documents should better align on the problem descriptions. >
>=C2=A0 =C2=A0 =C2=A0Here is the overview of the End-Node based Sticky S= ervice solution:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0- Each ANYCAST Edge Computing server either = learns or is informed
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 of the unicast Sticky Egress address= (Section 3). The goal of
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 the network is to deliver packets be= longing to one flow to the
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 same Sticky Egress address for the A= NYCAST address. Section 4.1
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 describes how Edge Computing Servers= discover their
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 corresponding Sticky Egress unicast = addresses.
>=C2=A0 =C2=A0 =C2=A0 =C2=A0- When an Edge Computing server sends data p= ackets to a UE (or
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 client), it inserts the Sticky-Dst-S= ubTLV (described in Section
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 4.2) into the packets' Destinati= on Option Header.
>=C2=A0 =C2=A0 =C2=A0 =C2=A0- UE (or client) needs to copy the Destinati= on Option Header from
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 the received packet to the next pack= et's Destination Header if
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 the next packet belongs to the same = flow as the previous packet.
>
> I was really confused by "next packet". I finally realized y= ou may be referring to response packets from the UE to the server, and the = "same flow" should be "same service". Better wording is= needed here.
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0- If the following conditions are true, the = ingress router
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 encapsulates the packet from the UE = in a tunnel whose outer
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 destination address is set to the St= icky Egress Address
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 extracted from the packet's Stic= ky-Dst-SubTLV:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 o The destination of the pack= et from the UE side matches
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0with one of the = Sticky Service ACLs configured on the
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ingress router o= f the LDN,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 o the packet header has the D= estination Option present with
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Sticky-Dst-SubTL= V.
>
> Wouldn't it be better for the UE to put in an SRH with one SID for= the server address and set the DA to be the egress router address? That wa= y you don't need the ACL or the DOH (the Sticky-Dst-SubTLV=C2=A0 inform= ation in the DOH is not for consumption by the server anyway), and you don&= #39;t even need tunneling or BGP (unless VPN is used - but that's ortho= gonal to this). Existing SRv6 function takes care of it.
>
> [Linda] 3GPP has rejected using SRH in the 5G Core. We can think about= using them in the N6 interface.
>
> Also, the Sticky-Dst-SubTLV in DOH of the server->UE traffic would = be better renamed as "return waypoint" for more generic purpose.<= br> > [Linda]=C2=A0 that is interesting suggestion.
>
> 4.1. Sticky Egress Address Discovery
>
>=C2=A0 =C2=A0 =C2=A0To an App server with ANYCAST address, the Sticky E= gress address is
>=C2=A0 =C2=A0 =C2=A0same as its default Gateway address.
>
>=C2=A0 =C2=A0 =C2=A0To prevent malicious UEs (or clients) sending DDOS = attacks to routers
>=C2=A0 =C2=A0 =C2=A0within 5G EC LDN, e.g. the Sticky Egress address th= at is encoded in
>=C2=A0 =C2=A0 =C2=A0the Destination option header in the packets sent b= ack to the UEs (or
>=C2=A0 =C2=A0 =C2=A0clients), a proxy Sticky Egress address can be enco= ded in the
>=C2=A0 =C2=A0 =C2=A0Destination option header. The proxy Sticky Egress = address is only
>=C2=A0 =C2=A0 =C2=A0recognizable by the 5G EC LDN ingress nodes, i.e. t= he Ra and Rb in
>=C2=A0 =C2=A0 =C2=A0the Figure 1, but not routable in other networks. T= he LDN ingress
>=C2=A0 =C2=A0 =C2=A0routers can translate the proxy Sticky Egress to a = routable address
>=C2=A0 =C2=A0 =C2=A0for the Sticky Egress node after the source address= es of the packets
>=C2=A0 =C2=A0 =C2=A0are authenticated.
>
> Why is the 4.1 title called "... discovery"? Does not seem t= o be about "discovery".
> [Linda] it is about remembering which Egress router was used for the f= low. Should it be "Sticky Egress Memory"?
>
>=C2=A0 =C2=A04.3. Expected behavior at the UE
>=C2=A0 =C2=A0 =C2=A0...
>=C2=A0 =C2=A0 =C2=A0Section 4 describes the network layer processing if= UEs do not
>=C2=A0 =C2=A0 =C2=A0perform the steps described here.
>
> Should be "Section 5".
>
> [Linda] Thank you.
>
> 5. Tunnel based Sticky Service Solutions 5.1. Ingress and Egress
> Routers Processing Behavior
>
>=C2=A0 =C2=A0 =C2=A0The solution assumes that both ingress routers and = egress routers
>=C2=A0 =C2=A0 =C2=A0support at least one type of tunnel and are configu= red with ACLs to
>=C2=A0 =C2=A0 =C2=A0filter out packets whose destination or source addr= esses match with
>=C2=A0 =C2=A0 =C2=A0the Sticky Service Identifier. The solution also as= sumes there are
>=C2=A0 =C2=A0 =C2=A0only limited number of Sticky Services to be suppor= ted.
>=C2=A0 =C2=A0 =C2=A0An ingress router needs to build a Sticky-Service-T= able, with the
>=C2=A0 =C2=A0 =C2=A0minimum following attributes. The Sticky-Service-Ta= ble is initialized
>=C2=A0 =C2=A0 =C2=A0to be empty.
>=C2=A0 =C2=A0 =C2=A0 =C2=A0- Sticky Service ID
>=C2=A0 =C2=A0 =C2=A0 =C2=A0- Flow Label
>=C2=A0 =C2=A0 =C2=A0 =C2=A0- Sticky Egress address
>=C2=A0 =C2=A0 =C2=A0 =C2=A0- Timer
>
>=C2=A0 =C2=A0 =C2=A0Editor's Note:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0When a UE moves from one 5G Site to another,= the same UE will have
>=C2=A0 =C2=A0 =C2=A0 =C2=A0a new IP address. "Flow Label + Sticky = Service ID" stays the same
>=C2=A0 =C2=A0 =C2=A0 =C2=A0when a UE is anchored to a new PSA. Therefor= e, this solution use
>=C2=A0 =C2=A0 =C2=A0 =C2=A0"Flow Label + Sticky Service ID" t= o identify a sticky flow. Since
>=C2=A0 =C2=A0 =C2=A0 =C2=A0the chance of different UEs sending packets = to the same ANYCAST
>=C2=A0 =C2=A0 =C2=A0 =C2=A0address using the same Flow Label is very lo= w, it is with high
>=C2=A0 =C2=A0 =C2=A0 =C2=A0probability that "Flow Label + Sticky S= ervice ID" can uniquely
>=C2=A0 =C2=A0 =C2=A0 =C2=A0identify a flow. When multiple UEs using the= same Flow Label
>=C2=A0 =C2=A0 =C2=A0 =C2=A0sending packets to the same ANYCAST address,= the solution described
>=C2=A0 =C2=A0 =C2=A0 =C2=A0in this section will stick the flows to the = same ANYCAST server
>=C2=A0 =C2=A0 =C2=A0 =C2=A0attached to the Sticky Egress router. This b= ehavior doesn't cause
>=C2=A0 =C2=A0 =C2=A0 =C2=A0any harm.
>
> It seems that the same flow label is used for traffic of the same serv= ice in both directions. So who will assign the flow label?
> [Linda] The "flow label" from the IPv6 header should be mana= ged by the hosts & servers.
>
> If two UEs of the same service happen to use the same flow label, then= sticky service is not guaranteed. For example, initially they're ancho= red at different UPFs, and UE1 traffic is sent to egress router 1 while UE2= traffic is sent to egress router 2. When UE 1 relocates to the same UPF as= UE 2's, its traffic will be sent to egress node 2 because the same flo= w label is used.
>
> Therefore, there should be a central controller to assign flow labels = based on UE id, and the UE id is not based on IP address (since it could ch= ange).
> [Linda] Since the "Flow Label" is randomly generated (by Hos= t OS), the chance of two UEs reaching the same service having the same Flow= Label is very small.=C2=A0 We can explore the option of getting the Contro= l Plane involved.
>
>=C2=A0 =C2=A0 =C2=A0Note: since there are only small number of Sticky s= ervices, the
>=C2=A0 =C2=A0 =C2=A0Sticky-Service-Table is not very large.
>
> With the above understanding, the table could get large?
> [Linda]?
>
>=C2=A0 =C2=A0 =C2=A0When an ingress router receives a packet from a UE = matching with one
>=C2=A0 =C2=A0 =C2=A0of the Sticky Service ACLs and there is no entry in= the Sticky-
>=C2=A0 =C2=A0 =C2=A0Service-Table matching the Flow Label and the Stick= y Service ID, the
>=C2=A0 =C2=A0 =C2=A0ingress router considers the packet to be the first= packet of the
>=C2=A0 =C2=A0 =C2=A0flow. There is no need to sticking the packet to an= y location. The
>=C2=A0 =C2=A0 =C2=A0ingress router uses its own algorithm to select the= optimal egress
>=C2=A0 =C2=A0 =C2=A0node as the Sticky Egress address for the ANYCAST a= ddress,
>=C2=A0 =C2=A0 =C2=A0encapsulates the packet with a tunnel that is suppo= rted by the egress
>=C2=A0 =C2=A0 =C2=A0node. The tunnel's destination address is set t= o the egress node
>=C2=A0 =C2=A0 =C2=A0address.
>
> If a UE was using egress router 1 and it relocates to a new UPF, the n= ew ingress router will likely have no corresponding entry for it? What if t= he new ingress router pick egress router 2?
> It seems that the ingress routers need to pre-exchange entries in the = table?
> I see it's discussed later that the routers do exchange the inform= ation. It should be mentioned up front when the table is introduced.
> [Linda] Would Adding a reference be enough?
>
>=C2=A0 =C2=A0 =C2=A0When an ingress router receives a packet in a tunne= l from any egress
>=C2=A0 =C2=A0 =C2=A0router and the packet's source address matches = with a Sticky Service
>=C2=A0 =C2=A0 =C2=A0ID, the egress router address is set as the Sticky = Egress address for
>=C2=A0 =C2=A0 =C2=A0the Sticky Service ID. The ingress router adds the = entry of "Sticky-
>=C2=A0 =C2=A0 =C2=A0Service-ID + Flow Label + the associated Sticky Egr= ess address +
>=C2=A0 =C2=A0 =C2=A0Timer" to the Sticky-Service-Table if the entr= y doesn't exist yet in
>=C2=A0 =C2=A0 =C2=A0the table. If the entry exists, the ingress router = refreshes the
>=C2=A0 =C2=A0 =C2=A0Timer of the entry in the table.
>
>=C2=A0 =C2=A0 =C2=A0When the ingress router receives the subsequent pac= kets of a flow
>=C2=A0 =C2=A0 =C2=A0from the 5G side matching with an Sticky Service ID= and the Sticky-
>=C2=A0 =C2=A0 =C2=A0Service ID exists in the Sticky-Service-Table, the = ingress router
>=C2=A0 =C2=A0 =C2=A0uses the Sticky Egress address found in the Sticky-= Service-Table to
>=C2=A0 =C2=A0 =C2=A0encapsulate the packet and refresh the Timer of the= entry. If the
>=C2=A0 =C2=A0 =C2=A0Sticky-Service ID doesn't exist in the table, t= he ingress router
>=C2=A0 =C2=A0 =C2=A0considers the packet as the first packet of a flow.=
>
> The above is what leads me to believe that the flow label is the same = in both directions.
> [Linda] they don't have to be the same, do they?
>
>=C2=A0 =C2=A05.3. Scenario 2: With communication with 5G system
>=C2=A0 =C2=A0 =C2=A0...
>=C2=A0 =C2=A0 =C2=A0The ingress and egress router processing are the sa= me as described in
>=C2=A0 =C2=A0 =C2=A0Section 5.1 except a flow is now uniquely identifie= d by the "Sticky
>=C2=A0 =C2=A0 =C2=A0Service ID" + "UE address" instead o= f "Sticky Service ID" + "Flow
>=C2=A0 =C2=A0 =C2=A0Label".
>
> This confirms my earlier understanding for scenario 1 that "there= should be a central controller to assign flow labels based on UE id, and t= he UE id is not based on IP address (since it could change)" and that = the table could get large.
>
> Of course now for scenario 2, you're not using the flow label any = more. While the table only contains entries that this ingress router actual= ly need, the following are still true:
> - The table could still get large (if the number of attached UEs for > the sticky services is large)
> - On demand fetching of the table entry may not be fast enough
>
> Additionally, instead of "scenario", "option" or &= quot;solution" would be a better wording.
> [Linda] Good suggestion!
>
> More importantly, this stateful flow steering based on the additional = table is just too heavy and complicated. Why not simply have the UEs suppor= t SRH so that traffic will be routed via the desired egress router using st= andard SRv6 mechanism?
> [Linda] It is not realistic for UEs (your smart phone) to support SRH.=
>
> Jeffrey
>
>
> -----Original Message-----
> From: Jeffrey (Zhaohui) Zhang
> Sent: Thursday, March 25, 2021 3:46 PM
> To: Linda Dunbar <linda.dunbar@futurewei.com= >; Kaippallimalil John
> <john.kaippallimalil@FUTUREWEI.COM>= ;; IPv6 List <ipv6@ietf.org>;
> idr@ietf. org <idr@ietf.org>
> Subject: questions about
> draft-dunbar-idr-5g-edge-compute-app-meta-data and
> draft-dunbar-6man-5g-edge-compute-sticky-service
>
> Hi Linda, John,
>
> I have the following questions.
>
> The two related drafts listed the following three problems respectivel= y:
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 1.3. Problem#1: ANYCAST in 5G EC Environmen= t.............. 6
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 1.4. Problem #2: Unbalanced Anycast Distrib= ution due to UE Mobility.................................................. = 7
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 1.5. Problem 3: Application Server Relocati= on............. 7
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 1.2. Problem #1: ANYCAST in 5G EC Environme= nt.............. 4
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 1.3. Problem #2: sticking to original App S= erver........... 5
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 1.4. Problem #3: Application Server Relocat= ion............. 5
>
> Why is problem #2 different in the two drafts? Is it true that none of= the two drafts address problem #3?
> The idr draft talk about "soft anchoring" problem and soluti= on - how is that different from the "sticky service"?
>
> Thanks.
> Jeffrey
>
> Juniper Business Use Only
>
> Juniper Business Use Only
>
> Juniper Business Use Only
> -------------------------------------------------------------------- > IETF IPv6 working group mailing list
> ipv6@ietf.org
> Administrative Requests: https://urldefense.com/v3/__https://nam11.safelink= s.protection.outlook.com/?url=3Dhttps*3A*2F*2Furldefense.com*2Fv3*2F__https= *3A*2F*2Fnam11.safelinks.protection.outlook.com*2F*3Furl*3Dhttps*3A*2F*2Fur= ldefense.com*2Fv3*2F__https*3A*2F*2Fnam11.safelinks.protection.outlook.com*= 2F*3Furl*3Dhttps*3A*2F*2Fwww.ietf.org*2Fmailman*2Flistinfo*2Fipv6*26amp*3Bd= ata*3D04*7C01*7Clinda.dunbar*40futurewei.com*7C4209e8a9acae47b96d1808d8f2d1= 6b8d*7C0fee8ff2a3b240189c753a1d5591fedc*7C1*7C0*7C637526328578769822*7CUnkn= own*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXV= CI6Mn0*3D*7C1000*26amp*3Bsdata*3DA39pqHVUBFsssO3DLSqrTUtPpcXAr*2F8pi*2Bmw*2= BtIJNME*3D*26amp*3Breserved*3D0__*3BJSUlJSUlJSUlJSUlJSUlJSUlJSUlJQ!!NEt6yMa= O-gk!QWI34EOzIdgzRLkkdD3rdv_fn4CLHXnnMvDpOOeQB4ELlElbfawu6WXv0nbjgi-z*24*26= amp*3Bdata*3D04*7C01*7Clinda.dunbar*40futurewei.com*7C2b126cc5be8541bd43cc0= 8d8f2d4a7ab*7C0fee8ff2a3b240189c753a1d5591fedc*7C1*7C0*7C637526342463012256= *7CUnknown*7CTWFpbGZsb3
=C2=A0d8eyJWIjoiMC4wLjAwMDAiLCJQIjoi__;JSUlJSUlJSUlJSUqKioqKioqKioqKioqKioq= KioqKioqKioqKioqKioqKioqKioqKioqKioqKiUlJSoqKioqKioqKio!!NEt6yMaO-gk!VemUe-= 2PHPVnuVFBu0NeOKMAx3tulW_X0zDToehm7avqQhBqbE3FDykd8nYVAqHW$
>
> V2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000%26amp%3Bsdata%3DaOw1DkDc=
> Qu0*2FmMi6RfWlUyRcLB2jRcsbBAhcpoaX5yE*3D%26amp%3Breserved%3D0__%3BJSUl=
> JSUlJSUlJSUqKioqKiolJSUqKioqKioqKioqKiolJSUqKioqJSUlJSUlJSUlJSUlJSUlJS=
> UlJQ!!NEt6yMaO-gk!Q-hLtDzPuot4CQsvyUhfrEcNgHIIBEdRDT4RgyHgVCE1f5Vt6Dlv=
> zYC-o7693kZ1%24&amp;data=3D04%7C01%7Clinda.dunbar%40futurewei.com%7C67c9
> 07b1d8e643cc5a5108d8f2d6f104%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C=
> 0%7C637526352290503133%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJ=
> QIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=3DBe%= 2FU8Mh
> JT%2BA5b83e8ZLFCeamlBcSoit3SJ6Xk3X9%2Bz8%3D&amp;reserved=3D0
> -------------------------------------------------------------------- >
> Juniper Business Use Only
>
> Juniper Business Use Only
>
> Juniper Business Use Only
>
> -------------------------------------------------------------------- > IETF IPv6 working group mailing list
> ipv6@ietf.org
> Administrative Requests:
> https://urldefense.com/v3/__https://na= m11.safelinks.protection.outlook.com/?url=3Dhttps*3A*2F*2Fwww__;JSUl!!NEt6y= MaO-gk!VemUe-2PHPVnuVFBu0NeOKMAx3tulW_X0zDToehm7avqQhBqbE3FDykd8ujmRF-e$ .
> ietf.org%2Fmailman%2Flistinfo%2Fipv6&amp;data=3D04= %7C01%7Clinda.dunbar%4
> 0futurewei.com%7Ca73842e8d0e7473e591d08d8f2e43b9= f%7C0fee8ff2a3b240189c
> 753a1d5591fedc%7C1%7C0%7C637526409383434704%7CUnknown%7CTWFpbGZsb3d8ey=
> JWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C100=
> 0&amp;sdata=3DqkqfI%2B3ZLo76yZFXDIxUxjfwyhA5MNJIqUwzUyTGXqc%3D&= ;amp;reser
> ved=3D0
> -------------------------------------------------------------------- >

Juniper Business Use Only

--------------------------------------------------------------------
IETF IPv6 working group mailing list
ipv6@ietf.org
Administrative Requests: https://www= .ietf.org/mailman/listinfo/ipv6
--------------------------------------------------------------------
--000000000000f23a2105bec13882--