RE: Questions/comments about draft-dunbar-6man-5g-edge-compute-sticky-service

[Linda Dunbar <[e-mail]>]

Jeffrey, 

Thank you very much for the constructive comments. 
Replies are inserted below:

-----Original Message-----
From: Jeffrey (Zhaohui) Zhang <[e-mail]> 
Sent: Friday, March 26, 2021 3:59 PM
To: Linda Dunbar <[e-mail]>; Kaippallimalil John <[e-mail]>; 'IPv6 List' <[e-mail]>
Subject: Questions/comments about draft-dunbar-6man-5g-edge-compute-sticky-service

Hi Linda, John,

   When a UE (User Equipment) initiates application packets using the
   destination address from a DNS reply or from its own cache, the
   packets from the UE are carried in a PDU session through 5G Core
   [5GC] to the 5G UPF-PSA (User Plan Function - PDU Session Anchor).
   The UPF-PSA decapsulate the 5G GTP outer header and forwards the
   packets from the UEs to the Ingress router of the Edge Computing (EC)
   Local Data Network (LDN). The LDN for 5G EC, which is the IP Networks
   from 5GC perspective, is responsible for forwarding the packets to
   the intended destinations.

A nit comment about "5G Core" above. When I first started learning 4G/5G It took me a while to realize the 3GPP "core network" concept in vastly different from what IETF people are used to. It's not about topology and now the "core network" functions are being more and more distributed into edges. Therefore, in this context it may be better to simply strike the "through 5G Core [5GC]" wording to reduce the confusion to some readers.

[Linda] That is very true. Removed the term per your suggestion. 5G Core refers to all the functions from Radio to UPF.  

  1.3. Problem #1: ANYCAST in 5G EC Environment

   Increasingly, ANYCAST is used extensively by various application
   providers and CDNs because it is possible to dynamically load balance
   across multiple locations of the same address based on network
   conditions. BGP is an integral part in the way IP anycast usually
   functions. Within BGP routing there are multiple routes for the same
   IP address which are pointing to different locations.

Not only BGP - but all IP routing protocols should work well with anycast. My understanding is that BGP being integral part here is really that the data network here is likely realized by VPNs over the same transport network. Is that a correct understanding?

[Linda] ANYCAST has traditionally been used for servers or loader balancers that are placed in geographically diverse locations, so that BGP alone is enough for the traffic in one region to be forwarded to one server.  But for the 5G Edge Computing where multiple Servers/load Balancers with the same ANYCAST addresses are placed close proximity, IGP is needed. 

Of course, BGP does have flexibility in providing better/more control of route selection than IGP does in the context of the companion draft-dunbar-idr-5g-edge-compute-app-meta-data.
[Linda] Correct. 

   But, having multiple locations for the same ANYCAST address in 5G
   Edge Computing environment can be problematic because all those edge
   computing Data Centers can be close in proximity.  There might not be
   any difference in the routing cost to reach the Application Servers
   in different Edge DCs.   Same routing cost to multiple ANYCAST
   locations can cause packets from one flow to be forwarded to
   different locations, which can cause service glitches.

As pointed out later in this same document, modern routers support "Flow Affinity" and should not cause packets of a flow on a specific router to be forwarded to different locations. The real problem is when a UE moves to a different location, the new router at that location may send it to a different egress router. However, that is the "sticky service" problem described in 1.4.
[Linda] Correct. 

From draft-dunbar-idr-5g-edge-compute-app-meta-data, I understand that on a specific router it needs to choose a location that can best serve an application based on some non-routing factors. If 1.3 is really for that purpose, it should be reworded accordingly. As I mentioned in an earlier email, the two documents should better align on the problem descriptions.

   Here is the overview of the End-Node based Sticky Service solution:
     - Each ANYCAST Edge Computing server either learns or is informed
        of the unicast Sticky Egress address (Section 3). The goal of
        the network is to deliver packets belonging to one flow to the
        same Sticky Egress address for the ANYCAST address. Section 4.1
        describes how Edge Computing Servers discover their
        corresponding Sticky Egress unicast addresses.
     - When an Edge Computing server sends data packets to a UE (or
        client), it inserts the Sticky-Dst-SubTLV (described in Section
        4.2) into the packets' Destination Option Header.
     - UE (or client) needs to copy the Destination Option Header from
        the received packet to the next packet's Destination Header if
        the next packet belongs to the same flow as the previous packet.

I was really confused by "next packet". I finally realized you may be referring to response packets from the UE to the server, and the "same flow" should be "same service". Better wording is needed here.

     - If the following conditions are true, the ingress router
        encapsulates the packet from the UE in a tunnel whose outer
        destination address is set to the Sticky Egress Address
        extracted from the packet's Sticky-Dst-SubTLV:
          o The destination of the packet from the UE side matches
             with one of the Sticky Service ACLs configured on the
             ingress router of the LDN,
          o the packet header has the Destination Option present with
             Sticky-Dst-SubTLV.

Wouldn't it be better for the UE to put in an SRH with one SID for the server address and set the DA to be the egress router address? That way you don't need the ACL or the DOH (the Sticky-Dst-SubTLV  information in the DOH is not for consumption by the server anyway), and you don't even need tunneling or BGP (unless VPN is used - but that's orthogonal to this). Existing SRv6 function takes care of it.

[Linda] 3GPP has rejected using SRH in the 5G Core. We can think about using them in the N6 interface. 

Also, the Sticky-Dst-SubTLV in DOH of the server->UE traffic would be better renamed as "return waypoint" for more generic purpose.
[Linda]  that is interesting suggestion. 

4.1. Sticky Egress Address Discovery

   To an App server with ANYCAST address, the Sticky Egress address is
   same as its default Gateway address.

   To prevent malicious UEs (or clients) sending DDOS attacks to routers
   within 5G EC LDN, e.g. the Sticky Egress address that is encoded in
   the Destination option header in the packets sent back to the UEs (or
   clients), a proxy Sticky Egress address can be encoded in the
   Destination option header. The proxy Sticky Egress address is only
   recognizable by the 5G EC LDN ingress nodes, i.e. the Ra and Rb in
   the Figure 1, but not routable in other networks. The LDN ingress
   routers can translate the proxy Sticky Egress to a routable address
   for the Sticky Egress node after the source addresses of the packets
   are authenticated.

Why is the 4.1 title called "... discovery"? Does not seem to be about "discovery".
[Linda] it is about remembering which Egress router was used for the flow. Should it be "Sticky Egress Memory"? 

 4.3. Expected behavior at the UE
   ...
   Section 4 describes the network layer processing if UEs do not
   perform the steps described here.

Should be "Section 5".

[Linda] Thank you. 

5. Tunnel based Sticky Service Solutions 5.1. Ingress and Egress Routers Processing Behavior

   The solution assumes that both ingress routers and egress routers
   support at least one type of tunnel and are configured with ACLs to
   filter out packets whose destination or source addresses match with
   the Sticky Service Identifier. The solution also assumes there are
   only limited number of Sticky Services to be supported.
   An ingress router needs to build a Sticky-Service-Table, with the
   minimum following attributes. The Sticky-Service-Table is initialized
   to be empty.
     - Sticky Service ID
     - Flow Label
     - Sticky Egress address
     - Timer

   Editor's Note:
     When a UE moves from one 5G Site to another, the same UE will have
     a new IP address. "Flow Label + Sticky Service ID" stays the same
     when a UE is anchored to a new PSA. Therefore, this solution use
     "Flow Label + Sticky Service ID" to identify a sticky flow. Since
     the chance of different UEs sending packets to the same ANYCAST
     address using the same Flow Label is very low, it is with high
     probability that "Flow Label + Sticky Service ID" can uniquely
     identify a flow. When multiple UEs using the same Flow Label
     sending packets to the same ANYCAST address, the solution described
     in this section will stick the flows to the same ANYCAST server
     attached to the Sticky Egress router. This behavior doesn't cause
     any harm.

It seems that the same flow label is used for traffic of the same service in both directions. So who will assign the flow label?
[Linda] The "flow label" from the IPv6 header should be managed by the hosts & servers. 

If two UEs of the same service happen to use the same flow label, then sticky service is not guaranteed. For example, initially they're anchored at different UPFs, and UE1 traffic is sent to egress router 1 while UE2 traffic is sent to egress router 2. When UE 1 relocates to the same UPF as UE 2's, its traffic will be sent to egress node 2 because the same flow label is used.

Therefore, there should be a central controller to assign flow labels based on UE id, and the UE id is not based on IP address (since it could change).
[Linda] Since the "Flow Label" is randomly generated (by Host OS), the chance of two UEs reaching the same service having the same Flow Label is very small.  We can explore the option of getting the Control Plane involved. 

   Note: since there are only small number of Sticky services, the
   Sticky-Service-Table is not very large.

With the above understanding, the table could get large?
[Linda]? 

   When an ingress router receives a packet from a UE matching with one
   of the Sticky Service ACLs and there is no entry in the Sticky-
   Service-Table matching the Flow Label and the Sticky Service ID, the
   ingress router considers the packet to be the first packet of the
   flow. There is no need to sticking the packet to any location. The
   ingress router uses its own algorithm to select the optimal egress
   node as the Sticky Egress address for the ANYCAST address,
   encapsulates the packet with a tunnel that is supported by the egress
   node. The tunnel's destination address is set to the egress node
   address.

If a UE was using egress router 1 and it relocates to a new UPF, the new ingress router will likely have no corresponding entry for it? What if the new ingress router pick egress router 2?
It seems that the ingress routers need to pre-exchange entries in the table?
I see it's discussed later that the routers do exchange the information. It should be mentioned up front when the table is introduced.
[Linda] Would Adding a reference be enough? 

   When an ingress router receives a packet in a tunnel from any egress
   router and the packet's source address matches with a Sticky Service
   ID, the egress router address is set as the Sticky Egress address for
   the Sticky Service ID. The ingress router adds the entry of "Sticky-
   Service-ID + Flow Label + the associated Sticky Egress address +
   Timer" to the Sticky-Service-Table if the entry doesn't exist yet in
   the table. If the entry exists, the ingress router refreshes the
   Timer of the entry in the table.

   When the ingress router receives the subsequent packets of a flow
   from the 5G side matching with an Sticky Service ID and the Sticky-
   Service ID exists in the Sticky-Service-Table, the ingress router
   uses the Sticky Egress address found in the Sticky-Service-Table to
   encapsulate the packet and refresh the Timer of the entry. If the
   Sticky-Service ID doesn't exist in the table, the ingress router
   considers the packet as the first packet of a flow.

The above is what leads me to believe that the flow label is the same in both directions.
[Linda] they don't have to be the same, do they? 

 5.3. Scenario 2: With communication with 5G system
   ...
   The ingress and egress router processing are the same as described in
   Section 5.1 except a flow is now uniquely identified by the "Sticky
   Service ID" + "UE address" instead of "Sticky Service ID" + "Flow
   Label".

This confirms my earlier understanding for scenario 1 that "there should be a central controller to assign flow labels based on UE id, and the UE id is not based on IP address (since it could change)" and that the table could get large.

Of course now for scenario 2, you're not using the flow label any more. While the table only contains entries that this ingress router actually need, the following are still true:
- The table could still get large (if the number of attached UEs for the sticky services is large)
- On demand fetching of the table entry may not be fast enough

Additionally, instead of "scenario", "option" or "solution" would be a better wording.
[Linda] Good suggestion! 

More importantly, this stateful flow steering based on the additional table is just too heavy and complicated. Why not simply have the UEs support SRH so that traffic will be routed via the desired egress router using standard SRv6 mechanism?
[Linda] It is not realistic for UEs (your smart phone) to support SRH. 

Jeffrey


-----Original Message-----
From: Jeffrey (Zhaohui) Zhang
Sent: Thursday, March 25, 2021 3:46 PM
To: Linda Dunbar <[e-mail]>; Kaippallimalil John <[e-mail]>; IPv6 List <[e-mail]>; idr@ietf. org <[e-mail]>
Subject: questions about draft-dunbar-idr-5g-edge-compute-app-meta-data and draft-dunbar-6man-5g-edge-compute-sticky-service

Hi Linda, John,

I have the following questions.

The two related drafts listed the following three problems respectively:

      1.3. Problem#1: ANYCAST in 5G EC Environment.............. 6
      1.4. Problem #2: Unbalanced Anycast Distribution due to UE Mobility.................................................. 7
      1.5. Problem 3: Application Server Relocation............. 7

      1.2. Problem #1: ANYCAST in 5G EC Environment.............. 4
      1.3. Problem #2: sticking to original App Server........... 5
      1.4. Problem #3: Application Server Relocation............. 5

Why is problem #2 different in the two drafts? Is it true that none of the two drafts address problem #3?
The idr draft talk about "soft anchoring" problem and solution - how is that different from the "sticky service"?

Thanks.
Jeffrey

Juniper Business Use Only