Re: [Idr] some comments about the draft-ietf-idr-5g-edge-service-metadata-14.txt

[Linda Dunbar <[e-mail]>]

Chang Wang,

Thank you very much for the comments and the suggestions.
Please see below of detailed resolutions to your comments.

Linda

From: linchangwang <[e-mail]>
Sent: Tuesday, December 19, 2023 8:21 AM
To: Linda Dunbar <[e-mail]>; [e-mail]
Cc: idr@ietf. org <[e-mail]>
Subject: [Idr] some comments about the draft-ietf-idr-5g-edge-service-metadata-14.txt

Hi Linda,

  Here are some comments on the draft-ietf-idr-5g-edge-service-metadata-14.txt for your consideration.


1.        1. BGP RR :

2.         It is recommended to add that in BGP RR networking, the next hop should not be changed, and the add-path feature should be used to advertise multiple routes with different next hops and metrics for the same prefix. The document does not mention the scenario of having BGP Route Reflectors (RRs) in the network. When BGP RRs are present, only the best path is reflected, and non-best paths are not sent. However, for anycast routes in such a service, it is necessary for BGP RRs to send all site routes, for example, by using the add-path feature to send the routes.

3.          BGP RR (Route Reflector) only advertises a single route, and in this document, it is required to advertise routes with the same prefix but different metrics and next hops. It is recommended to provide additional clarification in the document. The BGP RR mentioned here is different from the BGP RR controller mentioned in the document.



[Linda] RR is stated in the last paragraph of the Introduction:



“The extension is targeted for a single domain with RR controlling the propagation of the BGP UPDATE.  The edge service Metadata Path Attribute is only attached to the low latency services (routes) hosted in the 5G edge cloud sites, which are only a small subset of services initiated from UEs, not for UEs accessing many internet sites.”



Add-Path is for adding multiple paths to one NextHop. This draft is about selecting ONE optimal path to one of multiple nextHops.
4.
5.         2. Regarding the description of the Capacity Availability Index Metadata in section 4.3, there is a question:
6.            In the case where there are multiple servers within the same site, how are the updated site utilization rates from multiple servers handled?
[Linda] This has changed in v15. Since several people expressed confusion of the naming, the name has been changed to “Site Physical Availability Index Metadata” which indicates the percentage of impact on a group of routes associated with a common physical characteristic, for example, a pod, a row of server racks, a floor, or an entire DC.


7.         3. For Figure 3: Capacity Availability Index Sub-TLV, it is suggested to add a length field to the TLV.
8.             This TLV is associated with each prefix announcement, indicating the site ID associated with each prefix. Subsequently, we would like to notify fast switchover based on the percentage of available site IDs for a route. This logic is equivalent to adjusting the capacity of a site when the Route-Flag I is set to 0.
9.           However, this logic has some issues. We suggest announcing a single route, such as the minimum service IP address, to address this issue.


[Linda] this specific one has fixed Length. Please see the latest revision (-15).

10.
11.     4.There is a question about the Service Delay Prediction Sub-TLV.
12.        Is the delay value the actual delay, or is it calculated through Service Delay Prediction Based on Load Measurement?
13.        The delay value is typically represented more accurately by using minimum, maximum, and average to denote the delay.
[Linda] Please see the attached email proposing the actual delay Sub-TLV.


14.     5. Concerning Figure 5: Service Delay Prediction Raw Measurements Sub-TLV,
15.         the "total number of bytes to Edge service" is defined as 4 bytes here, but the range of values may not be sufficient.
16.         It should be defined as an 8-byte field.
17.         This type of statistic, which is essentially a packet statistic, is recommended to be converted into a universal cost value before being sent to the head-end.
[Linda] You are correct. Some people suggested to remove this sub-TLV.


18.      6. How is the preference Index used? For ECMP routes, are they first selected based on the Preference Index?
[Linda] Different services might have different preference index values configured for the same site. For
example, Service-A requires high computing power, Service-B requires high bandwidth among
its microservices, and Service-C requires high volume storage capacity. For a DC with relatively
low storage capacity but high bisectional bandwidth, its preference index value for Service-B is
higher and lower for Service-C. Site Preference Index can also be used to achieve stickiness for
some services.


It is out of the scope of this document how the preference index is determined or configured.
19.      7. In section 5.3, Figure 6: Metadata Influenced Decision：
20.       BGP should prefer multiple paths for the prefix, and there should not be a difference in the description of BGP's best path selection and the forwarding plane's best path    selection. BGP needs to prefer all these multiple next hops in order to update them to the forwarding plane. Through metadata data, BGP can update the load-sharing   ratios of these preferred next hops. Therefore, the update of next hop weights should also be through BGP to RIB and then to update the next hop weights in the forwarding plane.
[Linda] The ingress node BGP processing results in one optimal path  with consideration of many factors. Section 5.3 is for the purpose.

Best Regards
Changwang
-------------------------------------------------------------------------------------------------------------------------------------
本邮件及其附件含有新华三集团的保密信息，仅限于发送给上面地址中列出
的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、
或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本
邮件！
This e-mail and its attachments contain confidential information from New H3C, which is
intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
by phone or email immediately and delete it!

--- Begin Message ---

Cheng,

Do you think adding the following subsection to indicate the Resource Utilization for a Service (prefix) can address your comments?

4.1.2. Service Delay caused by Resource Utilization Anomalies.
One of the challenges for getting ultra-low latency services hosted in edge data centers is the extra service delays caused by higher-than-normal resource utilization of the hosting environment. Though out of the scope of this document, many cloud services employ queue-based systems for handling incoming requests and monitoring the length of queues to get insights into the workload that can highlight delays in processing requests when queues become unusually long. With advanced monitoring and sophisticated algorithms, some edge data centers can gauge or estimate the potential service delay caused by physical resource utilization. For those edge data centers, the following subTLV can be included in the Metadata Path Attribute to indicate the estimated service delay for the prefix and the overall resource utilization for the environment that hosts the prefix.
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Resource Utilization Sub-Type |   Length      |T| Reserved    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                  Resource utilization                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Delay caused by resource utilization anomalies               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+



  1.  Type (2 byte):
     *   Identifier for the type of subTLV.
  2.  Length (1 byte):
     *   Number of bytes in the Value field.
  3.  Value: including the following information:
     *   Resource Utilization Percentage (4 bytes):
        *   A percentage value indicating the level of resource utilization that triggered the delay.
     *   Delay caused by resource utilization anomalies (4 bytes):
        *   The duration of the delay caused by resource utilization anomalies, measured in milliseconds.

4.     Flag T: if set to 1, indicating both Resource Utilization Percentage and Delay caused by resource utilization anomalies are included. If set to 0, only the Resource Utilization Percentage is included.

Though out of the scope of this document, here are some exemplary practices that can be utilized to gauge or estimate service delay caused by resource utilization anomalies. Those practices are mainly to demonstrate that it is feasible to estimate service delay metrics.
Baseline Comparison:
Establishing a baseline for normal resource utilization is essential. Deviations from this baseline can be a strong indicator of abnormal behavior. Regularly comparing current resource utilization with historical data helps identify trends and anomalies.
Real-Time Monitoring:
Implementing real-time monitoring solutions enables prompt detection of spikes or sustained increases in resource utilization. Automated alerts can notify administrators when predefined thresholds are breached, facilitating swift intervention.
Machine Learning Algorithms:
Leveraging machine learning algorithms allows for the creation of predictive models that can anticipate resource utilization patterns. By training these models on historical data, the system can proactively identify potential service delays before they impact user experience.


Linda


From: Cheng Li <[e-mail]>
Sent: Tuesday, December 19, 2023 8:42 AM
To: Linda Dunbar <[e-mail]>; [e-mail]
Cc: idr@ietf. org <[e-mail]>
Subject: RE: Comments of draft-ietf-idr-5g-edge-service-metadata

Thank you for your quick reply! We are almost on the same page.  Please see my reply inline.

Respect,
Cheng


From: Linda Dunbar <[e-mail]<mailto:[e-mail]>>
Sent: Tuesday, December 19, 2023 12:33 AM
To: Cheng Li <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]>
Cc: idr@ietf. org <[e-mail]<mailto:[e-mail]>>
Subject: RE: Comments of draft-ietf-idr-5g-edge-service-metadata

Cheng,

Please see below for the resolutions. I snipped the text that you have agreed the changes to make it easier for the discussion.

Linda

From: Cheng Li <[e-mail]<mailto:[e-mail]>>
Sent: Thursday, December 14, 2023 10:50 AM
To: Linda Dunbar <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]>
Cc: idr@ietf. org <[e-mail]<mailto:[e-mail]>>
Subject: RE: Comments of draft-ietf-idr-5g-edge-service-metadata

Hi Linda,

Thank you for your reply, please see my reply inline.

Respect,
Cheng


From: Linda Dunbar <[e-mail]<mailto:[e-mail]>>
Sent: Tuesday, December 12, 2023 11:38 PM
To: Cheng Li <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]>
Cc: idr@ietf. org <[e-mail]<mailto:[e-mail]>>
Subject: RE: Comments of draft-ietf-idr-5g-edge-service-metadata

Cheng,

Thank you very much for the comments.
Please see below of the detailed resolutions to your comments.

Linda

From: Cheng Li <c.l=[e-mail]<mailto:c.l=[e-mail]>>
Sent: Monday, December 11, 2023 2:28 PM
To: [e-mail]<mailto:[e-mail]>
Cc: idr@ietf. org <[e-mail]<mailto:[e-mail]>>
Subject: Comments of draft-ietf-idr-5g-edge-service-metadata

Hi authors,

<sniped >

1.     Why the value range is 0-100? It is a 32 bit value, why no let it be 0-2**32-1? A deployment can choose a range as they want. The Ingress node only care about the value of one site is bigger or smaller than others, isn’t it?

I may suggest to make it open, and let deployment to choose the range they like. Otherwise, we still need to check the value before processing it.
[Linda] Sure, it can change to 0-2**32-1. Then, deployment probably can use a range to compare.
How about changing to the following:
Preference Index value: 1-2^31; the higher the value, the more preference the site.
[Cheng] why 2^31? The maximum is  2^32 -1? So we may not need to describe it? Just say that bigger value means higher priority? Then people can choose any value in 32 bits. Unless you want to make 0 as reserved for some reasons.

[Linda] Sure, it can be changed to  2^32 -1
[Cheng]thanks.

<snipped>

BTW, I will recommend to add a new sub-TLV to describe the available capacity of resource of a service? Because from the aspect of a service, it care how much resource it can access.
[Linda] is the new subTLV for a specific prefix? Or a group of prefixes?
[Cheng] for the prefix that it is associated to? It can be used for many prefixes if we want and BGP allow to do so. What do you think?
[Linda] I think per prefix is more reasonable as BGP UPDATE is to notify the information about a prefix. One prefix can represent a group of services.
[Cheng]Agree.


The whole resource of the datacenter or the site is important, but they are not all for this service.
We can use a unified value to present the capacity of the resource, while the unit is defined by the service itself, so that it can be standardized for many services instead of one or several ones.

# Service Delay Predication Value
I see this info can even be used for predict the load, or I misunderstand something? I will recommend to add a new sub-TLV of load or utilization independently.

[Linda] Is the loadUtilization per prefix?  Is it same as the earlier “resource of the service? Or is it different?
[Cheng] it is for per service associated with per prefix. The above may discuss the total resource of the service, and this is the load of it. Using this load(percentage) x capacity can get the real-time available resource?
[Linda] BGP UPDATE is about a route (a.k.a. a service) running on a server. The load utilization is about the server which can host many routes. So, the metrics is more about the underlay physical resource utilization to support the service. How does the egress know the information?
[Cheng]some agency will collect this information and send it to the egress router. This specific method is out of scope of this draft I think

The reason is a site may have their own means to measure the load/utilization of a service, and this percentage times capacity can tell us how much resource are available now.
This is important for the ingress to know if it can send more service packets to a site. We may not need to care about how the load is measured, but only need to distribute it to the ingress node.



The original delay is required for the ingress node to calculate the lowest delay from it to the instance inside a site.
The network delay can be obtained from existing means. The delay from the egress router attaching to the site and the instance inside the site can be measured by existing means as well, like TWAMP.
The Internal processing delay can be measured by existing tools. We may only care about the data value instead of how it is generated.
[Linda] correct.
[Cheng]Thanks, jitter or other type may be included as well. Let see how to define the flag. If we have different type of delay info, will they be included in the same time?
[Linda] Yes. Maybe using another bit in the Reserved ?
[Cheng]Yes, but if different flags are set in the same time, the order of the optional fields should be defined clearly.

Similar comments may be proposed in the past, sorry I did not follow to the discussion in time.

Hope my comments make sense.

Thanks and respect,
Cheng

--- End Message ---