Re: [alto] ALTO Draft ReCharter WG review

[Qin Wu <[e-mail]>]

Hi, Jensen and Qian:
Thanks for your clarification, , the interesting paper [2] you share,
I am wondering how ALTO server in your implementation get network info from BGP in each domain?
Do you configure ALTO server in each domain for consistent Cost Metric reporting? What if each ALTO server report different Cost metric in the multi-domain setting?

-Qin
发件人: Jensen Zhang [mailto:[e-mail]]
发送时间: 2021年3月5日 10:26
收件人: Qiao Xiang <[e-mail]>
抄送: Qin Wu <[e-mail]>; 刘鹏 <[e-mail]>; IETF ALTO <[e-mail]>
主题: Re: [alto] ALTO Draft ReCharter WG review

Hi Qin and Qiao,
Please see my comments inline.

On Thu, Mar 4, 2021 at 11:57 PM Qiao Xiang <[e-mail]<mailto:[e-mail]>> wrote:
Hi Qin,

Thank you so much for the feedback. Please see my responses inline.

On Thu, Mar 4, 2021 at 9:22 PM Qin Wu <[e-mail]<mailto:[e-mail]>> wrote:
Thanks Qiao for sharing your project on Unicorn and thought on multi-domain setting.
My impression in your implementation is each domain name and first ingress node in such domain will be carried in the ALTO response message.
First, for domain name, I do not believe we need that in the ALTO response message. Our setting here is each domain has an ALTO server, and the ALTO clients at the aggregator have separate connections to different ALTO servers. In this way, by maintaining a (domain, ALTO  server) mapping, the aggregator can differentiate responses from different domains.

Right, our implementation didn't carry each domain name in the ALTO message. The current ALTO protocol cannot carry on this info. And we also didn't do such an extension to ALTO. We just use a centralized aggregator (which can be considered as an ALTO client) to coordinate with the ALTO server of each domain. There is no communication between any two ALTO servers. The domain name may appear in the configuration message.

Second, for ingress node, the client needs to specify the ingress and the (srcIP, dstIP) pair in the query first so that the ALTO server knows what information to return. And I should also clarify that although we use ODL-ALTO for our system, but the path vector extension is not implemented exactly following the specification since it was not fully stabilized then.

Yes, and the ingress info only appears in the request message, not in the response message. We use the early design of the FCS extension [1] to do this. The request message should specify the flow-id of each flow.

[1] https://tools.ietf.org/html/draft-gao-alto-fcs-04


@Jensen is the core developer and can comment further on this, as well as the ODL implementation.

One thing I want to highlight is
Unicorn has already been deployed in several cities of USA in 2018 and implemented in the ODL open source.
I should clarify that this deployment is pre-production, and the demonstration scenario is at SC18 and SC19, where we are at the conference cities (Dallas and Denver), and orchestrate traffic from there back to a Caltech LHC site (we manually separate this site into two domains to create a 3-domain scenario for demonstration purpose.)

Quote from https://datatracker.ietf.org/doc/draft-xiang-alto-multidomain-analytics/
“
   The authors build an ALTO server on top of the OpenDaylight Software
   Defined Network controller.  The ALTO server collects the network
   state information from the OpenDaylight controller, e.g., topology,
   policy and traffic statistics, processes the collected information
   into resource abstraction, and sends the abstraction back to the ALTO
   client at the resource orchestrator.

The Unicorn framework has been deployed and
   demonstrated in small federation networks connecting Dallas, Texas,
   Los Angles, California, and Denver, Colorado at different
   conventions.  For example, in 2018, the federation in the
   demonstration is composed of three member networks.  Network 1 is in
   Dallas, Texas, and Network 2 and network 3 are in Los Angeles,
   California.  Network 1 is connected to network 2 through a layer-2
   WAN circuit with a 100 Gbps bandwidth, provisioned by several
   providers such as SCinet, CenturyLink and CENIC.  Network 1 is a
   temporal science network in the CMS experiment, while network 2 and 3
   are long-running CMS Tier-2 sites.
”
I believe your case require server to server communication for end-to-end interdomain routing.

Yes, the server-to-server communication for the e2e interdomain routing is helpful. However, we didn't use/extend ALTO to do this in our deployment. We used SFP (a BGP extension) [2] to do this. So any ALTO servers didn't communicate with each other. The limitation is that we must assume the ALTO server of each domain can provide consistent cost metrics.

[2] "SFP: Toward Interdomain Routing for SDN Networks", SIGCOMM 2018 Poster (https://dl.acm.org/doi/10.1145/3234200.3234207)

Thanks,
Jensen


Secondly, as for network information exposure in multi-domain setting, I think
1. 3GPP Network Exposure Function is a good example for network information exposure, but it is part of 5G core which enables data exchange between UE and application server and does not extend to other domain.
2. ZSM multi-domain network and service management can be another concrete example for multiple domain network information exposure which can be used to have a quick
response to network anomaly or reroute the traffic to the less congested path [https://www.etsi.org/deliver/etsi_gs/ZSM/001_099/002/01.01.01_60/gs_ZSM002v010101p.pdf].
3. PCE has similar design for multi-domain setting, which allows PCE to PCE communication.

Thank you again for the pointers. I'll take a look at ZSM and PCE soon.


Best
Qiao

-Qin
发件人: Qiao Xiang [mailto:[e-mail]<mailto:[e-mail]>]
发送时间: 2021年3月3日 0:18
收件人: 刘鹏 <[e-mail]<mailto:[e-mail]>>
抄送: Y. Richard Yang <[e-mail]<mailto:[e-mail]>>; IETF ALTO <[e-mail]<mailto:[e-mail]>>; Qin Wu <[e-mail]<mailto:[e-mail]>>
主题: Re: [alto] ALTO Draft ReCharter WG review

Hi Peng, Qin and Richard,

Very good discussion! Richard and I have been working with folks from CMS and ESNet (a large global multi-domain science network) to design network information exposure abstractions and mechanisms in multi-domain networks, with privacy requirements considered. The basic idea stems from the ALTO path-vector extension but goes beyond to take privacy into consideration. The following are some pointers.

[1] "Toward Fine-Grained, Privacy-Preserving, Efficient Multi-Domain Network Resource Discovery", IEEE JSAC, 2019. (https://ieeexplore.ieee.org/abstract/document/8756056)
[2] "Resource Orchestration for Multi-Domain, Exascale, Geo-Distributed Data Analytics", (https://datatracker.ietf.org/doc/draft-xiang-alto-multidomain-analytics/)

For the pointers above, the privacy requirement considered in this work is that the network information of multiple domains should be exposed to applications as a complete, unified aggregation, appearing as much as possible as from a single (virtual) network. We design a network information obfuscation mechanism so that the application is not able to associate any network resource bottleneck information to any domain, reducing the risk of exposing network vulnerability.

In addition, we also studied how to control the routing across multiple domains to achieve more flexible end-to-end interdomain routing. Essentially, we propose a mechanism that allows networks to expose their available interdomain routes, just as BGP looking glasses, so that applications can control them. In this setting, we consider the privacy setting where each network's BGP export policies are private, and design interesting algorithms for applications to select the best policy-compliant routes without knowing the export policies. The following is the pointer for this study:

[3] "Toward Optimal Software-Defined Interdomain Routing". INFOCOM 2020 (https://ieeexplore.ieee.org/abstract/document/9155486)

Above are our current efforts on extending ALTO to multi-domain settings. It would be great if we can know more about the industry efforts on network information exposure in multi-domain settings, and the privacy requirements of operators. This would be extremely helpful to push this extension forward! :-)



Best
Qiao

On Tue, Mar 2, 2021 at 1:14 PM 刘鹏 <[e-mail]<mailto:[e-mail]>> wrote:
Hi Richard,

Thank you. please see my reply inline below.


Peng Liu | 刘鹏
China Mobile | 移动研究院
mobile phone：[phone]
email:  [e-mail]<mailto:[e-mail]>

发件人: Y. Richard Yang<mailto:[e-mail]>
时间: 2021/03/02(星期二)07:36
收件人: 刘鹏<mailto:[e-mail]>;
抄送人: IETF ALTO<mailto:[e-mail]>;Qin Wu<mailto:[e-mail]>;
主题: Re: [alto] ALTO Draft ReCharter WG review
Dear Peng,

Thank you so much for the feedback. Please see below.

On Fri, Feb 26, 2021 at 9:23 PM 刘鹏 <[e-mail]<mailto:[e-mail]>> wrote:
Hi WG,

Here are some considerations of recharter:
I believe that the multi domain problem is worthy of attention.

It is good info.

At present, operators also research in it, which may involve guaranteeing end-to-end network service in the future, such as delay, bandwidth, etc. There are some researches on cross domain deterministic network in the industry, which need some support from management and control plane.

 Do you want to share some pointers?

[Peng] As Qin said, it is hard to collect information across network borders.
Just taking deterministic network as an example, it is hard to applying synchronization, unified forwarding strategy in multi domain, so there are some works need to be done with management plane. Due to the large scale and multi domains or operators, the management system may be distributed.
A potential way is to consider negotiating the forwarding time of each domain in advance and carrying time stamp in the message to control the forwarding path of each domain. While it needs some agreements like contracts to prevent one party from tampering with and denying the management content.
Beside this, there may be others use case. I'm not sure if Alto servers are willing to do those work, but it may be helpful to collect or configure some key information.

Who is the provider of Alto service is related to the deployment and cooperation mode. It may be difficult for operators to give too much detailed network information now. If the Alto service belongs to the operator, it may be used to help manage its own network. If Alto service belong to non operators, I think the issue of how to cooperate needs further discussion.

It looks that you want to consider both modes: multidomains but single operator (i.e., intra-cooperation) and multidomains and multiple operators. Regardless, I agree that it is important for the work to clarify on the privacy requirements.

[Peng] Yes, agree.

Richard



Regards,
Peng

Peng Liu | 刘鹏
China Mobile | 移动研究院
mobile phone：[phone]
email:  [e-mail]<mailto:[e-mail]>

发件人: Qin Wu<mailto:[e-mail]>
时间: 2021/02/22(星期一)21:45
收件人: IETF ALTO<mailto:[e-mail]>;
抄送人: alto-chairs<mailto:[e-mail]>;alto-ads<mailto:[e-mail]>;
主题: [alto] ALTO Draft ReCharter WG review
Hi, :
We have requested one hour session for ALTO WG meeting in the upcoming IETF 110, which is arranged on Friday, March 12, 14:30-15:30(UTC).
The goal is to boil down ALTO recharter and have consensus on charter contents in IETF 110.
To get this goal, an updated inline draft charter text for ALTO has just been posted to this list,

This charter has received a couple of rounds of informal review from WG members, chairs and our Ads from brief to deep thorough, 5 new chartered items have been listed.
We would like to solicit feedback on these new chartered items and your use case, deployment, idea corresponding to these new chartered items.
Sharing your past deployment story will also be appreciated.

============================================================================================
The ALTO working group was established in 2008 to devise a request/response protocol to allow a host to benefit from a server that is more cognizant of the network infrastructure than the host is.

The working group has developed an HTTP-based protocol and recent work has reported large-scale deployment of ALTO based solutions supporting applications such as content distribution networks (CDN).

ALTO is now proposed as a component for cloud-based interactive applications, large-scale data analytics, multi-cloud SD-WAN deployment, and distributed
computing. In all these cases, exposing network information such as abstract topologies and network function deployment location helps applications.

To support these emerging uses, extensions are needed, and additional functional and architectural features need to be considered as follows:

o Protocol extensions to support a richer and extensible set of policy attributes in ALTO information update request and response. Such policy attributes may indicate information dependency (e.g., ALTO path-cost/QoS properties with dependency on real-time network  indications), optimization criteria (e.g., lowest latency/throughput network performance objective), and constraints (e.g., relaxation bound of optimization criteria, domain or network node to be traversed, diversity and redundancy of paths).

o Protocol extensions for facilitating operational automation tasks and improving transport efficiency. In particular, extensions to provide "pub/sub" mechanisms to allow the client to request and receive a diverse types (such as event-triggered/sporadic, continuous), continuous, customized feed of publisher-generated information. Efforts developed in other working groups such as MQTT Publish / Subscribe Architecture, WebSub, Subscription to YANG Notifications will be considered, and issues such as scalability (e.g., using unicast or broadcast/multicast, and periodicity of object updates) should be considered.

o The working group will investigate the configuration, management, and operation of ALTO systems and may develop suitable data models.

o Extensions to ALTO services to support multi-domain settings. ALTO is currently specified for a single ALTO server in a single administrative domain, but a network may consist of
multiple domains and the potential information sources may not be limited to a certain domain. The working group will investigate extending the ALTO framework to (1) specify multi-ALTO-server protocol flow and usage guidelines when an ALTO service involves network paths spanning multiple domains with multiple ALTO servers, and (2) extend or introduce ALTO
services allowing east-west interfaces for multiple ALTO server integration and collaboration. The specifications and extensions should use existing services whenever possible. The specifications and extensions should consider realistic complexities including incremental deployment, dynamicity, and security issues such as access control, authorization (e.g., an ALTO server provides information for a network that the server has no authorization), and privacy protection in multi-domain settings.

o The working group will update RFC 7971 to provide operational considerations for recent protocol extensions (e.g., cost calendar, unified properties, and path vector) and new extensions that the WG develops. New considerations will include decisions about the set of information resources (e.g., what metrics to use), notification of changes either in proactive or reactive mode (e.g., pull the backend, or trigger just-in-time measurements), aggregation/processing of the collected information  (e.g., compute information and network information )according to the clients’ requests, and integration with new transport mechanisms (e.g., HTTP/2 and HTTP/3).

When the WG considers standardizing information that the ALTO server could provide, the following criteria are important
to ensure real feasibility:

- Can the ALTO server realistically provide (measure or derive) that information?

- Is it information that the ALTO client cannot find easily some other way?

- Is the distribution of the information allowed by the operator of the network? Does the exposure of the information introduce privacy and information leakage concerns?

Issues related to the specific content exchanged in systems that make use of ALTO are excluded from the WG's scope, as is the issue of dealing with enforcing the legality of the content. The WG will also not propose standards on how congestion is signaled, remediated, or avoided.

-Qin Wu (on behalf of chairs)
_______________________________________________
alto mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/alto


--
--
 =====================================
| Y. Richard Yang <[e-mail]<mailto:[e-mail]>>   |
| Professor of Computer Science       |
| http://www.cs.yale.edu/~yry/        |
 =====================================
_______________________________________________
alto mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/alto


--
Qiao Xiang
Professor,
School of Informatics,
Xiamen University


--
Qiao Xiang
Associate Research Scientist,
Department of Computer Science,
Yale University
_______________________________________________
alto mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/alto