Re: [alto] New Version Notification for draft-lachosrothenberg-alto-brokermdo-01.txt

"Y. Richard Yang" <> Mon, 09 July 2018 21:14 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 2D0EB1274D0 for <>; Mon, 9 Jul 2018 14:14:12 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.419
X-Spam-Status: No, score=-1.419 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, FREEMAIL_FORGED_FROMDOMAIN=0.249, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.25, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, WEIRD_PORT=0.001] autolearn=no autolearn_force=no
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id u8n2x8AySZnu for <>; Mon, 9 Jul 2018 14:14:07 -0700 (PDT)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 215A7130DF0 for <>; Mon, 9 Jul 2018 14:14:06 -0700 (PDT)
Received: by with SMTP id g6-v6so5730724lfb.11 for <>; Mon, 09 Jul 2018 14:14:06 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=QRWkvFgAz8yFkcwhteWooDiG0/hRN+UcmRV0jj+6oWk=; b=aEWuE9yjNjYvUcoDA1JCFjmgFbedvkbAZfVSl0AVeBEga/0URx8ytVB+HIugP2NkyU GXRElyprp+c+9QECFWEMHLtCUFPUf3OGeEJleSfzoIeYqbJjxqRuXfCgZk3R+R2TzVGT YtM1+nL7vcBpcvT4rXaivPpom54S0sRmlJ/OjXfixmDqH0XxlgjVW4Bxk7t1EjVOtRpE 05e9L4IVZOsusGYmF66pwRDm9tcGmZJhrqAhgPukM5uD44yxSNEuTQUtIoVLGD9U4GEa OclTww+4Wy/YIoZ09V9to1X/chFvQ9Tq79Mxkx2Vz+yKQuBjmGHMzk83tWoAOzBqq4wI fPAg==
X-Gm-Message-State: APt69E3ARMa3ufedgb6w7Tb8nKFoH4TY2jF5x9CWldRrRSPy1OxojT+l doTPSxBDJ8SvN8ilTTfQxyYAYZYL0pUiG6/qskg=
X-Google-Smtp-Source: AAOMgpfq4gNPd4eM8AwTsa5s/Xu/6EsHY9G4sNMNMyBOVUGZlOOPvDjOxPE9TfoMfOZyB1J9/k9FMq0X1T9v2fYmzHI=
X-Received: by 2002:a19:2c4f:: with SMTP id s76-v6mr807078lfs.25.1531170844122; Mon, 09 Jul 2018 14:14:04 -0700 (PDT)
MIME-Version: 1.0
References: <> <> <>
In-Reply-To: <>
From: "Y. Richard Yang" <>
Date: Mon, 9 Jul 2018 17:13:52 -0400
Message-ID: <>
To: Danny Alex Lachos Perez <>
Content-Type: multipart/alternative; boundary="0000000000004123870570977e58"
Archived-At: <>
Subject: Re: [alto] New Version Notification for draft-lachosrothenberg-alto-brokermdo-01.txt
X-Mailman-Version: 2.1.27
Precedence: list
List-Id: "Application-Layer Traffic Optimization \(alto\) WG mailing list" <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Mon, 09 Jul 2018 21:14:12 -0000

Danny, Christian,

Very interesting work. I finished a review of Sec. 6, and below is
the second part of my review.


6.  Proposed Approach

   The primary design goal for ALTO-based Broker-assisted Multi-domain
   Orchestration is to discover resource and topology information from
   different administrative domains involved in the federation, while
   also safeguarding the privacy and autonomy of every domain.

     [yry] Connect to all requirements above?

   In the architectural proposal shown in Figure 1, a broker component
   is conceived to be working as coordinator of a set of MdOs, whose key

     [yry] coordinator -> a coordinator; whose it is not clear righ away
     what "whose" refers to, MdOs or broker?

   components are: the Inter-domain Resource (IdR), the Inter-domain
   Topology (IdT) and the ALTO Server.

                              BROKER COMPONENT
      [yry] COMPONENTS

              |                                            |
              |             +-----------------+            |
              |             |                 |            |
          XXXXXXXXXXXXXXXXXXX ALTO SERVER(s)  |            |
          X   |             |                 +            |
          X   |             +----------------+\            |
          X   |            /                   \           |
          X   |           /                     \          |
          X   |  +------+/+-------+     +---------------+  |
          X   |  | Inter-domain   |     | Inter-domain  |  |
          X   |  | Topology (IdT) |     | Resource (IdR)|  |
          X   |  +-^------^-------+     +---^-------^---+  |
          X   |    |      |                 |       |      |
          X   +--------------------------------------------+
          X        |      |                 |       |
          X        |      |                 |       |
       +--X--------+---+--------------------+       |
       |               |  |                         |
       |               |  +------------+------------+---+
       |     MdO1      |               |                |
       |               +<------------->+                +---+
       +---------------+               |      MdO2      |   |
                                       |                |   |
                                       +-+--------------+   |
                                         |                  |
   Legend:                               |      MdO(n)      |
   XXX ALTO Protocol                     +------------------+

       Figure 1: Broker-assisted Multi-operator Network Architecture

    [yry] Does it help to label all, or key edges, not only the single
    XXX, on which protocol (protocols) will be used? You mentioned
    BGP/BGP-LS, ...

6.1.  Inter-domain Resource (IdR) Component

   It creates a hierarchical database that contains inter-domain
   resource information such as resource availability (i.e., CPU,
   memory, and storage), Virtual Network Functions (VNFs) and Physical
   Network Functions (PNFs) supported and Service Access Points (SAPs)
   to access those resources.  UNIFY [UNIFY.NFFG], TOSCA [TOSCA], ETSI-

     [yry] It can be ambiguous what Service Access Points refer to,
     only VNFs/PNFs or all? The idea of a hierarchical database appears to
    be not fully specified.

   NFV [ETSI-NFV-MANO], among other data models can be used to create
   the interface between IdR and MdOs.

      [yry] IdR not defined yet---appeared only in the section title.

6.2.  Inter-domain Topology (IdT) Component

   A hierarchical TED (Traffic Engineering Database) that contains
   inter-domain network topology information including additional key
   parameters (e.g., throughput and latency of links).  This information
   can be retrieved from each MdO through BGP-LS or REST interfaces.

6.3.  ALTO Server Functionalities

   The ALTO server component is the core of the broker layer.  Multiple
   logically centralized ALTO servers use the information collected from
   IdR and IdT modules to create and provide abstract maps with a

     [yry] module vs component

   simplified view, yet enough information about MdOs involved in the
   federation.  This information includes domain-level topology, storage
   resources, computation resources, networking resources and PNF/VNF

     [yry] Consistency: missing CPU, as above?

   As an ALTO client, each MdO sends ALTO service queries to the ALTO
   server.  This server provides aggregated inter-domain information
   exposed as set ALTO base services defined in [RFC7285], e.g., Network
   Map, Cost Map and ALTO extension services, e.g., Property
   Map [DRAFT-PM], Multi-Cost Map [RFC8189], Path Vector [DRAFT-PV].

   For example, when a source MdO receives a customer service request,
   it checks whether or not it can deliver the full service.  If it is
   unable to do so, the MdO consumes from the ALTO Server the Property
   Map service to have a clear global view of the resource information
   offered by other MdOs.  This information allows discovering which
   candidate MdOs may be contacted to deliver the remaining requirements
   of a requested end-to-end service deployment.  The connectivity
   information among discovered MdOs can be retrieved by a Cost Map
   service, responding, for instance, a path vector with the AS-level
   topology distance between the source MdO and candidate MdOs.

6.4.  Filtered Cost Map Extension

     [yry] A bit disruptive transition. Why only Filtered Cost Map?
     It helps to state the basic ALTO services that broker uses and
     state that this is the one needs extension---I actually think
     this should be a new service, instead of filtered cost map.

   The ALTO server MUST provide connectivity information for every SG
   link in the SG path for an E2E requirement.  This information is the
   AS-level topology distance in the form of path vector, and it
   includes all possible ways for each (source node, destination node)
   pair in the SG link.

     [yry] Is it necessary to limit to only AS-hop distance? I assume
     that other distances such as bw, latency can be useful as well.

   In this section, we introduce a non-normative overview of the
   Filtered Cost Map defined in Section 6.1 of [DRAFT-PV] [1].

   The specifications for the "Media Types", "HTTP method",
   "Capabilities" and "Uses" (described in Section 6.1 of [DRAFT-PV]
   [2]) are unchanged.

6.4.1.  Accept Input Parameters

   The ReqFilteredCostMap object in Section 6.1.2 of [DRAFT-PV] [3] is
   extended as follow:

     object {
       [NFFG sg;]
     } ReqFilteredCostMap;

      [yry] If sg is optional, the req can have an empty body. What is
      the response to an empty request? Non-optional makes more sense,
      to me.

      [yry] One key question that we should think: is it possible that
      NFFG is an example of a more *generic* query model, for example,
      waypoint model, so that the generic query is
        src -> dst: waypoints?

     object {
       JSONString nfs<1..*>;
       JSONString saps<1..*>;
       NextHops sg_links<1..*>;
       REQs reqs<1..*>;
     } NFFG;

     object {
       JSONNumber id;
       JSONString src-node;
       JSONString dst-node;
     } NextHops;

     object {
       JSONString id;
       JSONString src-node;
       JSONString dst-node;
       JSONNumber sg-path<1..*>;
     } REQs;

   sg:  If present, the ALTO Server MUST allow the request input to
      include an SG with a formatted body as an NFFG object.  An NFFG
      object contains NFs, SAPs, SG links representing logical
      connections between NFs, SAPs or both and E2E requirements as a
      list of ids of SG links.

   It is worth noting that further versions of this draft will define a
   more elaborated NFFG object to support extended parameters such as
   monitoring parameters, resource requirements, etc.

    [yry] Also, need consistency requirement, for example, src-node string
    appears in union of nfs, saps ...

6.4.2.  Response

   If the ALTO client includes the path vector cost mode in the "cost-
   type" or "multi-cost-types" field of the input parameter, the
   response for each SG link in each E2E requirement MUST be encoded as
   a JSONArray of JSONArrays of JSONStrings.  Anyone of the sub-arrays
   indicates a potential candidate path calculated as the per-domain
   topological distance corresponding to the amount of traversing

      [yry] The encoding is quite interesting, but without an example,
      and a bit explanation, it can be hard to understand the nested
      data structure. I feel that the model can be a generic model,
      such as waypoint, beyond SG; see above.

   Moreover, as defined in Section 6.3.6 of [DRAFT-PV] [4], If an ALTO
   client sends a request of the media type "application/alto-
   costmapfilter+json" and accepts "multipart/related", the ALTO server
   MUST provide path vector information along with the associated
   Property Map information (e.g., entry points of the corresponding
   foreign domains), in the same body of the response.

   Section 6.5.2 gives an example of the Filtered Cost Map query and the
   corresponding responses.

6.5.  Examples of Message Exchange

   This section list a couple of examples of the Property Map and
   Filtered Cost Map queries and the corresponding responses.  These
   responses are based on the information in Table 1 and Table 2 of a
   use case implementation described in Appendix A.

6.5.1.  Property Map Service

   In this example, the ALTO client wants to retrieve the entire
   Property Map for PID entities with the "entry-point", "cpu", "mem",
   "storage", "port" and "nf" properties.

     GET /propmap/full/inet-ucmspn HTTP/1.1
     Accept: application/alto-propmap+json,application/alto-error+json

     HTTP/1.1 200 OK
     Content-Length: ###
     Content-Type: application/alto-propmap+json
       "property-map": {
           "pid:AS1": {
               "entry-point": [ "" ],
               "cpu": [ "50.0" ],
               "mem": [ "60.0" ],
               "storage": [ "70.0" ],
               "port": [ "SAP1" ],
               "nf": [ "NF1", "NF3" ]
           "pid:AS2": {
               "entry-point": [ "" ],
               "cpu": [ "10.0" ],
               "mem": [ "20.0" ],
               "storage": [ "30.0" ],
               "nf": [ "NF2" ]
           "pid:AS3": {
               "entry-point": [ "" ],
               "cpu": [ "80.0" ],
               "mem": [ "90.0" ],
               "storage": [ "100.0" ],
               "port": [ "SAP2" ],
               "nf": [ "NF1", "NF3" ]

       [yry] A key issue that one may encounter in multi-domain
       aggregation is ambiguity of aggregated resources. Is the semantics
       that each resource is the *local* resource? If so, is there
       a naming/scoping requirement on naming entities?

       [yry] If you do hierarchical brokers and each broker can hide the
details, the resource aggregation can become ambiguous: (1) broker 1 can
       say it has 10 units of CPUs, and broker 2 can say the same, but they
       may aggregate the 10 from the same individual domain.

6.5.2.  Filtered Cost Map Service

   The following example uses the Filtered Cost Map service to request
   the path vector for a given E2E requirement.  The SG request
   information in Table 2 is used to describe the service, and it is
   composed of three NFs (NF1, NF2, and NF3) and two SAPs (SAP1 and
   SAP2).  Links connecting the NFs and SAPs ("sg_links" tag) are also
   included, followed by an E2E requirement ("reqs" tag) with
   information about the order in which NFs are traversed from SAP1 to

   Note that the request accepts "multipart/related" media type.  This
   means the ALTO server will include associated property information in
   the same response.

       POST /costmap/pv HTTP/1.1
       Accept: multipart/related, application/alto-costmap+json,
             application/alto-propmap+json, application/alto-error+json
       Content-Length: [TBD]
       Content-Type: application/alto-costmapfilter+json

         "cost-type": {
           "cost-mode": "array",
           "cost-metric": "ane-path"
         "sg": {
           "nfs": [ "NF1", "NF2", "NF3" ],
           "saps": [ "SAP1", "SAP2" ],
               "id": 2,
               "src-node": "SAP1",
               "dst-node": "NF1",

               "id": 2,
               "src-node": "NF1",
               "dst-node": "NF2",
               "id": 3,
               "src-node": "NF2",
               "dst-node": "NF3",
               "id": 4,
               "src-node": "NF3",
               "dst-node": "SAP2",
           "reqs": [
               "id": 1,
               "src-node": "SAP1",
               "dst-node": "SAP2",
               "sg-path": [ 1, 2, 3, 4 ]

   The ALTO server returns connectivity information for the E2E
   requirement provided by the ALTO Client request of the above example.
   Also, the response includes Property Map information for each element
   in the path vector.  In this case, it is retrieved a Property Map
   with the "entry-point" property, i.e., the URL of the MdO entry point
   for the corresponding network.

     HTTP/1.1 200 OK
     Content-Length: [TBD]
     Content-Type: multipart/related; boundary=example

     Content-Type: application/alto-endpointcost+json

       "meta": {
         "cost-type": {
            "cost-mode": "array",
            "cost-metric": "ane-path"

       "cost-map": {
         "SAP1": {
           "SAP2": {
               "SAP1": {
                   "NF1": [
                     [ "AS1" ], [ "AS1", "AS2", "AS3" ]
               "NF1": {
                   "NF2": [
                     [ "AS1", "AS2" ], [ "AS3", "AS2" ]
               "NF2": {
                   "NF3": [
                     [ "AS2", "AS1" ], [ "AS2", "AS3" ]
               "NF3": {
                   "SAP2": [
                     [ "AS1", "AS2", "AS3" ], [ "AS3" ]

     Content-Type: application/alto-propmap+json

       "property-map": {
         "pid:AS1": { "entry-point": "" },
         "pid:AS2": { "entry-point": "" },
         "pid:AS3": { "entry-point": "" }

   [yry] Very interesting design. It helps if you add some text to go over
   the meaning, such as [ "AS1" ], [ "AS1", "AS2", "AS3" ] are two

On Sun, Jul 8, 2018 at 9:47 PM Y. Richard Yang <>; wrote:

> Dear Danny, Christian,
> I started to read this draft. It addresses an important, timely issue.
> Below is the first part (on Sections 1-6) of my review, and I will send the
> second part on Sections 7-9 by tomorrow.
> Great work!
> Richard
> ====
> Existing proposals for the network service orchestration are
>    intrinsically conceived for single administrative domain scenarios.
>      [yry] Add citations right after existing proposals?
>    For example, in the standard service orchestration model described in
>    ETSI NFV MANO framework [ETSI-NFV-MANO], one orchestrator is supposed to
> work within one administrative domain.  The analysis of
>    orchestration and management of network Services over multiple
>      [yry] case, network vs Services
>    administrative domains have begun to be addressed by ETSI
>    in [ETSI-NFV-MANO-MDO].
>      [yry] The last sentence gives a related work. To make reading easier,
>      it may help to differentiate between proposals (first sentence) and
> analysis (this sentence).
>    All such proposals described above envision the potential
>    introduction of new business model approaches, including federation
>    models [PPP-5:2013] among administrative domains.  In this context,
>    this document considers each network operator involved in the
>    community advertises its abstracted capabilities (e.g., software/
>    hardware resources, physical/virtual network functions, etc.) to a
>    broker (i.e., 3rd party).  This latter, in its turn, provides or
>      [yry] latter -> broker
>    assists coordinate E2E network services spanning multi-domain
>    networks.
> 5.  Problem Statement and Challenges
>    The provision of a complete E2E network service requires chaining
>    services provided by multiple network operator with multiple
>      [yry] operator -> operators
>    technologies.  In this multi-domain environment, the orchestration
>    process will require an advertise mechanism through which single
>      [yry] advertise -> advertisement, single -> individual
>    domains can describe their capabilities, resources, and VNFs in an
>    interoperable manner.  Moreover, a discovery mechanism is also
>    necessary so that source domains can obtain candidate domains (with
>      [yry] Although one can infer what source/candidate domains mean,
>      it can help if you define them.
>    the corresponding connectivity information) which can provide a part
>    of the service and/or slide in an E2ENS requirement.
>      [yry] slide -> slice?
>    In order to the advertising and discovery process works in a proper
>    way, a number of challenges can be identified:
>      [yry] "In order to" -> "In order that"
>    Lack of Abstractions:  Multiple vendors with heterogeneous
>         technologies need an information model to adequately represent
>         in confidentiality-preserving fashion the resource and topology
>         information.
>    Scalability:  Involves the distribution of topology and resource
>         information in a peer-to-peer fashion (MdO-to-MdO).  Multi-
>      [yry] Note that the previous item mentions generic information model
>      but here mentions only topology and resource information. It can help
> if
>      the scoping is clarified up front: only topology/resource.
>         operator multi-domain environments where the information
>         distribution is advertised in a peer-to-peer model scales
>         linearly.  It means more MdO interconnections one has, the more
>       [yry] Why linearly? There can be ambiguity. Assume the total
>       transmission load of one domain grows linearly with n, which is
>       the number of domains. Total system scales w/ n^2. But there can be
> other peer-to-peer systems such as CHORD which publish
> (advertises) with a constant cost. It helps to specify some more details.
>         it "costs" to distribute.
>    Flexibility:  Considers that a distributed approach does not allow
>         domains without physical infrastructure (e.g., without BGP or
>         BGP-LS) to advertise resource capabilities and networking
>         resources.  Such procedures consist in deploying and configuring
>         physical peering points for these domains.
>      [yry] The description above is not clear to me yet. One can run BGP
> w/o a physical infrastructure. It helps if you can clarify this challenge.
>    Complexity:  Refers to the discovery mechanism to pre-select
>         candidate domains, accounting for resources and capabilities,
>         necessary for an E2E network service deployment.  An intrinsic
>         complexity exists in the process of assembling, logically
>         organizing, and enabling abstraction views of different
>         resources and capabilities in multi-domain scenarios.
>     [yry] A summary comment on the 4 items. Are they challenges, issues
>     or requirements? The wording appears to have multiple aspects: lacking
>     of abstractions seems to imply issues; flexibility seems to imply
>     requirements... Is the set complete, for example how about security,
> autonomy, privacy ...---some you touch right below?  One more comment is
> that the terms used (e.g., flexibility) can be too generic. If you go a bit
> more specific, for example, Discovery Flexibility, it may help.
> ====
> On Mon, Jul 2, 2018 at 8:25 PM Danny Alex Lachos Perez <
>>; wrote:
>> Dear WG members
>> This new version (-01) considers a section on benefits and open questions
>> (section 7) where we analyze the advantages and open issues in the
>> broker-assisted architecture. Moreover, we removed the Property Map
>> Extension section because the current Property Map draft [0] already
>> supports property values encoded as JSONArray.
>> Other changes include (i) update the Problem Statement and Challenges
>> section and (ii) many minor style and grammar edits.
>> Comments and feedback will be highly appreciated.
>> Thank you very much
>> [0]
>> Ss
>> Danny Lachos
>> On Mon, Jul 2, 2018 at 4:18 PM <>; wrote:
>>> A new version of I-D, draft-lachosrothenberg-alto-brokermdo-01.txt
>>> has been successfully submitted by Danny Alex Lachos Perez and posted to
>>> the
>>> IETF repository.
>>> Name:           draft-lachosrothenberg-alto-brokermdo
>>> Revision:       01
>>> Title:          ALTO-based Broker-assisted Multi-domain Orchestration
>>> Document date:  2018-07-02
>>> Group:          Individual Submission
>>> Pages:          22
>>> URL:
>>> Status:
>>> Htmlized:
>>> Htmlized:
>>> Diff:
>>> Abstract:
>>>    Evolving networking scenarios (e.g., 5G) demand new multiple
>>>    administrative domain (aka multi-domain) orchestration models.  This
>>>    document proposes the use of Application-Layer Traffic Optimization
>>>    (ALTO) services to offer topology and resources addressing network
>>>    service discovery and provisioning by multi-domain orchestrators.
>>>    The ALTO services with the proposed protocol extension offer
>>>    aggregated views on various types of resources contributing to a more
>>>    simple and scalable solution for resource and service discovery in
>>>    multi-domain, multi-technology environments.
>>> Please note that it may take a couple of minutes from the time of
>>> submission
>>> until the htmlized version and diff are available at
>>> The IETF Secretariat
>>> _______________________________________________
>> alto mailing list
> --
> --
>  =====================================
> | Y. Richard Yang <>;   |
> | Professor of Computer Science       |
> |        |
>  =====================================

| Y. Richard Yang <>;   |
| Professor of Computer Science       |
|        |