Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc

["Henderickx, Wim (Wim)" <[e-mail]>]

In-line with WH>




On 20/11/15 01:01, "BESS on behalf of [e-mail]" <[e-mail] on behalf of [e-mail]> wrote:

>2015-11-19, Henderickx, Wim (Wim):
>> WH> I vote for a an evolution of switches/TORs that have proper
>> support for this. I hope some HW vendors of TOR chips shime in, but I
>> am told the MPLS solution is possible in the next generation chips
>> they are working on.
>
>Well, it looks like the key questions are:
>- when would ToR chips support MPLS/MPLS/UDP ?  (the generation that has 
>been released recently but not present in most shipping ToRs yet, the 
>next one ?)
WH> the next generation TOR chips seem to support it, at least this is the info I got from 3 vendors
>- do we want an interim VXLAN-based solution ? (that will involve at 
>best a performance penalty with existing chips, and at worse impossible 
>to implement -- we haven't seen clear information in this thread)
WH> in my view no, since we also have the global VNID issue and a continuous tax and ASBR boxes for this.
>
>-Thomas
>
>
>>> Zhuangshunwan  :
>>>>
>>>> Hi Diego,
>>>>
>>>> Thanks for your comments. Pls see inline with [Vincent].
>>>>
>>>> Vincent
>>>>
>>>> *发件人:*BESS [mailto:[e-mail]] *代表 *Diego Garcia del
>>>> Rio *发送时间:*2015年11月18日14:25 *收件人:*[e-mail] *主题:*Re: [bess]
>>>> draft-hao-bess-inter-nvo3-vpn-optionc
>>>>
>>>> Some comments from my side, I think the draft makes quite a few
>>>> assumptions on specific implementation details that are way too
>>>> general to be considered widely available. Most of the TOR
>>>> devices today already pay a double-pass penalty when doing
>>>> routing of traffic in/out of vxlan-type tunnels. Only the newest
>>>> generation can route into tunnels without additional passes. And
>>>> are definitively limited in being able to arbitrary select UDP
>>>> ports on a per tunnel basis. In fact, most are even limited at
>>>> using more than one VNID per "service" of sorts. [Vincent]: Yes,
>>>> the new generation BCM chipset has already supported VXLAN
>>>> routing without additional passes. For OVS/TOR, VXLAN
>>>> encapsulation is more popular than MPLSoGRE/UDP, and has better
>>>> performance. The IP-addressed based implementation which would, I
>>>> assume, imply assigning one or more CIDRs to a loopback interface
>>>> on the ASBR-d is also quite arbitrary and does not seem like a
>>>> technically "clean" solution. (besides burning tons of IPs). As a
>>>> side-note, most PE-grade routers i've worked with were quite
>>>> limited in terms of IP addresses used for tunnel termination and
>>>> it wasn't that just a simple pool can be used. [Vincent]: I think
>>>> the larger VTEP IP address range on ASBR-d has no limitations.
>>>> For the hardware on ASBR-d, it has capability to terminate
>>>> multiple VXLAN tunnels with arbitrary destination VTEP IP
>>>> addresses. Wim's mentions on cases where the Application itself,
>>>> hosted in a datacenter, would be part of the option-C
>>>> interconnect, was dismissed without much discussion so far,
>>>> while, if we look in detail at the type of users which will even
>>>> consider a complex topology like model-C its most likely users
>>>> and operators very familiar with MPLS VPNs in the WAN. Those type
>>>> of operators will most likely be very interested in deploying
>>>> MPLS or WAN-grade applications (i.e., virtual-routers or other
>>>> VNFs) in the DC and thus its highly likely that the interconnect
>>>> would not terminate at the NVE itself but rather the TS (the
>>>> virtual machine). Also, the use of UDP ports at random would
>>>> imply quite complex logic on the ASBR-d IMHO. Im not saying its
>>>> impossible, but since the received packet now not only has to be
>>>> received on a random (though locally chosen) UDP port and this
>>>> information preserved in the pipeline to be able to do the
>>>> double-tunnel-stitching, it sounds quite complex. I am sure
>>>> someone in the list will mention this has already been
>>>> implemented somewhere, and I won't argue with that. And let's
>>>> not even bring into account what this would do to any DC
>>>> middlebox that now has to look at vxlan over almost any random
>>>> port. We have to go back to the "is it a 4 or is it a 6 in byte
>>>> x" heuristics to try to guess whether the packet is vxlan or just
>>>> something entirely different running over IP. [Vincent]: Using NP
>>>> or multi-core CPU hardware technology, it can be implemented
>>>> although deeper packet inspection is needed to perform UDP port
>>>> and MPLS stitching. In general I feel the proposed solution seems
>>>> to be fitting of a specific use-case which is not really detailed
>>>> in the draft and does not describe   a solution that would
>>>> "elegantly" solve the issues at hand. It just feels like we're
>>>> using any available bit-space to stuff data into places that do
>>>> not necesarily belong. Yes, MPLS encapsulations on virtual
>>>> switches are not yet fully available, and there can be some
>>>> performance penalty on the TORs, but the solutions are much
>>>> cleaner from a control and data plane point of view. Maybe I'm
>>>> too utopic. [Vincent]: I think pure VXLAN solution has its
>>>> scenario, it's general rather than specific. We can't require all
>>>> OVS/NVEs support VXLAN + MPLSoGRE at the same time. Best
>>>> regards, Diego
>>>> ---------------------------------------------------------------------------------
>>>>
>>>>
>Hi,
>>>> The problem we are trying to solve is to reduce data center
>>>> GW/ASBR-d's forwarding table size, the motivation is same as
>>>> traditional MPLS VPN option-C. Currently, the most common
>>>> practise on ASBR-d is to terminate VXLAN encapsulation, look up
>>>> local routing table, and then perform MPLS encapsulation to the
>>>> WAN network. ASBR-d needs to maintain all VM's MAC/IP. In
>>>> Option-C method, only transport layer information needed to be
>>>> maintained at GW/ASBR-d, the scalability will be greatly
>>>> enhanced. Traditonal Option-C is only for MPLS VPN network
>>>> interworking, because VXLAN is becoming pervasive in data center,
>>>> the solution in this draft was proposed for the heterogeneous
>>>> network interworking. The advantage of this solution is that only
>>>> VXLAN encapsulation is required for OVS/TOR. Unlike Wim's
>>>> solution, east-west bound traffic uses VXLAN encap, while
>>>> north-south bound traffic uses MPLSoGRE/UDP encap. There are two
>>>> solutions in this draft: 1. Using VXLAN tunnel destination IP for
>>>> stitching at ASBR-d. No data plane modification requirements on
>>>> OVS or TOR switches, only hardware changes on ASBR-d. ASBR-d
>>>> normally is router, it has capability to realize the hardware
>>>> changes. It will consume many IP addresses and the IP pool for
>>>> allocation needs to be configured on ASBR-d beforehand. 2. Using
>>>> VXLAN destination UDP port for stitching at ASBR-d. Compared with
>>>> solution 1, less IP address will be consumed for allocation. If
>>>> UDP port range is too large, we can combine with solution 1 and
>>>> 2. In this solution, both data plane modification changes are
>>>> needed at OVS/TOR and ASBR-d. ASBR-d also has capability to
>>>> realize the hardware changes. For OVS, it also can realize the
>>>> data plane changes. For TOR switch, it normally can't realize
>>>> this function.  This solution mainly focuses on pure software
>>>> based overlay network, it has more scalability. In public cloud
>>>> data center, software based overlay network is the majority
>>>> case. Whether using solution 1 or 2 depends on the operators real
>>>> envionment. So I think our solution has no flaws, it works fine.
>>>> Thanks, weiguo ________________________________ From: BESS
>>>> [[e-mail] <mailto:[e-mail]>] on behalf
>>>> of John E Drake [[e-mail] <mailto:[e-mail]>]
>>>> Sent: Wednesday, November 18, 2015 2:49 To: Henderickx, Wim
>>>> (Wim); EXT - [e-mail]
>>>> <mailto:[e-mail]>; BESS Subject: Re: [bess]
>>>> draft-hao-bess-inter-nvo3-vpn-optionc Hi, I think Wim has
>>>> conclusively demonstrated that this draft has fatal flaws and I
>>>> don’t support it.  I also agree with his suggestion that we first
>>>> figure out what problem we are trying to solve before solving
>>>> it. Yours Irrespectively, John From: BESS
>>>> [mailto:[e-mail] <mailto:[e-mail]>] On
>>>> Behalf Of Henderickx, Wim (Wim) Sent: Tuesday, November 17, 2015
>>>> 12:49 PM To: EXT - [e-mail]
>>>> <mailto:[e-mail]>; BESS Subject: Re: [bess]
>>>> draft-hao-bess-inter-nvo3-vpn-optionc — Snip — No, the spec as it
>>>> is can be implemented in its VXLAN variant with existing
>>>> vswitches (e.g. OVS allows to choose the VXLAN destination port,
>>>> ditto for the linux kernel stack). (ToR is certainly another
>>>> story, most of them not having a flexible enough VXLAN dataplane
>>>> nor support for any MPLS-over-IP.) WH> and how many ports
>>>> simultaneously would they support? For this to work every tenant
>>>> needs a different VXLAN UDP destination port/receive port. There
>>>> might be SW elements that could do some of this, but IETF defines
>>>> solutions which should be implemented across the board HW/SW/etc.
>>>> Even if some SW switches can do this, the proposal will impose so
>>>> many issues in HW/data-plane engines that I cannot be behind this
>>>> solution. To make this work generically we will have to make
>>>> changes anyhow. Given this, we better do it in the right way and
>>>> guide the industry to a solution which does not imply those
>>>> complexities. Otherwise we will stick with these specials forever
>>>> with all consequences (bugs, etc). - snip - From:
>>>> "[e-mail]
>>>> <mailto:[e-mail]><mailto:[e-mail]
>>>> <mailto:[e-mail]>>" <[e-mail]
>>>> <mailto:[e-mail]><mailto:[e-mail]
>>>> <mailto:[e-mail]>>> Organization: Orange Date:
>>>> Tuesday 17 November 2015 at 01:37 To: Wim Henderickx
>>>> <[e-mail]
>>>> <mailto:[e-mail]><mailto:[e-mail]
>>>>
>>>>
><mailto:[e-mail]>>>, BESS <[e-mail]
>>>> <mailto:[e-mail]><mailto:[e-mail]
>>>> <mailto:[e-mail]>>> Subject: Re: [bess]
>>>> draft-hao-bess-inter-nvo3-vpn-optionc Hi Wim, WG, 2015-11-16,
>>>> Henderickx, Wim (Wim): 2015-11-13, Henderickx, Wim (Wim): Thomas,
>>>> we can discuss forever and someone need to describe requirements,
>>>> but the current proposal I cannot agree to for the reasons
>>>> explained. TM> Well, although discussing forever is certainly not
>>>> the goal, the reasons for rejecting a proposal need to be
>>>> thoroughly understood. WH> my point is what is the real driver
>>>> for supporting a plain VXLAN data-plane here, the use cases I
>>>> have seen in this txt is always where an application behind a
>>>> NVE/TOR is demanding option c, but none of the NVE/TOR elements.
>>>> My understanding is that the applications  are contexts where
>>>> overlays are present is when workloads (VMs or baremetal) need to
>>>> be interconnected with VPNs. In these contexts, there can be
>>>> reasons to want Option C to reduce the state on ASBRs. In these
>>>> context, its not the workload (VM or baremetal) that would
>>>> typically handle VRFs, but really the vswitch or ToR. WH2> can it
>>>> not be all cases: TOR/vswitch/Application. I would make the
>>>> solution flexible to support all of these not? 2015-11-13,
>>>> Henderickx, Wim (Wim): TM> The right trade-off to make may in
>>>> fact depend on whether you prefer: (a) a new dataplane stitching
>>>> behavior on DC ASBRs (the behavior specified in this draft) or
>>>> (b) an evolution of the encaps on the vswitches and ToRs to
>>>> support MPLS/MPLS/(UDP or GRE) WH> b depends on the use case I
>>>> don't get what you mean by "b depends on the use case". WH> see
>>>> my above comment. If the real use case is an application behind
>>>> NVE/TOR requiring model C, than all the discussion on impact on
>>>> NVE/TOR is void. As such I want to have a discussion on the real
>>>> driver/requirement for option c interworking with an IP based
>>>> Fabric. Although I can agree than detailing requirements can
>>>> always help, I don't think one can assume a certain application
>>>> to dismiss the proposal. WH> for me the proposal is not
>>>> acceptable for the reasons explained: too much impact on the
>>>> data-planes I wrote the above based on the idea that the encap
>>>> used in MPLS/MPLS/(UDP or GRE), which hence has to be supported
>>>> on the ToRs and vswitches. Another possibility would be
>>>> service-label/middle-label/Ethernet assuming an L2 adjacency
>>>> between vswitches/ToRs and ASBRs, but this certainly does not
>>>> match your typical DC architecture. Or perhaps had you something
>>>> else in mind ? WH> see above. The draft right now also requires
>>>> changes in existing TOR/NVE so for me all this discussion/debate
>>>> is void. No, the spec as it is can be implemented in its VXLAN
>>>> variant with existing vswitches (e.g. OVS allows to choose the
>>>> VXLAN destination port, ditto for the linux kernel stack). (ToR
>>>> is certainly another story, most of them not having a flexible
>>>> enough VXLAN dataplane nor support for any MPLS-over-IP.) WH> and
>>>> how many ports simultaneously would they support? WH> and
>>>> depending on implementation you don’t need to change any of the
>>>> TOR/vswitches. Does this mean that for some implementations you
>>>> may not need to change any of the TOR/vswitches, but that for
>>>> some others you may ? WH> any proposal on the table requires
>>>> changes, so for me this is not a valid discussion See above, the
>>>> proposal in the draft does not necessarily need changes in
>>>> vswitches. Let me take a practical example : while I can quite
>>>> easily see how to implement the procedures in
>>>> draft-hao-bess-inter-nvo3-vpn-optionc based on current vswitch
>>>> implementations of VXLAN, the lack of MPLS/MPLS/(UDP, GRE)
>>>> support in commonplace vswitches seems to me as making that
>>>> alternate solution you suggest harder to implement. WH> I would
>>>> disagree to this. Tell me which switch/TOR handles multiple UDP
>>>> ports for VXLAN ? I mentioned _v_switches, and many do support a
>>>> variable destination port for VXLAN, which is sufficient to
>>>> implement what the draft proposes. -Thomas From: Thomas Morin
>>>> <[e-mail]
>>>> <mailto:[e-mail]><mailto:[e-mail]
>>>> <mailto:[e-mail]>>> Organization: Orange Date:
>>>> Friday 13 November 2015 at 09:57 To: Wim Henderickx
>>>> <[e-mail]
>>>> <mailto:[e-mail]><mailto:[e-mail]
>>>>
>>>>
><mailto:[e-mail]>>>
>>>> Cc: "[e-mail] <mailto:[e-mail]><mailto:[e-mail]
>>>> <mailto:[e-mail]>>" <[e-mail]
>>>> <mailto:[e-mail]><mailto:[e-mail]
>>>> <mailto:[e-mail]>>> Subject: Re: [bess]
>>>> draft-hao-bess-inter-nvo3-vpn-optionc Hi Wim, I agree on the
>>>> analysis that this proposal is restricted to implementations that
>>>> supports the chosen encap with non-IANA ports (which may be hard
>>>> to achieve for instance on hardware implementations, as you
>>>> suggest), or to context where managing multiple IPs would be
>>>> operationally viable. However, it does not seem obvious to me how
>>>> the alternative you propose [relying on 3-label option C with an
>>>> MPLS/MPLS/(UDP|GRE) encap] addresses the issue of whether the
>>>> encap behavior is supported or not (e.g. your typical ToR chipset
>>>> possibly may not support this kind of encap,  and even
>>>> software-based switches may not be ready to support that today).
>>>> My take is that having different options to adapt to various
>>>> implementations constraints we may have would have value. (+ one
>>>> question below on VXLAN...) -Thomas 2015-11-12, Henderickx, Wim
>>>> (Wim): On VXLAN/NVGRE, do you challenge the fact that they would
>>>> be used with a dummy MAC address that would be replaced by the
>>>> right MAC by a sender based on an ARP request when needed ? Is
>>>> the above the issue you had in mind about VXLAN and NVGRE ? WH>
>>>> yes I you don't mind me asking : why do you challenge that ?
>>>>
>
>_________________________________________________________________________________________________________________________
>
>Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
>pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
>a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
>France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci
>
>This message and its attachments may contain confidential or privileged information that may be protected by law;
>they should not be distributed, used or copied without authorization.
>If you have received this email in error, please notify the sender and delete this message and its attachments.
>As emails may be altered, France Telecom - Orange shall not be liable if this message was modified, changed or falsified.
>Thank you.
>
>_______________________________________________
>BESS mailing list
>[e-mail]
>https://www.ietf.org/mailman/listinfo/bess