Re: BFD over VXLAN: Trapping BFD Control packet at VTEP

[Dinesh Dutt <[e-mail]>]

Looks god to me Greg. Thank you for your hard work in this,

Dinesh

On Wed, Aug 7, 2019 at 9:25 AM Greg Mirsky <[e-mail]> wrote:

> Hi Dinesh, Joel, Sridhar, et al.,
> much appreciate the help you've given me sharing your expertise. I hope
> that the updates you will find in the attached diff and the working copy of
> the draft be closer to the acceptable solution for VTEP-VTEP BFD. Please
> note, that I'll shortly start a new discussion thread to address one of
> Carlos's questions on the ambiguity of the text on multiple concurrent
> sessions between the same pair of VTEPs.
> Please review the changes to Sections 4 and 6 and share your feedback,
> suggestions, and questions.
>
> Regards,
> Greg
>
> On Mon, Aug 5, 2019 at 6:03 PM Dinesh Dutt <[e-mail]> wrote:
>
>>
>>
>> On Mon, Aug 5, 2019 at 5:56 PM Greg Mirsky <[e-mail]> wrote:
>>
>>> Hi Dinesh,
>>> thank you for your expedient detailed response.
>>> I believe that the ability to run BFD session up to a tenant
>>> (VTEP-VTEP-tenant or tenant-tenant) was never in jeopardy from this
>>> specification.
>>> I'm trying to provide precise specification on what can be used ad the
>>> destination MAC and IP addresses in the inner frame/packet as I believe
>>> that likely will help to avoid interoperability issues.
>>> I'm interested to learn some more about the "martian checking" function.
>>> As you know, martian addresses have been used as destination IP address in
>>> LSP Ping and BFD over MPLS LSP and PW. I haven't heard that any silicon
>>> feature caused problems for operators using these tools.
>>>
>>
>> Interesting. I didn't know this aspect of use with MPLS ping. Did those
>> packets ever go through a firewall though? In any case, maybe suggest the
>> use of those addresses with a statement that this is how LSP does it, but
>> that other MAC/IP pairs are possible as long as the conditions of the
>> endpoint owning the MAC/IP was honored.
>>
>> Dinesh
>>
>>>
>>> Regards,
>>> Greg
>>>
>>> On Mon, Aug 5, 2019 at 3:59 PM Dinesh Dutt <[e-mail]> wrote:
>>>
>>>> Hi Greg,
>>>>
>>>> That we agree on the problem definition is the first step forward. Your
>>>> original document had my cases covered and so I was surprised by the track
>>>> this thread took. It doesn't matter, we're back on track.
>>>>
>>>> My recommendation is to not worry about specifying the precise MAC/IP
>>>> address used in the inner header. The addresses chosen MUST ensure that the
>>>> packet is trapped to the control plane of the VTEP and not escape to the
>>>> tenant if the BFD is to the VTEP. Any solution MUST also not preclude the
>>>> use of the BFD by tenant systems for that VNI. There are many ways an
>>>> implementer can choose to implement this. For example, the inner MAC
>>>> address is whatever the VTEP implementer would return if ARP'd for the IP
>>>> address used in the inner header in the given VNI. The implementer can pick
>>>> a fixed MAC address, one that they own etc. Multiple BFD sessions can be
>>>> run for testing path connectivity on more than one VNIs. Limits should be
>>>> in place to avoid overwhelming the receiver with BFD messages (you had
>>>> words about this in your currently published draft).  If the VNI is
>>>> irrelevant in the test i.e. only the VXLAN pipe at the VTEP is being
>>>> tested. the user can use any VNI active on the VTEP on which the VTEP owns
>>>> an IP address.
>>>>
>>>> I'm concerned about the use of 127/8 address only because of firewalls
>>>> or implementations that drop packets with these addresses as either the
>>>> source or destination. For example, on many merchant silicon, I don't
>>>> believe you can turn off martian checking and drops *only* for
>>>> VXLAN-encapsulated BFD packets. I don't know what the Linux kernel does
>>>> today on such packets, for example (or Hyper-V). I'd like a solution that
>>>> doesn't demand additional or new chip functionality or require additional
>>>> middle-box hole punch.
>>>>
>>>> Why do you feel you MUST to specify the MAC/IP address on the inner
>>>> packet? What am I missing here?
>>>>
>>>> Dinesh
>>>>
>>>> On Mon, Aug 5, 2019 at 3:04 PM Greg Mirsky <[e-mail]>
>>>> wrote:
>>>>
>>>>> Hi Dinesh,
>>>>> what do you see as the way forward? I agree, that the proposed text
>>>>> doesn't work for multi-VNI concurrent monitoring because these VNIs are
>>>>> tenant's VNIs. And in that case, we need to specify another mechanism to
>>>>> trap the BFD Control packet at VTEP. It seems that VTEP's Ethernet address
>>>>> must be used as the destination MAC address in the inner Ethernet frame.
>>>>> The destination IP address may be either VTEP's address of martian (I do
>>>>> prefer martian). Let me give it  try:
>>>>> NEW TEXT:
>>>>>
>>>>> To monitor continuity of the path between two VTEPs, an operator MUST
>>>>> select a VNI number to be used as Management VNI. Management VNI number
>>>>> MUST NOT be one of the tenant's VNIs to prevent sending VXLAN packets
>>>>> received on Management VNI to a tenant. VNI number 1 is RECOMMENDED as the
>>>>> default for Management VNI. [Ed.note: What we set the Destination MAC to?
>>>>> Can it be invalid MAC that MUST be ignored on receipt?]
>>>>>
>>>>> If an implementation supports concurrent monitoring of multiple VNIs,
>>>>> then the value of VNI number MAY be one of tenant's VNIs. The destination
>>>>> MAC address in the inner Ethernet frame encapsulating BFD Control packet
>>>>> MUST be MAC associated with the remote VTEP.
>>>>> The destination IP address of the inner IP packet MUST be selected
>>>>> from the range 127/8 for IPv4, and for IPv6 from the range
>>>>> 0:0:0:0:0:FFFF:7F00:0/104. The TTL value in the inner IP header MUST be set
>>>>> to 1.
>>>>>
>>>>> Regards,
>>>>> Greg
>>>>>
>>>>> On Sun, Aug 4, 2019 at 9:07 AM Dinesh Dutt <[e-mail]> wrote:
>>>>>
>>>>>> Hi Greg,
>>>>>>
>>>>>> Thanks for your clarifications. I agree with your sentiment on why
>>>>>> you're running BFD over VXLAN between VTEPs. I wasn't arguing against it at
>>>>>> all. All I was saying was pointing to the limitations of the use of
>>>>>> management VNI. I spoke to some operators who're running EVPN and mentioned
>>>>>> the discussion on this thread. They concur that they're using specific VNIs
>>>>>> to test connectivity over that VNI between VTEPs to ensure misconfiguration
>>>>>> doesn't lead to blackholes. My statements are based in real world operator
>>>>>> experience. And I was providing language that ensured packets didn't leak
>>>>>> across to tenants when they were destined to VTEPs.
>>>>>>
>>>>>> Dinesh
>>>>>>
>>>>>> On Sat, Aug 3, 2019 at 10:34 AM Greg Mirsky <[e-mail]>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Dinesh,
>>>>>>> many thanks for your detailed updates on how some implementations
>>>>>>> process VXLAN header and the inner Ethernet frame. These are very helpful
>>>>>>> in achieving the workable solution for the problem at hand.
>>>>>>> You've noted that a path between VTEPs may be monitored in the
>>>>>>> underlay network by merely establishing a BFD session. That is true, but by
>>>>>>> using BFD with VXLAN encapsulation between the pair of VTEPs we are
>>>>>>> extending the OAM domain by including, to some extent, VXLAN forwarding
>>>>>>> engine. Abstract in RFC 5880 defines the goal and the domain in which BFD
>>>>>>> protocol can detect a fault as:
>>>>>>>    This document describes a protocol intended to detect faults in
>>>>>>> the
>>>>>>>    bidirectional path between two forwarding engines, including
>>>>>>>    interfaces, data link(s), and to the extent possible the
>>>>>>> forwarding
>>>>>>>    engines themselves, with potentially very low latency.
>>>>>>> Thus, BFD in the underlay will exercise a part of IP forwarding
>>>>>>> engine while BFD with VXLAN encapsulation, ran between the same pair of
>>>>>>> VTEPs, extends the OAM domain. At the same time, defining BFD between
>>>>>>> tenant systems in outside the goal of this specification. But VXLAN BFD
>>>>>>> session between VTEPs may be useful in monitoring e2e path between tenants,
>>>>>>> as described in the update to -07:
>>>>>>>    At the same time, a service layer BFD session may be used between
>>>>>>> the
>>>>>>>    tenants of VTEPs IP1 and IP2 to provide end-to-end fault
>>>>>>> management.
>>>>>>>    In such case, for VTEPs BFD control packets of that session are
>>>>>>>    indistinguishable from data packets.  If end-to-end defect
>>>>>>> detection
>>>>>>>    is realized as the set of concatenated OAM domains, e.g., VM1-1 -
>>>>>>> IP1
>>>>>>>    -- IP2 - VM2-1, then the BFD session over VXLAN between VTEPs
>>>>>>> SHOULD
>>>>>>>    follow the procedures described in Section 6.8.17 [RFC5880].
>>>>>>> I've attached the current working version of the draft.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Greg
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Aug 2, 2019 at 5:43 PM Dinesh Dutt <[e-mail]> wrote:
>>>>>>>
>>>>>>>> What I mean is "How do you infer that it excludes the case I'm
>>>>>>>> talking about?".
>>>>>>>>
>>>>>>>> Dinesh
>>>>>>>>
>>>>>>>> On Fri, Aug 2, 2019 at 5:41 PM Dinesh Dutt <[e-mail]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> The abstract reads this: "
>>>>>>>>>
>>>>>>>>> This document describes the use of the Bidirectional Forwarding
>>>>>>>>>    Detection (BFD) protocol in point-to-point Virtual eXtensible Local
>>>>>>>>>    Area Network (VXLAN) tunnels forming up an overlay network."
>>>>>>>>>
>>>>>>>>> How do you infer what you said?
>>>>>>>>>
>>>>>>>>> Dinesh
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Aug 2, 2019 at 5:38 PM Joel M. Halpern <
>>>>>>>>> [e-mail]> wrote:
>>>>>>>>>
>>>>>>>>>> I am going by what the draft says its purpose is.  If you
>>>>>>>>>> (Dinesh) want
>>>>>>>>>> the draft to fulfill a different purpose, then either ask the
>>>>>>>>>> chairs to
>>>>>>>>>> take this draft back to the WG, or write a separate draft.
>>>>>>>>>> As currently written, the behavior Greg proposed meets the needs,
>>>>>>>>>> and
>>>>>>>>>> does so in a way that is consistent with VxLAN.
>>>>>>>>>>
>>>>>>>>>> Yours,
>>>>>>>>>> Joel
>>>>>>>>>>
>>>>>>>>>> On 8/2/2019 8:30 PM, Dinesh Dutt wrote:
>>>>>>>>>> > What is the stated purpose of this BFD session? The VTEP
>>>>>>>>>> reachability is
>>>>>>>>>> > determined by the underlay, I don't need VXLAN-encaped packet
>>>>>>>>>> for that.
>>>>>>>>>> > Do we agree?
>>>>>>>>>> >
>>>>>>>>>> > If I want to test the VXLAN encap/decap functionality alone,
>>>>>>>>>> picking any
>>>>>>>>>> > single VNI maybe fine. But is this all any network operator
>>>>>>>>>> wants? Why?
>>>>>>>>>> > In what situations has this been a problem? I suspect operators
>>>>>>>>>> also
>>>>>>>>>> > want to verify path continuity over a specific VNI. If you say
>>>>>>>>>> this is
>>>>>>>>>> > not defined by the document, I disagree because the current
>>>>>>>>>> version
>>>>>>>>>> > talks about controlling the number of BFD sessions between the
>>>>>>>>>> VTEPs
>>>>>>>>>> > (see section 3). More importantly, this is a real problem that
>>>>>>>>>> operators
>>>>>>>>>> > like to verify.
>>>>>>>>>> >
>>>>>>>>>> > Dinesh
>>>>>>>>>> >
>>>>>>>>>> > On Fri, Aug 2, 2019 at 5:08 PM Joel M. Halpern <
>>>>>>>>>> [e-mail]
>>>>>>>>>> > <mailto:[e-mail]>> wrote:
>>>>>>>>>> >
>>>>>>>>>> >     What is special about the management VNI is precisely that
>>>>>>>>>> it is NOT a
>>>>>>>>>> >     tenant VNI.  The VxLAN administration does know how it
>>>>>>>>>> allocates VNI to
>>>>>>>>>> >     tenants, and which VNI it has allocated.  In contrast, it
>>>>>>>>>> does not know
>>>>>>>>>> >     which IP addresses or MAC adddresses teh tenant is using or
>>>>>>>>>> may plan
>>>>>>>>>> >     to use.
>>>>>>>>>> >
>>>>>>>>>> >     Yours,
>>>>>>>>>> >     Joel
>>>>>>>>>> >
>>>>>>>>>> >     On 8/2/2019 6:41 PM, Dinesh Dutt wrote:
>>>>>>>>>> >      > The assumption of an IP address within any VNI is
>>>>>>>>>> suspect that way.
>>>>>>>>>> >      > What's special about a single VNI, the management VNI?
>>>>>>>>>> The VTEP IP
>>>>>>>>>> >      > address does not belong in reality in any VNI.
>>>>>>>>>> >      >
>>>>>>>>>> >      > Dinesh
>>>>>>>>>> >      >
>>>>>>>>>> >      > On Fri, Aug 2, 2019 at 3:17 PM Joel M. Halpern
>>>>>>>>>> >     <[e-mail] <mailto:[e-mail]>
>>>>>>>>>> >      > <mailto:[e-mail] <mailto:[e-mail]>>>
>>>>>>>>>> wrote:
>>>>>>>>>> >      >
>>>>>>>>>> >      >     Your response seems to miss two points:
>>>>>>>>>> >      >
>>>>>>>>>> >      >     First, the problem you describe is not what the
>>>>>>>>>> document says
>>>>>>>>>> >     it is
>>>>>>>>>> >      >     solving.  To the degree it discusses it at all, the
>>>>>>>>>> document
>>>>>>>>>> >     says "
>>>>>>>>>> >      >       In
>>>>>>>>>> >      >     most cases, a single BFD session is sufficient for
>>>>>>>>>> the given
>>>>>>>>>> >     VTEP to
>>>>>>>>>> >      >     monitor the reachability of a remote VTEP,
>>>>>>>>>> regardless of the
>>>>>>>>>> >     number of
>>>>>>>>>> >      >     VNIs in common. "
>>>>>>>>>> >      >
>>>>>>>>>> >      >     Second, you assume the existence of an IP address
>>>>>>>>>> for a VTEP
>>>>>>>>>> >     within a
>>>>>>>>>> >      >     VNI.  As with the MAC address, the VTEP does not
>>>>>>>>>> have an IP
>>>>>>>>>> >     address
>>>>>>>>>> >      >     within the VNI.  Some implementations may have
>>>>>>>>>> created such a
>>>>>>>>>> >     thing,
>>>>>>>>>> >      >     but
>>>>>>>>>> >      >     the general construct, as defined to date, does not
>>>>>>>>>> support such.
>>>>>>>>>> >      >
>>>>>>>>>> >      >     In short, you are requiring a behavior that violates
>>>>>>>>>> the
>>>>>>>>>> >     architectural
>>>>>>>>>> >      >     structure of overlay / underlay separation, and
>>>>>>>>>> common
>>>>>>>>>> >     usage.  And you
>>>>>>>>>> >      >     are doing so to support a use case that the working
>>>>>>>>>> group has not
>>>>>>>>>> >      >     indicated in the document as important.
>>>>>>>>>> >      >
>>>>>>>>>> >      >     Yours,
>>>>>>>>>> >      >     Joel
>>>>>>>>>> >      >
>>>>>>>>>> >      >     On 8/2/2019 5:01 PM, Dinesh Dutt wrote:
>>>>>>>>>> >      >      > Joel,
>>>>>>>>>> >      >      >
>>>>>>>>>> >      >      > You understood correctly.
>>>>>>>>>> >      >      >
>>>>>>>>>> >      >      > The VNIs may not share fate due to
>>>>>>>>>> misconfiguration. And I
>>>>>>>>>> >     strongly
>>>>>>>>>> >      >      > suspect someone will want to use BFD for that
>>>>>>>>>> because its
>>>>>>>>>> >     about
>>>>>>>>>> >      >     checking
>>>>>>>>>> >      >      > path continuity as stated by the draft. As long
>>>>>>>>>> as there's a
>>>>>>>>>> >      >     valid IP
>>>>>>>>>> >      >      > (because it's BFD) owned by the VTEP in that VNI,
>>>>>>>>>> you can
>>>>>>>>>> >     use BFD in
>>>>>>>>>> >      >      > that VNI. Thats all that you need to dictate.
>>>>>>>>>> That IP address
>>>>>>>>>> >      >     has a MAC
>>>>>>>>>> >      >      > address and you can use that on the inner frame.
>>>>>>>>>> That is
>>>>>>>>>> >     all normal
>>>>>>>>>> >      >      > VXLAN processing. The outer IP is always that of
>>>>>>>>>> the VTEP.
>>>>>>>>>> >      >      >
>>>>>>>>>> >      >      > Dinesh
>>>>>>>>>> >      >      >
>>>>>>>>>> >      >      > On Fri, Aug 2, 2019 at 11:03 AM Joel M. Halpern
>>>>>>>>>> >      >     <[e-mail] <mailto:[e-mail]>
>>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>
>>>>>>>>>> >      >      > <mailto:[e-mail] <mailto:
>>>>>>>>>> [e-mail]>
>>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>>>
>>>>>>>>>> wrote:
>>>>>>>>>> >      >      >
>>>>>>>>>> >      >      >     If I am reading your various emails correctly
>>>>>>>>>> Dinesh
>>>>>>>>>> >     (and I
>>>>>>>>>> >      >     may have
>>>>>>>>>> >      >      >     missed something) you are trying to use the
>>>>>>>>>> MAC address
>>>>>>>>>> >      >     because you
>>>>>>>>>> >      >      >     want
>>>>>>>>>> >      >      >     to be able to send these BFD packets over
>>>>>>>>>> arbitrary VNI to
>>>>>>>>>> >      >     monitor the
>>>>>>>>>> >      >      >     VNI.  That is not a requirement identified in
>>>>>>>>>> the
>>>>>>>>>> >     document.
>>>>>>>>>> >      >     It is not
>>>>>>>>>> >      >      >     even a problem I understand, since all the
>>>>>>>>>> VNI between an
>>>>>>>>>> >      >     ingress and
>>>>>>>>>> >      >      >     egress VTEP share fate.
>>>>>>>>>> >      >      >
>>>>>>>>>> >      >      >     Yours,
>>>>>>>>>> >      >      >     Joel
>>>>>>>>>> >      >      >
>>>>>>>>>> >      >      >     On 8/2/2019 1:44 PM, Dinesh Dutt wrote:
>>>>>>>>>> >      >      >      > Thanks for verifying this. On Linux and
>>>>>>>>>> hardware
>>>>>>>>>> >     routers
>>>>>>>>>> >      >     that I'm
>>>>>>>>>> >      >      >     aware
>>>>>>>>>> >      >      >      > of (Cisco circa 2012 and Cumulus), the
>>>>>>>>>> physical MAC
>>>>>>>>>> >     address is
>>>>>>>>>> >      >      >     reused
>>>>>>>>>> >      >      >      > across the VNIs on the VTEP. Did you check
>>>>>>>>>> on a non-VMW
>>>>>>>>>> >      >     device?
>>>>>>>>>> >      >      >     This is
>>>>>>>>>> >      >      >      > more for my own curiosity.
>>>>>>>>>> >      >      >      >
>>>>>>>>>> >      >      >      > To address the general case, can we not
>>>>>>>>>> define a
>>>>>>>>>> >      >     well-known (or
>>>>>>>>>> >      >      >     reserve
>>>>>>>>>> >      >      >      > one) unicast MAC address for use with
>>>>>>>>>> VTEP? If the MAC
>>>>>>>>>> >      >     address is
>>>>>>>>>> >      >      >      > configurable in BFD command, this can be
>>>>>>>>>> moot.
>>>>>>>>>> >      >      >      >
>>>>>>>>>> >      >      >      > Dinesh
>>>>>>>>>> >      >      >      >
>>>>>>>>>> >      >      >      > On Fri, Aug 2, 2019 at 10:27 AM Santosh P K
>>>>>>>>>> >      >      >      > <[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>>>
>>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>>>>> wrote:
>>>>>>>>>> >      >      >      >
>>>>>>>>>> >      >      >      >     I have cross checked point raised
>>>>>>>>>> about MAC address
>>>>>>>>>> >      >     usage. It is
>>>>>>>>>> >      >      >      >     possible that tenant could be using
>>>>>>>>>> physical MAC
>>>>>>>>>> >      >     address and
>>>>>>>>>> >      >      >     when a
>>>>>>>>>> >      >      >      >     packet comes with valid VNI with a MAC
>>>>>>>>>> address
>>>>>>>>>> >     that is
>>>>>>>>>> >      >     being
>>>>>>>>>> >      >      >     used by
>>>>>>>>>> >      >      >      >     tenant then packet will be sent to
>>>>>>>>>> that tenant.
>>>>>>>>>> >     This rules
>>>>>>>>>> >      >      >     out the
>>>>>>>>>> >      >      >      >     fact that we could use physical MAC
>>>>>>>>>> address as
>>>>>>>>>> >     inner
>>>>>>>>>> >      >     MAC to
>>>>>>>>>> >      >      >     ensure
>>>>>>>>>> >      >      >      >     packets get terminated at VTEP itself.
>>>>>>>>>> >      >      >      >
>>>>>>>>>> >      >      >      >     Thanks
>>>>>>>>>> >      >      >      >     Santosh P K
>>>>>>>>>> >      >      >      >
>>>>>>>>>> >      >      >      >     On Wed, Jul 31, 2019 at 11:00 AM
>>>>>>>>>> Santosh P K
>>>>>>>>>> >      >      >      >     <[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>>>
>>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>>>>>
>>>>>>>>>> >      >      >      >     wrote:
>>>>>>>>>> >      >      >      >
>>>>>>>>>> >      >      >      >         Joel,
>>>>>>>>>> >      >      >      >             Thanks for your inputs. I
>>>>>>>>>> checked
>>>>>>>>>> >      >     implementation within
>>>>>>>>>> >      >      >      >         Vmware. Perhaps I should have been
>>>>>>>>>> more clear
>>>>>>>>>> >      >     about MAC
>>>>>>>>>> >      >      >     address
>>>>>>>>>> >      >      >      >         space while checking internally. I
>>>>>>>>>> will cross
>>>>>>>>>> >      >     check again for
>>>>>>>>>> >      >      >      >         the same and get back on this list.
>>>>>>>>>> >      >      >      >
>>>>>>>>>> >      >      >      >         Thanks
>>>>>>>>>> >      >      >      >         Santosh P K
>>>>>>>>>> >      >      >      >
>>>>>>>>>> >      >      >      >         On Wed, Jul 31, 2019 at 10:54 AM
>>>>>>>>>> Joel M.
>>>>>>>>>> >     Halpern
>>>>>>>>>> >      >      >      >         <[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]> <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>>>> [e-mail]>
>>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>>
>>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]> <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>>>> [e-mail]>
>>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>> >      >      >      >
>>>>>>>>>> >      >      >      >             Sorry to ask a stupid
>>>>>>>>>> question.  Whose
>>>>>>>>>> >      >     implementation?
>>>>>>>>>> >      >      >      >
>>>>>>>>>> >      >      >      >             The reason I ask is that as
>>>>>>>>>> far as I
>>>>>>>>>> >     can tell,
>>>>>>>>>> >      >     since the
>>>>>>>>>> >      >      >      >             tenant does not
>>>>>>>>>> >      >      >      >             have any control access to the
>>>>>>>>>> VTEP,
>>>>>>>>>> >     there is no
>>>>>>>>>> >      >      >     reason for
>>>>>>>>>> >      >      >      >             the VTEP to
>>>>>>>>>> >      >      >      >             have a MAC address in the
>>>>>>>>>> tenant
>>>>>>>>>> >     space.  Yes, the
>>>>>>>>>> >      >      >     device has
>>>>>>>>>> >      >      >      >             a physical
>>>>>>>>>> >      >      >      >             MAC address.  But the tenant
>>>>>>>>>> could well be
>>>>>>>>>> >      >     using that MAC
>>>>>>>>>> >      >      >      >             address.  Yes,
>>>>>>>>>> >      >      >      >             they would be violating the
>>>>>>>>>> Ethernet spec.
>>>>>>>>>> >      >     But the whole
>>>>>>>>>> >      >      >      >             point of
>>>>>>>>>> >      >      >      >             segregation is not to care
>>>>>>>>>> about such
>>>>>>>>>> >     issues.
>>>>>>>>>> >      >      >      >
>>>>>>>>>> >      >      >      >             On the other hand, if you tell
>>>>>>>>>> me that
>>>>>>>>>> >     the VMWare
>>>>>>>>>> >      >      >      >             implementation has an
>>>>>>>>>> >      >      >      >             Ethernet address that is part
>>>>>>>>>> of the tenant
>>>>>>>>>> >      >     space, well,
>>>>>>>>>> >      >      >      >             they made up
>>>>>>>>>> >      >      >      >             this particular game.
>>>>>>>>>> >      >      >      >
>>>>>>>>>> >      >      >      >             Yours,
>>>>>>>>>> >      >      >      >             Joel
>>>>>>>>>> >      >      >      >
>>>>>>>>>> >      >      >      >             On 7/31/2019 1:44 PM, Santosh
>>>>>>>>>> P K wrote:
>>>>>>>>>> >      >      >      >              > I have checked with
>>>>>>>>>> implementation
>>>>>>>>>> >     in data
>>>>>>>>>> >      >     path.
>>>>>>>>>> >      >      >     When we
>>>>>>>>>> >      >      >      >             receive a
>>>>>>>>>> >      >      >      >              > packet with valid VNI then
>>>>>>>>>> lookup
>>>>>>>>>> >     for MAC will
>>>>>>>>>> >      >      >     happen and
>>>>>>>>>> >      >      >      >             it is VTEP own
>>>>>>>>>> >      >      >      >              > MAC then it will be trapped
>>>>>>>>>> to control
>>>>>>>>>> >      >     plane for
>>>>>>>>>> >      >      >      >             processing. I think we
>>>>>>>>>> >      >      >      >              > can have following options
>>>>>>>>>> >      >      >      >              > 1. Optional managment VNI
>>>>>>>>>> >      >      >      >              > 2. Mandatory inner MAC set
>>>>>>>>>> to VTEP mac
>>>>>>>>>> >      >      >      >              > 3. Inner IP TTL set to 1 to
>>>>>>>>>> avoid
>>>>>>>>>> >      >     forwarding of packet
>>>>>>>>>> >      >      >      >             via inner IP
>>>>>>>>>> >      >      >      >              > address.
>>>>>>>>>> >      >      >      >              >
>>>>>>>>>> >      >      >      >              >
>>>>>>>>>> >      >      >      >              > Thoughts?
>>>>>>>>>> >      >      >      >              >
>>>>>>>>>> >      >      >      >              > Thansk
>>>>>>>>>> >      >      >      >              > Santosh P K
>>>>>>>>>> >      >      >      >              >
>>>>>>>>>> >      >      >      >              > On Wed, Jul 31, 2019 at
>>>>>>>>>> 9:20 AM Greg
>>>>>>>>>> >     Mirsky
>>>>>>>>>> >      >      >      >             <[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>>>> [e-mail]>>
>>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]
>>>>>>>>>> >
>>>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>>>> [e-mail]>>>
>>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]> <mailto:
>>>>>>>>>> [e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>>>> [e-mail]>
>>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]
>>>>>>>>>> >>>>
>>>>>>>>>> >      >      >      >              > <mailto:
>>>>>>>>>> [e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>>>> [e-mail]>>
>>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]> <mailto:
>>>>>>>>>> [e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>>>
>>>>>>>>>> >      >      >      >             <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>>>> [e-mail]>>
>>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>>>>>> wrote:
>>>>>>>>>> >      >      >      >              >
>>>>>>>>>> >      >      >      >              >     Hi Dinesh,
>>>>>>>>>> >      >      >      >              >     thank you for your
>>>>>>>>>> consideration
>>>>>>>>>> >     of the
>>>>>>>>>> >      >      >     proposal and
>>>>>>>>>> >      >      >      >             questions. What
>>>>>>>>>> >      >      >      >              >     would you see as the
>>>>>>>>>> scope of
>>>>>>>>>> >     testing the
>>>>>>>>>> >      >      >      >             connectivity for the
>>>>>>>>>> >      >      >      >              >     specific VNI? If it is
>>>>>>>>>> >      >     tenant-to-tenant, then
>>>>>>>>>> >      >      >     VTEPs
>>>>>>>>>> >      >      >      >             will treat these
>>>>>>>>>> >      >      >      >              >     packets as regular user
>>>>>>>>>> frames. More
>>>>>>>>>> >      >     likely, these
>>>>>>>>>> >      >      >      >             could be Layer 2
>>>>>>>>>> >      >      >      >              >     OAM, e.g. CCM frames.
>>>>>>>>>> The reason
>>>>>>>>>> >     to use
>>>>>>>>>> >      >     127/8 for
>>>>>>>>>> >      >      >      >             IPv4, and
>>>>>>>>>> >      >      >      >              >
>>>>>>>>>>  0:0:0:0:0:FFFF:7F00:0/104 for
>>>>>>>>>> >     IPv6 is
>>>>>>>>>> >      >     to safeguard
>>>>>>>>>> >      >      >      >             from leaking
>>>>>>>>>> >      >      >      >              >     Ethernet frames with
>>>>>>>>>> BFD Control
>>>>>>>>>> >     packet
>>>>>>>>>> >      >     to a
>>>>>>>>>> >      >      >     tenant.
>>>>>>>>>> >      >      >      >              >     You've suggested using
>>>>>>>>>> a MAC
>>>>>>>>>> >     address to
>>>>>>>>>> >      >     trap the
>>>>>>>>>> >      >      >      >             control packet at
>>>>>>>>>> >      >      >      >              >     VTEP. What that address
>>>>>>>>>> could be? We
>>>>>>>>>> >      >     had proposed
>>>>>>>>>> >      >      >      >             using the
>>>>>>>>>> >      >      >      >              >     dedicated MAC and
>>>>>>>>>> VTEP's MAC and
>>>>>>>>>> >     both
>>>>>>>>>> >      >     raised
>>>>>>>>>> >      >      >     concerns
>>>>>>>>>> >      >      >      >             among VXLAN
>>>>>>>>>> >      >      >      >              >     experts. The idea of
>>>>>>>>>> using
>>>>>>>>>> >     Management
>>>>>>>>>> >      >     VNI may
>>>>>>>>>> >      >      >     be more
>>>>>>>>>> >      >      >      >             acceptable
>>>>>>>>>> >      >      >      >              >     based on its similarity
>>>>>>>>>> to the
>>>>>>>>>> >     practice
>>>>>>>>>> >      >     of using
>>>>>>>>>> >      >      >      >             Management VLAN.
>>>>>>>>>> >      >      >      >              >
>>>>>>>>>> >      >      >      >              >     Regards,
>>>>>>>>>> >      >      >      >              >     Greg
>>>>>>>>>> >      >      >      >              >
>>>>>>>>>> >      >      >      >              >     On Wed, Jul 31, 2019 at
>>>>>>>>>> 12:03 PM
>>>>>>>>>> >     Dinesh
>>>>>>>>>> >      >     Dutt
>>>>>>>>>> >      >      >      >             <[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]> <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>>>> >      >     <mailto:[e-mail] <mailto:[e-mail]>
>>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>>
>>>>>>>>>> >      >      >     <mailto:[e-mail] <mailto:
>>>>>>>>>> [e-mail]>
>>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>
>>>>>>>>>> >      >     <mailto:[e-mail] <mailto:[e-mail]>
>>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>>>
>>>>>>>>>> >      >      >      >              >     <mailto:
>>>>>>>>>> [e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>>> >      >     <mailto:[e-mail] <mailto:[e-mail]>>
>>>>>>>>>> >      >      >     <mailto:[e-mail] <mailto:
>>>>>>>>>> [e-mail]>
>>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>>
>>>>>>>>>> >      >     <mailto:[e-mail] <mailto:[e-mail]>
>>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>
>>>>>>>>>> >      >      >     <mailto:[e-mail] <mailto:
>>>>>>>>>> [e-mail]>
>>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>>>>>
>>>>>>>>>> >      >      >      >             wrote:
>>>>>>>>>> >      >      >      >              >
>>>>>>>>>> >      >      >      >              >         Hi Greg,
>>>>>>>>>> >      >      >      >              >
>>>>>>>>>> >      >      >      >              >         As long as the
>>>>>>>>>> inner MAC
>>>>>>>>>> >     address is
>>>>>>>>>> >      >     such
>>>>>>>>>> >      >      >     that the
>>>>>>>>>> >      >      >      >             packet is
>>>>>>>>>> >      >      >      >              >         trapped to the CPU,
>>>>>>>>>> it should be
>>>>>>>>>> >      >     fine for
>>>>>>>>>> >      >      >     use as
>>>>>>>>>> >      >      >      >             an inner MAC is
>>>>>>>>>> >      >      >      >              >         it not? Stating
>>>>>>>>>> that is
>>>>>>>>>> >     better than
>>>>>>>>>> >      >     trying to
>>>>>>>>>> >      >      >      >             force a management
>>>>>>>>>> >      >      >      >              >         VNI. What if
>>>>>>>>>> someone wants
>>>>>>>>>> >     to test
>>>>>>>>>> >      >      >     connectivity
>>>>>>>>>> >      >      >      >             on a specific
>>>>>>>>>> >      >      >      >              >         VNI? I would not
>>>>>>>>>> pick a
>>>>>>>>>> >     loopback IP
>>>>>>>>>> >      >      >     address for
>>>>>>>>>> >      >      >      >             this since that
>>>>>>>>>> >      >      >      >              >         address range is
>>>>>>>>>> host/node local
>>>>>>>>>> >      >     only. Is
>>>>>>>>>> >      >      >     there a
>>>>>>>>>> >      >      >      >             reason you're
>>>>>>>>>> >      >      >      >              >         not using the VTEP
>>>>>>>>>> IP as the
>>>>>>>>>> >     inner IP
>>>>>>>>>> >      >      >     address ?
>>>>>>>>>> >      >      >      >              >
>>>>>>>>>> >      >      >      >              >         Dinesh
>>>>>>>>>> >      >      >      >              >
>>>>>>>>>> >      >      >      >              >         On Wed, Jul 31,
>>>>>>>>>> 2019 at 5:48 AM
>>>>>>>>>> >      >     Greg Mirsky
>>>>>>>>>> >      >      >      >              >         <
>>>>>>>>>> [e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>>>> [e-mail]>>
>>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]> <mailto:
>>>>>>>>>> [e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>>>
>>>>>>>>>> >      >      >      >             <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>>>> [e-mail]>>
>>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>>>> <mailto:
>>>>>>>>>> [e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>>>> [e-mail]>>
>>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]> <mailto:
>>>>>>>>>> [e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>>>
>>>>>>>>>> >      >      >      >             <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>>>> [e-mail]>>
>>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>>>> >     <mailto:[e-mail]>>>>>> wrote:
>>>>>>>>>> >      >      >      >              >
>>>>>>>>>> >      >      >      >              >             Dear All,
>>>>>>>>>> >      >      >      >              >             thank you for
>>>>>>>>>> your comments,
>>>>>>>>>> >      >      >     suggestions on
>>>>>>>>>> >      >      >      >             this issue,
>>>>>>>>>> >      >      >      >              >             probably the
>>>>>>>>>> most
>>>>>>>>>> >     challenging
>>>>>>>>>> >      >     for this
>>>>>>>>>> >      >      >      >             specification. In the
>>>>>>>>>> >      >      >      >              >             course of our
>>>>>>>>>> discussions,
>>>>>>>>>> >      >     we've agreed to
>>>>>>>>>> >      >      >      >             abandon the
>>>>>>>>>> >      >      >      >              >             request to
>>>>>>>>>> allocate the
>>>>>>>>>> >      >     dedicated MAC
>>>>>>>>>> >      >      >     address
>>>>>>>>>> >      >      >      >             to be used as
>>>>>>>>>> >      >      >      >              >             the destination
>>>>>>>>>> MAC
>>>>>>>>>> >     address in
>>>>>>>>>> >      >     the inner
>>>>>>>>>> >      >      >      >             Ethernet frame.
>>>>>>>>>> >      >      >      >              >             Also, earlier
>>>>>>>>>> using VNI
>>>>>>>>>> >     0 was
>>>>>>>>>> >      >     changed from
>>>>>>>>>> >      >      >      >             mandatory to one
>>>>>>>>>> >      >      >      >              >             of the options
>>>>>>>>>> an
>>>>>>>>>> >      >     implementation may
>>>>>>>>>> >      >      >     offer to
>>>>>>>>>> >      >      >      >             an operator.
>>>>>>>>>> >      >      >      >              >             The most recent
>>>>>>>>>> >     discussion was
>>>>>>>>>> >      >     whether
>>>>>>>>>> >      >      >     VTEP's
>>>>>>>>>> >      >      >      >             MAC address
>>>>>>>>>> >      >      >      >              >             might be used
>>>>>>>>>> as the
>>>>>>>>>> >      >     destination MAC
>>>>>>>>>> >      >      >     address
>>>>>>>>>> >      >      >      >             in the inner
>>>>>>>>>> >      >      >      >              >             Ethernet frame.
>>>>>>>>>> As I
>>>>>>>>>> >     recall it, the
>>>>>>>>>> >      >      >     comments
>>>>>>>>>> >      >      >      >             from VXLAN
>>>>>>>>>> >      >      >      >              >             experts equally
>>>>>>>>>> split
>>>>>>>>>> >     with one
>>>>>>>>>> >      >     for it
>>>>>>>>>> >      >      >     and one
>>>>>>>>>> >      >      >      >             against. Hence
>>>>>>>>>> >      >      >      >              >             I would like to
>>>>>>>>>> propose
>>>>>>>>>> >     a new
>>>>>>>>>> >      >     text to
>>>>>>>>>> >      >      >     resolve
>>>>>>>>>> >      >      >      >             the issue. The
>>>>>>>>>> >      >      >      >              >             idea is to let
>>>>>>>>>> an
>>>>>>>>>> >     operator select
>>>>>>>>>> >      >      >     Management
>>>>>>>>>> >      >      >      >             VNI and use
>>>>>>>>>> >      >      >      >              >             that VNI in
>>>>>>>>>> VXLAN
>>>>>>>>>> >     encapsulation
>>>>>>>>>> >      >     of BFD
>>>>>>>>>> >      >      >      >             Control packets:
>>>>>>>>>> >      >      >      >              >             NEW TEXT:
>>>>>>>>>> >      >      >      >              >
>>>>>>>>>> >      >      >      >              >                 An operator
>>>>>>>>>> MUST
>>>>>>>>>> >     select a VNI
>>>>>>>>>> >      >      >     number to
>>>>>>>>>> >      >      >      >             be used as
>>>>>>>>>> >      >      >      >              >                 Management
>>>>>>>>>> VNI. VXLAN
>>>>>>>>>> >      >     packet for
>>>>>>>>>> >      >      >      >             Management VNI MUST NOT
>>>>>>>>>> >      >      >      >              >                 be sent to
>>>>>>>>>> a tenant. VNI
>>>>>>>>>> >      >     number 1 is
>>>>>>>>>> >      >      >      >             RECOMMENDED as the
>>>>>>>>>> >      >      >      >              >                 default for
>>>>>>>>>> >     Management VNI.
>>>>>>>>>> >      >      >      >              >
>>>>>>>>>> >      >      >      >              >             With that new
>>>>>>>>>> text, what
>>>>>>>>>> >     can be the
>>>>>>>>>> >      >      >     value of
>>>>>>>>>> >      >      >      >             the destination
>>>>>>>>>> >      >      >      >              >             MAC in the
>>>>>>>>>> inner Ethernet? I
>>>>>>>>>> >      >     tend to
>>>>>>>>>> >      >      >     believe
>>>>>>>>>> >      >      >      >             that it can be
>>>>>>>>>> >      >      >      >              >             anything and
>>>>>>>>>> ignored by the
>>>>>>>>>> >      >     reciever VTEP.
>>>>>>>>>> >      >      >      >             Also, if the
>>>>>>>>>> >      >      >      >              >             trapping is
>>>>>>>>>> based on VNI
>>>>>>>>>> >      >     number, the
>>>>>>>>>> >      >      >      >             destination IP address
>>>>>>>>>> >      >      >      >              >             of the inner IP
>>>>>>>>>> packet
>>>>>>>>>> >     can from
>>>>>>>>>> >      >     the range
>>>>>>>>>> >      >      >      >             127/8 for IPv4,
>>>>>>>>>> >      >      >      >              >             and for IPv6
>>>>>>>>>> from the range
>>>>>>>>>> >      >      >      >             0:0:0:0:0:FFFF:7F00:0/104. And
>>>>>>>>>> >      >      >      >              >             lastly, the TTL
>>>>>>>>>> to be
>>>>>>>>>> >     set to 1 (no
>>>>>>>>>> >      >      >     change here).
>>>>>>>>>> >      >      >      >              >
>>>>>>>>>> >      >      >      >              >             Much appreciate
>>>>>>>>>> your
>>>>>>>>>> >     comments,
>>>>>>>>>> >      >      >     questions, and
>>>>>>>>>> >      >      >      >             suggestions.
>>>>>>>>>> >      >      >      >              >
>>>>>>>>>> >      >      >      >              >             Best regards,
>>>>>>>>>> >      >      >      >              >             Greg
>>>>>>>>>> >      >      >      >              >
>>>>>>>>>> >      >      >      >
>>>>>>>>>> >      >      >
>>>>>>>>>> >      >
>>>>>>>>>> >
>>>>>>>>>>
>>>>>>>>>