Re: BFD over VXLAN: Trapping BFD Control packet at VTEP

[Greg Mirsky <[e-mail]>]

Hi Dinesh, Joel, Sridhar, et al.,
much appreciate the help you've given me sharing your expertise. I hope
that the updates you will find in the attached diff and the working copy of
the draft be closer to the acceptable solution for VTEP-VTEP BFD. Please
note, that I'll shortly start a new discussion thread to address one of
Carlos's questions on the ambiguity of the text on multiple concurrent
sessions between the same pair of VTEPs.
Please review the changes to Sections 4 and 6 and share your feedback,
suggestions, and questions.

Regards,
Greg

On Mon, Aug 5, 2019 at 6:03 PM Dinesh Dutt <[e-mail]> wrote:

>
>
> On Mon, Aug 5, 2019 at 5:56 PM Greg Mirsky <[e-mail]> wrote:
>
>> Hi Dinesh,
>> thank you for your expedient detailed response.
>> I believe that the ability to run BFD session up to a tenant
>> (VTEP-VTEP-tenant or tenant-tenant) was never in jeopardy from this
>> specification.
>> I'm trying to provide precise specification on what can be used ad the
>> destination MAC and IP addresses in the inner frame/packet as I believe
>> that likely will help to avoid interoperability issues.
>> I'm interested to learn some more about the "martian checking" function.
>> As you know, martian addresses have been used as destination IP address in
>> LSP Ping and BFD over MPLS LSP and PW. I haven't heard that any silicon
>> feature caused problems for operators using these tools.
>>
>
> Interesting. I didn't know this aspect of use with MPLS ping. Did those
> packets ever go through a firewall though? In any case, maybe suggest the
> use of those addresses with a statement that this is how LSP does it, but
> that other MAC/IP pairs are possible as long as the conditions of the
> endpoint owning the MAC/IP was honored.
>
> Dinesh
>
>>
>> Regards,
>> Greg
>>
>> On Mon, Aug 5, 2019 at 3:59 PM Dinesh Dutt <[e-mail]> wrote:
>>
>>> Hi Greg,
>>>
>>> That we agree on the problem definition is the first step forward. Your
>>> original document had my cases covered and so I was surprised by the track
>>> this thread took. It doesn't matter, we're back on track.
>>>
>>> My recommendation is to not worry about specifying the precise MAC/IP
>>> address used in the inner header. The addresses chosen MUST ensure that the
>>> packet is trapped to the control plane of the VTEP and not escape to the
>>> tenant if the BFD is to the VTEP. Any solution MUST also not preclude the
>>> use of the BFD by tenant systems for that VNI. There are many ways an
>>> implementer can choose to implement this. For example, the inner MAC
>>> address is whatever the VTEP implementer would return if ARP'd for the IP
>>> address used in the inner header in the given VNI. The implementer can pick
>>> a fixed MAC address, one that they own etc. Multiple BFD sessions can be
>>> run for testing path connectivity on more than one VNIs. Limits should be
>>> in place to avoid overwhelming the receiver with BFD messages (you had
>>> words about this in your currently published draft).  If the VNI is
>>> irrelevant in the test i.e. only the VXLAN pipe at the VTEP is being
>>> tested. the user can use any VNI active on the VTEP on which the VTEP owns
>>> an IP address.
>>>
>>> I'm concerned about the use of 127/8 address only because of firewalls
>>> or implementations that drop packets with these addresses as either the
>>> source or destination. For example, on many merchant silicon, I don't
>>> believe you can turn off martian checking and drops *only* for
>>> VXLAN-encapsulated BFD packets. I don't know what the Linux kernel does
>>> today on such packets, for example (or Hyper-V). I'd like a solution that
>>> doesn't demand additional or new chip functionality or require additional
>>> middle-box hole punch.
>>>
>>> Why do you feel you MUST to specify the MAC/IP address on the inner
>>> packet? What am I missing here?
>>>
>>> Dinesh
>>>
>>> On Mon, Aug 5, 2019 at 3:04 PM Greg Mirsky <[e-mail]>
>>> wrote:
>>>
>>>> Hi Dinesh,
>>>> what do you see as the way forward? I agree, that the proposed text
>>>> doesn't work for multi-VNI concurrent monitoring because these VNIs are
>>>> tenant's VNIs. And in that case, we need to specify another mechanism to
>>>> trap the BFD Control packet at VTEP. It seems that VTEP's Ethernet address
>>>> must be used as the destination MAC address in the inner Ethernet frame.
>>>> The destination IP address may be either VTEP's address of martian (I do
>>>> prefer martian). Let me give it  try:
>>>> NEW TEXT:
>>>>
>>>> To monitor continuity of the path between two VTEPs, an operator MUST
>>>> select a VNI number to be used as Management VNI. Management VNI number
>>>> MUST NOT be one of the tenant's VNIs to prevent sending VXLAN packets
>>>> received on Management VNI to a tenant. VNI number 1 is RECOMMENDED as the
>>>> default for Management VNI. [Ed.note: What we set the Destination MAC to?
>>>> Can it be invalid MAC that MUST be ignored on receipt?]
>>>>
>>>> If an implementation supports concurrent monitoring of multiple VNIs,
>>>> then the value of VNI number MAY be one of tenant's VNIs. The destination
>>>> MAC address in the inner Ethernet frame encapsulating BFD Control packet
>>>> MUST be MAC associated with the remote VTEP.
>>>> The destination IP address of the inner IP packet MUST be selected from
>>>> the range 127/8 for IPv4, and for IPv6 from the range
>>>> 0:0:0:0:0:FFFF:7F00:0/104. The TTL value in the inner IP header MUST be set
>>>> to 1.
>>>>
>>>> Regards,
>>>> Greg
>>>>
>>>> On Sun, Aug 4, 2019 at 9:07 AM Dinesh Dutt <[e-mail]> wrote:
>>>>
>>>>> Hi Greg,
>>>>>
>>>>> Thanks for your clarifications. I agree with your sentiment on why
>>>>> you're running BFD over VXLAN between VTEPs. I wasn't arguing against it at
>>>>> all. All I was saying was pointing to the limitations of the use of
>>>>> management VNI. I spoke to some operators who're running EVPN and mentioned
>>>>> the discussion on this thread. They concur that they're using specific VNIs
>>>>> to test connectivity over that VNI between VTEPs to ensure misconfiguration
>>>>> doesn't lead to blackholes. My statements are based in real world operator
>>>>> experience. And I was providing language that ensured packets didn't leak
>>>>> across to tenants when they were destined to VTEPs.
>>>>>
>>>>> Dinesh
>>>>>
>>>>> On Sat, Aug 3, 2019 at 10:34 AM Greg Mirsky <[e-mail]>
>>>>> wrote:
>>>>>
>>>>>> Hi Dinesh,
>>>>>> many thanks for your detailed updates on how some implementations
>>>>>> process VXLAN header and the inner Ethernet frame. These are very helpful
>>>>>> in achieving the workable solution for the problem at hand.
>>>>>> You've noted that a path between VTEPs may be monitored in the
>>>>>> underlay network by merely establishing a BFD session. That is true, but by
>>>>>> using BFD with VXLAN encapsulation between the pair of VTEPs we are
>>>>>> extending the OAM domain by including, to some extent, VXLAN forwarding
>>>>>> engine. Abstract in RFC 5880 defines the goal and the domain in which BFD
>>>>>> protocol can detect a fault as:
>>>>>>    This document describes a protocol intended to detect faults in the
>>>>>>    bidirectional path between two forwarding engines, including
>>>>>>    interfaces, data link(s), and to the extent possible the forwarding
>>>>>>    engines themselves, with potentially very low latency.
>>>>>> Thus, BFD in the underlay will exercise a part of IP forwarding
>>>>>> engine while BFD with VXLAN encapsulation, ran between the same pair of
>>>>>> VTEPs, extends the OAM domain. At the same time, defining BFD between
>>>>>> tenant systems in outside the goal of this specification. But VXLAN BFD
>>>>>> session between VTEPs may be useful in monitoring e2e path between tenants,
>>>>>> as described in the update to -07:
>>>>>>    At the same time, a service layer BFD session may be used between
>>>>>> the
>>>>>>    tenants of VTEPs IP1 and IP2 to provide end-to-end fault
>>>>>> management.
>>>>>>    In such case, for VTEPs BFD control packets of that session are
>>>>>>    indistinguishable from data packets.  If end-to-end defect
>>>>>> detection
>>>>>>    is realized as the set of concatenated OAM domains, e.g., VM1-1 -
>>>>>> IP1
>>>>>>    -- IP2 - VM2-1, then the BFD session over VXLAN between VTEPs
>>>>>> SHOULD
>>>>>>    follow the procedures described in Section 6.8.17 [RFC5880].
>>>>>> I've attached the current working version of the draft.
>>>>>>
>>>>>> Regards,
>>>>>> Greg
>>>>>>
>>>>>>
>>>>>> On Fri, Aug 2, 2019 at 5:43 PM Dinesh Dutt <[e-mail]> wrote:
>>>>>>
>>>>>>> What I mean is "How do you infer that it excludes the case I'm
>>>>>>> talking about?".
>>>>>>>
>>>>>>> Dinesh
>>>>>>>
>>>>>>> On Fri, Aug 2, 2019 at 5:41 PM Dinesh Dutt <[e-mail]> wrote:
>>>>>>>
>>>>>>>> The abstract reads this: "
>>>>>>>>
>>>>>>>> This document describes the use of the Bidirectional Forwarding
>>>>>>>>    Detection (BFD) protocol in point-to-point Virtual eXtensible Local
>>>>>>>>    Area Network (VXLAN) tunnels forming up an overlay network."
>>>>>>>>
>>>>>>>> How do you infer what you said?
>>>>>>>>
>>>>>>>> Dinesh
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Aug 2, 2019 at 5:38 PM Joel M. Halpern <[e-mail]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I am going by what the draft says its purpose is.  If you (Dinesh)
>>>>>>>>> want
>>>>>>>>> the draft to fulfill a different purpose, then either ask the
>>>>>>>>> chairs to
>>>>>>>>> take this draft back to the WG, or write a separate draft.
>>>>>>>>> As currently written, the behavior Greg proposed meets the needs,
>>>>>>>>> and
>>>>>>>>> does so in a way that is consistent with VxLAN.
>>>>>>>>>
>>>>>>>>> Yours,
>>>>>>>>> Joel
>>>>>>>>>
>>>>>>>>> On 8/2/2019 8:30 PM, Dinesh Dutt wrote:
>>>>>>>>> > What is the stated purpose of this BFD session? The VTEP
>>>>>>>>> reachability is
>>>>>>>>> > determined by the underlay, I don't need VXLAN-encaped packet
>>>>>>>>> for that.
>>>>>>>>> > Do we agree?
>>>>>>>>> >
>>>>>>>>> > If I want to test the VXLAN encap/decap functionality alone,
>>>>>>>>> picking any
>>>>>>>>> > single VNI maybe fine. But is this all any network operator
>>>>>>>>> wants? Why?
>>>>>>>>> > In what situations has this been a problem? I suspect operators
>>>>>>>>> also
>>>>>>>>> > want to verify path continuity over a specific VNI. If you say
>>>>>>>>> this is
>>>>>>>>> > not defined by the document, I disagree because the current
>>>>>>>>> version
>>>>>>>>> > talks about controlling the number of BFD sessions between the
>>>>>>>>> VTEPs
>>>>>>>>> > (see section 3). More importantly, this is a real problem that
>>>>>>>>> operators
>>>>>>>>> > like to verify.
>>>>>>>>> >
>>>>>>>>> > Dinesh
>>>>>>>>> >
>>>>>>>>> > On Fri, Aug 2, 2019 at 5:08 PM Joel M. Halpern <
>>>>>>>>> [e-mail]
>>>>>>>>> > <mailto:[e-mail]>> wrote:
>>>>>>>>> >
>>>>>>>>> >     What is special about the management VNI is precisely that
>>>>>>>>> it is NOT a
>>>>>>>>> >     tenant VNI.  The VxLAN administration does know how it
>>>>>>>>> allocates VNI to
>>>>>>>>> >     tenants, and which VNI it has allocated.  In contrast, it
>>>>>>>>> does not know
>>>>>>>>> >     which IP addresses or MAC adddresses teh tenant is using or
>>>>>>>>> may plan
>>>>>>>>> >     to use.
>>>>>>>>> >
>>>>>>>>> >     Yours,
>>>>>>>>> >     Joel
>>>>>>>>> >
>>>>>>>>> >     On 8/2/2019 6:41 PM, Dinesh Dutt wrote:
>>>>>>>>> >      > The assumption of an IP address within any VNI is suspect
>>>>>>>>> that way.
>>>>>>>>> >      > What's special about a single VNI, the management VNI?
>>>>>>>>> The VTEP IP
>>>>>>>>> >      > address does not belong in reality in any VNI.
>>>>>>>>> >      >
>>>>>>>>> >      > Dinesh
>>>>>>>>> >      >
>>>>>>>>> >      > On Fri, Aug 2, 2019 at 3:17 PM Joel M. Halpern
>>>>>>>>> >     <[e-mail] <mailto:[e-mail]>
>>>>>>>>> >      > <mailto:[e-mail] <mailto:[e-mail]>>>
>>>>>>>>> wrote:
>>>>>>>>> >      >
>>>>>>>>> >      >     Your response seems to miss two points:
>>>>>>>>> >      >
>>>>>>>>> >      >     First, the problem you describe is not what the
>>>>>>>>> document says
>>>>>>>>> >     it is
>>>>>>>>> >      >     solving.  To the degree it discusses it at all, the
>>>>>>>>> document
>>>>>>>>> >     says "
>>>>>>>>> >      >       In
>>>>>>>>> >      >     most cases, a single BFD session is sufficient for
>>>>>>>>> the given
>>>>>>>>> >     VTEP to
>>>>>>>>> >      >     monitor the reachability of a remote VTEP, regardless
>>>>>>>>> of the
>>>>>>>>> >     number of
>>>>>>>>> >      >     VNIs in common. "
>>>>>>>>> >      >
>>>>>>>>> >      >     Second, you assume the existence of an IP address for
>>>>>>>>> a VTEP
>>>>>>>>> >     within a
>>>>>>>>> >      >     VNI.  As with the MAC address, the VTEP does not have
>>>>>>>>> an IP
>>>>>>>>> >     address
>>>>>>>>> >      >     within the VNI.  Some implementations may have
>>>>>>>>> created such a
>>>>>>>>> >     thing,
>>>>>>>>> >      >     but
>>>>>>>>> >      >     the general construct, as defined to date, does not
>>>>>>>>> support such.
>>>>>>>>> >      >
>>>>>>>>> >      >     In short, you are requiring a behavior that violates
>>>>>>>>> the
>>>>>>>>> >     architectural
>>>>>>>>> >      >     structure of overlay / underlay separation, and common
>>>>>>>>> >     usage.  And you
>>>>>>>>> >      >     are doing so to support a use case that the working
>>>>>>>>> group has not
>>>>>>>>> >      >     indicated in the document as important.
>>>>>>>>> >      >
>>>>>>>>> >      >     Yours,
>>>>>>>>> >      >     Joel
>>>>>>>>> >      >
>>>>>>>>> >      >     On 8/2/2019 5:01 PM, Dinesh Dutt wrote:
>>>>>>>>> >      >      > Joel,
>>>>>>>>> >      >      >
>>>>>>>>> >      >      > You understood correctly.
>>>>>>>>> >      >      >
>>>>>>>>> >      >      > The VNIs may not share fate due to
>>>>>>>>> misconfiguration. And I
>>>>>>>>> >     strongly
>>>>>>>>> >      >      > suspect someone will want to use BFD for that
>>>>>>>>> because its
>>>>>>>>> >     about
>>>>>>>>> >      >     checking
>>>>>>>>> >      >      > path continuity as stated by the draft. As long as
>>>>>>>>> there's a
>>>>>>>>> >      >     valid IP
>>>>>>>>> >      >      > (because it's BFD) owned by the VTEP in that VNI,
>>>>>>>>> you can
>>>>>>>>> >     use BFD in
>>>>>>>>> >      >      > that VNI. Thats all that you need to dictate.
>>>>>>>>> That IP address
>>>>>>>>> >      >     has a MAC
>>>>>>>>> >      >      > address and you can use that on the inner frame.
>>>>>>>>> That is
>>>>>>>>> >     all normal
>>>>>>>>> >      >      > VXLAN processing. The outer IP is always that of
>>>>>>>>> the VTEP.
>>>>>>>>> >      >      >
>>>>>>>>> >      >      > Dinesh
>>>>>>>>> >      >      >
>>>>>>>>> >      >      > On Fri, Aug 2, 2019 at 11:03 AM Joel M. Halpern
>>>>>>>>> >      >     <[e-mail] <mailto:[e-mail]>
>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>
>>>>>>>>> >      >      > <mailto:[e-mail] <mailto:
>>>>>>>>> [e-mail]>
>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>>>
>>>>>>>>> wrote:
>>>>>>>>> >      >      >
>>>>>>>>> >      >      >     If I am reading your various emails correctly
>>>>>>>>> Dinesh
>>>>>>>>> >     (and I
>>>>>>>>> >      >     may have
>>>>>>>>> >      >      >     missed something) you are trying to use the
>>>>>>>>> MAC address
>>>>>>>>> >      >     because you
>>>>>>>>> >      >      >     want
>>>>>>>>> >      >      >     to be able to send these BFD packets over
>>>>>>>>> arbitrary VNI to
>>>>>>>>> >      >     monitor the
>>>>>>>>> >      >      >     VNI.  That is not a requirement identified in
>>>>>>>>> the
>>>>>>>>> >     document.
>>>>>>>>> >      >     It is not
>>>>>>>>> >      >      >     even a problem I understand, since all the VNI
>>>>>>>>> between an
>>>>>>>>> >      >     ingress and
>>>>>>>>> >      >      >     egress VTEP share fate.
>>>>>>>>> >      >      >
>>>>>>>>> >      >      >     Yours,
>>>>>>>>> >      >      >     Joel
>>>>>>>>> >      >      >
>>>>>>>>> >      >      >     On 8/2/2019 1:44 PM, Dinesh Dutt wrote:
>>>>>>>>> >      >      >      > Thanks for verifying this. On Linux and
>>>>>>>>> hardware
>>>>>>>>> >     routers
>>>>>>>>> >      >     that I'm
>>>>>>>>> >      >      >     aware
>>>>>>>>> >      >      >      > of (Cisco circa 2012 and Cumulus), the
>>>>>>>>> physical MAC
>>>>>>>>> >     address is
>>>>>>>>> >      >      >     reused
>>>>>>>>> >      >      >      > across the VNIs on the VTEP. Did you check
>>>>>>>>> on a non-VMW
>>>>>>>>> >      >     device?
>>>>>>>>> >      >      >     This is
>>>>>>>>> >      >      >      > more for my own curiosity.
>>>>>>>>> >      >      >      >
>>>>>>>>> >      >      >      > To address the general case, can we not
>>>>>>>>> define a
>>>>>>>>> >      >     well-known (or
>>>>>>>>> >      >      >     reserve
>>>>>>>>> >      >      >      > one) unicast MAC address for use with VTEP?
>>>>>>>>> If the MAC
>>>>>>>>> >      >     address is
>>>>>>>>> >      >      >      > configurable in BFD command, this can be
>>>>>>>>> moot.
>>>>>>>>> >      >      >      >
>>>>>>>>> >      >      >      > Dinesh
>>>>>>>>> >      >      >      >
>>>>>>>>> >      >      >      > On Fri, Aug 2, 2019 at 10:27 AM Santosh P K
>>>>>>>>> >      >      >      > <[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>>>
>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>>>>> wrote:
>>>>>>>>> >      >      >      >
>>>>>>>>> >      >      >      >     I have cross checked point raised about
>>>>>>>>> MAC address
>>>>>>>>> >      >     usage. It is
>>>>>>>>> >      >      >      >     possible that tenant could be using
>>>>>>>>> physical MAC
>>>>>>>>> >      >     address and
>>>>>>>>> >      >      >     when a
>>>>>>>>> >      >      >      >     packet comes with valid VNI with a MAC
>>>>>>>>> address
>>>>>>>>> >     that is
>>>>>>>>> >      >     being
>>>>>>>>> >      >      >     used by
>>>>>>>>> >      >      >      >     tenant then packet will be sent to that
>>>>>>>>> tenant.
>>>>>>>>> >     This rules
>>>>>>>>> >      >      >     out the
>>>>>>>>> >      >      >      >     fact that we could use physical MAC
>>>>>>>>> address as
>>>>>>>>> >     inner
>>>>>>>>> >      >     MAC to
>>>>>>>>> >      >      >     ensure
>>>>>>>>> >      >      >      >     packets get terminated at VTEP itself.
>>>>>>>>> >      >      >      >
>>>>>>>>> >      >      >      >     Thanks
>>>>>>>>> >      >      >      >     Santosh P K
>>>>>>>>> >      >      >      >
>>>>>>>>> >      >      >      >     On Wed, Jul 31, 2019 at 11:00 AM
>>>>>>>>> Santosh P K
>>>>>>>>> >      >      >      >     <[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>>>
>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>>>>>
>>>>>>>>> >      >      >      >     wrote:
>>>>>>>>> >      >      >      >
>>>>>>>>> >      >      >      >         Joel,
>>>>>>>>> >      >      >      >             Thanks for your inputs. I
>>>>>>>>> checked
>>>>>>>>> >      >     implementation within
>>>>>>>>> >      >      >      >         Vmware. Perhaps I should have been
>>>>>>>>> more clear
>>>>>>>>> >      >     about MAC
>>>>>>>>> >      >      >     address
>>>>>>>>> >      >      >      >         space while checking internally. I
>>>>>>>>> will cross
>>>>>>>>> >      >     check again for
>>>>>>>>> >      >      >      >         the same and get back on this list.
>>>>>>>>> >      >      >      >
>>>>>>>>> >      >      >      >         Thanks
>>>>>>>>> >      >      >      >         Santosh P K
>>>>>>>>> >      >      >      >
>>>>>>>>> >      >      >      >         On Wed, Jul 31, 2019 at 10:54 AM
>>>>>>>>> Joel M.
>>>>>>>>> >     Halpern
>>>>>>>>> >      >      >      >         <[e-mail]
>>>>>>>>> >     <mailto:[e-mail]> <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>>> [e-mail]>
>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>>
>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]> <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>>> [e-mail]>
>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>>>>
>>>>>>>>> wrote:
>>>>>>>>> >      >      >      >
>>>>>>>>> >      >      >      >             Sorry to ask a stupid
>>>>>>>>> question.  Whose
>>>>>>>>> >      >     implementation?
>>>>>>>>> >      >      >      >
>>>>>>>>> >      >      >      >             The reason I ask is that as far
>>>>>>>>> as I
>>>>>>>>> >     can tell,
>>>>>>>>> >      >     since the
>>>>>>>>> >      >      >      >             tenant does not
>>>>>>>>> >      >      >      >             have any control access to the
>>>>>>>>> VTEP,
>>>>>>>>> >     there is no
>>>>>>>>> >      >      >     reason for
>>>>>>>>> >      >      >      >             the VTEP to
>>>>>>>>> >      >      >      >             have a MAC address in the tenant
>>>>>>>>> >     space.  Yes, the
>>>>>>>>> >      >      >     device has
>>>>>>>>> >      >      >      >             a physical
>>>>>>>>> >      >      >      >             MAC address.  But the tenant
>>>>>>>>> could well be
>>>>>>>>> >      >     using that MAC
>>>>>>>>> >      >      >      >             address.  Yes,
>>>>>>>>> >      >      >      >             they would be violating the
>>>>>>>>> Ethernet spec.
>>>>>>>>> >      >     But the whole
>>>>>>>>> >      >      >      >             point of
>>>>>>>>> >      >      >      >             segregation is not to care
>>>>>>>>> about such
>>>>>>>>> >     issues.
>>>>>>>>> >      >      >      >
>>>>>>>>> >      >      >      >             On the other hand, if you tell
>>>>>>>>> me that
>>>>>>>>> >     the VMWare
>>>>>>>>> >      >      >      >             implementation has an
>>>>>>>>> >      >      >      >             Ethernet address that is part
>>>>>>>>> of the tenant
>>>>>>>>> >      >     space, well,
>>>>>>>>> >      >      >      >             they made up
>>>>>>>>> >      >      >      >             this particular game.
>>>>>>>>> >      >      >      >
>>>>>>>>> >      >      >      >             Yours,
>>>>>>>>> >      >      >      >             Joel
>>>>>>>>> >      >      >      >
>>>>>>>>> >      >      >      >             On 7/31/2019 1:44 PM, Santosh P
>>>>>>>>> K wrote:
>>>>>>>>> >      >      >      >              > I have checked with
>>>>>>>>> implementation
>>>>>>>>> >     in data
>>>>>>>>> >      >     path.
>>>>>>>>> >      >      >     When we
>>>>>>>>> >      >      >      >             receive a
>>>>>>>>> >      >      >      >              > packet with valid VNI then
>>>>>>>>> lookup
>>>>>>>>> >     for MAC will
>>>>>>>>> >      >      >     happen and
>>>>>>>>> >      >      >      >             it is VTEP own
>>>>>>>>> >      >      >      >              > MAC then it will be trapped
>>>>>>>>> to control
>>>>>>>>> >      >     plane for
>>>>>>>>> >      >      >      >             processing. I think we
>>>>>>>>> >      >      >      >              > can have following options
>>>>>>>>> >      >      >      >              > 1. Optional managment VNI
>>>>>>>>> >      >      >      >              > 2. Mandatory inner MAC set
>>>>>>>>> to VTEP mac
>>>>>>>>> >      >      >      >              > 3. Inner IP TTL set to 1 to
>>>>>>>>> avoid
>>>>>>>>> >      >     forwarding of packet
>>>>>>>>> >      >      >      >             via inner IP
>>>>>>>>> >      >      >      >              > address.
>>>>>>>>> >      >      >      >              >
>>>>>>>>> >      >      >      >              >
>>>>>>>>> >      >      >      >              > Thoughts?
>>>>>>>>> >      >      >      >              >
>>>>>>>>> >      >      >      >              > Thansk
>>>>>>>>> >      >      >      >              > Santosh P K
>>>>>>>>> >      >      >      >              >
>>>>>>>>> >      >      >      >              > On Wed, Jul 31, 2019 at 9:20
>>>>>>>>> AM Greg
>>>>>>>>> >     Mirsky
>>>>>>>>> >      >      >      >             <[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>>> [e-mail]>>
>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>
>>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>>> [e-mail]>>>
>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]> <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>>> [e-mail]>
>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]
>>>>>>>>> >>>>
>>>>>>>>> >      >      >      >              > <mailto:
>>>>>>>>> [e-mail]
>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>>> [e-mail]>>
>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]> <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>>>
>>>>>>>>> >      >      >      >             <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>>> [e-mail]>>
>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>>>>>> wrote:
>>>>>>>>> >      >      >      >              >
>>>>>>>>> >      >      >      >              >     Hi Dinesh,
>>>>>>>>> >      >      >      >              >     thank you for your
>>>>>>>>> consideration
>>>>>>>>> >     of the
>>>>>>>>> >      >      >     proposal and
>>>>>>>>> >      >      >      >             questions. What
>>>>>>>>> >      >      >      >              >     would you see as the
>>>>>>>>> scope of
>>>>>>>>> >     testing the
>>>>>>>>> >      >      >      >             connectivity for the
>>>>>>>>> >      >      >      >              >     specific VNI? If it is
>>>>>>>>> >      >     tenant-to-tenant, then
>>>>>>>>> >      >      >     VTEPs
>>>>>>>>> >      >      >      >             will treat these
>>>>>>>>> >      >      >      >              >     packets as regular user
>>>>>>>>> frames. More
>>>>>>>>> >      >     likely, these
>>>>>>>>> >      >      >      >             could be Layer 2
>>>>>>>>> >      >      >      >              >     OAM, e.g. CCM frames.
>>>>>>>>> The reason
>>>>>>>>> >     to use
>>>>>>>>> >      >     127/8 for
>>>>>>>>> >      >      >      >             IPv4, and
>>>>>>>>> >      >      >      >              >
>>>>>>>>>  0:0:0:0:0:FFFF:7F00:0/104 for
>>>>>>>>> >     IPv6 is
>>>>>>>>> >      >     to safeguard
>>>>>>>>> >      >      >      >             from leaking
>>>>>>>>> >      >      >      >              >     Ethernet frames with BFD
>>>>>>>>> Control
>>>>>>>>> >     packet
>>>>>>>>> >      >     to a
>>>>>>>>> >      >      >     tenant.
>>>>>>>>> >      >      >      >              >     You've suggested using a
>>>>>>>>> MAC
>>>>>>>>> >     address to
>>>>>>>>> >      >     trap the
>>>>>>>>> >      >      >      >             control packet at
>>>>>>>>> >      >      >      >              >     VTEP. What that address
>>>>>>>>> could be? We
>>>>>>>>> >      >     had proposed
>>>>>>>>> >      >      >      >             using the
>>>>>>>>> >      >      >      >              >     dedicated MAC and VTEP's
>>>>>>>>> MAC and
>>>>>>>>> >     both
>>>>>>>>> >      >     raised
>>>>>>>>> >      >      >     concerns
>>>>>>>>> >      >      >      >             among VXLAN
>>>>>>>>> >      >      >      >              >     experts. The idea of
>>>>>>>>> using
>>>>>>>>> >     Management
>>>>>>>>> >      >     VNI may
>>>>>>>>> >      >      >     be more
>>>>>>>>> >      >      >      >             acceptable
>>>>>>>>> >      >      >      >              >     based on its similarity
>>>>>>>>> to the
>>>>>>>>> >     practice
>>>>>>>>> >      >     of using
>>>>>>>>> >      >      >      >             Management VLAN.
>>>>>>>>> >      >      >      >              >
>>>>>>>>> >      >      >      >              >     Regards,
>>>>>>>>> >      >      >      >              >     Greg
>>>>>>>>> >      >      >      >              >
>>>>>>>>> >      >      >      >              >     On Wed, Jul 31, 2019 at
>>>>>>>>> 12:03 PM
>>>>>>>>> >     Dinesh
>>>>>>>>> >      >     Dutt
>>>>>>>>> >      >      >      >             <[e-mail]
>>>>>>>>> >     <mailto:[e-mail]> <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>>> >      >     <mailto:[e-mail] <mailto:[e-mail]>
>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>>
>>>>>>>>> >      >      >     <mailto:[e-mail] <mailto:
>>>>>>>>> [e-mail]>
>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>
>>>>>>>>> >      >     <mailto:[e-mail] <mailto:[e-mail]>
>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>>>
>>>>>>>>> >      >      >      >              >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>> >      >     <mailto:[e-mail] <mailto:[e-mail]>>
>>>>>>>>> >      >      >     <mailto:[e-mail] <mailto:
>>>>>>>>> [e-mail]>
>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>>
>>>>>>>>> >      >     <mailto:[e-mail] <mailto:[e-mail]>
>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>
>>>>>>>>> >      >      >     <mailto:[e-mail] <mailto:
>>>>>>>>> [e-mail]>
>>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>>>>>
>>>>>>>>> >      >      >      >             wrote:
>>>>>>>>> >      >      >      >              >
>>>>>>>>> >      >      >      >              >         Hi Greg,
>>>>>>>>> >      >      >      >              >
>>>>>>>>> >      >      >      >              >         As long as the inner
>>>>>>>>> MAC
>>>>>>>>> >     address is
>>>>>>>>> >      >     such
>>>>>>>>> >      >      >     that the
>>>>>>>>> >      >      >      >             packet is
>>>>>>>>> >      >      >      >              >         trapped to the CPU,
>>>>>>>>> it should be
>>>>>>>>> >      >     fine for
>>>>>>>>> >      >      >     use as
>>>>>>>>> >      >      >      >             an inner MAC is
>>>>>>>>> >      >      >      >              >         it not? Stating that
>>>>>>>>> is
>>>>>>>>> >     better than
>>>>>>>>> >      >     trying to
>>>>>>>>> >      >      >      >             force a management
>>>>>>>>> >      >      >      >              >         VNI. What if someone
>>>>>>>>> wants
>>>>>>>>> >     to test
>>>>>>>>> >      >      >     connectivity
>>>>>>>>> >      >      >      >             on a specific
>>>>>>>>> >      >      >      >              >         VNI? I would not
>>>>>>>>> pick a
>>>>>>>>> >     loopback IP
>>>>>>>>> >      >      >     address for
>>>>>>>>> >      >      >      >             this since that
>>>>>>>>> >      >      >      >              >         address range is
>>>>>>>>> host/node local
>>>>>>>>> >      >     only. Is
>>>>>>>>> >      >      >     there a
>>>>>>>>> >      >      >      >             reason you're
>>>>>>>>> >      >      >      >              >         not using the VTEP
>>>>>>>>> IP as the
>>>>>>>>> >     inner IP
>>>>>>>>> >      >      >     address ?
>>>>>>>>> >      >      >      >              >
>>>>>>>>> >      >      >      >              >         Dinesh
>>>>>>>>> >      >      >      >              >
>>>>>>>>> >      >      >      >              >         On Wed, Jul 31, 2019
>>>>>>>>> at 5:48 AM
>>>>>>>>> >      >     Greg Mirsky
>>>>>>>>> >      >      >      >              >         <
>>>>>>>>> [e-mail]
>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>>> [e-mail]>>
>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]> <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>>>
>>>>>>>>> >      >      >      >             <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>>> [e-mail]>>
>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>>>> <mailto:
>>>>>>>>> [e-mail]
>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>>> [e-mail]>>
>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]> <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>>>
>>>>>>>>> >      >      >      >             <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>>> [e-mail]>>
>>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>>> >     <mailto:[e-mail]>>>>>> wrote:
>>>>>>>>> >      >      >      >              >
>>>>>>>>> >      >      >      >              >             Dear All,
>>>>>>>>> >      >      >      >              >             thank you for
>>>>>>>>> your comments,
>>>>>>>>> >      >      >     suggestions on
>>>>>>>>> >      >      >      >             this issue,
>>>>>>>>> >      >      >      >              >             probably the most
>>>>>>>>> >     challenging
>>>>>>>>> >      >     for this
>>>>>>>>> >      >      >      >             specification. In the
>>>>>>>>> >      >      >      >              >             course of our
>>>>>>>>> discussions,
>>>>>>>>> >      >     we've agreed to
>>>>>>>>> >      >      >      >             abandon the
>>>>>>>>> >      >      >      >              >             request to
>>>>>>>>> allocate the
>>>>>>>>> >      >     dedicated MAC
>>>>>>>>> >      >      >     address
>>>>>>>>> >      >      >      >             to be used as
>>>>>>>>> >      >      >      >              >             the destination
>>>>>>>>> MAC
>>>>>>>>> >     address in
>>>>>>>>> >      >     the inner
>>>>>>>>> >      >      >      >             Ethernet frame.
>>>>>>>>> >      >      >      >              >             Also, earlier
>>>>>>>>> using VNI
>>>>>>>>> >     0 was
>>>>>>>>> >      >     changed from
>>>>>>>>> >      >      >      >             mandatory to one
>>>>>>>>> >      >      >      >              >             of the options an
>>>>>>>>> >      >     implementation may
>>>>>>>>> >      >      >     offer to
>>>>>>>>> >      >      >      >             an operator.
>>>>>>>>> >      >      >      >              >             The most recent
>>>>>>>>> >     discussion was
>>>>>>>>> >      >     whether
>>>>>>>>> >      >      >     VTEP's
>>>>>>>>> >      >      >      >             MAC address
>>>>>>>>> >      >      >      >              >             might be used as
>>>>>>>>> the
>>>>>>>>> >      >     destination MAC
>>>>>>>>> >      >      >     address
>>>>>>>>> >      >      >      >             in the inner
>>>>>>>>> >      >      >      >              >             Ethernet frame.
>>>>>>>>> As I
>>>>>>>>> >     recall it, the
>>>>>>>>> >      >      >     comments
>>>>>>>>> >      >      >      >             from VXLAN
>>>>>>>>> >      >      >      >              >             experts equally
>>>>>>>>> split
>>>>>>>>> >     with one
>>>>>>>>> >      >     for it
>>>>>>>>> >      >      >     and one
>>>>>>>>> >      >      >      >             against. Hence
>>>>>>>>> >      >      >      >              >             I would like to
>>>>>>>>> propose
>>>>>>>>> >     a new
>>>>>>>>> >      >     text to
>>>>>>>>> >      >      >     resolve
>>>>>>>>> >      >      >      >             the issue. The
>>>>>>>>> >      >      >      >              >             idea is to let an
>>>>>>>>> >     operator select
>>>>>>>>> >      >      >     Management
>>>>>>>>> >      >      >      >             VNI and use
>>>>>>>>> >      >      >      >              >             that VNI in VXLAN
>>>>>>>>> >     encapsulation
>>>>>>>>> >      >     of BFD
>>>>>>>>> >      >      >      >             Control packets:
>>>>>>>>> >      >      >      >              >             NEW TEXT:
>>>>>>>>> >      >      >      >              >
>>>>>>>>> >      >      >      >              >                 An operator
>>>>>>>>> MUST
>>>>>>>>> >     select a VNI
>>>>>>>>> >      >      >     number to
>>>>>>>>> >      >      >      >             be used as
>>>>>>>>> >      >      >      >              >                 Management
>>>>>>>>> VNI. VXLAN
>>>>>>>>> >      >     packet for
>>>>>>>>> >      >      >      >             Management VNI MUST NOT
>>>>>>>>> >      >      >      >              >                 be sent to a
>>>>>>>>> tenant. VNI
>>>>>>>>> >      >     number 1 is
>>>>>>>>> >      >      >      >             RECOMMENDED as the
>>>>>>>>> >      >      >      >              >                 default for
>>>>>>>>> >     Management VNI.
>>>>>>>>> >      >      >      >              >
>>>>>>>>> >      >      >      >              >             With that new
>>>>>>>>> text, what
>>>>>>>>> >     can be the
>>>>>>>>> >      >      >     value of
>>>>>>>>> >      >      >      >             the destination
>>>>>>>>> >      >      >      >              >             MAC in the inner
>>>>>>>>> Ethernet? I
>>>>>>>>> >      >     tend to
>>>>>>>>> >      >      >     believe
>>>>>>>>> >      >      >      >             that it can be
>>>>>>>>> >      >      >      >              >             anything and
>>>>>>>>> ignored by the
>>>>>>>>> >      >     reciever VTEP.
>>>>>>>>> >      >      >      >             Also, if the
>>>>>>>>> >      >      >      >              >             trapping is
>>>>>>>>> based on VNI
>>>>>>>>> >      >     number, the
>>>>>>>>> >      >      >      >             destination IP address
>>>>>>>>> >      >      >      >              >             of the inner IP
>>>>>>>>> packet
>>>>>>>>> >     can from
>>>>>>>>> >      >     the range
>>>>>>>>> >      >      >      >             127/8 for IPv4,
>>>>>>>>> >      >      >      >              >             and for IPv6
>>>>>>>>> from the range
>>>>>>>>> >      >      >      >             0:0:0:0:0:FFFF:7F00:0/104. And
>>>>>>>>> >      >      >      >              >             lastly, the TTL
>>>>>>>>> to be
>>>>>>>>> >     set to 1 (no
>>>>>>>>> >      >      >     change here).
>>>>>>>>> >      >      >      >              >
>>>>>>>>> >      >      >      >              >             Much appreciate
>>>>>>>>> your
>>>>>>>>> >     comments,
>>>>>>>>> >      >      >     questions, and
>>>>>>>>> >      >      >      >             suggestions.
>>>>>>>>> >      >      >      >              >
>>>>>>>>> >      >      >      >              >             Best regards,
>>>>>>>>> >      >      >      >              >             Greg
>>>>>>>>> >      >      >      >              >
>>>>>>>>> >      >      >      >
>>>>>>>>> >      >      >
>>>>>>>>> >      >
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>