Re: BFD over VXLAN: Trapping BFD Control packet at VTEP

[Dinesh Dutt <[e-mail]>]

On Mon, Aug 5, 2019 at 5:56 PM Greg Mirsky <[e-mail]> wrote:

> Hi Dinesh,
> thank you for your expedient detailed response.
> I believe that the ability to run BFD session up to a tenant
> (VTEP-VTEP-tenant or tenant-tenant) was never in jeopardy from this
> specification.
> I'm trying to provide precise specification on what can be used ad the
> destination MAC and IP addresses in the inner frame/packet as I believe
> that likely will help to avoid interoperability issues.
> I'm interested to learn some more about the "martian checking" function.
> As you know, martian addresses have been used as destination IP address in
> LSP Ping and BFD over MPLS LSP and PW. I haven't heard that any silicon
> feature caused problems for operators using these tools.
>

Interesting. I didn't know this aspect of use with MPLS ping. Did those
packets ever go through a firewall though? In any case, maybe suggest the
use of those addresses with a statement that this is how LSP does it, but
that other MAC/IP pairs are possible as long as the conditions of the
endpoint owning the MAC/IP was honored.

Dinesh

>
> Regards,
> Greg
>
> On Mon, Aug 5, 2019 at 3:59 PM Dinesh Dutt <[e-mail]> wrote:
>
>> Hi Greg,
>>
>> That we agree on the problem definition is the first step forward. Your
>> original document had my cases covered and so I was surprised by the track
>> this thread took. It doesn't matter, we're back on track.
>>
>> My recommendation is to not worry about specifying the precise MAC/IP
>> address used in the inner header. The addresses chosen MUST ensure that the
>> packet is trapped to the control plane of the VTEP and not escape to the
>> tenant if the BFD is to the VTEP. Any solution MUST also not preclude the
>> use of the BFD by tenant systems for that VNI. There are many ways an
>> implementer can choose to implement this. For example, the inner MAC
>> address is whatever the VTEP implementer would return if ARP'd for the IP
>> address used in the inner header in the given VNI. The implementer can pick
>> a fixed MAC address, one that they own etc. Multiple BFD sessions can be
>> run for testing path connectivity on more than one VNIs. Limits should be
>> in place to avoid overwhelming the receiver with BFD messages (you had
>> words about this in your currently published draft).  If the VNI is
>> irrelevant in the test i.e. only the VXLAN pipe at the VTEP is being
>> tested. the user can use any VNI active on the VTEP on which the VTEP owns
>> an IP address.
>>
>> I'm concerned about the use of 127/8 address only because of firewalls or
>> implementations that drop packets with these addresses as either the source
>> or destination. For example, on many merchant silicon, I don't believe you
>> can turn off martian checking and drops *only* for VXLAN-encapsulated BFD
>> packets. I don't know what the Linux kernel does today on such packets, for
>> example (or Hyper-V). I'd like a solution that doesn't demand additional or
>> new chip functionality or require additional middle-box hole punch.
>>
>> Why do you feel you MUST to specify the MAC/IP address on the inner
>> packet? What am I missing here?
>>
>> Dinesh
>>
>> On Mon, Aug 5, 2019 at 3:04 PM Greg Mirsky <[e-mail]> wrote:
>>
>>> Hi Dinesh,
>>> what do you see as the way forward? I agree, that the proposed text
>>> doesn't work for multi-VNI concurrent monitoring because these VNIs are
>>> tenant's VNIs. And in that case, we need to specify another mechanism to
>>> trap the BFD Control packet at VTEP. It seems that VTEP's Ethernet address
>>> must be used as the destination MAC address in the inner Ethernet frame.
>>> The destination IP address may be either VTEP's address of martian (I do
>>> prefer martian). Let me give it  try:
>>> NEW TEXT:
>>>
>>> To monitor continuity of the path between two VTEPs, an operator MUST
>>> select a VNI number to be used as Management VNI. Management VNI number
>>> MUST NOT be one of the tenant's VNIs to prevent sending VXLAN packets
>>> received on Management VNI to a tenant. VNI number 1 is RECOMMENDED as the
>>> default for Management VNI. [Ed.note: What we set the Destination MAC to?
>>> Can it be invalid MAC that MUST be ignored on receipt?]
>>>
>>> If an implementation supports concurrent monitoring of multiple VNIs,
>>> then the value of VNI number MAY be one of tenant's VNIs. The destination
>>> MAC address in the inner Ethernet frame encapsulating BFD Control packet
>>> MUST be MAC associated with the remote VTEP.
>>> The destination IP address of the inner IP packet MUST be selected from
>>> the range 127/8 for IPv4, and for IPv6 from the range
>>> 0:0:0:0:0:FFFF:7F00:0/104. The TTL value in the inner IP header MUST be set
>>> to 1.
>>>
>>> Regards,
>>> Greg
>>>
>>> On Sun, Aug 4, 2019 at 9:07 AM Dinesh Dutt <[e-mail]> wrote:
>>>
>>>> Hi Greg,
>>>>
>>>> Thanks for your clarifications. I agree with your sentiment on why
>>>> you're running BFD over VXLAN between VTEPs. I wasn't arguing against it at
>>>> all. All I was saying was pointing to the limitations of the use of
>>>> management VNI. I spoke to some operators who're running EVPN and mentioned
>>>> the discussion on this thread. They concur that they're using specific VNIs
>>>> to test connectivity over that VNI between VTEPs to ensure misconfiguration
>>>> doesn't lead to blackholes. My statements are based in real world operator
>>>> experience. And I was providing language that ensured packets didn't leak
>>>> across to tenants when they were destined to VTEPs.
>>>>
>>>> Dinesh
>>>>
>>>> On Sat, Aug 3, 2019 at 10:34 AM Greg Mirsky <[e-mail]>
>>>> wrote:
>>>>
>>>>> Hi Dinesh,
>>>>> many thanks for your detailed updates on how some implementations
>>>>> process VXLAN header and the inner Ethernet frame. These are very helpful
>>>>> in achieving the workable solution for the problem at hand.
>>>>> You've noted that a path between VTEPs may be monitored in the
>>>>> underlay network by merely establishing a BFD session. That is true, but by
>>>>> using BFD with VXLAN encapsulation between the pair of VTEPs we are
>>>>> extending the OAM domain by including, to some extent, VXLAN forwarding
>>>>> engine. Abstract in RFC 5880 defines the goal and the domain in which BFD
>>>>> protocol can detect a fault as:
>>>>>    This document describes a protocol intended to detect faults in the
>>>>>    bidirectional path between two forwarding engines, including
>>>>>    interfaces, data link(s), and to the extent possible the forwarding
>>>>>    engines themselves, with potentially very low latency.
>>>>> Thus, BFD in the underlay will exercise a part of IP forwarding engine
>>>>> while BFD with VXLAN encapsulation, ran between the same pair of VTEPs,
>>>>> extends the OAM domain. At the same time, defining BFD between tenant
>>>>> systems in outside the goal of this specification. But VXLAN BFD session
>>>>> between VTEPs may be useful in monitoring e2e path between tenants, as
>>>>> described in the update to -07:
>>>>>    At the same time, a service layer BFD session may be used between
>>>>> the
>>>>>    tenants of VTEPs IP1 and IP2 to provide end-to-end fault management.
>>>>>    In such case, for VTEPs BFD control packets of that session are
>>>>>    indistinguishable from data packets.  If end-to-end defect detection
>>>>>    is realized as the set of concatenated OAM domains, e.g., VM1-1 -
>>>>> IP1
>>>>>    -- IP2 - VM2-1, then the BFD session over VXLAN between VTEPs SHOULD
>>>>>    follow the procedures described in Section 6.8.17 [RFC5880].
>>>>> I've attached the current working version of the draft.
>>>>>
>>>>> Regards,
>>>>> Greg
>>>>>
>>>>>
>>>>> On Fri, Aug 2, 2019 at 5:43 PM Dinesh Dutt <[e-mail]> wrote:
>>>>>
>>>>>> What I mean is "How do you infer that it excludes the case I'm
>>>>>> talking about?".
>>>>>>
>>>>>> Dinesh
>>>>>>
>>>>>> On Fri, Aug 2, 2019 at 5:41 PM Dinesh Dutt <[e-mail]> wrote:
>>>>>>
>>>>>>> The abstract reads this: "
>>>>>>>
>>>>>>> This document describes the use of the Bidirectional Forwarding
>>>>>>>    Detection (BFD) protocol in point-to-point Virtual eXtensible Local
>>>>>>>    Area Network (VXLAN) tunnels forming up an overlay network."
>>>>>>>
>>>>>>> How do you infer what you said?
>>>>>>>
>>>>>>> Dinesh
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Aug 2, 2019 at 5:38 PM Joel M. Halpern <[e-mail]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I am going by what the draft says its purpose is.  If you (Dinesh)
>>>>>>>> want
>>>>>>>> the draft to fulfill a different purpose, then either ask the
>>>>>>>> chairs to
>>>>>>>> take this draft back to the WG, or write a separate draft.
>>>>>>>> As currently written, the behavior Greg proposed meets the needs,
>>>>>>>> and
>>>>>>>> does so in a way that is consistent with VxLAN.
>>>>>>>>
>>>>>>>> Yours,
>>>>>>>> Joel
>>>>>>>>
>>>>>>>> On 8/2/2019 8:30 PM, Dinesh Dutt wrote:
>>>>>>>> > What is the stated purpose of this BFD session? The VTEP
>>>>>>>> reachability is
>>>>>>>> > determined by the underlay, I don't need VXLAN-encaped packet for
>>>>>>>> that.
>>>>>>>> > Do we agree?
>>>>>>>> >
>>>>>>>> > If I want to test the VXLAN encap/decap functionality alone,
>>>>>>>> picking any
>>>>>>>> > single VNI maybe fine. But is this all any network operator
>>>>>>>> wants? Why?
>>>>>>>> > In what situations has this been a problem? I suspect operators
>>>>>>>> also
>>>>>>>> > want to verify path continuity over a specific VNI. If you say
>>>>>>>> this is
>>>>>>>> > not defined by the document, I disagree because the current
>>>>>>>> version
>>>>>>>> > talks about controlling the number of BFD sessions between the
>>>>>>>> VTEPs
>>>>>>>> > (see section 3). More importantly, this is a real problem that
>>>>>>>> operators
>>>>>>>> > like to verify.
>>>>>>>> >
>>>>>>>> > Dinesh
>>>>>>>> >
>>>>>>>> > On Fri, Aug 2, 2019 at 5:08 PM Joel M. Halpern <
>>>>>>>> [e-mail]
>>>>>>>> > <mailto:[e-mail]>> wrote:
>>>>>>>> >
>>>>>>>> >     What is special about the management VNI is precisely that it
>>>>>>>> is NOT a
>>>>>>>> >     tenant VNI.  The VxLAN administration does know how it
>>>>>>>> allocates VNI to
>>>>>>>> >     tenants, and which VNI it has allocated.  In contrast, it
>>>>>>>> does not know
>>>>>>>> >     which IP addresses or MAC adddresses teh tenant is using or
>>>>>>>> may plan
>>>>>>>> >     to use.
>>>>>>>> >
>>>>>>>> >     Yours,
>>>>>>>> >     Joel
>>>>>>>> >
>>>>>>>> >     On 8/2/2019 6:41 PM, Dinesh Dutt wrote:
>>>>>>>> >      > The assumption of an IP address within any VNI is suspect
>>>>>>>> that way.
>>>>>>>> >      > What's special about a single VNI, the management VNI? The
>>>>>>>> VTEP IP
>>>>>>>> >      > address does not belong in reality in any VNI.
>>>>>>>> >      >
>>>>>>>> >      > Dinesh
>>>>>>>> >      >
>>>>>>>> >      > On Fri, Aug 2, 2019 at 3:17 PM Joel M. Halpern
>>>>>>>> >     <[e-mail] <mailto:[e-mail]>
>>>>>>>> >      > <mailto:[e-mail] <mailto:[e-mail]>>>
>>>>>>>> wrote:
>>>>>>>> >      >
>>>>>>>> >      >     Your response seems to miss two points:
>>>>>>>> >      >
>>>>>>>> >      >     First, the problem you describe is not what the
>>>>>>>> document says
>>>>>>>> >     it is
>>>>>>>> >      >     solving.  To the degree it discusses it at all, the
>>>>>>>> document
>>>>>>>> >     says "
>>>>>>>> >      >       In
>>>>>>>> >      >     most cases, a single BFD session is sufficient for the
>>>>>>>> given
>>>>>>>> >     VTEP to
>>>>>>>> >      >     monitor the reachability of a remote VTEP, regardless
>>>>>>>> of the
>>>>>>>> >     number of
>>>>>>>> >      >     VNIs in common. "
>>>>>>>> >      >
>>>>>>>> >      >     Second, you assume the existence of an IP address for
>>>>>>>> a VTEP
>>>>>>>> >     within a
>>>>>>>> >      >     VNI.  As with the MAC address, the VTEP does not have
>>>>>>>> an IP
>>>>>>>> >     address
>>>>>>>> >      >     within the VNI.  Some implementations may have created
>>>>>>>> such a
>>>>>>>> >     thing,
>>>>>>>> >      >     but
>>>>>>>> >      >     the general construct, as defined to date, does not
>>>>>>>> support such.
>>>>>>>> >      >
>>>>>>>> >      >     In short, you are requiring a behavior that violates
>>>>>>>> the
>>>>>>>> >     architectural
>>>>>>>> >      >     structure of overlay / underlay separation, and common
>>>>>>>> >     usage.  And you
>>>>>>>> >      >     are doing so to support a use case that the working
>>>>>>>> group has not
>>>>>>>> >      >     indicated in the document as important.
>>>>>>>> >      >
>>>>>>>> >      >     Yours,
>>>>>>>> >      >     Joel
>>>>>>>> >      >
>>>>>>>> >      >     On 8/2/2019 5:01 PM, Dinesh Dutt wrote:
>>>>>>>> >      >      > Joel,
>>>>>>>> >      >      >
>>>>>>>> >      >      > You understood correctly.
>>>>>>>> >      >      >
>>>>>>>> >      >      > The VNIs may not share fate due to
>>>>>>>> misconfiguration. And I
>>>>>>>> >     strongly
>>>>>>>> >      >      > suspect someone will want to use BFD for that
>>>>>>>> because its
>>>>>>>> >     about
>>>>>>>> >      >     checking
>>>>>>>> >      >      > path continuity as stated by the draft. As long as
>>>>>>>> there's a
>>>>>>>> >      >     valid IP
>>>>>>>> >      >      > (because it's BFD) owned by the VTEP in that VNI,
>>>>>>>> you can
>>>>>>>> >     use BFD in
>>>>>>>> >      >      > that VNI. Thats all that you need to dictate.  That
>>>>>>>> IP address
>>>>>>>> >      >     has a MAC
>>>>>>>> >      >      > address and you can use that on the inner frame.
>>>>>>>> That is
>>>>>>>> >     all normal
>>>>>>>> >      >      > VXLAN processing. The outer IP is always that of
>>>>>>>> the VTEP.
>>>>>>>> >      >      >
>>>>>>>> >      >      > Dinesh
>>>>>>>> >      >      >
>>>>>>>> >      >      > On Fri, Aug 2, 2019 at 11:03 AM Joel M. Halpern
>>>>>>>> >      >     <[e-mail] <mailto:[e-mail]>
>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>
>>>>>>>> >      >      > <mailto:[e-mail] <mailto:
>>>>>>>> [e-mail]>
>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>>>
>>>>>>>> wrote:
>>>>>>>> >      >      >
>>>>>>>> >      >      >     If I am reading your various emails correctly
>>>>>>>> Dinesh
>>>>>>>> >     (and I
>>>>>>>> >      >     may have
>>>>>>>> >      >      >     missed something) you are trying to use the MAC
>>>>>>>> address
>>>>>>>> >      >     because you
>>>>>>>> >      >      >     want
>>>>>>>> >      >      >     to be able to send these BFD packets over
>>>>>>>> arbitrary VNI to
>>>>>>>> >      >     monitor the
>>>>>>>> >      >      >     VNI.  That is not a requirement identified in
>>>>>>>> the
>>>>>>>> >     document.
>>>>>>>> >      >     It is not
>>>>>>>> >      >      >     even a problem I understand, since all the VNI
>>>>>>>> between an
>>>>>>>> >      >     ingress and
>>>>>>>> >      >      >     egress VTEP share fate.
>>>>>>>> >      >      >
>>>>>>>> >      >      >     Yours,
>>>>>>>> >      >      >     Joel
>>>>>>>> >      >      >
>>>>>>>> >      >      >     On 8/2/2019 1:44 PM, Dinesh Dutt wrote:
>>>>>>>> >      >      >      > Thanks for verifying this. On Linux and
>>>>>>>> hardware
>>>>>>>> >     routers
>>>>>>>> >      >     that I'm
>>>>>>>> >      >      >     aware
>>>>>>>> >      >      >      > of (Cisco circa 2012 and Cumulus), the
>>>>>>>> physical MAC
>>>>>>>> >     address is
>>>>>>>> >      >      >     reused
>>>>>>>> >      >      >      > across the VNIs on the VTEP. Did you check
>>>>>>>> on a non-VMW
>>>>>>>> >      >     device?
>>>>>>>> >      >      >     This is
>>>>>>>> >      >      >      > more for my own curiosity.
>>>>>>>> >      >      >      >
>>>>>>>> >      >      >      > To address the general case, can we not
>>>>>>>> define a
>>>>>>>> >      >     well-known (or
>>>>>>>> >      >      >     reserve
>>>>>>>> >      >      >      > one) unicast MAC address for use with VTEP?
>>>>>>>> If the MAC
>>>>>>>> >      >     address is
>>>>>>>> >      >      >      > configurable in BFD command, this can be
>>>>>>>> moot.
>>>>>>>> >      >      >      >
>>>>>>>> >      >      >      > Dinesh
>>>>>>>> >      >      >      >
>>>>>>>> >      >      >      > On Fri, Aug 2, 2019 at 10:27 AM Santosh P K
>>>>>>>> >      >      >      > <[e-mail]
>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>>>
>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>>>>> wrote:
>>>>>>>> >      >      >      >
>>>>>>>> >      >      >      >     I have cross checked point raised about
>>>>>>>> MAC address
>>>>>>>> >      >     usage. It is
>>>>>>>> >      >      >      >     possible that tenant could be using
>>>>>>>> physical MAC
>>>>>>>> >      >     address and
>>>>>>>> >      >      >     when a
>>>>>>>> >      >      >      >     packet comes with valid VNI with a MAC
>>>>>>>> address
>>>>>>>> >     that is
>>>>>>>> >      >     being
>>>>>>>> >      >      >     used by
>>>>>>>> >      >      >      >     tenant then packet will be sent to that
>>>>>>>> tenant.
>>>>>>>> >     This rules
>>>>>>>> >      >      >     out the
>>>>>>>> >      >      >      >     fact that we could use physical MAC
>>>>>>>> address as
>>>>>>>> >     inner
>>>>>>>> >      >     MAC to
>>>>>>>> >      >      >     ensure
>>>>>>>> >      >      >      >     packets get terminated at VTEP itself.
>>>>>>>> >      >      >      >
>>>>>>>> >      >      >      >     Thanks
>>>>>>>> >      >      >      >     Santosh P K
>>>>>>>> >      >      >      >
>>>>>>>> >      >      >      >     On Wed, Jul 31, 2019 at 11:00 AM Santosh
>>>>>>>> P K
>>>>>>>> >      >      >      >     <[e-mail]
>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>>>
>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>>>>>
>>>>>>>> >      >      >      >     wrote:
>>>>>>>> >      >      >      >
>>>>>>>> >      >      >      >         Joel,
>>>>>>>> >      >      >      >             Thanks for your inputs. I checked
>>>>>>>> >      >     implementation within
>>>>>>>> >      >      >      >         Vmware. Perhaps I should have been
>>>>>>>> more clear
>>>>>>>> >      >     about MAC
>>>>>>>> >      >      >     address
>>>>>>>> >      >      >      >         space while checking internally. I
>>>>>>>> will cross
>>>>>>>> >      >     check again for
>>>>>>>> >      >      >      >         the same and get back on this list.
>>>>>>>> >      >      >      >
>>>>>>>> >      >      >      >         Thanks
>>>>>>>> >      >      >      >         Santosh P K
>>>>>>>> >      >      >      >
>>>>>>>> >      >      >      >         On Wed, Jul 31, 2019 at 10:54 AM
>>>>>>>> Joel M.
>>>>>>>> >     Halpern
>>>>>>>> >      >      >      >         <[e-mail]
>>>>>>>> >     <mailto:[e-mail]> <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>> [e-mail]>
>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>>
>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]> <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>> [e-mail]>
>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>>>>
>>>>>>>> wrote:
>>>>>>>> >      >      >      >
>>>>>>>> >      >      >      >             Sorry to ask a stupid question.
>>>>>>>> Whose
>>>>>>>> >      >     implementation?
>>>>>>>> >      >      >      >
>>>>>>>> >      >      >      >             The reason I ask is that as far
>>>>>>>> as I
>>>>>>>> >     can tell,
>>>>>>>> >      >     since the
>>>>>>>> >      >      >      >             tenant does not
>>>>>>>> >      >      >      >             have any control access to the
>>>>>>>> VTEP,
>>>>>>>> >     there is no
>>>>>>>> >      >      >     reason for
>>>>>>>> >      >      >      >             the VTEP to
>>>>>>>> >      >      >      >             have a MAC address in the tenant
>>>>>>>> >     space.  Yes, the
>>>>>>>> >      >      >     device has
>>>>>>>> >      >      >      >             a physical
>>>>>>>> >      >      >      >             MAC address.  But the tenant
>>>>>>>> could well be
>>>>>>>> >      >     using that MAC
>>>>>>>> >      >      >      >             address.  Yes,
>>>>>>>> >      >      >      >             they would be violating the
>>>>>>>> Ethernet spec.
>>>>>>>> >      >     But the whole
>>>>>>>> >      >      >      >             point of
>>>>>>>> >      >      >      >             segregation is not to care about
>>>>>>>> such
>>>>>>>> >     issues.
>>>>>>>> >      >      >      >
>>>>>>>> >      >      >      >             On the other hand, if you tell
>>>>>>>> me that
>>>>>>>> >     the VMWare
>>>>>>>> >      >      >      >             implementation has an
>>>>>>>> >      >      >      >             Ethernet address that is part of
>>>>>>>> the tenant
>>>>>>>> >      >     space, well,
>>>>>>>> >      >      >      >             they made up
>>>>>>>> >      >      >      >             this particular game.
>>>>>>>> >      >      >      >
>>>>>>>> >      >      >      >             Yours,
>>>>>>>> >      >      >      >             Joel
>>>>>>>> >      >      >      >
>>>>>>>> >      >      >      >             On 7/31/2019 1:44 PM, Santosh P
>>>>>>>> K wrote:
>>>>>>>> >      >      >      >              > I have checked with
>>>>>>>> implementation
>>>>>>>> >     in data
>>>>>>>> >      >     path.
>>>>>>>> >      >      >     When we
>>>>>>>> >      >      >      >             receive a
>>>>>>>> >      >      >      >              > packet with valid VNI then
>>>>>>>> lookup
>>>>>>>> >     for MAC will
>>>>>>>> >      >      >     happen and
>>>>>>>> >      >      >      >             it is VTEP own
>>>>>>>> >      >      >      >              > MAC then it will be trapped
>>>>>>>> to control
>>>>>>>> >      >     plane for
>>>>>>>> >      >      >      >             processing. I think we
>>>>>>>> >      >      >      >              > can have following options
>>>>>>>> >      >      >      >              > 1. Optional managment VNI
>>>>>>>> >      >      >      >              > 2. Mandatory inner MAC set to
>>>>>>>> VTEP mac
>>>>>>>> >      >      >      >              > 3. Inner IP TTL set to 1 to
>>>>>>>> avoid
>>>>>>>> >      >     forwarding of packet
>>>>>>>> >      >      >      >             via inner IP
>>>>>>>> >      >      >      >              > address.
>>>>>>>> >      >      >      >              >
>>>>>>>> >      >      >      >              >
>>>>>>>> >      >      >      >              > Thoughts?
>>>>>>>> >      >      >      >              >
>>>>>>>> >      >      >      >              > Thansk
>>>>>>>> >      >      >      >              > Santosh P K
>>>>>>>> >      >      >      >              >
>>>>>>>> >      >      >      >              > On Wed, Jul 31, 2019 at 9:20
>>>>>>>> AM Greg
>>>>>>>> >     Mirsky
>>>>>>>> >      >      >      >             <[e-mail]
>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>> [e-mail]>>
>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>
>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>> [e-mail]>>>
>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]> <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>> [e-mail]>
>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]
>>>>>>>> >>>>
>>>>>>>> >      >      >      >              > <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>> [e-mail]>>
>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]> <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>>>
>>>>>>>> >      >      >      >             <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>> [e-mail]>>
>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>>>>>> wrote:
>>>>>>>> >      >      >      >              >
>>>>>>>> >      >      >      >              >     Hi Dinesh,
>>>>>>>> >      >      >      >              >     thank you for your
>>>>>>>> consideration
>>>>>>>> >     of the
>>>>>>>> >      >      >     proposal and
>>>>>>>> >      >      >      >             questions. What
>>>>>>>> >      >      >      >              >     would you see as the
>>>>>>>> scope of
>>>>>>>> >     testing the
>>>>>>>> >      >      >      >             connectivity for the
>>>>>>>> >      >      >      >              >     specific VNI? If it is
>>>>>>>> >      >     tenant-to-tenant, then
>>>>>>>> >      >      >     VTEPs
>>>>>>>> >      >      >      >             will treat these
>>>>>>>> >      >      >      >              >     packets as regular user
>>>>>>>> frames. More
>>>>>>>> >      >     likely, these
>>>>>>>> >      >      >      >             could be Layer 2
>>>>>>>> >      >      >      >              >     OAM, e.g. CCM frames. The
>>>>>>>> reason
>>>>>>>> >     to use
>>>>>>>> >      >     127/8 for
>>>>>>>> >      >      >      >             IPv4, and
>>>>>>>> >      >      >      >              >     0:0:0:0:0:FFFF:7F00:0/104
>>>>>>>> for
>>>>>>>> >     IPv6 is
>>>>>>>> >      >     to safeguard
>>>>>>>> >      >      >      >             from leaking
>>>>>>>> >      >      >      >              >     Ethernet frames with BFD
>>>>>>>> Control
>>>>>>>> >     packet
>>>>>>>> >      >     to a
>>>>>>>> >      >      >     tenant.
>>>>>>>> >      >      >      >              >     You've suggested using a
>>>>>>>> MAC
>>>>>>>> >     address to
>>>>>>>> >      >     trap the
>>>>>>>> >      >      >      >             control packet at
>>>>>>>> >      >      >      >              >     VTEP. What that address
>>>>>>>> could be? We
>>>>>>>> >      >     had proposed
>>>>>>>> >      >      >      >             using the
>>>>>>>> >      >      >      >              >     dedicated MAC and VTEP's
>>>>>>>> MAC and
>>>>>>>> >     both
>>>>>>>> >      >     raised
>>>>>>>> >      >      >     concerns
>>>>>>>> >      >      >      >             among VXLAN
>>>>>>>> >      >      >      >              >     experts. The idea of using
>>>>>>>> >     Management
>>>>>>>> >      >     VNI may
>>>>>>>> >      >      >     be more
>>>>>>>> >      >      >      >             acceptable
>>>>>>>> >      >      >      >              >     based on its similarity
>>>>>>>> to the
>>>>>>>> >     practice
>>>>>>>> >      >     of using
>>>>>>>> >      >      >      >             Management VLAN.
>>>>>>>> >      >      >      >              >
>>>>>>>> >      >      >      >              >     Regards,
>>>>>>>> >      >      >      >              >     Greg
>>>>>>>> >      >      >      >              >
>>>>>>>> >      >      >      >              >     On Wed, Jul 31, 2019 at
>>>>>>>> 12:03 PM
>>>>>>>> >     Dinesh
>>>>>>>> >      >     Dutt
>>>>>>>> >      >      >      >             <[e-mail]
>>>>>>>> >     <mailto:[e-mail]> <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>>
>>>>>>>> >      >     <mailto:[e-mail] <mailto:[e-mail]>
>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>>
>>>>>>>> >      >      >     <mailto:[e-mail] <mailto:
>>>>>>>> [e-mail]>
>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>
>>>>>>>> >      >     <mailto:[e-mail] <mailto:[e-mail]>
>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>>>
>>>>>>>> >      >      >      >              >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>> >      >     <mailto:[e-mail] <mailto:[e-mail]>>
>>>>>>>> >      >      >     <mailto:[e-mail] <mailto:
>>>>>>>> [e-mail]>
>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>>
>>>>>>>> >      >     <mailto:[e-mail] <mailto:[e-mail]>
>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>
>>>>>>>> >      >      >     <mailto:[e-mail] <mailto:
>>>>>>>> [e-mail]>
>>>>>>>> >     <mailto:[e-mail] <mailto:[e-mail]>>>>>>
>>>>>>>> >      >      >      >             wrote:
>>>>>>>> >      >      >      >              >
>>>>>>>> >      >      >      >              >         Hi Greg,
>>>>>>>> >      >      >      >              >
>>>>>>>> >      >      >      >              >         As long as the inner
>>>>>>>> MAC
>>>>>>>> >     address is
>>>>>>>> >      >     such
>>>>>>>> >      >      >     that the
>>>>>>>> >      >      >      >             packet is
>>>>>>>> >      >      >      >              >         trapped to the CPU,
>>>>>>>> it should be
>>>>>>>> >      >     fine for
>>>>>>>> >      >      >     use as
>>>>>>>> >      >      >      >             an inner MAC is
>>>>>>>> >      >      >      >              >         it not? Stating that
>>>>>>>> is
>>>>>>>> >     better than
>>>>>>>> >      >     trying to
>>>>>>>> >      >      >      >             force a management
>>>>>>>> >      >      >      >              >         VNI. What if someone
>>>>>>>> wants
>>>>>>>> >     to test
>>>>>>>> >      >      >     connectivity
>>>>>>>> >      >      >      >             on a specific
>>>>>>>> >      >      >      >              >         VNI? I would not pick
>>>>>>>> a
>>>>>>>> >     loopback IP
>>>>>>>> >      >      >     address for
>>>>>>>> >      >      >      >             this since that
>>>>>>>> >      >      >      >              >         address range is
>>>>>>>> host/node local
>>>>>>>> >      >     only. Is
>>>>>>>> >      >      >     there a
>>>>>>>> >      >      >      >             reason you're
>>>>>>>> >      >      >      >              >         not using the VTEP IP
>>>>>>>> as the
>>>>>>>> >     inner IP
>>>>>>>> >      >      >     address ?
>>>>>>>> >      >      >      >              >
>>>>>>>> >      >      >      >              >         Dinesh
>>>>>>>> >      >      >      >              >
>>>>>>>> >      >      >      >              >         On Wed, Jul 31, 2019
>>>>>>>> at 5:48 AM
>>>>>>>> >      >     Greg Mirsky
>>>>>>>> >      >      >      >              >         <
>>>>>>>> [e-mail]
>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>> [e-mail]>>
>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]> <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>>>
>>>>>>>> >      >      >      >             <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>> [e-mail]>>
>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>>>> <mailto:
>>>>>>>> [e-mail]
>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>> [e-mail]>>
>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]> <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>>>
>>>>>>>> >      >      >      >             <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>> >      >     <mailto:[e-mail] <mailto:
>>>>>>>> [e-mail]>>
>>>>>>>> >      >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>
>>>>>>>> >      >     <mailto:[e-mail]
>>>>>>>> >     <mailto:[e-mail]>>>>>> wrote:
>>>>>>>> >      >      >      >              >
>>>>>>>> >      >      >      >              >             Dear All,
>>>>>>>> >      >      >      >              >             thank you for
>>>>>>>> your comments,
>>>>>>>> >      >      >     suggestions on
>>>>>>>> >      >      >      >             this issue,
>>>>>>>> >      >      >      >              >             probably the most
>>>>>>>> >     challenging
>>>>>>>> >      >     for this
>>>>>>>> >      >      >      >             specification. In the
>>>>>>>> >      >      >      >              >             course of our
>>>>>>>> discussions,
>>>>>>>> >      >     we've agreed to
>>>>>>>> >      >      >      >             abandon the
>>>>>>>> >      >      >      >              >             request to
>>>>>>>> allocate the
>>>>>>>> >      >     dedicated MAC
>>>>>>>> >      >      >     address
>>>>>>>> >      >      >      >             to be used as
>>>>>>>> >      >      >      >              >             the destination
>>>>>>>> MAC
>>>>>>>> >     address in
>>>>>>>> >      >     the inner
>>>>>>>> >      >      >      >             Ethernet frame.
>>>>>>>> >      >      >      >              >             Also, earlier
>>>>>>>> using VNI
>>>>>>>> >     0 was
>>>>>>>> >      >     changed from
>>>>>>>> >      >      >      >             mandatory to one
>>>>>>>> >      >      >      >              >             of the options an
>>>>>>>> >      >     implementation may
>>>>>>>> >      >      >     offer to
>>>>>>>> >      >      >      >             an operator.
>>>>>>>> >      >      >      >              >             The most recent
>>>>>>>> >     discussion was
>>>>>>>> >      >     whether
>>>>>>>> >      >      >     VTEP's
>>>>>>>> >      >      >      >             MAC address
>>>>>>>> >      >      >      >              >             might be used as
>>>>>>>> the
>>>>>>>> >      >     destination MAC
>>>>>>>> >      >      >     address
>>>>>>>> >      >      >      >             in the inner
>>>>>>>> >      >      >      >              >             Ethernet frame.
>>>>>>>> As I
>>>>>>>> >     recall it, the
>>>>>>>> >      >      >     comments
>>>>>>>> >      >      >      >             from VXLAN
>>>>>>>> >      >      >      >              >             experts equally
>>>>>>>> split
>>>>>>>> >     with one
>>>>>>>> >      >     for it
>>>>>>>> >      >      >     and one
>>>>>>>> >      >      >      >             against. Hence
>>>>>>>> >      >      >      >              >             I would like to
>>>>>>>> propose
>>>>>>>> >     a new
>>>>>>>> >      >     text to
>>>>>>>> >      >      >     resolve
>>>>>>>> >      >      >      >             the issue. The
>>>>>>>> >      >      >      >              >             idea is to let an
>>>>>>>> >     operator select
>>>>>>>> >      >      >     Management
>>>>>>>> >      >      >      >             VNI and use
>>>>>>>> >      >      >      >              >             that VNI in VXLAN
>>>>>>>> >     encapsulation
>>>>>>>> >      >     of BFD
>>>>>>>> >      >      >      >             Control packets:
>>>>>>>> >      >      >      >              >             NEW TEXT:
>>>>>>>> >      >      >      >              >
>>>>>>>> >      >      >      >              >                 An operator
>>>>>>>> MUST
>>>>>>>> >     select a VNI
>>>>>>>> >      >      >     number to
>>>>>>>> >      >      >      >             be used as
>>>>>>>> >      >      >      >              >                 Management
>>>>>>>> VNI. VXLAN
>>>>>>>> >      >     packet for
>>>>>>>> >      >      >      >             Management VNI MUST NOT
>>>>>>>> >      >      >      >              >                 be sent to a
>>>>>>>> tenant. VNI
>>>>>>>> >      >     number 1 is
>>>>>>>> >      >      >      >             RECOMMENDED as the
>>>>>>>> >      >      >      >              >                 default for
>>>>>>>> >     Management VNI.
>>>>>>>> >      >      >      >              >
>>>>>>>> >      >      >      >              >             With that new
>>>>>>>> text, what
>>>>>>>> >     can be the
>>>>>>>> >      >      >     value of
>>>>>>>> >      >      >      >             the destination
>>>>>>>> >      >      >      >              >             MAC in the inner
>>>>>>>> Ethernet? I
>>>>>>>> >      >     tend to
>>>>>>>> >      >      >     believe
>>>>>>>> >      >      >      >             that it can be
>>>>>>>> >      >      >      >              >             anything and
>>>>>>>> ignored by the
>>>>>>>> >      >     reciever VTEP.
>>>>>>>> >      >      >      >             Also, if the
>>>>>>>> >      >      >      >              >             trapping is based
>>>>>>>> on VNI
>>>>>>>> >      >     number, the
>>>>>>>> >      >      >      >             destination IP address
>>>>>>>> >      >      >      >              >             of the inner IP
>>>>>>>> packet
>>>>>>>> >     can from
>>>>>>>> >      >     the range
>>>>>>>> >      >      >      >             127/8 for IPv4,
>>>>>>>> >      >      >      >              >             and for IPv6 from
>>>>>>>> the range
>>>>>>>> >      >      >      >             0:0:0:0:0:FFFF:7F00:0/104. And
>>>>>>>> >      >      >      >              >             lastly, the TTL
>>>>>>>> to be
>>>>>>>> >     set to 1 (no
>>>>>>>> >      >      >     change here).
>>>>>>>> >      >      >      >              >
>>>>>>>> >      >      >      >              >             Much appreciate
>>>>>>>> your
>>>>>>>> >     comments,
>>>>>>>> >      >      >     questions, and
>>>>>>>> >      >      >      >             suggestions.
>>>>>>>> >      >      >      >              >
>>>>>>>> >      >      >      >              >             Best regards,
>>>>>>>> >      >      >      >              >             Greg
>>>>>>>> >      >      >      >              >
>>>>>>>> >      >      >      >
>>>>>>>> >      >      >
>>>>>>>> >      >
>>>>>>>> >
>>>>>>>>
>>>>>>>