Re: BFD over VXLAN: Trapping BFD Control packet at VTEP

[Dinesh Dutt <[e-mail]>]

I understand his point now. It arose in my head because of a difference in
perspective. On a hardware switch, pretty much every implementation I've
seen reuses one of the device's MAC address across all the local VNIs. Even
in VMW's case, Even on a server, NIC's MAC address is by default reused
across all VNIs. It seems some software switches have taken liberties with
this.

Given this information, if we want to ensure any proposal we come up with
will work with existing packet switching silicon and software switch
implementation,  I also would prefer not to define something that'll be
overridden or worked around soon enough by a new draft. With these
assumptions, I see the problem definition as:

   - checking VTEP-to-VTEP connectivity in the overlay. Checking
   VTEP-to-VTEP connectivity in the underlay is a multi-hop BFD scenario.
   - I suspect whoever wants to check connectivity in the overlay will also
   want to check the connectivity in a specific VNI(s), Otherwise why are they
   checking this in the overlay? There maybe existing mechanisms such as IEEE
   OAM frames to check overlay in a specific VNI, but given my past
   experience, I suspect people will want a single unified mechanism using
   VXLAN (at least I see this in the data center use case).

I believe you;ve implicitly excluded the case of bullet 2 currently. I'm
not in favor of that for the reasons I stated above.

I see the primary solution principles as:

   - BFD packet addressed to the VTEP MUST not leak beyond the VTEP..
   - BFD packets exchanged between endpoints behind VTEPs MUST work as
   well.

So, on the critical question of how do we address the VTEP, I see we have
three possibilities:

   - Pick a MAC/IP pair that belongs to the VTEP in the VNI that you wish
   to send the packet in. These three parameters MUST be configurable: {VNI,
   MAC, IP}. The default VNI for this can be VNI 1.
   - Pick a management VNI and use that VNI to trap control packets to the
   CPU. I don't like this option because we're burning a VNI for just this
   purpose.
   - Pick a well-defined MAC/IP for a given VNI. This is the solution that
   tries to use the 127/8 address for example. But this is a Martian packet,
   is it not? I don't like it for this reason.

Does this make sense?

Dinesh

On Thu, Aug 1, 2019 at 5:40 AM Greg Mirsky <[e-mail]> wrote:

> Hi Dinesh,
> as I understand, Sridhar is on vacation. In the meantime, this is his
> response to the question on using VTEP's MAC address as the destination MAC
> in the inner Ethernet frame:
>
>
> T. Sridhar
> Sun, Jun 30, 7:13 PM
> to Reshad, Martin, [e-mail], Matthew, Sam, Jeffrey
> Reshad,
>
> Sorry - could not respond earlier.
>
> Joel is right - the draft does impose a requirement on the inner MAC being
> the same as the VTEP MAC and eats into the tenant MAC space which is not a
> desirable approach. You *could* use VNI0 as the one only VNI to specify
> this but that's not ideal either since we did not impose any restriction on
> the VNI space.
>
> Btw,  I also think I should modify the comment I had made below about
> using the inner IP to be the same as the VTEP IP since there is no
> guarantee that the inner MAC frame is an IP packet.
>
> Thanks,
> Sridhar
>
>
>
> On 6/26/19, 6:59 PM, "Reshad Rahman (rrahman)" <[e-mail]> wrote:
>
>     Thank you Sridhar. One concern which came up is wrt the use of the
> destination VTEP MAC address as dest MAC in the inner IP header (see
> attached email). Could you please comment on that aspect?
>
>     Regards,
>     Reshad.
>
> I understand that RFC 7348 maybe is not clear on that issue. I'd like to
> understand how the existing implementations behave, process VXLAN header
> and the inner Ethernet frame to minimize changes BFD over VXLAN may impose
> on the implementation.
>
> Regards,
> Greg
>
>
> On Thu, Aug 1, 2019 at 12:48 AM Dinesh Dutt <[e-mail]> wrote:
>
>> I don't understand his objection. My recommendation is to understand that
>> before we propose new text. I fear otherwise that we'll have a new draft in
>> a few months to address the issue of using non-mgmt VNI.
>>
>> Dinesh
>> On Jul 31, 2019, 12:07 PM -0700, Greg Mirsky <[e-mail]>,
>> wrote:
>>
>> Hi Dinesh,
>> if I recall correctly, T.Sridhar has noted that VTEP's MAC must not be
>> used as the destination MAC address in the inner Ethernet frame.
>>
>>
>>
>> Also, I should have been more precise in the proposed text, please see
>> the updated version to stress that the management VNI MUST NOT be one of
>> the tenant's VNIs:
>> NEW TEXT:
>>
>> An operator MUST select a VNI number to be used as Management VNI.
>> Management VNI number MUST NOT be one of the tenant's VNIs to prevent
>> sending VXLAN packets received on Management VNI to a tenant. VNI number 1
>> is RECOMMENDED as the default for Management VNI.
>>
>>
>> On Wed, Jul 31, 2019 at 2:25 PM Dinesh Dutt <[e-mail]> wrote:
>>
>>> Hi Greg,
>>>
>>> On Wed, Jul 31, 2019 at 9:20 AM Greg Mirsky <[e-mail]>
>>> wrote:
>>>
>>>> Hi Dinesh,
>>>> thank you for your consideration of the proposal and questions. What
>>>> would you see as the scope of testing the connectivity for the specific
>>>> VNI? If it is tenant-to-tenant, then VTEPs will treat these packets as
>>>> regular user frames. More likely, these could be Layer 2 OAM, e.g. CCM
>>>> frames. The reason to use 127/8 for IPv4, and 0:0:0:0:0:FFFF:7F00:0/104 for
>>>> IPv6 is to safeguard from leaking Ethernet frames with BFD Control packet
>>>> to a tenant.
>>>> You've suggested using a MAC address to trap the control packet at
>>>> VTEP. What that address could be? We had proposed using the dedicated MAC
>>>> and VTEP's MAC and both raised concerns among VXLAN experts. The idea of
>>>> using Management VNI may be more acceptable based on its similarity to the
>>>> practice of using Management VLAN.
>>>>
>>>
>>> If you use the inner IP address as the VTEP IP address, then use the MAC
>>> address that the VTEP would respond with when replying to an ARP for that
>>> VTEP IP address. If a VXLAN expert disagrees with this, could you kindly
>>> tell me who it is so that I can understand their disagreement? So this
>>> handles the case where the VNI is not a user-tenant VNI. If the VNI used in
>>> the BFD packet is a user-tenant VNI, then the receiving VTEP MUST have an
>>> IP address in that VNI (mapped to a VRF) else you cannot use that VNI in
>>> the BFD packet. Why won't this combination address all the cases you've
>>> listed? What am I missing? Define VNI 1 as a possible use, not VNI 0. I
>>> objected to VNI 0 because there are too many switching siicon out there and
>>> some of them will not be able to handle this scenario.
>>>
>>> Dinesh
>>>
>>>>
>>>> Regards,
>>>> Greg
>>>>
>>>> On Wed, Jul 31, 2019 at 12:03 PM Dinesh Dutt <[e-mail]> wrote:
>>>>
>>>>> Hi Greg,
>>>>>
>>>>> As long as the inner MAC address is such that the packet is trapped to
>>>>> the CPU, it should be fine for use as an inner MAC is it not? Stating that
>>>>> is better than trying to force a management VNI. What if someone wants to
>>>>> test connectivity on a specific VNI? I would not pick a loopback IP address
>>>>> for this since that address range is host/node local only. Is there a
>>>>> reason you're not using the VTEP IP as the inner IP address ?
>>>>>
>>>>> Dinesh
>>>>>
>>>>> On Wed, Jul 31, 2019 at 5:48 AM Greg Mirsky <[e-mail]>
>>>>> wrote:
>>>>>
>>>>>> Dear All,
>>>>>> thank you for your comments, suggestions on this issue, probably the
>>>>>> most challenging for this specification. In the course of our discussions,
>>>>>> we've agreed to abandon the request to allocate the dedicated MAC address
>>>>>> to be used as the destination MAC address in the inner Ethernet frame.
>>>>>> Also, earlier using VNI 0 was changed from mandatory to one of the options
>>>>>> an implementation may offer to an operator. The most recent discussion was
>>>>>> whether VTEP's MAC address might be used as the destination MAC address in
>>>>>> the inner Ethernet frame. As I recall it, the comments from VXLAN experts
>>>>>> equally split with one for it and one against. Hence I would like to
>>>>>> propose a new text to resolve the issue. The idea is to let an operator
>>>>>> select Management VNI and use that VNI in VXLAN encapsulation of BFD
>>>>>> Control packets:
>>>>>> NEW TEXT:
>>>>>>
>>>>>> An operator MUST select a VNI number to be used as Management VNI.
>>>>>> VXLAN packet for Management VNI MUST NOT be sent to a tenant. VNI number 1
>>>>>> is RECOMMENDED as the default for Management VNI.
>>>>>>
>>>>>> With that new text, what can be the value of the destination MAC in
>>>>>> the inner Ethernet? I tend to believe that it can be anything and ignored
>>>>>> by the reciever VTEP. Also, if the trapping is based on VNI number, the
>>>>>> destination IP address of the inner IP packet can from the range 127/8 for
>>>>>> IPv4, and for IPv6 from the range 0:0:0:0:0:FFFF:7F00:0/104. And lastly,
>>>>>> the TTL to be set to 1 (no change here).
>>>>>>
>>>>>> Much appreciate your comments, questions, and suggestions.
>>>>>>
>>>>>> Best regards,
>>>>>> Greg
>>>>>>
>>>>>