Re: [nvo3] RFC 7637 on NVGRE: Network Virtualization Using Generic Routing Encapsulation

Sure, probably all of the hardware implementations have some limits on their ability to handle the full breadth of Geneve options. Geneve was intentionally designed to be very future proof and support limits beyond what I would realistically expect people to implement/use in the near future. The more interesting question is whether it is possible to support a useful set for what we need today. Unfortunately, I can’t really give specifics for different implementations (in the cases that I know) but let me try to make some generalizations and you can follow up with individual vendors if you need more details.

NICs generally don’t have much of an issue and seem to have largely settled on supporting 64 bytes of options with offloads for the current generation. I don’t really see it as an issue if the parser can’t looker deeper than this. Assuming that 64 option bytes is enough for data packets, the control plane would simply configure senders to not use more than that. And in the edge cases where we need to use more than that (perhaps a control packet), the packets would still flow, just without using offloads.

Switching ASICs are more complicated and varied, as you say. There are definitely implementations that can support parsing and generating a set of TLVs at full multi-terabit data rates (16 or 32 bytes are popular here) and undoubtedly some that are just supporting VXLAN-like capabilities with a few additional features. Even the latter is useful since it allows transitioning to a single protocol with incremental use of additional functionality, especially since in some cases this can be enabled with existing chips that are already deployed.

One final point – most of these limitations are around the length of the header, not the variable nature of it. So even if Geneve as a protocol is capable of supporting more than what is used today, the baseline is probably pretty similar to the alternatives and the future flexibility is hugely useful. Without a doubt, implementing variable length headers is more effort than fixed length but based on the above implementations, it seems clearly doable if the benefit is there. Given the number of extensions that we have seem to VXLAN, this type of extensibility seems to be essential.

Jesse

From: Diego Garcia del Rio
Date: Sunday, November 1, 2015 at 11:40 AM
To: Jesse Gross
Cc: Tom Herbert, Pankaj Garg, "nvo3@ietf.org<mailto:nvo3@ietf.org>", "Manish Kumar (manishkr)", Dino Farinacci, Lucy yong
Subject: Re: [nvo3] RFC 7637 on NVGRE: Network Virtualization Using Generic Routing Encapsulation

Thanks for the references. But out of all the switching asics, which are arguably the most constrained in terms parsing flexibility and need for performance, how many support arbitrary tlvs push/pop? Geneve in its most basic form without options looks very similar to vxlan and thus I'm not surprised we can see several of the products claiming support for it. But when we start adding arbitrary protocol types and variable options I'm not so sure that those implementations can claim full support. Even for the nics and all the offloads, it comes a moment where the parsing engines can't look deep enough into the packet and then we have a problem.

I think there is a reason why there is quite some pushback from the hardware side against variable length options.

My two cents,
Diego

On Oct 30, 2015, at 21:29, Jesse Gross <jgross@vmware.com<mailto:jgross@vmware.com>> wrote:

From: Tom Herbert
Date: Saturday, October 31, 2015 at 9:40 AM
To: Jesse Gross
Cc: Pankaj Garg, "nvo3@ietf.org<mailto:nvo3@ietf.org>", "Manish Kumar (manishkr)", Dino Farinacci, Lucy yong
Subject: Re: [nvo3] RFC 7637 on NVGRE: Network Virtualization Using Generic Routing Encapsulation

To follow up on Pankaj’s mention of ecosystem support, one comment about the
viability of TLVs is that whether they are a useful extension mechanism is
mostly based on the implementer’s perception. If they are seen as an add-on
that is not really core functionality (as in IPv4 and IPv6), then sure,
people won’t bother to support them. However, in the case of Geneve, they
are obviously the major goal of the protocol and they are being implemented,
in both software and hardware.

Jesse,

Your point that TLVs are major goal of Geneve would be much stronger
if you could reference defined TLVs that are critical to the protocol
function. Maybe I'm missing something, but I can't find any defined
TLVs for Geneve on the web at all.

Here is an example from OVN:
https://github.com/openvswitch/ovs/blob/master/ovn/ovn-architecture.7.xml#L1014

OVN is actually a pretty useful reference in general because it is an open source network virtualization implementation that is using Geneve as its preferred encapsulation.

And here is a list of implementations of Geneve that I am aware of, from my presentation at the meeting in Prague plus a few additional ones that I became aware of since then. We don’t really need to speculate what people might do, given that there are a number of implementations already from different vendors. (This is all public.):

Controller:
Open Virtual Networking (OVN)

Software Endpoint:
Open vSwitch
Linux

Debugging Tool:
Wireshark
tcpdump
libpcap

NIC:
Intel XL710
Mellanox ConnectX-4
Broadcom NetXtreme C-Series
QLogic 578xx
Netronome NFP-6xxx

Switching ASIC:
Broadcom Trident 2+/DNX
Cavium XPliant
Mellanox Spectrum
Intel Red Rock Canyon
Centec GoldenGate
Marvell Prestera
_______________________________________________
nvo3 mailing list
nvo3@ietf.org<mailto:nvo3@ietf.org>
https://www.ietf.org/mailman/listinfo/nvo3