Re: [Stackevo-discuss] On boundaries and interfaces in transport protocol evolution (was Re: [tsvwg] draft-byrne-opsec-udp-advisory)

[Tom Herbert <[e-mail]>]

Hi Brian,

Some replies inline...

> I've skimmed the draft but need to look into GUE more deeply. Is there code I can play with somewhere?
>
http://lwn.net/Articles/615044/ for GUE in Linux
https://lwn.net/Articles/614348/ for FOU which allows direct
encapsulation of any IP protocol over UDP (GUE and FOU share code
base)

> I'm only passingly familiar with the situation here -- I doubt any of the TCP/IP stack code from the kernel that was around the last time I dug into it (1997) is still there. But I will observe that an interface with fifty hardpoints isn't an interface, it's an arbitrary line drawn through what is essentially an integrated design.
>
For the most part the interfaces are clean. SCTP and DCCP similarly
interact with the rest of the stack although probably not as intensely
since TCP tends to be the first focus for optimizations.

> Protocol offloading is pretty much the opposite of what we're trying to do here -- offloads only work for ossified protocols pretty much by definition, so we'd have to give these up in the near term for new protocols.
>
Not really, many good protocol offloads are generic mechanisms that
permit broad application. For instance, if implemented properly HW
checksum offload can work on any transport layer protocol checksum
even if device does not know how to parse the the transport or even
network layer or any used encapsulation protocols. CRC could be
similarly offloaded (like for SCTP) at least on transmit. For UDP
encapsulations, we are requesting that vendors implement offloads that
are agnostic to (the many :-) ) different encapsulation protocols (see
http://people.netfilter.org/pablo/netdev0.1/papers/UDP-Encapsulation-in-Linux.pdf).

>> The
>> stack interacts with packet steering mechanisms (deliver packets to
>> right CPU),
>
> This may be exposing my ignorance, but packet steering works on (a function of) the 5-tuple, no? One of the points we're conceding here is that the ports are de facto part of the network layer, and appear at fixed offsets from the end of the network header.
>
The 5-tuple hash for TCP and UDP is what most devices have
implemented, however IMO there is a better solution moving forward. As
I pointed out before, not all packets contain port numbers (IP
fragments, IPIP, GRE/IP, MPLS/IP, etc.). Also, for ECMP and RSS
hardware needs to parse the transport protocol so currently we are
effectively limited from using anything other than TCP or UDP in the
DC for high packet rate applications. I have come to believe that the
real fix for these problems (as opposed to just blindly encapsulating
everything in UDP) is to use IPv6 flow labels for load balancing (a la
RFC6438). We've implemented the necessary kernel support, and are
actively requesting HW vendors to use the field as input to ECMP and
RSS (several already do). We intend to enable automatic flow label
generation by default in Linux after due diligence, so in the not too
distant future expect to see a lot of IPv6 packets on the Internet
with non-zero flow labels!

Tom