Re: [Stackevo-discuss] Scope of stackevo and ossification in DC

[Tom Herbert <[e-mail]>]

>> Similar to the use of protocols on the Internet we are hitting the
>> transport protocol ossification problem in the data center.
>> Specifically, performance optimizations in networking devices only
>> support TCP or UDP, and without these optimizations this negatively
>> impacts our use of other protocols.
>
> Right. But this is a fundamental problem, I think. NIC offloads reach pretty deeply into the transport protocol, and as such won't work with new transports whether they're encrypted or not until those new transports. A question: for NIC offload, how much of the win comes from segmentation offloading, and how much comes from other trickery? If the biggest win really is bundling a bunch of packets into a single context switch, then how would the performance of the current offload architecture compare with a smart library on top of approaches like netmap?
>
Segmentation offload (RX and TX) is considered win because it reduces
the number of packets that need to be processed through various layers
of the stack. This becomes really evident in deep layering such as we
see with network virtualization. However, most of the benefits can be
achieved with software mechanisms and LRO (RX segmentation offload) is
pretty controversial since the device is compressing TCP headers and
that has had some insidious effects. Checksum offload and RSS are the
critical offloads we need.

>> One example of this is the need
>> for fine grained ECMP which has become driver behind many of the
>> foo-over-UDP proposals (e.g. MPLS/UDP, GRE/UDP, ...).
>
> So this is a separate issue -- ECMP is a (semi-elegant) hack, predicated on the assumption that things on a five-tuple need to stay together and things on separate five-tuples don't. NAT + TCP (any reordering-intolerant transport, really) makes this assumption more or less hold. Driving it in the opposite direction -- using knowledge that there's ECMP on path to do cheap traffic engineering -- leads to the unintended consequences that foo-over-udp brings with it.
>
> What you really want architecturally is a way for the network layer (at a gateway) to explicitly say "keep these packets together" and "it's okay to split these packets apart". It'd be even better if we had a way to request/measure/enforce actual path diversity without manually managing tunnels, but this is sadly explicitly a non-feature of our routing protocols. In any case this seems to have a harder incremental deployment story than simple transport state exposure.
>
IPv6 flow label for ECMP (RFC6438) solves the problem of ECMP/RSS.
With the use of this, devices don't need to parse beyond the IPv6
header to switch packets and we don't need to have the overhead of UDP
encapsulation just for the purpose of getting good ECMP.

>> This problem is likely a proper subset of the general problem, but
>> might be more amenable to some "simpler" solutions. Is this within
>> scope of stackevo?
>
> It very much seems to be, yes. Let's keep this discussion going on this list...
>
Protocol ossification is also now in the vernacular of Linux
networking: https://lwn.net/Articles/667059/

Tom