Re: [tsvwg] RDMA Support by UDP FRAG Option

[Joseph Touch <[e-mail]>]

> On Jun 19, 2021, at 1:40 PM, Tom Herbert <[e-mail]> wrote:
> 
> On Sat, Jun 19, 2021 at 12:56 PM Joseph Touch <[e-mail]> wrote:
>> 
>> Tom,
>> 
>>> On Jun 19, 2021, at 10:11 AM, Tom Herbert <[e-mail]> wrote:
>>> 
>>> ...
>>> There is another serious problem with transport checksums and use of
>>> the UDP surplus area. Most of this discussion has presumed that the
>>> UDP checksum is the one in the packet being offloaded, but that may
>>> not be the case. Consider the case where a sender sends a TCP packet
>>> that is encapsulated GRE/UDP or VXLAN (a very common use case in
>>> virtual networks where VMs send packets on their virtual networks).
>>> The stack will attempt to offload the innermost checksum which is the
>>> TCP checksum.
>> 
>> I’m assuming this means TCP in GRE/UDP (it seems ambiguous).
>> 
>>> The TCP checksum is the one offloaded regardless of
>>> whether or not the outer UDP checksum is zero (if it's non-zero then
>>> the stack would set it using local checksum offload (LCO)).
>> 
>> Can you explain this? Packets are both defined and processed from the outside in; to do anything else might not yield a meaningful result.
>> 
> It is described in
> https://www.kernel.org/doc/html/latest/networking/checksum-offloads.html.
> 
>>> The
>>> offloaded TCP checksum computation would start the computation at the
>>> first byte of the TCP header through the end of the whole packet.
>> 
>> If TCP interprets “end” to mean anything beyond what the UDP header indicates, it is that TCP stack that is broken.
>> 
> The assumption is that there are no bits in the packet beyond the
> transport layer on transmit. This is a valid assumption since there
> are currently no use cases where this is true;

That is the fundamental error; this is not a valid assumption.

> UDP options would be
> the first instance of supporting trailers.

TCP has had trailers too, at one point.

> On the receive side, stacks
> will properly handle checksum offload with data in the surplu space.
> 
>> (Presumably we’re talking about non-UDP fragmented packets; with UDP fragmentation, the TCP operation should never happen until the UDP fragments are reassembled).
>> 
>>> So
>>> unless the surplus area is properly checksummed then the computed TCP
>>> checksum will be invalid and the packet will be dropped at the
>>> receiver.
>> 
>> See above.
>> 
>>> This is not just a problem for offload for offload, I
>>> believe that this wouldn't work properly in existing software stacks
>>> without some major changes.
>> 
>> Same problem, same answer.
>> 
>>> So to make all uses of transport checksum computation and offload
>>> reasonably robust, when the UDP surplus area is being used both the
>>> UDP checksum and checksum over the surplus area MUST always be set.
>> 
>> If protocol stacks that try to peek ahead in layers don’t follow the rules, there’s not much we can do, ever.
>> 
>>> FYI, here is some nice background checksum offload is
>>> https://www.kernel.org/doc/html/latest/networking/checksum-offloads.html
>> 
>> There is an error in draft-herbert-remotecsumoffload in Section 2.1; it states that the UDP checksum is over the upper layer packet length (it is not). RFC768 defines the UDP checksum over the pseudoheader, UDP header, and data (not referring to the IP packet except for pseudo header info). In fact, it also refers to the pseudoheader as using the UDP Length, not the IP length (adjusted or not).
>> 
>> The same error appears in the draft-herbert-vxlan-rco in the definition of packet_csum.
>> 
>> If either doc reflects how offloading is implemented, then those are bugs that should be fixed.
>> 
>> In TCPM we’ve identified a number of additional issues with TCP offload, notably regarding how they incorrectly coalesce packets with different TCP headers.
>> 
>> If we’re not calling out these behaviors as the bugs they are, there’s little point in doing much of anything in the IETF.
>> 
> You're welcome to call these behavior bugs if you want, but the fact
> is they are correct behavior and robust behavior for all currently
> defined IETF protocols as evidenced by the fact that the Internet runs
> on billions of devices with these behaviors.

“Currently works” is not the same as “correct”.

Correct follows the spec; the above notes do not. They were never in WG docs or that would have been presumably corrected.

> The problem you are
> hitting is that we have over thirty years of deployment and
> implementation experience with Internet protocols that follow some
> basic principles and conventions like use protocol headers and not
> trailers.

It’s not about headers vs. trailers. It’s about whether you follow the specs or not.

> So while a protocol that diverges from those principles and
> conventions might be academically correct on paper, in deployment it
> may be replete with a myriad of issues which is what we see when
> delving into the details of how UDP options interact with real stacks
> and devices. If the goal is to produce a deployable and performant
> protocol, which I believe is the purpose of IETF, then we need to take
> realities of deployment and implementation into account.

We need to start bu not continuing to refer to false claims in unpublished drafts.

When we find errors, we should fix them, not propagate them.

Joe