Re: [tsvwg] RDMA Support by UDP FRAG Option

[Tom Herbert <[e-mail]>]

On Sat, Jun 19, 2021 at 12:56 PM Joseph Touch <[e-mail]> wrote:
>
> Tom,
>
> > On Jun 19, 2021, at 10:11 AM, Tom Herbert <[e-mail]> wrote:
> >
> > ...
> > There is another serious problem with transport checksums and use of
> > the UDP surplus area. Most of this discussion has presumed that the
> > UDP checksum is the one in the packet being offloaded, but that may
> > not be the case. Consider the case where a sender sends a TCP packet
> > that is encapsulated GRE/UDP or VXLAN (a very common use case in
> > virtual networks where VMs send packets on their virtual networks).
> > The stack will attempt to offload the innermost checksum which is the
> > TCP checksum.
>
> I’m assuming this means TCP in GRE/UDP (it seems ambiguous).
>
> > The TCP checksum is the one offloaded regardless of
> > whether or not the outer UDP checksum is zero (if it's non-zero then
> > the stack would set it using local checksum offload (LCO)).
>
> Can you explain this? Packets are both defined and processed from the outside in; to do anything else might not yield a meaningful result.
>
It is described in
https://www.kernel.org/doc/html/latest/networking/checksum-offloads.html.

> > The
> > offloaded TCP checksum computation would start the computation at the
> > first byte of the TCP header through the end of the whole packet.
>
> If TCP interprets “end” to mean anything beyond what the UDP header indicates, it is that TCP stack that is broken.
>
The assumption is that there are no bits in the packet beyond the
transport layer on transmit. This is a valid assumption since there
are currently no use cases where this is true; UDP options would be
the first instance of supporting trailers. On the receive side, stacks
will properly handle checksum offload with data in the surplu space.

> (Presumably we’re talking about non-UDP fragmented packets; with UDP fragmentation, the TCP operation should never happen until the UDP fragments are reassembled).
>
> > So
> > unless the surplus area is properly checksummed then the computed TCP
> > checksum will be invalid and the packet will be dropped at the
> > receiver.
>
> See above.
>
> > This is not just a problem for offload for offload, I
> > believe that this wouldn't work properly in existing software stacks
> > without some major changes.
>
> Same problem, same answer.
>
> > So to make all uses of transport checksum computation and offload
> > reasonably robust, when the UDP surplus area is being used both the
> > UDP checksum and checksum over the surplus area MUST always be set.
>
> If protocol stacks that try to peek ahead in layers don’t follow the rules, there’s not much we can do, ever.
>
> > FYI, here is some nice background checksum offload is
> > https://www.kernel.org/doc/html/latest/networking/checksum-offloads.html
>
> There is an error in draft-herbert-remotecsumoffload in Section 2.1; it states that the UDP checksum is over the upper layer packet length (it is not). RFC768 defines the UDP checksum over the pseudoheader, UDP header, and data (not referring to the IP packet except for pseudo header info). In fact, it also refers to the pseudoheader as using the UDP Length, not the IP length (adjusted or not).
>
> The same error appears in the draft-herbert-vxlan-rco in the definition of packet_csum.
>
> If either doc reflects how offloading is implemented, then those are bugs that should be fixed.
>
> In TCPM we’ve identified a number of additional issues with TCP offload, notably regarding how they incorrectly coalesce packets with different TCP headers.
>
> If we’re not calling out these behaviors as the bugs they are, there’s little point in doing much of anything in the IETF.
>
You're welcome to call these behavior bugs if you want, but the fact
is they are correct behavior and robust behavior for all currently
defined IETF protocols as evidenced by the fact that the Internet runs
on billions of devices with these behaviors. The problem you are
hitting is that we have over thirty years of deployment and
implementation experience with Internet protocols that follow some
basic principles and conventions like use protocol headers and not
trailers. So while a protocol that diverges from those principles and
conventions might be academically correct on paper, in deployment it
may be replete with a myriad of issues which is what we see when
delving into the details of how UDP options interact with real stacks
and devices. If the goal is to produce a deployable and performant
protocol, which I believe is the purpose of IETF, then we need to take
realities of deployment and implementation into account.

Tom

> Joe