Re: [tsvwg] RDMA Support by UDP FRAG Option

[Tom Herbert <[e-mail]>]

On Sat, Jun 19, 2021, 2:58 PM Joseph Touch <[e-mail]> wrote:

>
>
> > On Jun 19, 2021, at 1:40 PM, Tom Herbert <[e-mail]> wrote:
> >
> > On Sat, Jun 19, 2021 at 12:56 PM Joseph Touch <[e-mail]>
> wrote:
> >>
> >> Tom,
> >>
> >>> On Jun 19, 2021, at 10:11 AM, Tom Herbert <[e-mail]> wrote:
> >>>
> >>> ...
> >>> There is another serious problem with transport checksums and use of
> >>> the UDP surplus area. Most of this discussion has presumed that the
> >>> UDP checksum is the one in the packet being offloaded, but that may
> >>> not be the case. Consider the case where a sender sends a TCP packet
> >>> that is encapsulated GRE/UDP or VXLAN (a very common use case in
> >>> virtual networks where VMs send packets on their virtual networks).
> >>> The stack will attempt to offload the innermost checksum which is the
> >>> TCP checksum.
> >>
> >> I’m assuming this means TCP in GRE/UDP (it seems ambiguous).
> >>
> >>> The TCP checksum is the one offloaded regardless of
> >>> whether or not the outer UDP checksum is zero (if it's non-zero then
> >>> the stack would set it using local checksum offload (LCO)).
> >>
> >> Can you explain this? Packets are both defined and processed from the
> outside in; to do anything else might not yield a meaningful result.
> >>
> > It is described in
> > https://www.kernel.org/doc/html/latest/networking/checksum-offloads.html
> .
> >
> >>> The
> >>> offloaded TCP checksum computation would start the computation at the
> >>> first byte of the TCP header through the end of the whole packet.
> >>
> >> If TCP interprets “end” to mean anything beyond what the UDP header
> indicates, it is that TCP stack that is broken.
> >>
> > The assumption is that there are no bits in the packet beyond the
> > transport layer on transmit. This is a valid assumption since there
> > are currently no use cases where this is true;
>
> That is the fundamental error; this is not a valid assumption.
>
> > UDP options would be
> > the first instance of supporting trailers.
>
> TCP has had trailers too, at one point.
>
> > On the receive side, stacks
> > will properly handle checksum offload with data in the surplu space.
> >
> >> (Presumably we’re talking about non-UDP fragmented packets; with UDP
> fragmentation, the TCP operation should never happen until the UDP
> fragments are reassembled).
> >>
> >>> So
> >>> unless the surplus area is properly checksummed then the computed TCP
> >>> checksum will be invalid and the packet will be dropped at the
> >>> receiver.
> >>
> >> See above.
> >>
> >>> This is not just a problem for offload for offload, I
> >>> believe that this wouldn't work properly in existing software stacks
> >>> without some major changes.
> >>
> >> Same problem, same answer.
> >>
> >>> So to make all uses of transport checksum computation and offload
> >>> reasonably robust, when the UDP surplus area is being used both the
> >>> UDP checksum and checksum over the surplus area MUST always be set.
> >>
> >> If protocol stacks that try to peek ahead in layers don’t follow the
> rules, there’s not much we can do, ever.
> >>
> >>> FYI, here is some nice background checksum offload is
> >>>
> https://www.kernel.org/doc/html/latest/networking/checksum-offloads.html
> >>
> >> There is an error in draft-herbert-remotecsumoffload in Section 2.1; it
> states that the UDP checksum is over the upper layer packet length (it is
> not). RFC768 defines the UDP checksum over the pseudoheader, UDP header,
> and data (not referring to the IP packet except for pseudo header info). In
> fact, it also refers to the pseudoheader as using the UDP Length, not the
> IP length (adjusted or not).
> >>
> >> The same error appears in the draft-herbert-vxlan-rco in the definition
> of packet_csum.
> >>
> >> If either doc reflects how offloading is implemented, then those are
> bugs that should be fixed.
> >>
> >> In TCPM we’ve identified a number of additional issues with TCP
> offload, notably regarding how they incorrectly coalesce packets with
> different TCP headers.
> >>
> >> If we’re not calling out these behaviors as the bugs they are, there’s
> little point in doing much of anything in the IETF.
> >>
> > You're welcome to call these behavior bugs if you want, but the fact
> > is they are correct behavior and robust behavior for all currently
> > defined IETF protocols as evidenced by the fact that the Internet runs
> > on billions of devices with these behaviors.
>
> “Currently works” is not the same as “correct”.
>
> Correct follows the spec; the above notes do not. They were never in WG
> docs or that would have been presumably corrected.
>
> > The problem you are
> > hitting is that we have over thirty years of deployment and
> > implementation experience with Internet protocols that follow some
> > basic principles and conventions like use protocol headers and not
> > trailers.
>
> It’s not about headers vs. trailers. It’s about whether you follow the
> specs or not.
>
> > So while a protocol that diverges from those principles and
> > conventions might be academically correct on paper, in deployment it
> > may be replete with a myriad of issues which is what we see when
> > delving into the details of how UDP options interact with real stacks
> > and devices. If the goal is to produce a deployable and performant
> > protocol, which I believe is the purpose of IETF, then we need to take
> > realities of deployment and implementation into account.
>
> We need to start bu not continuing to refer to false claims in unpublished
> drafts.
>
> When we find errors, we should fix them, not propagate them.
>

Joe,

A nice thing about an open source project like Linux is that anyone who
thinks there's a bug they can submit a patch and it will be accepted *if*
you can justify the patch to the maintainers. Good luck, if you want to
take that on!

Tom


> Joe