Re: [tsvwg] RDMA Support by UDP FRAG Option

["C. M. Heard" <[e-mail]>]

On Fri, Jun 18, 2021 at 2:36 AM Gorry Fairhurst wrote:

> On 17/06/2021 18:30, Tom Herbert wrote:
> > On Thu, Jun 17, 2021 at 7:41 AM Gorry Fairhurst wrote:
> >> When [the] dust has settled, I expect we should see an updated
> >> fragmentation text.  I suspect we should consider the final format for
> >> suitability for offload (perhaps Tom Herbert would look?) and for ease
> >> of processing in message nbuffere (Tom Jones offered to look).
> >
> > Please look at draft-herbert-udp-space-hdr-01 for my proposal as to
> > how the UDP surplus space should be formatted. If there is interest, I
> > could update draft to include considerations for UDP options
> > fragmentation and reassembly offload as well as header/data split
> > which is needed for zero-copy receive (i.e. packet headers are DMA'ed
> > into one set of buffers to be processed by the stack, payload is
> > DMA'ed to another set of buffers to be processed by the application).
>
>
> Thanks, I saw that a while ago, thanks for reminding people.
>

Indeed. I wanted to undertake a detailed analysis of the effects of the
primary competing formats on checksum offload, and I was looking around for
the treatment in Appendix A of draft-herbert-udp-space-hdr-01,  especially
the discussion of how common offload hardware works, when the reminder came
in.

Since draft-herbert-udp-space-hdr-01 does not take into account the OCS
pseudo-header (which is one 16-bit word consisting of the surplus area
length), I'm going to do the analysis from scratch in order to account for
this point. My main conclusions are that for the purposes of checksum
offload, IT DOESN'T MATTER whether OCS is at a fixed position in an option
header or in a TLV. The differences are very minor.

What follows is a fairly detailed comparison of how the offload logic
differs between the format proposed in draft-ietf-tsvwg-udp-options-12 and
that in draft-herbert-udp-space-hdr-01. Those who do not care to pick
through the entire analysis should skip down to the CONCLUSIONS section
below.

Tom's draft describes typical hardware support for transmit checksums as
follows:

   In generic checksum offload, for each packet the host indicates to
   the device the starting offset where the checksum calculation begins
   and the offset of the field to write the resultant checksum. The
   extent of the checksum coverage is assumed to be the end of the
   packet. In particular, this means that even if the UDP checksum is
   being offloaded, the UDP surplus space is included in the device's
   computation.


As the draft-herbert-udp-space-hdr-01 notes, there are (in general) two
transmit checksums to be computed: the UDP checksum, and the OCS. The idea
is to use the hardware support to compute whichever checksum covers more
data and relegate the other to the host CPU.

When the packet is a standard UDP datagram with an options trailer in the
surplus area, OCS is typically the one that covers fewer bytes, so the host
CPU prepares and formats the option trailer either as described in
draft-ietf-tsvwg-udp-options-12 or as in draft-herbert-udp-space-hdr-01 and
computes the OCS as specified in draft-ietf-tsvwg-udp-options-12 (i.e.,
including the 1-word pseudo-header). Then the host CPU calculates a
modified UDP pseudo-header where the length is the IP payload length (not
the UDP length) and puts this in the UDP checksum field. Finally, the host
instructs the offload engine to calculate the Internet checksum over the
entire IP payload and store the complement in the UDP checksum field.
Clearly, nothing in the offload logic depends on the details of how the
trailer is formatted, only that the 1s complement sum of all 16-bit words
therein adds up to the 1s complement of the trailer length.

When the packet is a UDP fragment (including a degenerate one that is both
an initial and terminal fragment), the host CPU calculates the UDP header
checksum in the standard way (including using the UDP length of 8 in the
pseudo-header). It then formats the trailer area either as described in
draft-ietf-tsvwg-udp-options-12 or as in draft-herbert-udp-space-hdr-01 and
populates the OCS field with the surplus area length (i.e., IP payload
length - 8). Note that in the case when OCS is a TLV
(draft-ietf-tsvwg-udp-options-12) the issue of alignment may arise: the OCS
pseudo-header needs to be byte-swapped if the OCS checksum field is on an
odd byte boundary. This can be avoided by judicious use of NOPs. The host
CPU then instructs the offload engine to calculate the Internet checksum
over the surplus area and store the complement in the OCS checksum field.
Some offload hardware may not be able to store at an arbitrary byte
boundary, but any hardware alignment requirements can again be accommodated
by judicious use of NOPs. In this case some details of the offload logic
are dependent on the surplus area format, but it is clear, I think, that
either of the competing formats draft-ietf-tsvwg-udp-options-12 or
draft-herbert-udp-space-hdr-01) can be accommodated.

It's easy to accommodate optional UDP checksum (UDP CS=0) and
(conditionally) optional OCS: just omit the unneeded checksum
computation(s). Conditionally optional OCS could be accommodated in the
format proposed in draft-herbert-udp-space-hdr-01 by allowing OCS (therein
called USH checksum) equal to zero to indicate absence.

 The draft describes typical hardware support for receive checksums as
follows:

   In the most generic form of receive checksum offload, a device
   performs a running checksum calculation across a packet as it is
   received. That is, it performs a running ones complement addition
   over two byte words as they are received. The device then provides
   the computed value, referred to as the "checksum complete" value, to
   the host in the meta data (receive descriptor) for the packet. The
   host can use this value to verify one or more packet checksums
   contained in the packet.


In order to use the raw checksum provided in this way. the host CPU needs
to do the following:

1.) Compensate for any lower layer headers that may have been included in
the raw checksum. This could
the IPv4/IPv6 header and any IPv6 extension headers that the NIC does not
strip off. I'm not going to dwell further on this point because the effects
are the same under the different approaches to UDP options.

2.) Determine whether the received packet is a standard UDP packet with an
options trailer and also whether the UCP checksum is present or absent, and
if the UDP checksum is absent, whether OCS is also absent. Determining
whether the a standard UDP packet with an options trailer or a UDP fragment
(possibly a degenerate one that is both an initial and terminal fragment)
is easy: just check to see if UDP Length = 8. Determining whether the UDP
checksum is present is easy: just check to see if UDP checksum = 0.
Determining whether OCS is present is a bit more difficult with the TLV
format in draft-ietf-tsvwg-udp-options-12, but not much so, PROVIDED THAT
OCS IS ALWAYS THE FIRST OPTION, apart from any NOPs that may be present for
alignment.

Let's consider first the case where both checksums are present. The first
step for the host CPU is to compute the ones complement sum of a
pseudo-header that is similar to the standard UDP pseudo-header but uses
the IP payload length in place of the UDP length. When that is added to the
raw checksum coming from the offload engine (following compensation for
lower layer headers) the result will be zero if both the UDP checksum and
the OCS are correct or both have errors that happen to cancel
(see draft-fairhurst-udp-options-cco-00). So if this check passes, the host
CPU needs to separately verify either the UDP checksum or the OCS in order
to ensure that both are valid.

When the packet is a standard UDP datagram with an options trailer in the
surplus area, OCS is usually the cheaper one to verify, and the calculation
required to do so is very little different whether the trailer uses the TLV
format in draft-ietf-tsvwg-udp-options-12 or the format in
draft-herbert-udp-space-hdr-01. Assertions to the contrary notwithstanding,
IT IS NOT NECESSARY TO PARSE the OCS TLV in order to perform the OCS
validation. One need only compute the Internet checksum over the trailer
area using the trailer length as the pseudo-header, same as for the trailer
format described in draft-herbert-udp-space-hdr-01.

When the packet is a UDP fragment (including a degenerate one that is both
an initial and terminal fragment), the host CPU validates the UDP header
checksum in the standard way (including using the UDP length of 8 in the
pseudo-header). There is no difference in the two formats for this.

If UDP CS=0 and OCS is present, then the host CPU needs to validate OCS
only. When the packet is a standard UDP datagram with an options trailer in
the surplus area, the cheapest thing to do is likely just to calculate OCS
directly. When the packet is a UDP fragment, then the host CPU can
profitably use the output of the offload engine by adding the 1's
complement of the 1's complement checksum of the UDP header and then adding
the OCS pseudo-header. In both cases the amount of work is identical (or
nearly so) for both formats.

CONCLUSIONS:

1.) For checksum offload, the differences in the packet formats in
draft-ietf-tsvwg-udp-options-12 and format in
draft-herbert-udp-space-hdr-01 vary only at the edges. For the former it is
necessary to impose a constraint on the placement of the OCS TLV in order
for a receiver to safely determine whether it is present in the case where
UDP CS=0. For the latter this placement constraint is automatic, but on the
other hand there is now an alignment requirement when the options are in a
trailer, which is something that the receiver needs to check. To me, that's
six of one or half a dozen of the other, a wash. I do concede that there is
a question of what happens if an OCS TLV shows up later in the packet when
UDP CS = 0. My sense is that the receiver should just discard the option in
that case and not check OCS.

2.) In both cases the receive logic would be cleaner if we reduced this to
two cases: UDP CS<>0 and OCS is required (as in
draft-ietf-tsvwg-udp-options-12) or UCP CS=0 and OCS is not required (and
ignored if present). Then there would be no need to constrain the placement
of the OCS TLV (since, as I point out above, it's always possible to check
it without parsing the option TLVs). The alignment annoyance with the
herbert-udp-space-hdr-01 trailer format remains. I personally have trouble
seeing a lot of value in UDP CS=0 with OCS present, no matter how hard I
squint; but if the WG really wants it, I'm OK to leave it in.

3.) If both checksums are always present whenever either is, a substantive
simplification suggests itself, namely, not to worry about offsetting
errors in UDP CS and OCS. That weakens the check, but not by a lot, and it
streamlines both implementations that use offload and those that do not.
Again, though, this is just a suggestion; it is not a point on which I feel
strongly.

Gorry also wrote:

> I think it would be good for the WG to focus on how to finish
> draft-ietf-tsvwg-udp-options, but I do seem some opportunities to use
> some of these ideas for making the fragment header - because that also
> places all data in the "option".
>

I agree that we need to focus on getting this deliverable finished. I'm
rapidly coming to the conclusion that to some extent at least -- and
certainly for checksum offload -- the disagreements that Joe, Tom, and I
have about the packet formats amount largely to bike-shedding. I'm going to
shut up for a while, review the 50+ messages in the "A review of
draft-ietf-tsvwg-udp-options-12," and come back with some substantive
suggestions for convergence.

Mike Heard