Re: [tsvwg] A review of draft-ietf-tsvwg-udp-options-12

[Tom Herbert <[e-mail]>]

On Tue, Jun 15, 2021 at 8:12 AM Joseph Touch <[e-mail]> wrote:
>
> Mike,
>
> Zero-copy networking has been around since roughly 1991 (James Sterbenz PhD dissertation on “Axon” was arguably one of the first, the IBM 360 had zero copy for memory storage transfers back in the 1960s), and it has been used in networking stacks since that time. Yes, it’s more prevalent in datacenter and HPC (high performance computing) environments, where RDMA is one variant, but it can be supported in at least some network cards that offload packet processing.
>
> It requires few per-packet decisions to confirm support, which is why some simple decisions we can make - the don’t constrain us - help.Those below include making OCS come first and enabling FRAG to come immediately after, so the entire TLV chain between the two doesn’t need to be walked. I agree we should not design in ways that complicate processing for non-zerocopy endpoints, but if the work is either small (as it was previously) or effectively none (as it would be with the structure below), there’s no utility in ignoring it.
>
> It might be useful to note that zero copy support exists in most modern OSes, including Linux, macOS, and Windows.
>
Joe,

Please see the paper that Mike referred to, that describes concisely
how zero copy receive is supported in Linux. Receive zero was not easy
to pull off and does require a constrained environment. Zero copy send
was easier as we have splice and MSG_ZEROCOPY has been added as the
deck describes.

IMO, if you want to make UDP options seamlessly integrate and get
ubiquitous support of UDP options in the OS and hardware (leverage
checksum offload, zero copy, hardware fragmentation/reassembly,
offload authentication/crypto, header/data split, etc.), the best
direction would be to make UDP with options look like TCP as much as
possible at least with regards to packet format. i.e. use the same
option format, have a header length field, use headers instead of
trailers, etc.

Tom

> Joe
>
> On Jun 14, 2021, at 10:10 PM, C. M. Heard <[e-mail]> wrote:
>
> On Sun, Jun 13, 2021 at 9:31 PM Joseph Touch wrote:
>>
>> On Jun 13, 2021, at 7:20 PM, C. M. Heard wrote:
>>>
>>> I for one would appreciate further discussion of these last points. I admit that I have failed to grasp Joe's message on the RDMA thread, and I would appreciate some time to think about it .
>>
>>
>> Sure - here’s how it all works. Note that this is relevant mostly for long transfers with persistent UDP fragmentation; if that is assumed to be ‘adjusted’ at the app layer (as QUIC does), then we don’t need zero-copy support...
>>
>> - right now, UDP data can be zero-copied when received into user space, starting with the user data
>> - if we add options, UDP data can still be zero-copied because it hasn’t moved (it still begins the payload
>> - however, fragments are different because (esp given the merging of frag and lite) they don’t start at the beginning of data
>> - they always start after OCS (which I think we should make fit the uniform KIND/LEN/OCS format of 4 bytes)
>> - if the FRAG comes next, then we can move the frag content around a little and still support zero-copy
>>
>> notably, we move the first 10 bytes of the fragment to the end
>> 4 for OCS
>> 6 for FRAG (assuming FRAG includes KIND/OPTLEN/FRAGOFFSET/ID/FRAGLEN)
>> that way we can zero-copy the frag packet into place, then just copy those last 8 bytes over OCS and the FRAG header
>>
>> This method assumes that we try to keep FRAG early in the packet - preferably right after OCS. The later it comes, the more additional bytes we need to move to “fix” the copy (beyond the 8 bytes noted above).
>>
>> —
>>
>> This method is the only reason we would want to allow options after non-terminal fragments - basically to keep the fragment toward the front of the packet, using the rule that post-noninitial frag options still operate on the fragment, rather than waiting for reassembly. The exception is the terminal fragment, where post-terminal fragment options operate on the reassembled packet.
>
>
> I'm not understanding this AT ALL, and I apologize if there is well-known stuff of which I am embarrassingly ignorant. That being said:
>
> EVERY description of a zero-copy receive describes something involving MTUs and highly constrained header length that allow the user data in a TCP segment or UDP packet to be mapped to one or more kernel pages. Here is one example:
>
> PATH to TCP 4K MTU and RX zerocopy
>
> In every case that I have found, the solutions apply only to a highly constrained environment, such as a data center, and not over the Internet writ large. Some even involve requiring the application to process the transport headers, which is surely not an outcome that we wish in general.
>
> If I am wrong -- and it would most assuredly not be the first time -- I am eager to be disabused, preferably with a complete and open description of a zero-copy technology without such shortcomings.
>
> But if my conclusions are substantially correct, I don't think that TSVWG should expend effort on zero copy for UDP fragment reassembly. Transport options for UDP need to apply across the general Internet.
>
> NOTE: the unfavorable conclusions that I make about zero-copy do NOT apply to checksum offload; the advantages and applicability of that technology (especially with OCS now defined to be an equivalent to the CCO proposal) are readily apparent, even though they are not realizable in every implementation.
>
> Thanks
>
> Mike Heard
>
>
>
>
>
>>
>>
>
>