Re: [storm] DDP messages ordering

[Elena Gurevich <[e-mail]>]

Hi Tom,  that you for your responses you are the only person that reacts to my question :(.

If we back to my question:
according to RFC 5040
-------------------------------------------------------------
5.2 RDMA Read Operation
The RDMA Read operation MUST consist of a single RDMA Read Request Message and a single RDMA Read Response Message.
-------------------------------------------------------------

And "the last flag" you mentioned relates to DDP layer to sign last segment of the same DDP message ( RDMA message is mapped one-by-one to DDP message).

Personally I think that for transmitter it will be more simple to allow interleaving of DDP segments of different RDMA messages related to different transmit queues: SQ and IRRQ, but for receiver this behavior complicates the processing.

So if we allow this behavior RFC 5041 must more accurate define this rule, or I still miss something.

Best regards,
Elena



-----Original Message-----
From: Tom Talpey [mailto:[e-mail]]
Sent: Monday, September 28, 2015 7:17 PM
To: Elena Gurevich; [e-mail]
Subject: Re: [storm] DDP messages ordering

I was traveling last week so this is a delayed response. I want to say that I am not answering in the role of STORM WG co-chair.
This is simply my interpretation of the specifications.

The behavior you observed is allowed by the RDMA Read processing rules. In RFC5040 section 4.5, the RDMA Read Response is defined to be simply a DDP message. The key is that the sender (i.e. the RDMA Read responder) can and does respond to a single RDMA Read with multiple Responses at the RDMAP layer. The "Last" flag, defined in
RFC5041 section 4.1, is used to indicate the final Response in such a sequence. The RDMA Read Requestor simply processes the responses until the Last flag is received, at which point it completes the RDMA Read operation back to the application.

Because each RDMA Read Response is a separate DDP message, it is perfectly legal for the Responder to interleave other DDP messages while processing the RDMA Read. There is no requirement that DDP pause the stream until the Last flag is sent.

I'd actually expect most RNICs to do this for large RDMA Reads, for example if they are larger than a single Ethernet frame, or any other size if desired. Because the responding RNIC needs to perform multiple PCI Reads, and because RNICs don't usually buffer tagged data, it will be very convenient for them to issue interim RDMA Read Responses.

Does this help?

Tom.

On 9/17/2015 7:53 AM, Elena Gurevich wrote:
> Hello Tom,
>
> Please reference exact place that explicitly or implicitly allows/forbids the behavior in question.
>
> Best regards,
> Lena
>
> -----Original Message-----
> From: storm [mailto:[e-mail]] On Behalf Of Elena
> Gurevich
> Sent: Thursday, September 10, 2015 3:03 PM
> To: Tom Talpey; [e-mail]
> Subject: Re: [storm] DDP messages ordering
>
> Hello Tom,
>
> Section 5.5 just specifies ordering rules of RDMAP requester and responder.
> According to such rules RDMAP generates RDMA messages and passes them to DDP layer one by one for future processing.
> RDMAP's granularity and visibility is a particular RDMA message.
> DDP layer gets RDMA messages from RDMAP layer maps them one-by-one to DDP messages performs segmentation and passed them to transport layer.
> So DDP segments interleaving/rules cannot be part of RDMAP , it is clearly DDP layer behavior.
> And according to RFC 5041 Section 5.3 and already mentioned by you "first rule" cannot interleave DDP segments of different operations.
>
> Best regards,
> Lena
>
> -----Original Message-----
> From: Tom Talpey [mailto:[e-mail]]
> Sent: Wednesday, September 09, 2015 6:54 PM
> To: Elena Gurevich; [e-mail]
> Subject: Re: [storm] DDP messages ordering
>
> On 9/9/2015 9:07 AM, Elena Gurevich wrote:
>> Hello,
>>
>> During my testing of some iWARP adaptor I discovered that RNIC interleaves DDP segments of Send and Read Response messages.
>> According to your previous response this behavior is forbidden, is it ?
>
> Not necessarily, because an RDMA Read response is special. It's injected into the DDP stream within the RNIC by RDMAP, and therefore it follows the RDMAP rules. See section 5.5 of RFC5040.
>
> Tom.
>
>>
>> Best regards,
>> Lena
>>
>> -----Original Message-----
>> From: storm [mailto:[e-mail]] On Behalf Of Tom Talpey
>> Sent: Tuesday, September 08, 2015 5:15 PM
>> To: [e-mail]
>> Subject: Re: [storm] DDP messages ordering
>>
>> On 9/8/2015 6:02 AM, Elena Gurevich wrote:
>>> Hello,
>>>
>>> RFC 5041 section 5.3 stands that
>>>
>>> ----------------------------------
>>>
>>> 5.3 Ordering Among DDP Messages
>>>
>>> Messages passed through the DDP MUST conform to the ordering rules
>>>
>>> defined in this section.
>>>
>>> At the Data Source, DDP:
>>>
>>> * MUST transmit DDP Messages in the order they were submitted to
>>>
>>> the DDP layer,
>>>
>>> * SHOULD transmit DDP Segments within a DDP Message in increasing
>>>
>>> MO order for Untagged DDP Messages, and in increasing TO order
>>>
>>> for Tagged DDP Messages.
>>>
>>> ----------------------------------
>>>
>>> Does this mean that transmitter MUST not interleave DDM segments
>>> related to consequent DDP messages ?
>>
>> It depends on what you mean by "the transmitter". It's certainly possible that network delivery and TCP retransmission can reorder segments. But the first rule requires that DDP not interleave two separate operations when passing them to the TCP transport.
>>
>> Tom.
-------------------------------------------------------------------------------------------------------------------------------------------------
This email and any files transmitted and/or attachments with it are confidential and proprietary information of
Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager. This message contains confidential
information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named
addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately
by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not
the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on
the contents of this information is strictly prohibited.
------------------------------------------------------------------------------------------------------------------------------------------------