Re: [storm] Send vs. Immediate

[Tom Talpey <[e-mail]>]

On 9/8/2015 1:55 AM, Elena Gurevich wrote:
> Thanks you again,
>
> If so, the statement "Not applicable" in cell "Ordering at Remote Peer" for RDMA Write followed by Send operation in Appendix B of RFC5040 is inaccurate
> And should be changed as "Send is completed after RDMA Write placed and delivered".

I disagree, it is accurate because the table is defining the behavior
as visible from the *local* peer, i.e. the initiator. You will note
that the only messages with anything but "not applicable" are the
RDMA Read responses.

Note the last words of the first sentence:

    Appendix B.  Ordering and Completion Table

    The following table summarizes the ordering relationships that are
    defined in Section 5.5, "Ordering and Completions", from the
    standpoint of the local peer issuing the two Operations.

The state of the *remote* peer depends upon it taking completions,
which is not something the local peer can determine without an
upper layer exchange.

Tom.

>
> Best regards,
> Elena
>
> -----Original Message-----
> From: Tom Talpey [mailto:[e-mail]]
> Sent: Monday, September 07, 2015 6:25 PM
> To: Elena Gurevich; [e-mail]
> Subject: Re: [storm] Send vs. Immediate
>
>   > So why we need Immediate Data operation and cannot just reuse Send operation ?
>
> Using either one is a choice by the upper layer (RDMA consumer), and it's absolutely possible to just reuse a Send. As mentioned, many upper layers simply use Send. Others, for various reasons, use Immediate data. DDP, and the extended RDMAP (RFC7306) support either one.
>
>
> On 9/7/2015 5:48 AM, Elena Gurevich wrote:
>> Hello Tom, thanks for a prompt response, my question was little bit different.
>>
>> I failed to found differences in behavior between RDMA Write followed by Immediate Data and RDMA Write followed by Send.
>> Both consume untagged buffer of queue #0 - so at data sink this buffer should be preposted in both cases.
>>
>> According to RFCs except DDP message format both operation has to be processed similarly as at data Source as at data Sink.
>>
>> According to ordering and completion rules, specified in RFC 5040 and  7306:
>>
>> First          |Second       | Placement         | Placement       | Ordering
>> Operation| Operation | Guarantee at     | Guarantee at   | Guarantee at
>>                    |                  | Remote Peer     | Local Peer        | Remote Peer
>> -------------+---------------+----------------------+--------------------+----------------
>> RDMA      | Send          | No placement   | Not applicable | Not applicable
>> Write       |                   | guarantee. If     |                         |
>>                   |                   | guarantee is      |                         |
>>                   |                   | necessary, see   |                         |
>>                   |                   | footnote 1.         |                         |
>> -------------+---------------+----------------------+--------------------+------------------
>> RDMA     | Immediate| No Placement   | Not                   | Immediate Data
>> Write      | Data          | Guarantee         | Applicable        | is Completed
>>                  |                   |                           |                          | after RDMA
>>                  |                   |                           |                          | Write is Placed
>>                  |                   |                           |                          | and Delivered
>>
>> But even if the cell  "Ordering Guarantee at Remote Peer" is different for RDMA Write followed by Immediate Data and RDMA Write followed by Send.
>> At the end behavior has to be the same, because of specification of
>> RFC5041 Section 5.4. ( see details in my original e-mail)
>>
>> So why we need Immediate Data operation and cannot just reuse Send operation ?
>>
>> Best regards,
>> Lena
>>
>> -----Original Message-----
>> From: storm [mailto:[e-mail]] On Behalf Of Tom Talpey
>> Sent: Sunday, September 06, 2015 4:34 PM
>> To: [e-mail]
>> Subject: Re: [storm] Send vs. Immediate
>>
>> On 9/6/2015 5:58 AM, Elena Gurevich wrote:
>>> Hello dear authors,
>>>
>>> I am new in iWARP and need your help to clear my  misunderstanding of RFCs.
>>>
>>> RFC5041 Section 5.4 stands that:
>>>
>>> "At the Data Sink, DDP MUST Deliver a DDP Message if and only if all
>>> of the following are true:
>>>
>>> * the last DDP Segment of the DDP Message had its Last flag set,
>>>
>>> * all of the DDP Segments of the DDP Message have been Placed,
>>>
>>> * all preceding DDP Messages have been Placed, and
>>>
>>> * each preceding DDP Message has been Delivered to the ULP."
>>>
>>> Let's assume that data sink receives sequence of "rdma_write" and send"
>>> messages.
>>>
>>> According to RFC even if miss ordering happens "send" can be
>>> delivered to RDMAP only after last segment of rdma_write arrives and is placed.
>>>
>>> So RDMAP layer completes "send" only after full rdma_write assembly.
>>
>> These statements are correct, the Send is delivered only after Placing all the previous RDMA Write segments. This is a key behavior upon which most upper layers depend.
>>
>>>
>>> If so why we need to generate new "Immediate request type" and cannot
>>> reuse 8 bytes length "send" ?
>>
>> I don't understand the question. An Immediate is another behavior, and
>> which is used by upper layers rather differently. Also, what
>> "8 bytes" are you referring to?
>>
>> Perhaps you mean, why doesn't the upper layer use an RDMA Write with Immediate? Some upper layers do, I believe Lustre does, for example, but the original iWARP RDMAP (RFC5040) protocol did not support that operation. The RFC7306 extension, published last year, adds it.
>>
>> Other upper layers, for example, iSER, NFS/RDMA, SMB Direct, etc, need a larger, separate message to complete an operation, and so use a full Send. These protocols use iWARP, without Immediates.
>>
> -------------------------------------------------------------------------------------------------------------------------------------------------
> This email and any files transmitted and/or attachments with it are confidential and proprietary information of
> Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed.
> If you have received this email in error please notify the system manager. This message contains confidential
> information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named
> addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately
> by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not
> the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on
> the contents of this information is strictly prohibited.
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
>
>