Re: [storm] Send vs. Immediate

[Elena Gurevich <[e-mail]>]

Hello Tom, thanks for a prompt response, my question was little bit different.

I failed to found differences in behavior between RDMA Write followed by Immediate Data and RDMA Write followed by Send.
Both consume untagged buffer of queue #0 - so at data sink this buffer should be preposted in both cases.

According to RFCs except DDP message format both operation has to be processed similarly as at data Source as at data Sink.

According to ordering and completion rules, specified in RFC 5040 and  7306:

First          |Second       | Placement         | Placement       | Ordering
Operation| Operation | Guarantee at     | Guarantee at   | Guarantee at
                 |                  | Remote Peer     | Local Peer        | Remote Peer
-------------+---------------+----------------------+--------------------+----------------
RDMA      | Send          | No placement   | Not applicable | Not applicable
Write       |                   | guarantee. If     |                         |
                |                   | guarantee is      |                         |
                |                   | necessary, see   |                         |
                |                   | footnote 1.         |                         |
-------------+---------------+----------------------+--------------------+------------------
RDMA     | Immediate| No Placement   | Not                   | Immediate Data
Write      | Data          | Guarantee         | Applicable        | is Completed
               |                   |                           |                          | after RDMA
               |                   |                           |                          | Write is Placed
               |                   |                           |                          | and Delivered

But even if the cell  "Ordering Guarantee at Remote Peer" is different for RDMA Write followed by Immediate Data and RDMA Write followed by Send.
At the end behavior has to be the same, because of specification of RFC5041 Section 5.4. ( see details in my original e-mail)

So why we need Immediate Data operation and cannot just reuse Send operation ?

Best regards,
Lena

-----Original Message-----
From: storm [mailto:[e-mail]] On Behalf Of Tom Talpey
Sent: Sunday, September 06, 2015 4:34 PM
To: [e-mail]
Subject: Re: [storm] Send vs. Immediate

On 9/6/2015 5:58 AM, Elena Gurevich wrote:
> Hello dear authors,
>
> I am new in iWARP and need your help to clear my  misunderstanding of RFCs.
>
> RFC5041 Section 5.4 stands that:
>
> "At the Data Sink, DDP MUST Deliver a DDP Message if and only if all
> of the following are true:
>
> * the last DDP Segment of the DDP Message had its Last flag set,
>
> * all of the DDP Segments of the DDP Message have been Placed,
>
> * all preceding DDP Messages have been Placed, and
>
> * each preceding DDP Message has been Delivered to the ULP."
>
> Let's assume that data sink receives sequence of "rdma_write" and send"
> messages.
>
> According to RFC even if miss ordering happens "send" can be delivered
> to RDMAP only after last segment of rdma_write arrives and is placed.
>
> So RDMAP layer completes "send" only after full rdma_write assembly.

These statements are correct, the Send is delivered only after Placing all the previous RDMA Write segments. This is a key behavior upon which most upper layers depend.

>
> If so why we need to generate new "Immediate request type" and cannot
> reuse 8 bytes length "send" ?

I don't understand the question. An Immediate is another behavior, and which is used by upper layers rather differently. Also, what
"8 bytes" are you referring to?

Perhaps you mean, why doesn't the upper layer use an RDMA Write with Immediate? Some upper layers do, I believe Lustre does, for example, but the original iWARP RDMAP (RFC5040) protocol did not support that operation. The RFC7306 extension, published last year, adds it.

Other upper layers, for example, iSER, NFS/RDMA, SMB Direct, etc, need a larger, separate message to complete an operation, and so use a full Send. These protocols use iWARP, without Immediates.

_______________________________________________
storm mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/storm
-------------------------------------------------------------------------------------------------------------------------------------------------
This email and any files transmitted and/or attachments with it are confidential and proprietary information of
Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager. This message contains confidential
information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named
addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately
by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not
the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on
the contents of this information is strictly prohibited.
------------------------------------------------------------------------------------------------------------------------------------------------