RE: draft-lubashev-quic-partial-reliability

["Lubashev, Igor" <[e-mail]>]

  *   This proposal requires out of order delivery be supported by QUIC streams

It does not.  All delivery, whether of data or gaps-in-data, is strictly in-order.  I can see a receiver API look something like:

class Stream {
      …
size_t receive(void *buf, size_t buf_size) throw QuickException;
size_t receive(size_t &expired_bytes, void *buf, size_t buf_size) throw QuickException;
…
};

The version of receive with expired_bytes will return how many bytes have expired followed by the data received in buf.  Clients that do not expect missing messages will just call the version of receive() w/o expired_bytes and will get an exception if any data is expired.   (APIs that are event callback based will just declare a new stream event type “StreamExpiredData”.)




  *   Also, the receiver needs to be able to determine what the message boundaries are if there are gaps in the stream.

It will not.  Since sender keeps track of where message boundaries are and controls minimum retransmittable offset, the receiver is guaranteed that any data after a gap corresponds to the start of a new message.
(MAY in Section 6 “Receiver Interface and Behavior” is not a MUST to allow for an optimization of keeping data that has arrived after MIN_STREAM_DATA due to reordering, if the application has not yet been notified about the data gap.)



  *   Can a STREAM frame contain the end of one message and the beginning of another in a single frame?

Yes, it can, since nothing is changed in the behavior of STREAM frames.  The assumption is that the application can figure out itself how long each fully-delivered message is.



  *   In section 4, you discuss the separation of Unsent bytes from Sent data, but I'm still confused on the motivation there, since separating them seems to make flow control more complex.

I guess I failed to be clear here, as Unsent Bytes was designed to NOT complicate flow control accounting (in fact, it has no effect on flow control accounting at all).

Imagine an HD video conferencing app has sent 10MB of data to a stream (maybe in multiple messages).  Now, say, it was only able to transmit 1MB out of those 10MB when a new key frame is received.

If it has to use normal stream and connection flow control credits to advance this stream by the additional 9MB, it may might not be able to do so w/o violating MAX_STREAM_DATA or MAX_DATA.  The sender would need to “skip” those 9MB over several RTTs, if the receiver is only willing to commit, say, 2MB of buffers to this stream/connection (it is not aware that the sender has 9MB of octets to skip).  Moreover, this would be blocking all other streams from making progress by closing down the connection window.

Unsent Bytes was designed to solve this program, as it *does not count toward stream or connection flow control* (so it does not violate them and does not reduce connection window), but it is a hint to open the stream and connection windows to accommodate skipping byte octets (since they do not require any buffer space).  The stream would be able to resume in at most 1 rtt, and it would not block other streams meanwhile.

However, I now see a problem – while opening the stream window by 9MB+ is ok (since any octets at offset lower than Min Stream Offset are to be discarded), there is no requirement that the extra connection window is used for that stream only.  A malicious client can render connection window meaningless by sending MIN_STREAM_DATA with a huge Unsent Bytes and then never sending on that stream again.  Oops!  I will fix this!




  *   I believe there is another challenge, which is indicating to the receiver when they should stop expecting to receive data.  Conveniently, this is exactly what MIN_STREAM_DATA does, but I think it's worth noting in the introduction.

I meant exactly this by “notifying the transport and the peer when data previously enqueued for transmission no longer needs to be transmitted“ (that’s what “and the peer” was for) but said it very awkwardly.  Will fix by focusing only on the peer (interaction between the app and local transport does not concern wire protocol).



  *   Igor



From: Ian Swett [mailto:[e-mail]]
Sent: Tuesday, December 19, 2017 3:20 PM
To: Lucas Pardue <[e-mail]>
Cc: Lubashev, Igor <[e-mail]>; [e-mail]
Subject: Re: draft-lubashev-quic-partial-reliability

Thanks for writing this up Igor, it's an interesting idea, and I like how it allows any stream besides 0 to be partially reliable.  I'm a bit concerned(or possibly confused) about the effort to change how flow control works, see comments below.  I believe this proposal implicitly assumes that it's easier and more efficient to use streams for ordering and make the application deal with interpreting out of order data than the current model of making applications reorder a large number of small streams?  That may or may not be true in a given application, but I can imagine cases when it is true.

At the beginning you say:
"The key to partial reliablity is notifying the transport and the peer
 when data previously enqueued for transmission no longer needs to be
 transmitted."

I believe there is another challenge, which is indicating to the receiver when they should stop expecting to receive data.  Conveniently, this is exactly what MIN_STREAM_DATA does, but I think it's worth noting in the introduction.

In terms of the downsides of using streams as is, you say:
"Hence, a message-per-stream
 approach requires each message to contain an extra header portion to
 associate the message with a logical application stream.  In case of
 short messages, this approach introduces a significant overhead due
 to STREAM frames and message headers.  It also places the burden on
 the application to reorder data arriving on multiple QUIC streams."

In practice, I believe a lot of applications using partial reliability today have such headers, since they're commonly designed with UDP in mind.

In section 4, you discuss the separation of Unsent bytes from Sent data, but I'm still confused on the motivation there, since separating them seems to make flow control more complex.  Currently the connection flow control used is the sum of the largest byte offsets of all streams on a connection, and now that would be dramatically different, correct?  Can you add more background on why these are separated and how they are applied to various limits?  If I read correctly, MIN_STREAM_DATA frame is effectively able to increase a peer's flow control limits by declaring previously used credits up for grabs?

This proposal requires out of order delivery be supported by QUIC streams, which is worth explicitly noting, maybe in section 5, since it's currently not required.  Also, the receiver needs to be able to determine what the message boundaries are if there are gaps in the stream.  How easy that is would depend upon what is being sent on the stream.  It might deserve some discussion about how one might do this and what restrictions should be imposed on how the sender bundles data, if that ends up being necessary?  ie: Can a STREAM frame contain the end of one message and the beginning of another in a single frame?


On Tue, Dec 19, 2017 at 10:46 AM, Lucas Pardue <[e-mail]<mailto:[e-mail]>> wrote:
Igor wrote:

The bookkeeping is simple enough and the runtime resources required is just one uint64_t per steam, so I do not see a problem for constrained devices.
Ultimately, partial reliability is a feature that an application either needs or does not. If an application needs it, it most likely requires it. In any case, an application can do its own negotiation if it desires to do so. For example, it can use stream 1 for that.
Now I understand this is connection-wide, I generally agree. However, for something like conventional HTTP/QUIC, the reliable guarantee assurances are required by the mapping. If partial reliability is default enabled transport feature, applications like HTTP/QUIC must restrict or disable it – I’m not sure if the editors/drafts have broached that subject yet (I’ve certainly had some feedback on my draft about such restrictions).

I did not fully understand the last question. Are you asking whether it is possible for the connection to negotiate just a single stream (in each direction) that would be partially reliable and, therefore, not send a stream id with MIN_STREAM_DATA? I could actually think of a version of MIN_STREAM_DATA that has an implied stream id -- the stream id of a prior/following STREAM frame in the same packet. That would be an optimization that I am happy to add, if there is enough support for it.

Apologies, I hadn’t drunk any coffee when first replying and got myself confused. All QUIC frames of this nature should normally include a Stream ID, which identifies the stream it is sent on. I think your suggestion relies on serial ordering/processing (even inside a packet), which might be a hard sell.

Separately, in terms of asymmetry, I was thinking that you might always want (as a policy) clients to transmit with full reliability but provide data to them with partial reliability. I think this is again an application mapping concern.

Regards,
Lucas