Re: flow control and DATAGRAM

[Jana Iyengar <[e-mail]>]

Hey Tommy,


> I'm not suggesting any changes to when the packet gets
> ack'ed—specifically, I was responding to Martin's hypothetical of having an
> ACK to a DATAGRAM frame meaning that the application had processed the
> frame. My impression is that the ACK to a packet containing a DATAGRAM
> frame means:
> - The packet made it across the network to the QUIC endpoint on the other
> side
>

... and was processed by the QUIC receiver (not necessarily by the
application). This is the same semantic as the ack of a regular frame.


> - The QUIC implementation will deliver the DATAGRAM frame to application
> (it won't drop it locally)
>

In which case you will need flow control. If you agree, then we're on the
same page so far.


> Having a separate outstanding data limit for the DATAGRAM "stream" is an
> interesting solution to the space. It would then have the nice property of
> not looking like traditional flow control. It could even be measured in
> number of frames, rather than bytes (depending on what the limiting factors
> are).
>

In terms of flow control, I don't think DATAGRAM flow control is any
different than sending stream data on any stream and then canceling it --
there are no retransmissions, but the sender still accounts for it in flow
control. My thought here was basically to have exactly the same flow
control as stream-level flow control, and allow for DATAGRAM bytes to fall
within connection-level flow control as normal.

The simplest scheme would be to reserve a stream ID for DATAGRAM data --
this could be 0 or 2^62-1 -- and then MAX_STREAM_DATA uses this stream ID
for flow control of DATAGRAM octets. Alternatively, define a new frame
called MAX_DATAGRAM_DATA that is carries the largest allowed offset that
can be sent as a DATAGRAM, and introduce an offset field in the DATAGRAM
frame.

- jana


> Thanks,
> Tommy
>
> On Oct 29, 2018, at 12:37 PM, Jana Iyengar <[e-mail]> wrote:
>
> Tommy,
>
> Changing the semantics of an acknowledgment to include delivery up to the
> application is a fundamental change to the QUIC machinery, and it doesn't
> work. First, an ACK frame acknowledges packets, and you can't have
> different semantics of an acknowledgment for different frames that are
> carried in the same packet. Second, it interferes with RTT measurement, and
> it conflates flow control with congestion control, which gets messy. (This
> conflation is an interesting problem to consider theoretically, but not one
> for us at this time IMO.)
>
> I am wondering if applying a stream-level flow control for DATAGRAMs makes
> sense instead. Meaning that you treat DATAGRAMs as a separate stream for
> flow control purposes. You might benefit from having an offset in the
> DATAGRAM frame for this purpose.
>
> - jana
>
> On Mon, Oct 29, 2018 at 8:21 AM Tommy Pauly <[e-mail]> wrote:
>
>> Hi Martin, Ian,
>>
>> Yes, very good points!
>>
>> My tendency would be to prefer what Ian's implementation does of passing
>> these DATAGRAM frames up immediately to the application. I don't think that
>> the acknowledgment needs to indicate that the frame was processed by the
>> application, but merely that it has been delivered to the application (that
>> is, the application doesn't get to do anything with the frame that can
>> influence the acknowledgment).
>>
>> The current draft indicates that the content of the DATAGRAM frames
>> contributes to the limit used for MAX_DATA, and that if that amount is
>> reached, the frames are blocked along with STREAM data. I think this works
>> fine for the sender, while the receiver gets into the discussion you
>> present. On the sender side, reaching MAX_DATA could mean dropping the
>> DATAGRAM frames when unable to send more (and sending BLOCKED instead).
>> Since the frames are unreliable, they can be dropped in this situation
>> without violating the API contract.
>>
>> On the receiver side, I agree that queuing the DATAGRAM frames to let the
>> application drive flow control in the way it does for STREAM frames adds
>> complexity and diminishes the utility of the frame and ACKs.. However, I
>> can imagine taking a fairly simplistic approach in which the data limit is
>> automatically increased upon reception of the frame (and the frame is
>> immediately passed to the application). This allows the initial_max_data to
>> put a cap on the amount of data in a given flight of DATAGRAMS, and allow
>> the size of a flight of DATAGRAM frames to be limited by the amount of room
>> left over from STREAM data that may be consuming the connection-wide flow
>> control.
>>
>> Perhaps this approach needs a clearer name other than "flow control",
>> since it has a somewhat different meaning in effect.
>>
>> As for ACKs, if we never discard on the receiver side, the ACK is pretty
>> useful for detecting if there was network-based packet loss.
>>
>> Thanks,
>> Tommy
>>
>> On Oct 29, 2018, at 5:32 AM, Ian Swett <[e-mail]> wrote:
>>
>> Good catch Martin, I missed that in the draft as well, and I also think
>> it's impossibly with the proposed design.
>>
>> And yes, I think Martin's proposed solution is likely the only practical
>> one.  In my implementation, the frame is passed up to the application
>> immediately, so technically QUIC processed it, and it's the application's
>> job to decide what to do with it.
>>
>> On Mon, Oct 29, 2018 at 1:16 AM Martin Thomson <[e-mail]>
>> wrote:
>>
>>> Hi Tommy,
>>>
>>> Your slides - <
>>> https://github.com/quicwg/wg-materials/blob/master/ietf103/IETF103-QUIC-Datagram.pdf
>>> >
>>> - say that DATAGRAM frames respect connection-level flow control.  I
>>> missed that in the draft, and I don't know how they can do that in the
>>> face of packet loss, especially when you don't necessarily retransmit
>>> lost DATAGRAM frames.
>>>
>>> For that to work, you would need a bunch more machinery to make the
>>> connection-level flow control sync between endpoints in the case that
>>> packets are lost.  A disagreement about how much flow control is used
>>> causes things to break down badly.  Ian and I discussed this point at
>>> the last meeting and quickly agreed that while it might be nice to
>>> have flow control for this stuff, the increase in complexity is
>>> considerable and (at the time) we thought it wouldn't be worth it.
>>>
>>> The problem that introduces is that you could end up having too many
>>> DATAGRAM frames arrive.  The receiver has to drop something at the
>>> point that it can't handle them.  And we say that when you acknowledge
>>> something, you processed it.  That's tricky.
>>>
>>> It might be easier to say that a QUIC acknowledgment for a DATAGRAM
>>> frame doesn't mean that it was received and processed by an
>>> application.  An endpoint might discard these frames before passing
>>> them on to applications if it doesn't have space.  In other words,
>>> acknowledgment of DATAGRAM means that QUIC got it, not that the
>>> application got it.  Sadly, that means that the QUIC acknowledgment
>>> machinery doesn't help the application that uses DATAGRAM all that
>>> much.  Also, the lower bound on reliability is 0, which isn't the best
>>> thing ever.
>>>
>>> Hard choices, I know.  I don't have a good design for maintaining
>>> connection-level flow control (or any back pressure mechanism with
>>> equivalent properties) that doesn't add both complexity and overhead.
>>>
>>> Cheers,
>>> Martin
>>>
>>>
>>
>