Re: flow control and DATAGRAM

David Schinazi <> Wed, 31 October 2018 20:36 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 812B7130DC8 for <>; Wed, 31 Oct 2018 13:36:33 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -0.009
X-Spam-Status: No, score=-0.009 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, HTTPS_HTTP_MISMATCH=1.989, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 7Adt2JGiZpp4 for <>; Wed, 31 Oct 2018 13:36:29 -0700 (PDT)
Received: from ( [IPv6:2607:f8b0:4864:20::62b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 42A31130D7A for <>; Wed, 31 Oct 2018 13:36:29 -0700 (PDT)
Received: by with SMTP id g59-v6so3824984plb.10 for <>; Wed, 31 Oct 2018 13:36:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=0a8yaUoFu4x/q1y1fx1TKUVJe/Kn0ck+5talidqV42Q=; b=sh+td4JuonOs+6iovUNB5M44PdYqCyeYRiIUSgXNc3IhudRTaPJ+vQNIZyzdlDB5kw 71713OoM5xXJg2TmsOb4ubaksEalMqX8FLQPvBIdC7CtN1NQJN32RDs9uy0DC46tKtHw e2CJ5EspIT1OmoXBzgXLIpyCuesGX7f4PtQjZmYYTb/V5N4BHYKOlLzxoro24fSQAynO 8FpnefAS0aOpi3qlCiPDZX7Iags7jgTgaH78x/AAjA6lV/K3w7zzuPcCBLx1+D6Myu72 XlVaGyCWrlIQAIsLWXWxrt8loRkaVL3gQwHmK6IivT4PM4TmekDdEy1ldU3rT9PZSFfj 56UQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=0a8yaUoFu4x/q1y1fx1TKUVJe/Kn0ck+5talidqV42Q=; b=RgSSxsGWnle8gDUUt0jqq9qBGQFd8/VFDId7+ivwCUCvhQ1RLNnyw8Elmg8HwdRy0v UAHp5lIWvHDrsKRNBA9eN/EAaJeIlZ5BP2GMOhih6tEfc2tt+yjFnzBd+MIeZzI/oQze AqWaVO7HZ3W2aJjxIq8gIVhZcnj8Z0chP/kpChtqZC/DNQCIVW2xpciO1Tbu3dX2ASR0 57s9zR+DFVlB23N2voOQsuM48prgq6FV/SqlxDgUt3pWVSvV+7hy/rO6Wj9iQK18eJvn P6Cnr/f3BqevUP5ns1vE4+jccD5hTOKwBax2x6UUpXVhZ4elb7JoXS7xERy25J13377z cvjQ==
X-Gm-Message-State: AGRZ1gJcGFfcSM2b3iXvKxVePz8elbz1K5TJSBVy1YouTfmuIBmEgc4D 2ujgzBzlHiXHEz0B3dM3AFLUjua0/zXCEI9SkjQ=
X-Google-Smtp-Source: AJdET5dRfxRMsixKdHYpD9dAxSMnzoqg4R3UQ860sG4nidC4wPHNSjxX8Ba4qm3Le9cZTaAgH4dsXvZT645zWgGJB40=
X-Received: by 2002:a17:902:6a8b:: with SMTP id n11-v6mr4789976plk.16.1541018188735; Wed, 31 Oct 2018 13:36:28 -0700 (PDT)
MIME-Version: 1.0
References: <> <> <> <> <> <> <> <> <>
In-Reply-To: <>
From: David Schinazi <>
Date: Wed, 31 Oct 2018 13:36:17 -0700
Message-ID: <>
Subject: Re: flow control and DATAGRAM
To: Ian Swett <>
Cc:, QUIC <>,,, Tommy Pauly <>
Content-Type: multipart/alternative; boundary="000000000000bb684905798c417e"
Archived-At: <>
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 31 Oct 2018 20:36:34 -0000

I agree.

Another use case is VPN, where one could want to have the control traffic
(e.g. what IKEv2 does today) be sent reliably with flow control and have
the data path (e.g. what ESP does today) be sent unreliably without flow
control. As with ESP today, flow control is handled by upper layer
protocols (e.g. TCP) inside the encryption/encapsulation.

I think it might be best to remove flow control from this proposal
entirely, and if later someone has a use-case for flow-controlled datagrams
we can define a new different extension then.


On Wed, Oct 31, 2018 at 9:01 AM Ian Swett <ianswett=> wrote:

> If you add flow control, there's little benefit over a unidirectional
> stream, so adding flow control fundamentally breaks this design.
> My goal is to use this for cases when I want to use a single QUIC
> connection to replace DTLS or RTP + something reliable.  WebRTC is the
> obvious use case, but not the only one.  DTLS and RTP don't have flow
> control, so adding it just make the application mapping a lot more complex.
> On Tue, Oct 30, 2018 at 12:23 AM Jana Iyengar <> wrote:
>> Igor,
>> I had thought that this was similar to canceling streams, but you're
>> right. Canceling a stream gets rid of stream flow control state entirely,
>> where in this case, the receiver cannot discard flow control state.
>> I don't have any great ideas right now, but I fear that not having flow
>> control simply punts the problem down the road. FWIW, this may be quite
>> fine for this extension, and we can surely evolve it to have flow control
>> when we have a way to do it.
>> - jana
>> On Mon, Oct 29, 2018 at 8:54 PM Lubashev, Igor <>
>> wrote:
>>> Jana,
>>> > The simplest scheme would be to reserve a stream ID for DATAGRAM data
>>> -- [...]
>>> > -- and then MAX_STREAM_DATA uses this
>>> > stream ID for flow control of DATAGRAM octets.
>>> There are many dragons that way. Flow control indicates commitment to
>>> buffer data. When sender does not retransmit, when does receiver give up
>>> its commitment to wait for (and buffer) earlier data so it can advance
>>> MAX_STREAM_DATA to allow the sender to send more data? Can the signal from
>>> sender tolerate loss/reordering vs STREAM frames (assuming the datagram can
>>> be split for sending in multiple QUIC packets)? These dragons can be
>>> conquered (see my partially reliable streams draft, v2 or v3), but this is
>>> not simple. :(
>>> - Igor
>>> -----Original Message-----
>>> *From:* Jana Iyengar []
>>> *Received:* Monday, 29 Oct 2018, 10:24PM
>>> *To:* []
>>> *CC:* Ian Swett []; QUIC WG []; Martin
>>> Thomson [ <>]
>>> *Subject:* Re: flow control and DATAGRAM
>>> Hey Tommy,
>>>> I'm not suggesting any changes to when the packet gets
>>>> ack'ed—specifically, I was responding to Martin's hypothetical of having an
>>>> ACK to a DATAGRAM frame meaning that the application had processed the
>>>> frame. My impression is that the ACK to a packet containing a DATAGRAM
>>>> frame means:
>>>> - The packet made it across the network to the QUIC endpoint on the
>>>> other side
>>> ... and was processed by the QUIC receiver (not necessarily by the
>>> application). This is the same semantic as the ack of a regular frame.
>>>> - The QUIC implementation will deliver the DATAGRAM frame to
>>>> application (it won't drop it locally)
>>> In which case you will need flow control. If you agree, then we're on
>>> the same page so far.
>>>> Having a separate outstanding data limit for the DATAGRAM "stream" is
>>>> an interesting solution to the space. It would then have the nice property
>>>> of not looking like traditional flow control. It could even be measured in
>>>> number of frames, rather than bytes (depending on what the limiting factors
>>>> are).
>>> In terms of flow control, I don't think DATAGRAM flow control is any
>>> different than sending stream data on any stream and then canceling it --
>>> there are no retransmissions, but the sender still accounts for it in flow
>>> control. My thought here was basically to have exactly the same flow
>>> control as stream-level flow control, and allow for DATAGRAM bytes to fall
>>> within connection-level flow control as normal.
>>> The simplest scheme would be to reserve a stream ID for DATAGRAM data --
>>> this could be 0 or 2^62-1 -- and then MAX_STREAM_DATA uses this stream ID
>>> for flow control of DATAGRAM octets. Alternatively, define a new frame
>>> called MAX_DATAGRAM_DATA that is carries the largest allowed offset that
>>> can be sent as a DATAGRAM, and introduce an offset field in the DATAGRAM
>>> frame.
>>> - jana
>>>> Thanks,
>>>> Tommy
>>>> On Oct 29, 2018, at 12:37 PM, Jana Iyengar <> wrote:
>>>> Tommy,
>>>> Changing the semantics of an acknowledgment to include delivery up to
>>>> the application is a fundamental change to the QUIC machinery, and it
>>>> doesn't work. First, an ACK frame acknowledges packets, and you can't have
>>>> different semantics of an acknowledgment for different frames that are
>>>> carried in the same packet. Second, it interferes with RTT measurement, and
>>>> it conflates flow control with congestion control, which gets messy. (This
>>>> conflation is an interesting problem to consider theoretically, but not one
>>>> for us at this time IMO.)
>>>> I am wondering if applying a stream-level flow control for DATAGRAMs
>>>> makes sense instead. Meaning that you treat DATAGRAMs as a separate stream
>>>> for flow control purposes. You might benefit from having an offset in the
>>>> DATAGRAM frame for this purpose.
>>>> - jana
>>>> On Mon, Oct 29, 2018 at 8:21 AM Tommy Pauly <> wrote:
>>>>> Hi Martin, Ian,
>>>>> Yes, very good points!
>>>>> My tendency would be to prefer what Ian's implementation does of
>>>>> passing these DATAGRAM frames up immediately to the application. I don't
>>>>> think that the acknowledgment needs to indicate that the frame was
>>>>> processed by the application, but merely that it has been delivered to the
>>>>> application (that is, the application doesn't get to do anything with the
>>>>> frame that can influence the acknowledgment).
>>>>> The current draft indicates that the content of the DATAGRAM frames
>>>>> contributes to the limit used for MAX_DATA, and that if that amount is
>>>>> reached, the frames are blocked along with STREAM data. I think this works
>>>>> fine for the sender, while the receiver gets into the discussion you
>>>>> present. On the sender side, reaching MAX_DATA could mean dropping the
>>>>> DATAGRAM frames when unable to send more (and sending BLOCKED instead).
>>>>> Since the frames are unreliable, they can be dropped in this situation
>>>>> without violating the API contract.
>>>>> On the receiver side, I agree that queuing the DATAGRAM frames to let
>>>>> the application drive flow control in the way it does for STREAM frames
>>>>> adds complexity and diminishes the utility of the frame and ACKs.. However,
>>>>> I can imagine taking a fairly simplistic approach in which the data limit
>>>>> is automatically increased upon reception of the frame (and the frame is
>>>>> immediately passed to the application). This allows the initial_max_data to
>>>>> put a cap on the amount of data in a given flight of DATAGRAMS, and allow
>>>>> the size of a flight of DATAGRAM frames to be limited by the amount of room
>>>>> left over from STREAM data that may be consuming the connection-wide flow
>>>>> control.
>>>>> Perhaps this approach needs a clearer name other than "flow control",
>>>>> since it has a somewhat different meaning in effect.
>>>>> As for ACKs, if we never discard on the receiver side, the ACK is
>>>>> pretty useful for detecting if there was network-based packet loss.
>>>>> Thanks,
>>>>> Tommy
>>>>> On Oct 29, 2018, at 5:32 AM, Ian Swett <> wrote:
>>>>> Good catch Martin, I missed that in the draft as well, and I also
>>>>> think it's impossibly with the proposed design.
>>>>> And yes, I think Martin's proposed solution is likely the only
>>>>> practical one.  In my implementation, the frame is passed up to the
>>>>> application immediately, so technically QUIC processed it, and it's the
>>>>> application's job to decide what to do with it.
>>>>> On Mon, Oct 29, 2018 at 1:16 AM Martin Thomson <
>>>>> <>> wrote:
>>>>>> Hi Tommy,
>>>>>> Your slides - <
>>>>>> <>
>>>>>> >
>>>>>> - say that DATAGRAM frames respect connection-level flow control.  I
>>>>>> missed that in the draft, and I don't know how they can do that in the
>>>>>> face of packet loss, especially when you don't necessarily retransmit
>>>>>> lost DATAGRAM frames.
>>>>>> For that to work, you would need a bunch more machinery to make the
>>>>>> connection-level flow control sync between endpoints in the case that
>>>>>> packets are lost.  A disagreement about how much flow control is used
>>>>>> causes things to break down badly.  Ian and I discussed this point at
>>>>>> the last meeting and quickly agreed that while it might be nice to
>>>>>> have flow control for this stuff, the increase in complexity is
>>>>>> considerable and (at the time) we thought it wouldn't be worth it.
>>>>>> The problem that introduces is that you could end up having too many
>>>>>> DATAGRAM frames arrive.  The receiver has to drop something at the
>>>>>> point that it can't handle them.  And we say that when you acknowledge
>>>>>> something, you processed it.  That's tricky.
>>>>>> It might be easier to say that a QUIC acknowledgment for a DATAGRAM
>>>>>> frame doesn't mean that it was received and processed by an
>>>>>> application.  An endpoint might discard these frames before passing
>>>>>> them on to applications if it doesn't have space.  In other words,
>>>>>> acknowledgment of DATAGRAM means that QUIC got it, not that the
>>>>>> application got it.  Sadly, that means that the QUIC acknowledgment
>>>>>> machinery doesn't help the application that uses DATAGRAM all that
>>>>>> much.  Also, the lower bound on reliability is 0, which isn't the best
>>>>>> thing ever.
>>>>>> Hard choices, I know.  I don't have a good design for maintaining
>>>>>> connection-level flow control (or any back pressure mechanism with
>>>>>> equivalent properties) that doesn't add both complexity and overhead.
>>>>>> Cheers,
>>>>>> Martin