Re: flow control and DATAGRAM

Jana Iyengar <jri.ietf@gmail.com> Wed, 07 November 2018 12:23 UTC

Return-Path: <jri.ietf@gmail.com>
X-Original-To: quic@ietfa.amsl.com
Delivered-To: quic@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6D2A512D4F0 for <quic@ietfa.amsl.com>; Wed, 7 Nov 2018 04:23:09 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.01
X-Spam-Level:
X-Spam-Status: No, score=-0.01 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, HTTPS_HTTP_MISMATCH=1.989, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aFeNpcKLwdPK for <quic@ietfa.amsl.com>; Wed, 7 Nov 2018 04:23:05 -0800 (PST)
Received: from mail-lf1-x12e.google.com (mail-lf1-x12e.google.com [IPv6:2a00:1450:4864:20::12e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 0CC2812D4EA for <quic@ietf.org>; Wed, 7 Nov 2018 04:23:05 -0800 (PST)
Received: by mail-lf1-x12e.google.com with SMTP id p17so11308939lfh.4 for <quic@ietf.org>; Wed, 07 Nov 2018 04:23:04 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=N5iRtRS46waFEJLnNqWTR23C3gXZQn4cEktnJ/7CytY=; b=CXznBtvnVCvjUO02a9EyKK1BHkGV+7XhNMuoxh2uMjKMD+Ui+argqpB6y4Ee3KpHNm 9fqTbfcRIPUlE4fr5jG2r5Nkk6RWh1QvDNqUVT9oJU7pNHXY+dCRABTRBkIY86KoS4JL ka3u7nNozPugVHhFVWjrTDyUxFAaG6jUPyQtFHvyaTL+xIohDdsIcMGLtQ76XNxv8Xrs gusBORQtVLZztXyLkyxC01erdbbwJGO6GZLG2hxuvYxLMAeRtFDuFLE3kkdmor0N2mIg a4hOXA8ntGvlgHYaa2zoznxPxJYl+HzM1/nRZdFJdOz6nDLrJJE/djxxH6Lm6P5sbJir tEnQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=N5iRtRS46waFEJLnNqWTR23C3gXZQn4cEktnJ/7CytY=; b=Fb3sA2SuzfQyqBgUVgal/Z2qVU4EOuXjCaDqJCjGHYBsclJAIxA5si8t49cjqqCWmH t61I2puJRlofmjAl+hpcO0GsM+JJyKGf5the4AhaGaWzGnLfhFFY5UiX9ggS05FwjSXe yhQ4uwM3texB+XKepFDpj4UuayQL5ZFDqact2hp64ET51X1tbpXPOSslAGlbTqt5PDeB C2gPdZWeWzgzfg4T4AAD96AfRGFZjrqWStFk7wxs6yEcYPUHCqcpx+0u9YGX8voFHq4W A2kIFLOHqO6ogxCVvS7IsXPb60uHAL0pP2fF2lQrrL7MposcTs0+4tPc4fEs6llWv1Pa kx9w==
X-Gm-Message-State: AGRZ1gKIjCV6Hjtude7Ae5IZKakmGjmyIM/99bb7FnUEgD3xxQZ3hv1b obfJlDPHry2k46QwyY/5RNkOlr16WO3ozuZAuI0=
X-Google-Smtp-Source: AJdET5ebGEDTHPvCbinaVJ4Wgg+ejoYHDK3MiSj2bBdTwNzHfI2gtnaHlGRpOtv+9X00ZEWI1MUqDeWoACnmu22jTYw=
X-Received: by 2002:a19:2395:: with SMTP id j143mr1015658lfj.107.1541593383052; Wed, 07 Nov 2018 04:23:03 -0800 (PST)
MIME-Version: 1.0
References: <CABkgnnU26aYD=TybuD0FZGYtfEa6np-Sk3Jo6t7LRp0wzKh3Lg@mail.gmail.com> <CAKcm_gMDOyJC0AwOo2knN6AMxVbySjGsrLvjcC9A8UA9xcvcWw@mail.gmail.com> <19D3595B-8845-45B6-A60D-9E934DD49FAC@apple.com> <CACpbDcfk+GXb3aL5LG0wQM87thRGO5Y4Q+9cbXf5YuW1=jWCig@mail.gmail.com> <B7BE2454-2A4D-4323-B0A5-0D73BD4B819F@apple.com> <344a8a30b95441af988b675c2f887965@usma1ex-dag1mb6.msg.corp.akamai.com> <CY4PR22MB098366E6A90123689A855766DACE0@CY4PR22MB0983.namprd22.prod.outlook.com> <MWHPR15MB1821E1BE32DF76B098FE970DB6C40@MWHPR15MB1821.namprd15.prod.outlook.com>
In-Reply-To: <MWHPR15MB1821E1BE32DF76B098FE970DB6C40@MWHPR15MB1821.namprd15.prod.outlook.com>
From: Jana Iyengar <jri.ietf@gmail.com>
Date: Wed, 07 Nov 2018 19:22:51 +0700
Message-ID: <CACpbDcds453b7HzwY2nDEGBrZsmPo4OH9ZQCXuZy-f9KZYTpGA@mail.gmail.com>
Subject: Re: flow control and DATAGRAM
To: Subodh Iyengar <subodh@fb.com>
Cc: Mike Bishop <mbishop@evequefou.be>, "Lubashev, Igor" <ilubashe@akamai.com>, tpauly@apple.com, Ian Swett <ianswett@google.com>, QUIC WG <quic@ietf.org>, Martin Thomson <martin.thomson@gmail.com>
Content-Type: multipart/alternative; boundary="000000000000fc29e5057a122de0"
Archived-At: <https://mailarchive.ietf.org/arch/msg/quic/MhmL6JWFeADDOTuQtnts_s8wmAs>
X-BeenThere: quic@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <quic.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/quic>, <mailto:quic-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/quic/>
List-Post: <mailto:quic@ietf.org>
List-Help: <mailto:quic-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/quic>, <mailto:quic-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 07 Nov 2018 12:23:10 -0000

+1. You definitely want to ack these frames too.

On Wed, Nov 7, 2018 at 12:02 PM Subodh Iyengar <subodh@fb.com> wrote:

> Slight aside wrt regard to ACKs, I think you have to ACK the DATAGRAM
> frame if you want to subject DATAGRAM to congestion control, otherwise you
> rely on loss recovery to recover the inflight bytes. So a DATAGRAM frame
> with optional ACKs seems not great given the current state of congestion
> control.
>
>
> Subodh
>
> ------------------------------
> *From:* QUIC <quic-bounces@ietf.org> on behalf of Mike Bishop <
> mbishop@evequefou.be>
> *Sent:* Wednesday, October 31, 2018 9:08:24 PM
> *To:* Lubashev, Igor; Tommy Pauly; Jana Iyengar
> *Cc:* Ian Swett; QUIC WG; Martin Thomson
> *Subject:* RE: flow control and DATAGRAM
>
>
> Technically, an implementation could choose not to ACK without ignoring
> all frames in the packet.  Our design for frames says that they can be
> sent/received multiple times without error, so a spuriously-retransmitted
> reliable frame from that packet is wasted data, but not an error.  However,
> it might not be totally clear to application designers that, while ACKs can
> tell you which datagrams definitely did arrive, they can never tell you
> which ones definitely didn’t.
>
>
>
> But it sounds like where this is going is the realization that you have to
> have flow control if you’re going to prohibit the remote implementation
> from dropping datagrams after ACKing them; but if you do allow the
> implementation to do that, then ACKs tell you nothing concrete about
> DATAGRAMs that might have been in those packets.
>
>
>
> *From:* QUIC <quic-bounces@ietf.org> *On Behalf Of * Lubashev, Igor
> *Sent:* Monday, October 29, 2018 4:29 PM
> *To:* Tommy Pauly <tpauly@apple.com>; Jana Iyengar <jri.ietf@gmail.com>
> *Cc:* Ian Swett <ianswett@google.com>; QUIC WG <quic@ietf.org>; Martin
> Thomson <martin.thomson@gmail.com>
> *Subject:* RE: flow control and DATAGRAM
>
>
>
> > ACK to a packet containing a DATAGRAM frame means:
>
> > - The packet made it across the network to the QUIC endpoint on the
> other side
>
> > - The QUIC implementation will deliver the DATAGRAM frame to application
> (it won't drop it locally)
>
> >
>
> > Does that make sense?
>
>
>
> This only makes sense, if there is an effective flow control for
> DATAGRAMs.  The spec should be implementable by poll/read QUIC
> implementations as well as by callback-on-network-receive-thread
> implementations.  Unless there is an effective flow control, a buffering
> QUIC implementation must be able to discard DATAGRAMs, if the receiver app
> is not catching up.  The receiver would then have to ignore all
> non-DATAGRAM frames in that packet, since it would be unable to ACK the
> packet.  If the receiver is processing DATAGRAMs and STREAMs at different
> speeds, deliberate discards of packets containing DATAGRAMs would interfere
> with the congestion control, effectively HOL blocking STREAMs.
>
>
>
> An effective flow control requires sender and receiver to synchronize
> their view of the flow control counter (MAX_DATA for connection flow
> control).  Relying on ACKs is very hard, since the receiver cannot know how
> much data the sender has sent (unless the sender is explicit about it).
>
>
>
> If flow control for DATAGRAMs is desired, you can use mechanics such as in
> https://tools.ietf.org/html/draft-lubashev-quic-partial-reliability-02
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dlubashev-2Dquic-2Dpartial-2Dreliability-2D02&d=DwMGaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=h3Ju9EBS7mHtwg-wAyN7fQ&m=7-2U7tgFm9FFGYQbRf9Nbscngq456CsvamxwAvNBDeE&s=ym3xrptVe9JbKdR-VQGI5V7nGaEzgbyg5BBlnxPLJxY&e=>
> or https://tools.ietf.org/html/draft-lubashev-quic-partial-reliability-03
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dlubashev-2Dquic-2Dpartial-2Dreliability-2D03&d=DwMGaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=h3Ju9EBS7mHtwg-wAyN7fQ&m=7-2U7tgFm9FFGYQbRf9Nbscngq456CsvamxwAvNBDeE&s=J-GHsot00B89AF-s9_5e1ErCuqDg_8cXmWGidGXyP9I&e=>
> (https://github.com/igorlord/draft-lubashev-quic-partial-reliability).
> (This would be treating DATAGRAMs as belonging to a partially reliable
> stream, or streams.  Except you would not need to ensure ordering of
> DATAGRAMs in that stream.)
>
>
>
>    - Igor
>
>
>
>
>
> *From:* Tommy Pauly <tpauly@apple.com>
> *Sent:* Monday, October 29, 2018 4:01 PM
> *To:* Jana Iyengar <jri.ietf@gmail.com>
> *Cc:* Ian Swett <ianswett@google.com>; QUIC WG <quic@ietf.org>; Martin
> Thomson <martin.thomson@gmail.com>
> *Subject:* Re: flow control and DATAGRAM
>
>
>
> Hi Jana,
>
>
>
> I'm not suggesting any changes to when the packet gets
> ack'ed—specifically, I was responding to Martin's hypothetical of having an
> ACK to a DATAGRAM frame meaning that the application had processed the
> frame. My impression is that the ACK to a packet containing a DATAGRAM
> frame means:
>
> - The packet made it across the network to the QUIC endpoint on the other
> side
>
> - The QUIC implementation will deliver the DATAGRAM frame to application
> (it won't drop it locally)
>
>
>
> Does that make sense?
>
>
>
> Having a separate outstanding data limit for the DATAGRAM "stream" is an
> interesting solution to the space. It would then have the nice property of
> not looking like traditional flow control. It could even be measured in
> number of frames, rather than bytes (depending on what the limiting factors
> are).
>
>
>
> Thanks,
>
> Tommy
>
>
>
> On Oct 29, 2018, at 12:37 PM, Jana Iyengar <jri.ietf@gmail.com> wrote:
>
>
>
> Tommy,
>
>
>
> Changing the semantics of an acknowledgment to include delivery up to the
> application is a fundamental change to the QUIC machinery, and it doesn't
> work. First, an ACK frame acknowledges packets, and you can't have
> different semantics of an acknowledgment for different frames that are
> carried in the same packet. Second, it interferes with RTT measurement, and
> it conflates flow control with congestion control, which gets messy. (This
> conflation is an interesting problem to consider theoretically, but not one
> for us at this time IMO.)
>
>
>
> I am wondering if applying a stream-level flow control for DATAGRAMs makes
> sense instead. Meaning that you treat DATAGRAMs as a separate stream for
> flow control purposes. You might benefit from having an offset in the
> DATAGRAM frame for this purpose.
>
>
>
> - jana
>
>
>
> On Mon, Oct 29, 2018 at 8:21 AM Tommy Pauly <tpauly@apple.com> wrote:
>
> Hi Martin, Ian,
>
>
>
> Yes, very good points!
>
>
>
> My tendency would be to prefer what Ian's implementation does of passing
> these DATAGRAM frames up immediately to the application. I don't think that
> the acknowledgment needs to indicate that the frame was processed by the
> application, but merely that it has been delivered to the application (that
> is, the application doesn't get to do anything with the frame that can
> influence the acknowledgment).
>
>
>
> The current draft indicates that the content of the DATAGRAM frames
> contributes to the limit used for MAX_DATA, and that if that amount is
> reached, the frames are blocked along with STREAM data. I think this works
> fine for the sender, while the receiver gets into the discussion you
> present. On the sender side, reaching MAX_DATA could mean dropping the
> DATAGRAM frames when unable to send more (and sending BLOCKED instead).
> Since the frames are unreliable, they can be dropped in this situation
> without violating the API contract.
>
>
>
> On the receiver side, I agree that queuing the DATAGRAM frames to let the
> application drive flow control in the way it does for STREAM frames adds
> complexity and diminishes the utility of the frame and ACKs.. However, I
> can imagine taking a fairly simplistic approach in which the data limit is
> automatically increased upon reception of the frame (and the frame is
> immediately passed to the application). This allows the initial_max_data to
> put a cap on the amount of data in a given flight of DATAGRAMS, and allow
> the size of a flight of DATAGRAM frames to be limited by the amount of room
> left over from STREAM data that may be consuming the connection-wide flow
> control.
>
>
>
> Perhaps this approach needs a clearer name other than "flow control",
> since it has a somewhat different meaning in effect.
>
>
>
> As for ACKs, if we never discard on the receiver side, the ACK is pretty
> useful for detecting if there was network-based packet loss.
>
>
>
> Thanks,
>
> Tommy
>
>
>
> On Oct 29, 2018, at 5:32 AM, Ian Swett <ianswett@google.com> wrote:
>
>
>
> Good catch Martin, I missed that in the draft as well, and I also think
> it's impossibly with the proposed design.
>
>
>
> And yes, I think Martin's proposed solution is likely the only practical
> one.  In my implementation, the frame is passed up to the application
> immediately, so technically QUIC processed it, and it's the application's
> job to decide what to do with it.
>
>
>
> On Mon, Oct 29, 2018 at 1:16 AM Martin Thomson <martin.thomson@gmail.com>
> wrote:
>
> Hi Tommy,
>
> Your slides - <
> https://github.com/quicwg/wg-materials/blob/master/ietf103/IETF103-QUIC-Datagram.pdf
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_quicwg_wg-2Dmaterials_blob_master_ietf103_IETF103-2DQUIC-2DDatagram.pdf&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=-JAVyKK4LrDFUn-TzI96ZFlrFZYozg4GFVttnjIjGdg&s=tqUQZW-hP0GwZ7x1otPkSYpA3axGVizYu0a4JlLou6Q&e=>
> >
> - say that DATAGRAM frames respect connection-level flow control.  I
> missed that in the draft, and I don't know how they can do that in the
> face of packet loss, especially when you don't necessarily retransmit
> lost DATAGRAM frames.
>
> For that to work, you would need a bunch more machinery to make the
> connection-level flow control sync between endpoints in the case that
> packets are lost.  A disagreement about how much flow control is used
> causes things to break down badly.  Ian and I discussed this point at
> the last meeting and quickly agreed that while it might be nice to
> have flow control for this stuff, the increase in complexity is
> considerable and (at the time) we thought it wouldn't be worth it.
>
> The problem that introduces is that you could end up having too many
> DATAGRAM frames arrive.  The receiver has to drop something at the
> point that it can't handle them.  And we say that when you acknowledge
> something, you processed it.  That's tricky.
>
> It might be easier to say that a QUIC acknowledgment for a DATAGRAM
> frame doesn't mean that it was received and processed by an
> application.  An endpoint might discard these frames before passing
> them on to applications if it doesn't have space.  In other words,
> acknowledgment of DATAGRAM means that QUIC got it, not that the
> application got it.  Sadly, that means that the QUIC acknowledgment
> machinery doesn't help the application that uses DATAGRAM all that
> much.  Also, the lower bound on reliability is 0, which isn't the best
> thing ever.
>
> Hard choices, I know.  I don't have a good design for maintaining
> connection-level flow control (or any back pressure mechanism with
> equivalent properties) that doesn't add both complexity and overhead.
>
> Cheers,
> Martin
>
>
>
>
>