Re: flow control and DATAGRAM

Tommy Pauly <tpauly@apple.com> Mon, 29 October 2018 15:21 UTC

Return-Path: <tpauly@apple.com>
X-Original-To: quic@ietfa.amsl.com
Delivered-To: quic@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id EA97C130FDE for <quic@ietfa.amsl.com>; Mon, 29 Oct 2018 08:21:49 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.47
X-Spam-Level:
X-Spam-Status: No, score=-2.47 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.47, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=apple.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CgSyUtebUtTl for <quic@ietfa.amsl.com>; Mon, 29 Oct 2018 08:21:47 -0700 (PDT)
Received: from nwk-aaemail-lapp01.apple.com (nwk-aaemail-lapp01.apple.com [17.151.62.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 8819E130FF0 for <quic@ietf.org>; Mon, 29 Oct 2018 08:21:47 -0700 (PDT)
Received: from pps.filterd (nwk-aaemail-lapp01.apple.com [127.0.0.1]) by nwk-aaemail-lapp01.apple.com (8.16.0.22/8.16.0.22) with SMTP id w9TFLg5l064953; Mon, 29 Oct 2018 08:21:42 -0700
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=mime-version : content-type : sender : from : message-id : subject : date : in-reply-to : cc : to : references; s=20180706; bh=JIy9OVUr8FO+NAGFgzAAn/b3CJXxpz6NS/rMFYg23T4=; b=C44moYFPQaO2uLeGnrWLfAuiBkKVbD3knSWpiMbLMISr4DSxfSRg2B0iDOwLdH4N3i9F DovX2K/dHScFQzLGKZf8QLbqGcPZepgHlVCUO221oFmXAEor4FA/NA2hY4g/0/mOJor3 9Hu+DZ1ENFQy6kE/TjHMcDs9guUg9iB/mDcdU+UzNANcHknkAmCT54CANiniC0QTR6nd N5tiHoCZLfUaQxJ1Ztla2O3NUGqGVZ/CPkAIA+wYdNT9It43f0e1Y8V0acZwsa2XW9UW YOQOPLC+0A2rSmpHhXMrriawBKZGNhCEBs5B/RWCowt08MIBwDqm+Dj9Rr9zZdkb/QW1 YQ==
Received: from ma1-mtap-s02.corp.apple.com (ma1-mtap-s02.corp.apple.com [17.40.76.6]) by nwk-aaemail-lapp01.apple.com with ESMTP id 2ncpy5rbev-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Mon, 29 Oct 2018 08:21:42 -0700
MIME-version: 1.0
Content-type: multipart/alternative; boundary="Boundary_(ID_vYp3Wjs6KJBd5xKQoiEXWQ)"
Received: from nwk-mmpp-sz10.apple.com (nwk-mmpp-sz10.apple.com [17.128.115.122]) by ma1-mtap-s02.corp.apple.com (Oracle Communications Messaging Server 8.0.2.3.20180614 64bit (built Jun 14 2018)) with ESMTPS id <0PHD00A7I9C24W70@ma1-mtap-s02.corp.apple.com>; Mon, 29 Oct 2018 08:21:39 -0700 (PDT)
Received: from process_viserion-daemon.nwk-mmpp-sz10.apple.com by nwk-mmpp-sz10.apple.com (Oracle Communications Messaging Server 8.0.2.3.20180614 64bit (built Jun 14 2018)) id <0PHD00B008CZD900@nwk-mmpp-sz10.apple.com>; Mon, 29 Oct 2018 08:21:38 -0700 (PDT)
X-Va-A:
X-Va-T-CD: 1fbd3678d9e2d523e4297dddc27688bd
X-Va-E-CD: c49bf73f2750f6e382c68706c9cc0d07
X-Va-R-CD: 22b8f453b4f47490ba2e2176fe96324e
X-Va-CD: 0
X-Va-ID: 2d1b6828-93dd-4059-b941-32d74e9c085f
X-V-A:
X-V-T-CD: 8d71b7585ae9b1e749a466ceb4b88582
X-V-E-CD: c49bf73f2750f6e382c68706c9cc0d07
X-V-R-CD: 22b8f453b4f47490ba2e2176fe96324e
X-V-CD: 0
X-V-ID: 1a93c611-a911-455d-9b7c-b1d597d0a615
Received: from process_milters-daemon.nwk-mmpp-sz10.apple.com by nwk-mmpp-sz10.apple.com (Oracle Communications Messaging Server 8.0.2.3.20180614 64bit (built Jun 14 2018)) id <0PHD00M00953MX00@nwk-mmpp-sz10.apple.com>; Mon, 29 Oct 2018 08:21:38 -0700 (PDT)
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2018-10-29_11:,, signatures=0
Received: from [17.234.42.146] (unknown [17.234.42.146]) by nwk-mmpp-sz10.apple.com (Oracle Communications Messaging Server 8.0.2.3.20180614 64bit (built Jun 14 2018)) with ESMTPSA id <0PHD00IK19C14S70@nwk-mmpp-sz10.apple.com>; Mon, 29 Oct 2018 08:21:38 -0700 (PDT)
Sender: tpauly@apple.com
From: Tommy Pauly <tpauly@apple.com>
Message-id: <19D3595B-8845-45B6-A60D-9E934DD49FAC@apple.com>
Subject: Re: flow control and DATAGRAM
Date: Mon, 29 Oct 2018 08:21:33 -0700
In-reply-to: <CAKcm_gMDOyJC0AwOo2knN6AMxVbySjGsrLvjcC9A8UA9xcvcWw@mail.gmail.com>
Cc: IETF QUIC WG <quic@ietf.org>
To: Ian Swett <ianswett@google.com>, Martin Thomson <martin.thomson@gmail.com>
References: <CABkgnnU26aYD=TybuD0FZGYtfEa6np-Sk3Jo6t7LRp0wzKh3Lg@mail.gmail.com> <CAKcm_gMDOyJC0AwOo2knN6AMxVbySjGsrLvjcC9A8UA9xcvcWw@mail.gmail.com>
X-Mailer: Apple Mail (2.3445.100.36)
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-10-29_11:, , signatures=0
Archived-At: <https://mailarchive.ietf.org/arch/msg/quic/ARudWu6xkuq9CZZtumztvvaj-z0>
X-BeenThere: quic@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <quic.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/quic>, <mailto:quic-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/quic/>
List-Post: <mailto:quic@ietf.org>
List-Help: <mailto:quic-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/quic>, <mailto:quic-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 29 Oct 2018 15:21:50 -0000

Hi Martin, Ian,

Yes, very good points!

My tendency would be to prefer what Ian's implementation does of passing these DATAGRAM frames up immediately to the application. I don't think that the acknowledgment needs to indicate that the frame was processed by the application, but merely that it has been delivered to the application (that is, the application doesn't get to do anything with the frame that can influence the acknowledgment).

The current draft indicates that the content of the DATAGRAM frames contributes to the limit used for MAX_DATA, and that if that amount is reached, the frames are blocked along with STREAM data. I think this works fine for the sender, while the receiver gets into the discussion you present. On the sender side, reaching MAX_DATA could mean dropping the DATAGRAM frames when unable to send more (and sending BLOCKED instead). Since the frames are unreliable, they can be dropped in this situation without violating the API contract. 

On the receiver side, I agree that queuing the DATAGRAM frames to let the application drive flow control in the way it does for STREAM frames adds complexity and diminishes the utility of the frame and ACKs. However, I can imagine taking a fairly simplistic approach in which the data limit is automatically increased upon reception of the frame (and the frame is immediately passed to the application). This allows the initial_max_data to put a cap on the amount of data in a given flight of DATAGRAMS, and allow the size of a flight of DATAGRAM frames to be limited by the amount of room left over from STREAM data that may be consuming the connection-wide flow control. 

Perhaps this approach needs a clearer name other than "flow control", since it has a somewhat different meaning in effect.

As for ACKs, if we never discard on the receiver side, the ACK is pretty useful for detecting if there was network-based packet loss.

Thanks,
Tommy 

> On Oct 29, 2018, at 5:32 AM, Ian Swett <ianswett@google.com>; wrote:
> 
> Good catch Martin, I missed that in the draft as well, and I also think it's impossibly with the proposed design.
> 
> And yes, I think Martin's proposed solution is likely the only practical one.  In my implementation, the frame is passed up to the application immediately, so technically QUIC processed it, and it's the application's job to decide what to do with it.
> 
> On Mon, Oct 29, 2018 at 1:16 AM Martin Thomson <martin.thomson@gmail.com <mailto:martin.thomson@gmail.com>> wrote:
> Hi Tommy,
> 
> Your slides - <https://github.com/quicwg/wg-materials/blob/master/ietf103/IETF103-QUIC-Datagram.pdf <https://github.com/quicwg/wg-materials/blob/master/ietf103/IETF103-QUIC-Datagram.pdf>>
> - say that DATAGRAM frames respect connection-level flow control.  I
> missed that in the draft, and I don't know how they can do that in the
> face of packet loss, especially when you don't necessarily retransmit
> lost DATAGRAM frames.
> 
> For that to work, you would need a bunch more machinery to make the
> connection-level flow control sync between endpoints in the case that
> packets are lost.  A disagreement about how much flow control is used
> causes things to break down badly.  Ian and I discussed this point at
> the last meeting and quickly agreed that while it might be nice to
> have flow control for this stuff, the increase in complexity is
> considerable and (at the time) we thought it wouldn't be worth it.
> 
> The problem that introduces is that you could end up having too many
> DATAGRAM frames arrive.  The receiver has to drop something at the
> point that it can't handle them.  And we say that when you acknowledge
> something, you processed it.  That's tricky.
> 
> It might be easier to say that a QUIC acknowledgment for a DATAGRAM
> frame doesn't mean that it was received and processed by an
> application.  An endpoint might discard these frames before passing
> them on to applications if it doesn't have space.  In other words,
> acknowledgment of DATAGRAM means that QUIC got it, not that the
> application got it.  Sadly, that means that the QUIC acknowledgment
> machinery doesn't help the application that uses DATAGRAM all that
> much.  Also, the lower bound on reliability is 0, which isn't the best
> thing ever.
> 
> Hard choices, I know.  I don't have a good design for maintaining
> connection-level flow control (or any back pressure mechanism with
> equivalent properties) that doesn't add both complexity and overhead.
> 
> Cheers,
> Martin
>