Re: flow control and DATAGRAM

Tommy Pauly <tpauly@apple.com> Wed, 07 November 2018 12:34 UTC

Return-Path: <tpauly@apple.com>
X-Original-To: quic@ietfa.amsl.com
Delivered-To: quic@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1451812DDA3 for <quic@ietfa.amsl.com>; Wed, 7 Nov 2018 04:34:42 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.481
X-Spam-Level:
X-Spam-Status: No, score=-0.481 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.47, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, HTTPS_HTTP_MISMATCH=1.989, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=apple.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id w9gZWZddQ_em for <quic@ietfa.amsl.com>; Wed, 7 Nov 2018 04:34:38 -0800 (PST)
Received: from nwk-aaemail-lapp01.apple.com (nwk-aaemail-lapp01.apple.com [17.151.62.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 6398112D4F0 for <quic@ietf.org>; Wed, 7 Nov 2018 04:34:38 -0800 (PST)
Received: from pps.filterd (nwk-aaemail-lapp01.apple.com [127.0.0.1]) by nwk-aaemail-lapp01.apple.com (8.16.0.22/8.16.0.22) with SMTP id wA7CW9GW042983; Wed, 7 Nov 2018 04:34:27 -0800
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=mime-version : content-type : sender : from : message-id : subject : date : in-reply-to : cc : to : references; s=20180706; bh=K1EToGvRjOMrfE20uM0N6sYASlEnlotzFgWtrmh3nZE=; b=WyoOIHP+WtZ4pFavEiTMR4jUl4BbQt6ryBhxrY9yktzKJoHGM74F/SM+DyJHafd978HB zvdhGm/OeY+WMWYVRS7O5AG3JcEXis3s5Q4HftzCjxPcaK9v+7Wffhf36T7FmXTEoaqn mLg2fwHWbTLpiJooNfYmzpTd8Y/EvOeULKMFu6fPDjkuW0RMsv6azYKl9oKYnxV5+wtW OLvWkCG5013xaymnnRo1GSetAr6rBSDI7QsUKyw3a77fnN+fO8S+F3fd39oHW2/Hrg7v ZyfaeP+exN+KtZ8HkIMNhbxQFgPgTPWuRWvp4fYSgxOsJGNv+O1H7Fihbq4xiGys7CSQ vg==
Received: from mr2-mtap-s02.rno.apple.com (mr2-mtap-s02.rno.apple.com [17.179.226.134]) by nwk-aaemail-lapp01.apple.com with ESMTP id 2njuus2e61-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Wed, 07 Nov 2018 04:34:27 -0800
MIME-version: 1.0
Content-type: multipart/alternative; boundary="Boundary_(ID_a45uWfR16X8P8y2BG6KvJg)"
Received: from ma1-mmpp-sz10.apple.com (ma1-mmpp-sz10.apple.com [17.171.128.150]) by mr2-mtap-s02.rno.apple.com (Oracle Communications Messaging Server 8.0.2.3.20180614 64bit (built Jun 14 2018)) with ESMTPS id <0PHT00LZAPLFM720@mr2-mtap-s02.rno.apple.com>; Wed, 07 Nov 2018 04:34:27 -0800 (PST)
Received: from process_viserion-daemon.ma1-mmpp-sz10.apple.com by ma1-mmpp-sz10.apple.com (Oracle Communications Messaging Server 8.0.2.3.20180614 64bit (built Jun 14 2018)) id <0PHT00700PDTD000@ma1-mmpp-sz10.apple.com>; Wed, 07 Nov 2018 04:34:27 -0800 (PST)
X-Va-A:
X-Va-T-CD: a5deab3aa8128cfa8ffecae1f592b6eb
X-Va-E-CD: c49bf73f2750f6e382c68706c9cc0d07
X-Va-R-CD: 22b8f453b4f47490ba2e2176fe96324e
X-Va-CD: 0
X-Va-ID: 1bfa751e-f9e8-4354-aa5b-8cbe2b6fc983
X-V-A:
X-V-T-CD: aefb07da2fd6938664ce892a3812191f
X-V-E-CD: c49bf73f2750f6e382c68706c9cc0d07
X-V-R-CD: 22b8f453b4f47490ba2e2176fe96324e
X-V-CD: 0
X-V-ID: c6477c44-e869-4939-b8f8-9638e6d36147
Received: from process_milters-daemon.ma1-mmpp-sz10.apple.com by ma1-mmpp-sz10.apple.com (Oracle Communications Messaging Server 8.0.2.3.20180614 64bit (built Jun 14 2018)) id <0PHT00K00NOTE400@ma1-mmpp-sz10.apple.com>; Wed, 07 Nov 2018 04:34:26 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2018-11-07_10:,, signatures=0
Received: from [17.234.131.25] (unknown [17.234.131.25]) by ma1-mmpp-sz10.apple.com (Oracle Communications Messaging Server 8.0.2.3.20180614 64bit (built Jun 14 2018)) with ESMTPSA id <0PHT001ZNPL3TL30@ma1-mmpp-sz10.apple.com>; Wed, 07 Nov 2018 04:34:26 -0800 (PST)
Sender: tpauly@apple.com
From: Tommy Pauly <tpauly@apple.com>
Message-id: <6C37BE03-337F-414F-8177-CDC2E8E5E877@apple.com>
Subject: Re: flow control and DATAGRAM
Date: Wed, 07 Nov 2018 19:34:11 +0700
In-reply-to: <CACpbDcds453b7HzwY2nDEGBrZsmPo4OH9ZQCXuZy-f9KZYTpGA@mail.gmail.com>
Cc: Subodh Iyengar <subodh@fb.com>, Ian Swett <ianswett@google.com>, "Lubashev, Igor" <ilubashe@akamai.com>, Mike Bishop <mbishop@evequefou.be>, QUIC WG <quic@ietf.org>, Martin Thomson <martin.thomson@gmail.com>
To: Jana Iyengar <jri.ietf@gmail.com>
References: <CABkgnnU26aYD=TybuD0FZGYtfEa6np-Sk3Jo6t7LRp0wzKh3Lg@mail.gmail.com> <CAKcm_gMDOyJC0AwOo2knN6AMxVbySjGsrLvjcC9A8UA9xcvcWw@mail.gmail.com> <19D3595B-8845-45B6-A60D-9E934DD49FAC@apple.com> <CACpbDcfk+GXb3aL5LG0wQM87thRGO5Y4Q+9cbXf5YuW1=jWCig@mail.gmail.com> <B7BE2454-2A4D-4323-B0A5-0D73BD4B819F@apple.com> <344a8a30b95441af988b675c2f887965@usma1ex-dag1mb6.msg.corp.akamai.com> <CY4PR22MB098366E6A90123689A855766DACE0@CY4PR22MB0983.namprd22.prod.outlook.com> <MWHPR15MB1821E1BE32DF76B098FE970DB6C40@MWHPR15MB1821.namprd15.prod.outlook.com> <CACpbDcds453b7HzwY2nDEGBrZsmPo4OH9ZQCXuZy-f9KZYTpGA@mail.gmail.com>
X-Mailer: Apple Mail (2.3445.100.36)
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-11-07_10:, , signatures=0
Archived-At: <https://mailarchive.ietf.org/arch/msg/quic/EyxiZ5l6I5b7J2nVer6WgD9KgWE>
X-BeenThere: quic@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <quic.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/quic>, <mailto:quic-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/quic/>
List-Post: <mailto:quic@ietf.org>
List-Help: <mailto:quic-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/quic>, <mailto:quic-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 07 Nov 2018 12:34:42 -0000

Yes, totally agreed. The current document does require all DATAGRAM frames to be acked, and I think this is the correct scheme to maintain.

Thanks,
Tommy

> On Nov 7, 2018, at 7:22 PM, Jana Iyengar <jri.ietf@gmail.com> wrote:
> 
> +1. You definitely want to ack these frames too.
> 
> On Wed, Nov 7, 2018 at 12:02 PM Subodh Iyengar <subodh@fb.com <mailto:subodh@fb.com>> wrote:
> Slight aside wrt regard to ACKs, I think you have to ACK the DATAGRAM frame if you want to subject DATAGRAM to congestion control, otherwise you rely on loss recovery to recover the inflight bytes. So a DATAGRAM frame with optional ACKs seems not great given the current state of congestion control.
> 
> 
> 
> Subodh
> 
> 
> 
> From: QUIC <quic-bounces@ietf.org <mailto:quic-bounces@ietf.org>> on behalf of Mike Bishop <mbishop@evequefou.be <mailto:mbishop@evequefou.be>>
> Sent: Wednesday, October 31, 2018 9:08:24 PM
> To: Lubashev, Igor; Tommy Pauly; Jana Iyengar
> Cc: Ian Swett; QUIC WG; Martin Thomson
> Subject: RE: flow control and DATAGRAM
>  
> Technically, an implementation could choose not to ACK without ignoring all frames in the packet.  Our design for frames says that they can be sent/received multiple times without error, so a spuriously-retransmitted reliable frame from that packet is wasted data, but not an error.  However, it might not be totally clear to application designers that, while ACKs can tell you which datagrams definitely did arrive, they can never tell you which ones definitely didn’t.
> 
>  
> But it sounds like where this is going is the realization that you have to have flow control if you’re going to prohibit the remote implementation from dropping datagrams after ACKing them; but if you do allow the implementation to do that, then ACKs tell you nothing concrete about DATAGRAMs that might have been in those packets.
> 
>  
> From: QUIC <quic-bounces@ietf.org <mailto:quic-bounces@ietf.org>> On Behalf Of Lubashev, Igor
> Sent: Monday, October 29, 2018 4:29 PM
> To: Tommy Pauly <tpauly@apple.com <mailto:tpauly@apple.com>>; Jana Iyengar <jri.ietf@gmail.com <mailto:jri.ietf@gmail.com>>
> Cc: Ian Swett <ianswett@google.com <mailto:ianswett@google.com>>; QUIC WG <quic@ietf.org <mailto:quic@ietf..org>>; Martin Thomson <martin.thomson@gmail..com <mailto:martin.thomson@gmail.com>>
> Subject: RE: flow control and DATAGRAM
> 
>  
> > ACK to a packet containing a DATAGRAM frame means:
> 
> > - The packet made it across the network to the QUIC endpoint on the other side
> 
> > - The QUIC implementation will deliver the DATAGRAM frame to application (it won't drop it locally)
> 
> > 
> 
> > Does that make sense?
> 
>  
> This only makes sense, if there is an effective flow control for DATAGRAMs.  The spec should be implementable by poll/read QUIC implementations as well as by callback-on-network-receive-thread implementations.  Unless there is an effective flow control, a buffering QUIC implementation must be able to discard DATAGRAMs, if the receiver app is not catching up.  The receiver would then have to ignore all non-DATAGRAM frames in that packet, since it would be unable to ACK the packet.  If the receiver is processing DATAGRAMs and STREAMs at different speeds, deliberate discards of packets containing DATAGRAMs would interfere with the congestion control, effectively HOL blocking STREAMs.
> 
>  
> An effective flow control requires sender and receiver to synchronize their view of the flow control counter (MAX_DATA for connection flow control).  Relying on ACKs is very hard, since the receiver cannot know how much data the sender has sent (unless the sender is explicit about it).
> 
>  
> If flow control for DATAGRAMs is desired, you can use mechanics such as in https://tools.ietf.org/html/draft-lubashev-quic-partial-reliability-02 <https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dlubashev-2Dquic-2Dpartial-2Dreliability-2D02&d=DwMGaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=h3Ju9EBS7mHtwg-wAyN7fQ&m=7-2U7tgFm9FFGYQbRf9Nbscngq456CsvamxwAvNBDeE&s=ym3xrptVe9JbKdR-VQGI5V7nGaEzgbyg5BBlnxPLJxY&e=> or https://tools.ietf.org/html/draft-lubashev-quic-partial-reliability-03 <https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dlubashev-2Dquic-2Dpartial-2Dreliability-2D03&d=DwMGaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=h3Ju9EBS7mHtwg-wAyN7fQ&m=7-2U7tgFm9FFGYQbRf9Nbscngq456CsvamxwAvNBDeE&s=J-GHsot00B89AF-s9_5e1ErCuqDg_8cXmWGidGXyP9I&e=> (https://github.com/igorlord/draft-lubashev-quic-partial-reliability <https://github.com/igorlord/draft-lubashev-quic-partial-reliability>).  (This would be treating DATAGRAMs as belonging to a partially reliable stream, or streams.  Except you would not need to ensure ordering of DATAGRAMs in that stream.)
> 
>  
> Igor
>  
>  
> From: Tommy Pauly <tpauly@apple.com <mailto:tpauly@apple.com>> 
> Sent: Monday, October 29, 2018 4:01 PM
> To: Jana Iyengar <jri.ietf@gmail.com <mailto:jri.ietf@gmail.com>>
> Cc: Ian Swett <ianswett@google.com <mailto:ianswett@google.com>>; QUIC WG <quic@ietf.org <mailto:quic@ietf..org>>; Martin Thomson <martin.thomson@gmail..com <mailto:martin.thomson@gmail.com>>
> Subject: Re: flow control and DATAGRAM
> 
>  
> Hi Jana,
> 
>  
> I'm not suggesting any changes to when the packet gets ack'ed—specifically, I was responding to Martin's hypothetical of having an ACK to a DATAGRAM frame meaning that the application had processed the frame. My impression is that the ACK to a packet containing a DATAGRAM frame means:
> 
> - The packet made it across the network to the QUIC endpoint on the other side
> 
> - The QUIC implementation will deliver the DATAGRAM frame to application (it won't drop it locally)
> 
>  
> Does that make sense?
> 
>  
> Having a separate outstanding data limit for the DATAGRAM "stream" is an interesting solution to the space. It would then have the nice property of not looking like traditional flow control. It could even be measured in number of frames, rather than bytes (depending on what the limiting factors are).
> 
>  
> Thanks,
> 
> Tommy
> 
>  
> On Oct 29, 2018, at 12:37 PM, Jana Iyengar <jri.ietf@gmail.com <mailto:jri.ietf@gmail.com>> wrote:
> 
>  
> Tommy,
> 
>  
> Changing the semantics of an acknowledgment to include delivery up to the application is a fundamental change to the QUIC machinery, and it doesn't work. First, an ACK frame acknowledges packets, and you can't have different semantics of an acknowledgment for different frames that are carried in the same packet. Second, it interferes with RTT measurement, and it conflates flow control with congestion control, which gets messy. (This conflation is an interesting problem to consider theoretically, but not one for us at this time IMO.)
> 
>  
> I am wondering if applying a stream-level flow control for DATAGRAMs makes sense instead. Meaning that you treat DATAGRAMs as a separate stream for flow control purposes. You might benefit from having an offset in the DATAGRAM frame for this purpose.
> 
>  
> - jana
> 
>  
> On Mon, Oct 29, 2018 at 8:21 AM Tommy Pauly <tpauly@apple.com <mailto:tpauly@apple.com>> wrote:
> 
> Hi Martin, Ian,
> 
>  
> Yes, very good points!
> 
>  
> My tendency would be to prefer what Ian's implementation does of passing these DATAGRAM frames up immediately to the application. I don't think that the acknowledgment needs to indicate that the frame was processed by the application, but merely that it has been delivered to the application (that is, the application doesn't get to do anything with the frame that can influence the acknowledgment).
> 
>  
> The current draft indicates that the content of the DATAGRAM frames contributes to the limit used for MAX_DATA, and that if that amount is reached, the frames are blocked along with STREAM data. I think this works fine for the sender, while the receiver gets into the discussion you present. On the sender side, reaching MAX_DATA could mean dropping the DATAGRAM frames when unable to send more (and sending BLOCKED instead). Since the frames are unreliable, they can be dropped in this situation without violating the API contract. 
> 
>  
> On the receiver side, I agree that queuing the DATAGRAM frames to let the application drive flow control in the way it does for STREAM frames adds complexity and diminishes the utility of the frame and ACKs.. However, I can imagine taking a fairly simplistic approach in which the data limit is automatically increased upon reception of the frame (and the frame is immediately passed to the application). This allows the initial_max_data to put a cap on the amount of data in a given flight of DATAGRAMS, and allow the size of a flight of DATAGRAM frames to be limited by the amount of room left over from STREAM data that may be consuming the connection-wide flow control. 
> 
>  
> Perhaps this approach needs a clearer name other than "flow control", since it has a somewhat different meaning in effect.
> 
>  
> As for ACKs, if we never discard on the receiver side, the ACK is pretty useful for detecting if there was network-based packet loss.
> 
>  
> Thanks,
> 
> Tommy 
> 
>  
> On Oct 29, 2018, at 5:32 AM, Ian Swett <ianswett@google.com <mailto:ianswett@google.com>> wrote:
> 
>  
> Good catch Martin, I missed that in the draft as well, and I also think it's impossibly with the proposed design.
> 
>  
> And yes, I think Martin's proposed solution is likely the only practical one.  In my implementation, the frame is passed up to the application immediately, so technically QUIC processed it, and it's the application's job to decide what to do with it.
> 
>  
> On Mon, Oct 29, 2018 at 1:16 AM Martin Thomson <martin.thomson@gmail.com <mailto:martin.thomson@gmail.com>> wrote:
> 
> Hi Tommy,
> 
> Your slides - <https://github.com/quicwg/wg-materials/blob/master/ietf103/IETF103-QUIC-Datagram.pdf <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_quicwg_wg-2Dmaterials_blob_master_ietf103_IETF103-2DQUIC-2DDatagram.pdf&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=-JAVyKK4LrDFUn-TzI96ZFlrFZYozg4GFVttnjIjGdg&s=tqUQZW-hP0GwZ7x1otPkSYpA3axGVizYu0a4JlLou6Q&e=>>
> - say that DATAGRAM frames respect connection-level flow control.  I
> missed that in the draft, and I don't know how they can do that in the
> face of packet loss, especially when you don't necessarily retransmit
> lost DATAGRAM frames.
> 
> For that to work, you would need a bunch more machinery to make the
> connection-level flow control sync between endpoints in the case that
> packets are lost.  A disagreement about how much flow control is used
> causes things to break down badly.  Ian and I discussed this point at
> the last meeting and quickly agreed that while it might be nice to
> have flow control for this stuff, the increase in complexity is
> considerable and (at the time) we thought it wouldn't be worth it.
> 
> The problem that introduces is that you could end up having too many
> DATAGRAM frames arrive.  The receiver has to drop something at the
> point that it can't handle them.  And we say that when you acknowledge
> something, you processed it.  That's tricky.
> 
> It might be easier to say that a QUIC acknowledgment for a DATAGRAM
> frame doesn't mean that it was received and processed by an
> application.  An endpoint might discard these frames before passing
> them on to applications if it doesn't have space.  In other words,
> acknowledgment of DATAGRAM means that QUIC got it, not that the
> application got it.  Sadly, that means that the QUIC acknowledgment
> machinery doesn't help the application that uses DATAGRAM all that
> much.  Also, the lower bound on reliability is 0, which isn't the best
> thing ever.
> 
> Hard choices, I know.  I don't have a good design for maintaining
> connection-level flow control (or any back pressure mechanism with
> equivalent properties) that doesn't add both complexity and overhead.
> 
> Cheers,
> Martin
> 
>  
>