Re: ack delay in rx packets

Subodh Iyengar <subodh@fb.com> Wed, 25 April 2018 16:48 UTC

Return-Path: <prvs=7653103cd5=subodh@fb.com>
X-Original-To: quic@ietfa.amsl.com
Delivered-To: quic@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id AE4111200C5 for <quic@ietfa.amsl.com>; Wed, 25 Apr 2018 09:48:12 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.731
X-Spam-Level:
X-Spam-Status: No, score=-0.731 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, HTTPS_HTTP_MISMATCH=1.989, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, T_DKIMWL_WL_HIGH=-0.01, T_DKIMWL_WL_MED=-0.01] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=fb.com header.b=lH3DDrYW; dkim=pass (1024-bit key) header.d=fb.onmicrosoft.com header.b=PoHQW8SV
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GU3SClTD-QeK for <quic@ietfa.amsl.com>; Wed, 25 Apr 2018 09:48:10 -0700 (PDT)
Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C6939128C0A for <quic@ietf.org>; Wed, 25 Apr 2018 09:48:10 -0700 (PDT)
Received: from pps.filterd (m0044008.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w3PGiUap019084; Wed, 25 Apr 2018 09:48:07 -0700
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : mime-version; s=facebook; bh=s50CQA4YA9hximDi8m7S1WfhnBbDd9bQ+V1cXuSTR7E=; b=lH3DDrYWXqstGElUlGz8XTROrD0TztZzwk2jAtQR0p9tVWstOpxhdKjdlx1ag7fd08Sl p78FnYuqyFpdIEKvO6vN4WwxJe150n364THsb8uMEyYDtkzdyu6mpKOTVgZxcez7JBqm P8/CxRocKXcyYmfVVDDI+MMdolbO7GvK6ss=
Received: from maileast.thefacebook.com ([199.201.65.23]) by mx0a-00082601.pphosted.com with ESMTP id 2hjucsre48-1 (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NOT); Wed, 25 Apr 2018 09:48:07 -0700
Received: from NAM03-CO1-obe.outbound.protection.outlook.com (192.168.183.28) by o365-in.thefacebook.com (192.168.177.30) with Microsoft SMTP Server (TLS) id 14.3.361.1; Wed, 25 Apr 2018 12:48:05 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.onmicrosoft.com; s=selector1-fb-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=s50CQA4YA9hximDi8m7S1WfhnBbDd9bQ+V1cXuSTR7E=; b=PoHQW8SVntwGuGQrX1XAFBG9OOK8QRnowfG60BeFNxiVqrWKYwyPfA5rB7W4uasM4diPDDBMeqZH/szSoPF2TwdmDiOBeC7yXfT50pHyUogosubaamWkdNr1LxUPVRVSRDYNA3Ad3o9J1OQ1a0keFrFCliKIxRuEcvyHlOPQdY8=
Received: from MWHPR15MB1821.namprd15.prod.outlook.com (10.174.255.137) by MWHPR15MB1520.namprd15.prod.outlook.com (10.173.235.17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.696.15; Wed, 25 Apr 2018 16:48:03 +0000
Received: from MWHPR15MB1821.namprd15.prod.outlook.com ([fe80::4569:2fbb:3aab:6126]) by MWHPR15MB1821.namprd15.prod.outlook.com ([fe80::4569:2fbb:3aab:6126%17]) with mapi id 15.20.0696.020; Wed, 25 Apr 2018 16:48:03 +0000
From: Subodh Iyengar <subodh@fb.com>
To: Ian Swett <ianswett@google.com>
CC: "ianswett=40google.com@dmarc.ietf.org" <ianswett=40google.com@dmarc.ietf.org>, Jana Iyengar <jri.ietf@gmail.com>, IETF QUIC WG <quic@ietf.org>
Subject: Re: ack delay in rx packets
Thread-Topic: ack delay in rx packets
Thread-Index: AQHT3KYtFrc5wMFDcECav0DMEKPj+aQRoboAgAAEmYCAAABT34AAAESigAAF1wCAAACaBg==
Date: Wed, 25 Apr 2018 16:48:03 +0000
Message-ID: <MWHPR15MB18219D3154C51A7B10C2D3F8B68F0@MWHPR15MB1821.namprd15.prod.outlook.com>
References: <MWHPR15MB1821770E698C99AAD160864EB68F0@MWHPR15MB1821.namprd15.prod.outlook.com> <CACpbDcd-N+KLECz9z1JQJwE_JXqq_4rNzP9i9Aq9itfRhGaPgA@mail.gmail.com> <CAKcm_gNU-yFCk0QzCrLivb96Wy07VnFpH59Vdut2kR_Ff9fjTg@mail.gmail.com> <MWHPR15MB18211154D0F967B0A31FC0EDB68F0@MWHPR15MB1821.namprd15.prod.outlook.com> <MWHPR15MB1821A5550632E66D8133F54CB68F0@MWHPR15MB1821.namprd15.prod.outlook.com>, <CAKcm_gOp9er6UR8e-RDjFMBT5Xtvuz1EexYPwfQWwYtQ7Ax=3A@mail.gmail.com>
In-Reply-To: <CAKcm_gOp9er6UR8e-RDjFMBT5Xtvuz1EexYPwfQWwYtQ7Ax=3A@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [2620:10d:c090:180::1:b93]
x-ms-publictraffictype: Email
x-microsoft-exchange-diagnostics: 1; MWHPR15MB1520; 7:4M3HBVOZznOInoViAbgYGQgD6vxqBfcjH2Z43PGgfFoNt8H8XcMTVl8dYOQKf6Ay6GbQ58JAhj2L3/AFjqnktqApY4sxwQXJfhfZco6LlpRdKvk9lT1q4XNywIyMmfZrGj7a2oN6eN266y6OJahMmd5PQWOWq6bBGcgcv4YJ/IDCI2A1faszHvGvbomKNxwORJwPI1g7N1O8bJbPySTDNhs4f5IpQOOsXrj43CWZwfNANBoNZW67m2yAleQgGgh7; 20:sfilPjVztrS0cMck/aEutozlDesbjXwm8lT0X1VNygcrFvT1pa0bCkxVs/fZ9s3UA4ZdwGoEi8S50sAjZCV3FzbFS6Vt6vL2LG6nY2JuAERw/SwQb/81ed54kU0Ht43Uv0u+XDfl9GDqdleBH7f8UDcoUQIxjjidko618hslLUk=
x-ms-exchange-antispam-srfa-diagnostics: SOS;
x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060)(7193020); SRVR:MWHPR15MB1520;
x-ms-traffictypediagnostic: MWHPR15MB1520:
x-microsoft-antispam-prvs: <MWHPR15MB15208488E92598837CC782FCB68F0@MWHPR15MB1520.namprd15.prod.outlook.com>
x-exchange-antispam-report-test: UriScan:(158342451672863)(10436049006162)(85827821059158)(265273979862326)(67672495146484)(211936372134217)(153496737603132);
x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(3231232)(11241501184)(944501410)(52105095)(93006095)(93001095)(10201501046)(3002001)(6041310)(20161123564045)(20161123558120)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(6072148)(201708071742011); SRVR:MWHPR15MB1520; BCL:0; PCL:0; RULEID:; SRVR:MWHPR15MB1520;
x-forefront-prvs: 06530126A4
x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(39380400002)(346002)(376002)(396003)(366004)(39860400002)(189003)(199004)(99286004)(478600001)(106356001)(606006)(76176011)(14454004)(81156014)(966005)(81166006)(7696005)(6436002)(2900100001)(33656002)(8676002)(6606003)(186003)(6916009)(97736004)(476003)(7736002)(19627405001)(6116002)(55016002)(53936002)(236005)(11346002)(54906003)(446003)(9686003)(68736007)(486006)(6306002)(2906002)(6506007)(6246003)(229853002)(54896002)(5250100002)(5660300001)(86362001)(59450400001)(3280700002)(102836004)(39060400002)(4326008)(74316002)(316002)(53546011)(46003)(105586002)(3660700001)(8936002)(25786009)(93886005); DIR:OUT; SFP:1102; SCL:1; SRVR:MWHPR15MB1520; H:MWHPR15MB1821.namprd15.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1;
received-spf: None (protection.outlook.com: fb.com does not designate permitted sender hosts)
x-microsoft-antispam-message-info: ZHiV24tS5c3HbZ4k744cPHx1Z2o70qPTYiLTfBV4TwEiYovo7CCFwOAui+MF6opEEFB50jpffOtu74EFl29e+fvg9MiabCI107YYtHJEEDx2yWAMf+pQ7b6ma491krKfckMqpXtuWAqESjCPXuSMOeMiwf/ut4KBPezY+n7f1hxNDS8ls+bn5wA9obZr5y0W
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative; boundary="_000_MWHPR15MB18219D3154C51A7B10C2D3F8B68F0MWHPR15MB1821namp_"
MIME-Version: 1.0
X-MS-Office365-Filtering-Correlation-Id: 5f19f119-b5be-4fdf-a223-08d5aacc500e
X-MS-Exchange-CrossTenant-Network-Message-Id: 5f19f119-b5be-4fdf-a223-08d5aacc500e
X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Apr 2018 16:48:03.4261 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR15MB1520
X-OriginatorOrg: fb.com
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-04-25_05:, , signatures=0
X-Proofpoint-Spam-Reason: safe
X-FB-Internal: Safe
Archived-At: <https://mailarchive.ietf.org/arch/msg/quic/HkFp3pXFFhysRpfId7lVIX2e08o>
X-BeenThere: quic@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <quic.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/quic>, <mailto:quic-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/quic/>
List-Post: <mailto:quic@ietf.org>
List-Help: <mailto:quic-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/quic>, <mailto:quic-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 25 Apr 2018 16:48:13 -0000

It's possible that both sides will TLP at the same time assuming their RTT estimates are the same and both sides are tails. To see whether this happens I'll have to actually look at data in practice to see whether we see whether there is a delay between the server's TLP/RTO and the client's TLP/RTO.

There is also another class case which in which it might be useful to retransmit the ack without a new packet where one side does not have a tail but the other does. For example, in the case of a revproxy where there is an upstream server. The client has sent a tail request to the revproxy, which it forwards to a webserver.


There are 2 cases here:

  1.  The webserver has some processing delay which is < 2 * RTT, so the response comes back and we have new data to send. The server could also bundle a duplicate ack with this packet.
  2.  The response is larger than the request, so we could have bundled some dup ACKs with the response.


It is possible that the client would set an ACK timer < RTO timer and then send an ACK for this new packet, however if a server strictly does not ACK acks,  the client would still have to wait for the RTO timer.


Subodh

________________________________
From: Ian Swett <ianswett@google.com>
Sent: Wednesday, April 25, 2018 9:32:40 AM
To: Subodh Iyengar
Cc: ianswett=40google.com@dmarc.ietf.org; Jana Iyengar; IETF QUIC WG
Subject: Re: ack delay in rx packets

I'm not sure it would be much better, but it's worth thinking through the case you're describing.

It must be that no 'retransmittable'(I hate that term) packets are being received, but you're sending packets.  If it's a one-way transfer, then you're receiving ACKs for the data you're sending at a regular rate.  If you bundled an ack with outgoing data every sometimes, the ack would have a new largest acked of the received ACK packets, so all is well.

If it's a long pause, such as a TLP or RTO, then you've waited more than an RTT and received no packets to acknowledge, but then you decide to bundle an ACK?  At that point, it's likely the peer has already TLP'd around the same time, so you're probably not saving them a retransmission.

It had never occurred to me to bundle an ACK in this case, but doing so does have the side effect of creating useless RTT estimates.  We could have a special value(already disliking this) for max_ack_delay which indicates the ACK should not be used for RTT estimates, but I'm not sure it's valuable to bundle an ACK with TLPs and RTOs.

On Wed, Apr 25, 2018 at 12:20 PM Subodh Iyengar <subodh@fb.com<mailto:subodh@fb.com>> wrote:

@Ian Swett<mailto:ianswett@google.com> sending ACKs in response to packets or delay seems fine, and an implementation might choose to do that. If we have more opportunities to send an ACK and avoid an RTO on the peer, wouldn't that be better? Is the objective of sending packets only during timer or recv data to simplify the ack delay calculation logic?


Subodh

________________________________
From: Subodh Iyengar
Sent: Wednesday, April 25, 2018 9:11:15 AM
To: Ian Swett; Jana Iyengar
Cc: IETF QUIC WG
Subject: Re: ack delay in rx packets


My concern is this situation: a "receiver" had not seen the original ACK (the packet was lost), then the sender rtos and sends a large ack delay to compensate for the rto, the max filter for max ack delay would inflate the delay for future packet.


if latest_rtt - min_rtt  > ack_delay:

   max_ack_delay = max(max_ack_delay, ack_delay)


latest_rtt in this case would be rto_timer + rtt because the sender sends an ack after an rto.


The rtt cal So max ack delay would become the rto timer. Is this a correct characterization?


Subodh


________________________________
From: QUIC <quic-bounces@ietf.org<mailto:quic-bounces@ietf.org>> on behalf of Ian Swett <ianswett=40google.com@dmarc.ietf.org<mailto:40google.com@dmarc.ietf.org>>
Sent: Wednesday, April 25, 2018 9:09:39 AM
To: Jana Iyengar
Cc: Subodh Iyengar; IETF QUIC WG
Subject: Re: ack delay in rx packets

Jana's right about ignoring the ack for RTT calculation if it doesn't acknowledge a new packet, see section 3.1: https://tools.ietf.org/html/draft-ietf-quic-recovery-11#section-3.1<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dietf-2Dquic-2Drecovery-2D11-23section-2D3.1&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=h3Ju9EBS7mHtwg-wAyN7fQ&m=vQP8b9xGAVAynYcZHdgaeS3tK7f1xcKRczLJy-IILfA&s=z__D443_wlVvybsHUk1ua_Ve_0v1RqsNH09OhpeVA7g&e=>

If it did acknowledge a new packet because the previous ACK was lost, then you end up with either a useless RTT, a useless max ack delay, or both.  Because of issues like this, I prefer sending acks in response to packets, or within an alarm of packet receipt, rather than at other times.

On Wed, Apr 25, 2018 at 11:53 AM Jana Iyengar <jri.ietf@gmail.com<mailto:jri.ietf@gmail.com>> wrote:
If the sender has seen a previous ack for the same largest received, then it should basically ignore this new ack. If not, the ack delay will be accurate... What concern do you see with having a large ack delay?

On Wed, Apr 25, 2018, 8:08 AM Subodh Iyengar <subodh@fb.com<mailto:subodh@fb.com>> wrote:

We ran into very common cases where we might have all our data to send already outstanding, so when a TLP and RTO alarm fires, we have no new data to send, so we resend data that was in previous packets.


In the time that the RTO fires, we might not have received any new packets, however want to send ACKs in this new packet as well. Since no new packets were received, this ACK will have the same largest acked as the previous packet.


One question came up as to what the ack delay of an ACK in such a rx packet should be. We could technically set it to the time since receipt of the packet, however with the new max_ack_delay computations, adding this might screw up our estimate of the normal ack delay, so I don't really want to do that. Is there a better option here, or does max_ack_delay not cover this case?


Subodh