Re: [tcpPrague] TSO burst sizing causing TCP Prague unfairness on high capacity links ?

Ashutosh Srivastava <as12738@nyu.edu> Wed, 17 June 2020 03:24 UTC

Return-Path: <as12738@nyu.edu>
X-Original-To: tcpprague@ietfa.amsl.com
Delivered-To: tcpprague@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id F0BDD3A0D94 for <tcpprague@ietfa.amsl.com>; Tue, 16 Jun 2020 20:24:27 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.099
X-Spam-Level:
X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=nyu.edu header.b=fgiI8rRP; dkim=pass (2048-bit key) header.d=nyu-edu.20150623.gappssmtp.com header.b=EcvB/QfS
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id q8Wkb1HgIsD8 for <tcpprague@ietfa.amsl.com>; Tue, 16 Jun 2020 20:24:26 -0700 (PDT)
Received: from mx0b-00256a01.pphosted.com (mx0a-00256a01.pphosted.com [148.163.150.240]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 3FDE63A0E05 for <tcpprague@ietf.org>; Tue, 16 Jun 2020 20:24:19 -0700 (PDT)
Received: from pps.filterd (m0094546.ppops.net [127.0.0.1]) by mx0b-00256a01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 05H3OJJ7063002 for <tcpprague@ietf.org>; Tue, 16 Jun 2020 23:24:19 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nyu.edu; h=mime-version : references : in-reply-to : from : date : message-id : subject : to : content-type; s=20180315; bh=n4wFB9UO/rnyXbwxdCAfpnoxz5Gu24F8YdBmVJaW5As=; b=fgiI8rRP3nbdMQ7YRGtoIkbNrDdACbOx1QHu6qehMP+6pLw4m1yqZHmPW0JY5UqcYbdW 0O8kgtGyXHXo9Z5XrF1C0sJwZ6sV7QU62pTeGbziBEGQnbbJdY3CX6rQK9+O9/1iXt8o TlErlif4OzR9ATnl27/LSmujTusLXnr15FZ5byhYJ4HU7vAIfj5Bm+GLS43C62/U7gMc kO1B0e3aeB4eyq2NrtIp7+6sqOC7XQ7oPAVPp414wr3m0Qr0pjDlNEMteD8b5pGLybiz PyaRZNpMmXVpziR4v61oGP3etg61pKBEHNvRc2v4fBXHHx8uEpBLvEbKDyWqR95PU94i 6Q==
Received: from mail-io1-f72.google.com (mail-io1-f72.google.com [209.85.166.72]) by mx0b-00256a01.pphosted.com with ESMTP id 31q64kk81w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for <tcpprague@ietf.org>; Tue, 16 Jun 2020 23:24:18 -0400
Received: by mail-io1-f72.google.com with SMTP id l204so874816ioa.4 for <tcpprague@ietf.org>; Tue, 16 Jun 2020 20:24:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nyu-edu.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=n4wFB9UO/rnyXbwxdCAfpnoxz5Gu24F8YdBmVJaW5As=; b=EcvB/QfSJN5EFeZyH9CIi2Tm5nqWcH4Ln00woX4x/2VAVvYzkyXoRLiSaSdC4ca2wb WygQHPo3iTK8AiCjw7NyiHVQB2AkNQ21hTlM66vyW8lYfamO0nEvOoy0ooHI1Ghfy+cv jjnoRa2qxnL9/1gwp195gr33iOmDHdzdx8yaQasaLkkTxjAJbuI1pBFiIMOayo0SUR8+ loo1NQ+iUhdkJmme0vrzUDXXMq/sXfUWPfUe+Fyd5lx49MZh8q/XNmNTf6yceDo9Tdst GoOz7sMnKl1P7KyVLcwQ0zXpLfnLmVBeiXo5t0e2OtrDc+rI8NL3Xwzo+9Fthnjin1LD NwdA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=n4wFB9UO/rnyXbwxdCAfpnoxz5Gu24F8YdBmVJaW5As=; b=bTKy/cZP8/z3wMLF0pOg7U8Bkam9VwisTyiwCzJmeYRwDbCEcTPfhNwM6kcVyYQiJQ SxDvXPg5KHhb2UJgmF6a9+Nuw0wlzH1fsts7rPrVFZJ3wB5OiqLK0DdGdoZKdEJY6NHq 8qJzGoAEncw2qosnFbIzK3V3A8ewNv9fpMS2+1KF4N78ijo5xe5jGEgYR4+dhPw4s2Jb VwbfsYmOKZ/R+2RELJkGPbse2LwOuRkumC60QkTFkwPZhbP9J3B++lyuSUkPAyuShsAC 8HIDXnyMZIwR6tMqaecQVpcuydulDfZ+hXx2bAuNjPpYutAzHmGzvRLaLFo0mKuGhIwg aA1Q==
X-Gm-Message-State: AOAM533JQYSw/z1fhjNzsj0TyMPHfpwpn5xPzEKjKSN6qMpRSjrWOhhp uZT9xOTsp7wKcBCZy3/klQt35Zhhzh6zkBCuwHqnf7+D5YFRbMHQNjuQRZWNNsxZHt9nfBGN6Fr 1cFnbGCYQ3JxzchtV5n6L32zruA==
X-Received: by 2002:a05:6e02:c62:: with SMTP id f2mr6524648ilj.43.1592364257895; Tue, 16 Jun 2020 20:24:17 -0700 (PDT)
X-Google-Smtp-Source: ABdhPJwREUnsEh5knZIKtbUIUXlvSdYhMlus/cvFcpTFUNSyD8HorsEkLbtUek/9WngxgQKJxTDzT5bhg3lFZZDB+/4=
X-Received: by 2002:a05:6e02:c62:: with SMTP id f2mr6524632ilj.43.1592364257677; Tue, 16 Jun 2020 20:24:17 -0700 (PDT)
MIME-Version: 1.0
References: <CAJyCXab5M=hUaORAeQs5NO3W-rDYPe6r5j6Wyx6q=Bxz4GEzvA@mail.gmail.com> <AM0PR07MB46418F029644832BBC06D260E0830@AM0PR07MB4641.eurprd07.prod.outlook.com>
In-Reply-To: <AM0PR07MB46418F029644832BBC06D260E0830@AM0PR07MB4641.eurprd07.prod.outlook.com>
From: Ashutosh Srivastava <as12738@nyu.edu>
Date: Tue, 16 Jun 2020 23:24:06 -0400
Message-ID: <CAJyCXaafO_uzoBBd0z6pWupqNeSsJMoUHgBVNhtev9Fg1Hh6cg@mail.gmail.com>
To: "Tilmans, Olivier (Nokia - BE/Antwerp)" <olivier.tilmans@nokia-bell-labs.com>, tcpprague@ietf.org
Content-Type: multipart/alternative; boundary="000000000000eeabdb05a83f3177"
X-Orig-IP: 209.85.166.72
X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 spamscore=0 clxscore=1015 phishscore=0 impostorscore=0 mlxlogscore=999 mlxscore=0 malwarescore=0 bulkscore=0 adultscore=0 lowpriorityscore=0 cotscore=-2147483648 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2004280000 definitions=main-2006170026
Archived-At: <https://mailarchive.ietf.org/arch/msg/tcpprague/usJHTaVYtVu3feScH4XMC-CxMQc>
Subject: Re: [tcpPrague] TSO burst sizing causing TCP Prague unfairness on high capacity links ?
X-BeenThere: tcpprague@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." <tcpprague.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tcpprague>, <mailto:tcpprague-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tcpprague/>
List-Post: <mailto:tcpprague@ietf.org>
List-Help: <mailto:tcpprague-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tcpprague>, <mailto:tcpprague-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 17 Jun 2020 03:24:28 -0000

Hi Olivier,

Please find comments inline.

Could you confirm that this is also happening with the tip of the current
> Branch? I.e., 3b63cc0
>

Yes, I observe these unfairness issues with the tip of the current branch (
3b63cc0). This happens both in the case of a flow entering after some gap
or with multiple flows starting together.

- Confirm that this happens with both the internal TCP pacing and the fq
> qdisc as pacer on the data sender
>
Yes , it happens with both  with internal TCP pacing and fq pacing.

- Confirm that this happens both with and without gro/gso/tso/lro on the
> endhosts (data sender and receiver, and also the aqm node as fq does not do
> gro splitting prior to enqueue)?
>

This *does not happen* when I turn off gro/gso/tso on all the nodes (end
hosts, AQM) in my experiment.

- Confirm that the problem persists if you increase the base RTT--a simple
> netem qdisc on the reverse path to add 2-5ms should be sufficient, do not
> forget to disable gro/gso as that poorly interacts with netem.
>

The problem persists even with some added delay.


> - Log the reported CE marks by the AQM, as well as the
> delivered_ce/received_ce counters throughout the experiment on the data
> sender/receiver?
>

I have uploaded  additional data for experiments with commit 3b63cc0 on the
google drive I shared earlier :
https://drive.google.com/drive/folders/1pLC0dcMF0-M1cgtw9IhoFiYOMvFJ-cc7?usp=sharing

The columns of the AQM csv file are : timestamp, dropped packets, queue
backlog (bytes), queue backlog (packets) , CE_marks
Columns of the ss data file at the sender are : timestamp, ID, cwnd, SRTT
(ms), delivered, delivered_ce, pacing rate (Mbps)

I have also added the raw data files. The experiment settings were the same
as in my last mail.

Thank you,
Ashutosh.