Re: [tcpPrague] TSO burst sizing causing TCP Prague unfairness on high capacity links ?

Ashutosh Srivastava <> Wed, 17 June 2020 03:24 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id F0BDD3A0D94 for <>; Tue, 16 Jun 2020 20:24:27 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.099
X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key) header.b=fgiI8rRP; dkim=pass (2048-bit key) header.b=EcvB/QfS
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id q8Wkb1HgIsD8 for <>; Tue, 16 Jun 2020 20:24:26 -0700 (PDT)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 3FDE63A0E05 for <>; Tue, 16 Jun 2020 20:24:19 -0700 (PDT)
Received: from pps.filterd ( []) by ( with SMTP id 05H3OJJ7063002 for <>; Tue, 16 Jun 2020 23:24:19 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; h=mime-version : references : in-reply-to : from : date : message-id : subject : to : content-type; s=20180315; bh=n4wFB9UO/rnyXbwxdCAfpnoxz5Gu24F8YdBmVJaW5As=; b=fgiI8rRP3nbdMQ7YRGtoIkbNrDdACbOx1QHu6qehMP+6pLw4m1yqZHmPW0JY5UqcYbdW 0O8kgtGyXHXo9Z5XrF1C0sJwZ6sV7QU62pTeGbziBEGQnbbJdY3CX6rQK9+O9/1iXt8o TlErlif4OzR9ATnl27/LSmujTusLXnr15FZ5byhYJ4HU7vAIfj5Bm+GLS43C62/U7gMc kO1B0e3aeB4eyq2NrtIp7+6sqOC7XQ7oPAVPp414wr3m0Qr0pjDlNEMteD8b5pGLybiz PyaRZNpMmXVpziR4v61oGP3etg61pKBEHNvRc2v4fBXHHx8uEpBLvEbKDyWqR95PU94i 6Q==
Received: from ( []) by with ESMTP id 31q64kk81w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for <>; Tue, 16 Jun 2020 23:24:18 -0400
Received: by with SMTP id l204so874816ioa.4 for <>; Tue, 16 Jun 2020 20:24:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=n4wFB9UO/rnyXbwxdCAfpnoxz5Gu24F8YdBmVJaW5As=; b=EcvB/QfSJN5EFeZyH9CIi2Tm5nqWcH4Ln00woX4x/2VAVvYzkyXoRLiSaSdC4ca2wb WygQHPo3iTK8AiCjw7NyiHVQB2AkNQ21hTlM66vyW8lYfamO0nEvOoy0ooHI1Ghfy+cv jjnoRa2qxnL9/1gwp195gr33iOmDHdzdx8yaQasaLkkTxjAJbuI1pBFiIMOayo0SUR8+ loo1NQ+iUhdkJmme0vrzUDXXMq/sXfUWPfUe+Fyd5lx49MZh8q/XNmNTf6yceDo9Tdst GoOz7sMnKl1P7KyVLcwQ0zXpLfnLmVBeiXo5t0e2OtrDc+rI8NL3Xwzo+9Fthnjin1LD NwdA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=n4wFB9UO/rnyXbwxdCAfpnoxz5Gu24F8YdBmVJaW5As=; b=bTKy/cZP8/z3wMLF0pOg7U8Bkam9VwisTyiwCzJmeYRwDbCEcTPfhNwM6kcVyYQiJQ SxDvXPg5KHhb2UJgmF6a9+Nuw0wlzH1fsts7rPrVFZJ3wB5OiqLK0DdGdoZKdEJY6NHq 8qJzGoAEncw2qosnFbIzK3V3A8ewNv9fpMS2+1KF4N78ijo5xe5jGEgYR4+dhPw4s2Jb VwbfsYmOKZ/R+2RELJkGPbse2LwOuRkumC60QkTFkwPZhbP9J3B++lyuSUkPAyuShsAC 8HIDXnyMZIwR6tMqaecQVpcuydulDfZ+hXx2bAuNjPpYutAzHmGzvRLaLFo0mKuGhIwg aA1Q==
X-Gm-Message-State: AOAM533JQYSw/z1fhjNzsj0TyMPHfpwpn5xPzEKjKSN6qMpRSjrWOhhp uZT9xOTsp7wKcBCZy3/klQt35Zhhzh6zkBCuwHqnf7+D5YFRbMHQNjuQRZWNNsxZHt9nfBGN6Fr 1cFnbGCYQ3JxzchtV5n6L32zruA==
X-Received: by 2002:a05:6e02:c62:: with SMTP id f2mr6524648ilj.43.1592364257895; Tue, 16 Jun 2020 20:24:17 -0700 (PDT)
X-Google-Smtp-Source: ABdhPJwREUnsEh5knZIKtbUIUXlvSdYhMlus/cvFcpTFUNSyD8HorsEkLbtUek/9WngxgQKJxTDzT5bhg3lFZZDB+/4=
X-Received: by 2002:a05:6e02:c62:: with SMTP id f2mr6524632ilj.43.1592364257677; Tue, 16 Jun 2020 20:24:17 -0700 (PDT)
MIME-Version: 1.0
References: <> <>
In-Reply-To: <>
From: Ashutosh Srivastava <>
Date: Tue, 16 Jun 2020 23:24:06 -0400
Message-ID: <>
To: "Tilmans, Olivier (Nokia - BE/Antwerp)" <>,
Content-Type: multipart/alternative; boundary="000000000000eeabdb05a83f3177"
X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 spamscore=0 clxscore=1015 phishscore=0 impostorscore=0 mlxlogscore=999 mlxscore=0 malwarescore=0 bulkscore=0 adultscore=0 lowpriorityscore=0 cotscore=-2147483648 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2004280000 definitions=main-2006170026
Archived-At: <>
Subject: Re: [tcpPrague] TSO burst sizing causing TCP Prague unfairness on high capacity links ?
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 17 Jun 2020 03:24:28 -0000

Hi Olivier,

Please find comments inline.

Could you confirm that this is also happening with the tip of the current
> Branch? I.e., 3b63cc0

Yes, I observe these unfairness issues with the tip of the current branch (
3b63cc0). This happens both in the case of a flow entering after some gap
or with multiple flows starting together.

- Confirm that this happens with both the internal TCP pacing and the fq
> qdisc as pacer on the data sender
Yes , it happens with both  with internal TCP pacing and fq pacing.

- Confirm that this happens both with and without gro/gso/tso/lro on the
> endhosts (data sender and receiver, and also the aqm node as fq does not do
> gro splitting prior to enqueue)?

This *does not happen* when I turn off gro/gso/tso on all the nodes (end
hosts, AQM) in my experiment.

- Confirm that the problem persists if you increase the base RTT--a simple
> netem qdisc on the reverse path to add 2-5ms should be sufficient, do not
> forget to disable gro/gso as that poorly interacts with netem.

The problem persists even with some added delay.

> - Log the reported CE marks by the AQM, as well as the
> delivered_ce/received_ce counters throughout the experiment on the data
> sender/receiver?

I have uploaded  additional data for experiments with commit 3b63cc0 on the
google drive I shared earlier :

The columns of the AQM csv file are : timestamp, dropped packets, queue
backlog (bytes), queue backlog (packets) , CE_marks
Columns of the ss data file at the sender are : timestamp, ID, cwnd, SRTT
(ms), delivered, delivered_ce, pacing rate (Mbps)

I have also added the raw data files. The experiment settings were the same
as in my last mail.

Thank you,