[tcpm] Re: Some comments on trimming in TCP

"Costin RAICIU (76898)" <costin.raiciu@upb.ro> Wed, 29 July 2026 08:38 UTC

Return-Path: <costin.raiciu@upb.ro>
X-Original-To: tcpm@mail2.ietf.org
Delivered-To: tcpm@mail2.ietf.org
Received: from localhost (localhost [127.0.0.1]) by mail2.ietf.org (Postfix) with ESMTP id 5A7D61204F2DB for <tcpm@mail2.ietf.org>; Wed, 29 Jul 2026 01:38:56 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ietf.org; s=ietf1; t=1785314336; bh=jL4wWMcK4Q8rxlScRAikpgfQLUDlwFK848e7cLNApWA=; h=From:To:Subject:Date:References:In-Reply-To; b=lgkCrggvAumIyPv4WYnwmmGNPz8gkbpd0vbx5NRBU7zGTTK1ZXWFYJZqO6hlsHYVq 1zGIiQEDUvv3QSNkzwbOFIsPqxlDDEs3LlfwrtzawMoZoK0gnMQIeQqNXUSCKmh/wR j8vgsK4ALzJm1XwgPndOaM75wV8KAn3bhJqqbLlA=
X-Virus-Scanned: amavisd-new at ietf.org
X-Spam-Flag: NO
X-Spam-Score: -2.098
X-Spam-Level:
X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: mail2.ietf.org (amavisd-new); dkim=pass (2048-bit key) header.d=upb.ro
Received: from mail2.ietf.org ([166.84.6.31]) by localhost (mail2.ietf.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SxqBJh5ZKDlP for <tcpm@mail2.ietf.org>; Wed, 29 Jul 2026 01:38:55 -0700 (PDT)
Received: from AS8PR04CU009.outbound.protection.outlook.com (mail-westeuropeazon11021141.outbound.protection.outlook.com [52.101.70.141]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature ECDSA (P-256) server-digest SHA256) (No client certificate requested) by mail2.ietf.org (Postfix) with ESMTPS id A71C31204F2D2 for <tcpm@ietf.org>; Wed, 29 Jul 2026 01:38:55 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=W4hU6PbBqbvJI038MEwA7VxvXCvPdV0Yur+nuZT0DA5dv9HCLZRjD0Kx+EM6bMlvuc1+Bh7Mz+tL4gZR8eDmqLgX9cLGf+2vBWh6L2xLfBx7ZV+rIZPXk6QjFFc8HOYw7zxRNx8tFE9i7z3DsZ+UysdeNlD58D16mzbZ2z/CgsNiDtuUmRilDA4A4jAYiPJ68G803zetF0kxN1VEshsLDmwtLbJ0evZXH1H9hg5V2frzTenCE4yZoXF68sFUNyheIjDl04bvQEk5P3EuX24fL4BsRbQm9wNttn7PLfnj14dVkIZcNFJAN89CNl8SwtEGURP+anQe3J1Uxf7xWFXMIg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jL4wWMcK4Q8rxlScRAikpgfQLUDlwFK848e7cLNApWA=; b=fGYlDbDNcNVXM2X+3cAq9y6y8llsK3UEjqGMRJ+e4s26XQmvHFXvsM22p1YZNT/oifJctbON3TqvCEd6K9BoV4P6SbyxrH98pYFtvBsP7pMjLR97fY9r7K4CwK9JspfoulBdtGG74e4o/suLU77fElGegPblYSlpV8TnYxCNhiSdp2p6FaNP83ZJyx4rAKdfAcJgABGhFDsFtWnOYQkmmxZRgQ73O+GZp5xIH2uyMxWFBnoChnDXT5aW0ZVDtWUdTlPBqPn8E+arZnYqVhMJ5jmo6El+A5lb/AMkUWk3ML0GxbUj3VgU/zPnLzLMSfl0Nz1LdrM3u03HbXjNcoZKfA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=upb.ro; dmarc=pass action=none header.from=upb.ro; dkim=pass header.d=upb.ro; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=upb.ro; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jL4wWMcK4Q8rxlScRAikpgfQLUDlwFK848e7cLNApWA=; b=k72gieOQuGNenLDbcf8OSV8fTgEy3emQflTEd6I8K1oOPwQVYQatP/HY9/No/kK4d+WOgncXejKYjGzXoLTpRjFjxsQM0uWm9mQ2Ptmc8CExXp5fk4+RT+/5CCdwdesRCYgEnlCl+trzJXIsO+o9PW8/BEA3fRWn3CNkPywPKePYz1SXbp6CLHnqtWOv/fNDZ/BVNcUEkSQpB9p4AQ+tx3Z9Oa0Um2jVuv4eiKMOvBaBnyaOai5hsWqyYRxqOSs6N4cbXjWNSPNSUB3siWgU6NlCJ1udgDlx8qoKm3CKP46w89AGk5J9Z3coHNqEYaUpsIvsuBCj+1f4R0c1NhJM6A==
Received: from PA4PR01MB10231.eurprd01.prod.exchangelabs.com (2603:10a6:102:268::10) by MRWPR01MB12664.eurprd01.prod.exchangelabs.com (2603:10a6:501:8e::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.245.13; Wed, 29 Jul 2026 08:38:47 +0000
Received: from PA4PR01MB10231.eurprd01.prod.exchangelabs.com ([fe80::ec1f:9f4:5a11:171b]) by PA4PR01MB10231.eurprd01.prod.exchangelabs.com ([fe80::ec1f:9f4:5a11:171b%4]) with mapi id 15.21.0270.009; Wed, 29 Jul 2026 08:38:47 +0000
From: "Costin RAICIU (76898)" <costin.raiciu@upb.ro>
To: "tcpm@ietf.org" <tcpm@ietf.org>
Thread-Topic: [tcpm] Some comments on trimming in TCP
Thread-Index: AQHdHyZTqwkOncyw3UOxIPSMd3+gyraELP4A
Date: Wed, 29 Jul 2026 08:38:47 +0000
Message-ID: <52F14420-3D89-46A1-85AF-708EF89A6FCC@upb.ro>
References: <8FC600C2-1D00-4EBB-B873-1FCF09FEA537@upb.ro>
In-Reply-To: <8FC600C2-1D00-4EBB-B873-1FCF09FEA537@upb.ro>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=upb.ro;
x-ms-publictraffictype: Email
x-ms-traffictypediagnostic: PA4PR01MB10231:EE_|MRWPR01MB12664:EE_
x-ms-office365-filtering-correlation-id: 84f34447-d00e-43b7-d320-08deed4cce7b
x-ms-exchange-senderadcheck: 1
x-ms-exchange-antispam-relay: 0
x-microsoft-antispam: BCL:0;ARA:13230040|4022899009|1800799024|366016|786006|23010399003|376014|56012099006|10067099003|11063799006|3023799007|22082099003|18002099003|38070700021;
x-microsoft-antispam-message-info: fbLQ6SIzyzRbdFUpw519YTluQoAZBYo4UMoDJ81Y0M7MKzhEAhmLRuMBhm0e5uQxMUnYGK032ECk1RK0pEzVjXeqhDNCQva6QW7IsS4FIyRTFn1z1/kSKumAwPwVROAoLry+ybNZddZuf4i4xuT/7e5TL1hC0ajFHe1Uvz8EUx1WRC/slpqjH+NQfBE1G2kCrz0aB8f3CsN3Re6fKmewRHS1N7RX0Ky9r+u3kdAX2G5DEmrOe5JDLCTEHzoCT4wCXVvkvqHbzGOAreBAXAt3EMk6uY8jHW6whQQte68DBklokIV2uCbD2sei8y/btG4cAEoIrPWXaJbRPuop6KWOHRwoFQLV/eRhuUfFIrM9vWEGen5Sg2s0Yrv8IAaPLDKh6UZW1ZWEQwxva+SAh16avbDzGKnWVf5/V/V57ggS0nRfJ4vcr4Pvf3GUySz1caUK/NVkzoH6m2rc6NSbwUkoCD3ByI+ArH8yCQKYWTkqvvsayVml4QeqIVIaLZDA8/6MLIx8Yh7XtyDmPqDdOUopq6naC2ZkatcLHbTTSDHH/Qn5ChETIexqH2qMAXU4K/WvLgpLqrt0IxNiWySpVAN2aw89ACBrr3I8mTHypw8q8JWsEcL64kz+5S4t1HPGE9wi59E2DOM/a8TTh026y4BlW21qX73v1i0+pNNzqEIeWLwmeusG7ZcQGaOQgjHjpxSSXE7UdhrdpbtHc+I0Zk+Xfo6jwsbMugNOW4qOWdxO3fs=
x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PA4PR01MB10231.eurprd01.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(4022899009)(1800799024)(366016)(786006)(23010399003)(376014)(56012099006)(10067099003)(11063799006)(3023799007)(22082099003)(18002099003)(38070700021);DIR:OUT;SFP:1102;
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: Shm1YQ0+s/YovxmiWyHE6Vh2Tf8naCTn1kfj+hu+U+y6FycS9MJHJMu2U3z8r1rXaSDCKPRejoynBpHRXiW6lagMtByGwlD0UMDV+Pt91lSZto2c1O0nsDT+8IuxlmmdyrxN3qZPZH0duklGIsqnn7heWtLVHfAUJ0blkoo0PDbrvIIvQ2oIuJqEkiXdMjxZ0JLDbIKntgJnXddoE7bNRSL7k8zD3VBXzCPo/3GWmVsNgcTrB7dMXtzdl2O5ZkGf387BHA9dOtODK7Fj4v82ZJA9r58Dwn62PsPrxOyokHcUkMiqGlSEqXF8YmL7JaP6rWyWS3OZSQ24B/WqhGMVnfxnoRHjuhQzHeZssXYCpu058O+z6EuB2Ba2eNmO++VHuUTcxj8EqCaK3LLnTLOpBLDFN4dlxPr4Q/wKHZA0IhICYtrp2bdNiKHcjgq3hq3ELcPHc+PngL2Bsdk9YVjG7OIP+7Rm5Dv0utnK9eiO+bXcDRm+/kVFRkfdum0TICAba3XXGi+oUbHgOPde5LjI1zf69RCBLqxmWpYYO1BTVtpCKcLqxvJOtpbpHj4je4xKV0cZ+OSn2cAnubZ5w9/DdPPGGPsXeLeXFPOQMj1bW94u/XH9fNOS4P0eTVGBVDR/MNE97CsFVHtl6QIuF82/UsSimnp9F0VxFS3rnwgj6YE6xRf6NXG8GHtGt7qR5S05bbPCYaF2Lljf+gTgLcRXhvVc319mJMT6y/oKRMpCh1omPWk3LoIGbEMew6MykVpyvwLyADqScQ0dq1rOdqHGKtgyY1/0EVEr4rKDxbxBS1iFBRsMAmHgcPL17PDIhkq5dIO/D2CvUGQBnwne9JMzBNwtAakb+rX+l35kHd5caFKNUUc3RG6iaQv0NCbpccl/q1gmqC2/YDEk36joadbcdrXygLwwriMD5qbvk/M4S/MV7y6tT6rA/XlP1k1Kx7AoW9g/wpGJsllcFmiFKOchyQqx8ViYk8RUVxMiT3nE5e0RcDvZlRWWHR4EgCfdqUFVfRSiNjn3jrWx+elUv8a3vUWAIKhCVbJBgVPLuklB/M96iYdWI5aWMVuwXSf/gs2uuXwu8LcVhSXZqUbtX3M3Qcytsq6+e33OczzvAaLUl2wO0kgX/K+kuxjJghlO/aUCKWqaFdaaiME5Aji0rjve9VcP2rd9Qf/EbqZsiZSLA0uTuLnA5kYtsjebFAgvi3rJph9e3JdFSeQRNLUMVh49fWsWHyG30+yLxAoD5GqG+xCz8c3T45l7p90eMRZ028LdIAS/PTwww00VAzj9XcCi3s9aB+79piF/x+CLGViXKyXG0vKH6s7nT2fPzStrWtnGk6WFiy8AEhBPILNazyYek3Dj05L7SyzeBNRp8T8E/aArZZp/fexjtT/2B1RnvOWd0cePORld7Ds+abR+ZDD+OFA2cwAdcC+0aQScmMLN967Qr34ZzInWzwEKJTKfLDeLg+4R7G+A8hztWIkAszgRi8OV0XYDlx/iMgQvXx1iExlOzOPymXGJwRlzPDEKpxdbAH4YxqW4Z83fs50CEgwVZP0rpBUqK9PQFqW2jlTFAC95LwjG3EW1PCSKtRsYEHuJDR7booCjcdh2qUyR+zP/yxZC14YbSIHXqlO/02tjozjd/XgnFlCoFwA+n341lmC3D+w6faFLAyFJIEOvUC3AU/TRPMP7DAtN1GdUdRkVa5Xte95x0QgLXSEyhAgZWmZOcniFElfW4gG1jL3jXewfFg==
Content-Type: text/plain; charset="utf-8"
Content-ID: <320F4675BFF762489D63240F268A128C@eurprd01.prod.exchangelabs.com>
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-OriginatorOrg: upb.ro
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PA4PR01MB10231.eurprd01.prod.exchangelabs.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 84f34447-d00e-43b7-d320-08deed4cce7b
X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Jul 2026 08:38:47.0494 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 2d8cc8ba-8dda-4334-9e5c-fac2092e9bac
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: vjOKoAHe1YcR+HyegKGioOFzbi2pVveZPZAyJU8NF2YqNxejIvjgDCXEpOuNkxtC49ITr1b2Wb0XnJf7LhwoPA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MRWPR01MB12664
Message-ID-Hash: VIGDHPAZTJOKILSGOB2NFMMWADNUCWGF
X-Message-ID-Hash: VIGDHPAZTJOKILSGOB2NFMMWADNUCWGF
X-MailFrom: costin.raiciu@upb.ro
X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-tcpm.ietf.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header
X-Mailman-Version: 3.3.9rc6
Precedence: list
Subject: [tcpm] Re: Some comments on trimming in TCP
List-Id: TCP Maintenance and Minor Extensions Working Group <tcpm.ietf.org>
Archived-At: <https://mailarchive.ietf.org/arch/msg/tcpm/hDz98y-FOOXZJyt_t1T2ri8eQdI>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tcpm>
List-Help: <mailto:tcpm-request@ietf.org?subject=help>
List-Owner: <mailto:tcpm-owner@ietf.org>
List-Post: <mailto:tcpm@ietf.org>
List-Subscribe: <mailto:tcpm-join@ietf.org>
List-Unsubscribe: <mailto:tcpm-leave@ietf.org>

Two additional points:

5. Trimming and ECN co-existence:

Trimming is orthogonal to ECN. ECN marking is performed when the queue size is below the trim threshold; trimming happens when it exceeds the trim threshold. 

The trimming spec says that trimmed packets SHOULD NOT be subject of ECN marking either at the trimming switch or subsequent switches. 
This ensures that the CE bits of the original packet are preserved, indicating (additional) congestion at a hop prior to the trim point. Congestion controllers could take this extra information into account when reducing the rate, but current controllers do not (e.g. NSCC).

6. Aggressive buffer sizing with trimming makes networks more predictable

A key benefit of trimming is that it allows operating networks with aggressively small trim thresholds, on the order of 1BDP.   This is in contrast to existing best practices where alpha sharing is used to cope with incast (e.g. in DCTCP) and to allow the CC to reduce the rate while avoiding massive packet loss. 

The current approach has the downside that queues can grow to many BDPs in transient congestion, which has two effects: (1) significantly increases latency for unrelated “victim" traffic going through the same queues and (2) create switch buffer pressure that results in losses on other traffic classes too (e.g. higher priority ones).

With queues capped at ~1BDP, such performance coupling between different flows and different traffic classes will be reduced.

Best.
Costin

> On 29 Jul 2026, at 09:48, Costin RAICIU (76898) <costin.raiciu@upb.ro> wrote:
> 
> Hi all,
> 
> Just re-joined the tcpm mailing list, catching up with the activity related to the trimming draft.
> 
> Here are some further clarifications addressing the more substantive comments (if I missed major point, please let me know).
> 
> 1. Design of trimming and scope of usage
> 
> The key design goals for trimming were to be transport protocol independent and simple to implement in hardware. Also, trimmed packets should look like normal IP packets (with valid checksum and IP length) to avoid them getting dropped downstream of the trim point. Using two DSCP code points for signalling was the simplest approach we could think of that works in datacenters, hence this is the approach we took at the UEC. Using trimming in the Internet at large was not a stated goal of that effort, hence we didn’t have to worry about the implications of using DSCPs across different administrative domains.
> 
> We are fully aware that using DSCP codepoints is not the most elegant design, but given the constraints this was the result.
> 
> Trimming as standardised is now available in switching hardware from major vendors, and is a feature on the roadmap for all future devices as far as I know. 
> Currently, trimming is getting deployed in backend (GPU scale-out) networks where the transport is MRC and in the future UEC. But the same switching hardware is also used for the frontend network which connects the CPUs and storage.
> 
> 2. Is TCP relevant for AI datacenters?
> 
> TCP is quite heavily used in AI datacenters, especially in the frontend network. Some hyperscalers use it for storage (NVMe over TCP).  When training across buildings (called scale-across, RTTs of hundreds of us to ms) some people are planning to use TCP or MPTCP for this purpose.
> 
> 3. What is the major advantage for having TCP support trimming?
> 
> Trimming enables deterministic, one-RTT loss detection in almost all practical cases including app-limited traffic, sprayed traffic, etc. All the current loss detection mechanisms have some corner case they do not cover (e.g. they assume in-order delivery, they require the ack clock to work after the lost packet, or they need packet timers at the end of a burst, etc, etc).  Trimming can however still fail to detect losses when the number of trims is so large that they overflow the trim queue (e.g. very large many to one traffic lasting for one RTT or more).
> 
> Trimming helps most in incast-like workloads, for instance storage traffic. When incast exists (synchronised many to one traffic) you have synchronised bursts of loss which are difficult to handle with heuristic loss detection. Trimming really helps here.
> 
> In general, trimming enables: (1) line-rate start for connections because the cost of getting it wrong is very small (retransmissions are cheap if you know exactly what was lost). This will provide much better performance because you can essentially  skip slow start for datacenters, and start at line rate instead; (2) packet spraying where each packet is randomly placed on a separate path by ECMP hashing in the switches for instance. Packet spraying reduces flow collisions and improves network utilisation; it is absolutely critical for GPU networking were flows quickly reach line rate, but it will help CPU networks too were TCP is used. Compared to Multipath TCP, TCP with packet spraying is much simpler (much less state, single CWND, single sequence space) and handles short flows too - whereas MPTCP requires an additional RTT to setup subsequent sub flows, sprayed TCP can use multipath from the get.
> 
> Note that we are not aiming to standardise sprayed TCP or line-rate start in TCP in this work - but adding trimming opens the door for there enhancements in the future.
> 
> 4. TCP Trimming design optimisations.
> 
> We are happy to use the feedback and collective wisdom of the WG here. 
> 
> Re Yoshi’s comment about signaling multiple trims in a NACK packet: generating a NACK for each trim is the simplest option from an implementation point of view, but it does generate more control packets. We could allow multiple trims per NACK and implementations can then decide how many they want to encode per NACK.
> 
> Finally, I will add that we have a Linux kernel implementation of TCP trimming support that we are happy to release open-source and will work to upstream to the Linux kernel if this standardisation effort progresses.
> 
> Thanks.
> Costin
> 
> 
> _______________________________________________
> tcpm mailing list -- tcpm@ietf.org
> To unsubscribe send an email to tcpm-leave@ietf.org