Re: [tsvwg] [tcpm] Tech review of draft-grimes-tcpm-tcpsce-01.txt

Jonathan Morton <> Fri, 13 March 2020 01:38 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 4C1E03A0CFA; Thu, 12 Mar 2020 18:38:26 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.848
X-Spam-Status: No, score=-1.848 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id ey666xyJJUFr; Thu, 12 Mar 2020 18:38:24 -0700 (PDT)
Received: from ( [IPv6:2a00:1450:4864:20::12a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 12EFE3A0CFB; Thu, 12 Mar 2020 18:38:23 -0700 (PDT)
Received: by with SMTP id j15so6536422lfk.6; Thu, 12 Mar 2020 18:38:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=SVOe7G79ETy802Q7s4AZmW4PCSZdyiT+9tzK8b41Fxk=; b=k34znCiQBAIGBVhIHYIe87wDSI6if59zNbqFqBeTyfGDM2zNbgQ+3X3UzfNkfuo2da L56VqZIFrj6hbdJa3cBE7wsNpJhWZ3FXdcoAv14QPMPFB3GFBIThReHEQO6HFwiaWQ+o GZcordu8eRiYBfYKkYHvwnkcIvp2YYFISvHiEXbbQensNkeFm9t0O+RyjaIPpEpuXvai XRZixyuLK9KDbQMUHdMxNnCGUIXHwBFwT+DaknfaTB79Yho5xvxXeh8mHGyfDANOb2rs LzsTAx6V7YbRjL0YEsoOjmmsw2Zl6eJ/Bl/HLPpQm8nspNVjzsjLQQzZckXXJQNxgzlx v1hA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=SVOe7G79ETy802Q7s4AZmW4PCSZdyiT+9tzK8b41Fxk=; b=WpjWXEMHiPxy9Get9u/fYVmzYaazHoCuU+j82VJjAQ+Sn19tsEHMihAPk8jy6Yy2d5 Xxlh9GRzAvm0RQYQoDfVI7b1w8cXdxNB9ivBr3pWbPyJO1k8qcFxYDFlLqDMhZ0C56NQ ua4112Ey4aWV2PGs4q/E0PxwOLC6yJeWTbt6XZN41c2bziirhG0iWNG66ppWja3hyNun Yiewrmo28Bzk4N63IeFgthcfIieJkMDh21xHJq4+q8pW5z4AlnYkehygQnqmhGzO9usp zkzlrVBcKpeuraiYuDF3uqxD71om013Pr6j8v3VLLG1dNgIcIKOiM9WPdSYOeeghJwe5 dCfQ==
X-Gm-Message-State: ANhLgQ0EmMT7uhRZlSJqCo78DWc0p2ptcVU8ezDm9fcMzvAK8pMPXJto B/0RZNBNLtzJNAkb87csx2kGhkbC
X-Google-Smtp-Source: =?utf-8?q?ADFU+vvlT8zevfdrZYkRe5Fy7JQMHTZBqrTfgTr6W65x?= =?utf-8?q?VCcI4EPiOrCAK/myQ50LVCaJnC5BhtAoxg=3D=3D?=
X-Received: by 2002:a19:98a:: with SMTP id 132mr6764852lfj.171.1584063501925; Thu, 12 Mar 2020 18:38:21 -0700 (PDT)
Received: from jonathartonsmbp.lan ( []) by with ESMTPSA id f4sm28490210ljo.79.2020. (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 12 Mar 2020 18:38:21 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\))
From: Jonathan Morton <>
In-Reply-To: <>
Date: Fri, 13 Mar 2020 03:38:19 +0200
Content-Transfer-Encoding: quoted-printable
Message-Id: <>
References: <> <> <> <> <>
To: Bob Briscoe <>
X-Mailer: Apple Mail (2.3445.9.1)
Archived-At: <>
Subject: Re: [tsvwg] [tcpm] Tech review of draft-grimes-tcpm-tcpsce-01.txt
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Transport Area Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Fri, 13 Mar 2020 01:38:27 -0000

> On 13 Mar, 2020, at 2:03 am, Bob Briscoe <> wrote:
> Reminder: In Nov'19,  I carefully unpicked the descriptions of all the ideas in your draft, asking questions on the list where necessary, and provided a detailed review of whether it does what it says it aims to do (setting aside whether I agree with what it aims to do).
> The draft hasn't been updated since.
> The draft said more assessment was needed of each. It rejected one idea itself. I analyzed the other three, and concluded that they all had major problems. I spent some time structuring my review to make it easy to follow and included an exec summary and table assessing each idea against the goals given in the draft.
> You were going to "examine my arguments in detail".

Which we did, and concluded there wasn't a way (at that time) of answering your points in detail without sparking a massive and counterproductive flamewar.  So I'm not going to do that, but I have managed to extract a few salient points which I *can* address.

> The draft contains four tentative ideas for different semantics for the proposed feedback. There isn't any mechanism for negotiating which scheme to use, so I assume the intent is to eventually choose one.

Your assumption would be incorrect.

SCE is designed to accommodate flexibility in several areas, including the precise algorithms used to control the send rate, marking schedule, and feedback.  This draft, of course, covers the feedback.  Any algorithm which succeeds in reflecting the received SCE marks as ESCE flags with reasonable accuracy is acceptable.  The draft describes several such algorithms and tries to characterise their relative advantages, including the extent to which they maintain a desired Delayed Ack ratio.

The scheme actually implemented in our Linux-based reference code is the fourth algorithm, labelled Advanced.  This precisely matches segments' marks with their acks' flags, while still permitting Delayed Acks to function at both low and high marking ratios.  Really, this is the ideal.  The other schemes were proposed as being potentially easier to implement in some TCP stacks, with lesser performance.  (We might delete the Dithered scheme from a future version.)

Ack thinning, as you describe it, is something that happens on the reverse path, after the ack stream has left the receiver.  SCE's feedback scheme in TCP is unreliable, so naive ack thinning (that is, discarding acks without considering the potential loss of information in flags and/or options) can indeed lead to a temporary bias in the feedback signal as seen at the sender.  Acks can also be lost for other reasons, such as AQM action in the event of congestion, and this would have a similar effect in principle.

The important point is that SCE can tolerate this because the control loop remains closed, even if the feedback signal becomes systematically biased.  The AQM at the bottleneck is part of that control loop; it will adjust the marking rate as much as is needed to control the queue.  So even if the signal is so heavily biased by thinning that it appears to saturate at 100% or 0%, the system will be driven to a state where that bias is reduced, or even becomes impossible because it has reached 0% or 100% respectively.

This is part of what we mean when we say that SCE can tolerate a 100% relative error in the feedback.  The other part is that even if ESCE feedback fails entirely, we have a built-in failsafe in the form of CE marks and the reliable ECE/CWR feedback mechanism, as defined in RFC-3168.  This would cause a Multiplicative Decrease, which might technically be overkill for the actual congestion situation, but is conservative and safe.

So the result is a system of minimal complexity that can be implemented easily, requires minimal resources (one spare header bit) on the wire, and achieves closed-loop control even under less than ideal network conditions.

This is engineering, not an ivory tower.

As a final point, there is in fact a way to perform Ack Thinning which preserves nearly all of the information in the lost acks, by being selective about which acks are actually dropped.  This does require more careful inspection of consecutive acks, to see if the information in the first is adequately reflected in the information present in the second.  An example implementation of this is found in net/sched/sch_cake.c in our reference codebase, and completely avoids the bias effects that you postulate.

 - Jonathan Morton