Re: [Lsr] Congestion (flow) control thoughts. Thu, 30 April 2020 16:16 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 8B8C03A0C03 for <>; Thu, 30 Apr 2020 09:16:01 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.499
X-Spam-Status: No, score=-1.499 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FORGED_FROMDOMAIN=0.25, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.25, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=no autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 1DjVacfdqNSa for <>; Thu, 30 Apr 2020 09:15:59 -0700 (PDT)
Received: from ( [IPv6:2607:f8b0:4864:20::52a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 6A7F83A0BEB for <>; Thu, 30 Apr 2020 09:15:59 -0700 (PDT)
Received: by with SMTP id d17so2972942pgo.0 for <>; Thu, 30 Apr 2020 09:15:59 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=sender:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=sPrcvshB7vYO+PhDfwyWJHtyTJ28sE07TiPtUK3qCpc=; b=coYUHVDysZjB6r6+ezgR/2aarlbh6jnSJphH4ZnSZAJ7yHobpEmcPSDWETNhTbZKYZ 7u3lPPIyLTEw2r2j/yg/Qjk1u4Lyaw9MjP9FX6lkG1OynQJao1MfrpK3EcPLlSfG7oo4 QkQzfF76SnXF+XpGPQczEmrVE2p4pBUh/11O7+ZwdgG340cq6GS6eihwmymJonwYFFF3 F6TY9PDszN2fmJPuQZmyt7bZ7o+MJiT5s5M00Y3jeLkeu/1yKrgPO80fcC+mNZCXyqLA G9K3mD8lr4dBG/fnKb69+UuIYxV6FXKAwLhgFVYc4ldS4K9tvti8/tBP7EPdt4xlv0/4 hoaQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:sender:mime-version:subject:from:in-reply-to :date:cc:content-transfer-encoding:message-id:references:to; bh=sPrcvshB7vYO+PhDfwyWJHtyTJ28sE07TiPtUK3qCpc=; b=XQiawCCZLfuhD7IOh8Ihtf8FZ+2pyHlBdenAsw5xwqvnXrgJFv7iyyQTsi47aLSQ/O b2mFwTvY4eeIC9/OFcB/p3UdEzZX7IOyiHuZFgqB3Hx7dq0VyeoCfvuu6y0TmnHz1Pun 3FWhMW9ej1CFQw1NIdc0UAlDfCIAWzUReoQuYt1yX3HE3pDAgqUG9orl+IRzIanC2RlA d4ZL4czr3SRfKImTtgnchkMLDMYklKxQPqQB97FRt8wJ8CAPhmmg2915QvdqdXwKoa84 snTKybe150El+U3UGCK4Y/uqzv0uRIyJ7ECkvz6z3ZWs5xYfjbW+iryQaBhOlUseVs91 YPgQ==
X-Gm-Message-State: AGi0PubBhbLRPiAuuuB+i2F/YY4Q5tBFn4gAd1oAANtX+CdJmZSgJkSG aBqPBPVg2kGF9iwm/FsDo+g=
X-Google-Smtp-Source: APiQypIbfpGfQloGh9ZHRLpVwkLYuG6N6VDre+147MnihtUvA7bZQavNmZ4bZXgfjFKeFKKet+zcPw==
X-Received: by 2002:a05:6a00:c8:: with SMTP id e8mr82915pfj.206.1588263358766; Thu, 30 Apr 2020 09:15:58 -0700 (PDT)
Received: from [] ([]) by with ESMTPSA id b12sm206900pfd.165.2020. (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Apr 2020 09:15:57 -0700 (PDT)
Sender: Tony Li <>
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.\))
In-Reply-To: <>
Date: Thu, 30 Apr 2020 09:15:55 -0700
Cc: Christian Hopps <>, "" <>
Content-Transfer-Encoding: quoted-printable
Message-Id: <>
References: <> <>
To: "Gengxuesong (Geng Xuesong)" <>
X-Mailer: Apple Mail (2.3608.
Archived-At: <>
Subject: Re: [Lsr] Congestion (flow) control thoughts.
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Link State Routing Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Thu, 30 Apr 2020 16:16:08 -0000

Hi Xuesong,

> In congestion control of layer 4, it is assumed that there is a bottleneck in the network, and the ideal rate of the transmitters equals to a fair share of the bandwidth in the bottleneck. The flows in the network change all the time and so as to the ideal transmitting rate. There are some methods to detect the bottleneck, for example detecting packet loss, setting ECN, RTT and so on. Considering that the goal of cc is to maximize the throughput and minimize the queuing delay, the throughput and delay could be used to compare different cc algorithms.

I consider flow control and congestion control to be separate, but similar problems.

Flow control is about creating a single control loop between a single transmitter and single receiver.

Congestion control is about creating multiple interacting control loops between multiple transmitters and multiple receivers.

> The problem of mine is that for CC of ISIS flooding, where is the bottleneck?(maybe the receiver capability) What method of detecting could be used? (In my understanding, we have two options at this stage: one from the receiver side and the other from the transmitter side) What is the criteria of comparing? (still a little confusing for me )

In the case of modern IS-IS, we have gigabit links everywhere, the bandwidth greatly exceeds our needs, and flooding only happens between peers, so link resources are never the issue.  Thus, to my mind, congestion only happens when a node is engaged in flooding on multiple interfaces concurrently. Effectively, congestion control can be seen as flow control with shared resources, or as flow control with varying resources.

The resources of concern vary depending on the internals of both the sender and receiver.  Some that come to mind are:

- CPU. Both transmitter and receiver have finite cycles available. This is typically shared across multiple interfaces, so can congest.

- Process level buffer space. There may be a fixed number of packet buffers within the IS-IS process.  These can be depleted.

- Kernel buffers. 

- Internal control fabric. In some multi-chassis designs, the line card chassis is remote from the CPU running IS-IS. The network between the two can congest.

- Line card CPU. Again in the case of a multi-chassis system, control packets may flow through a CPU on the line card.  This can congest due to cycles or buffers.

- Forwarding plane buffers. Modern design for many systems has a forwarding plane ASIC doing a great deal of our packet handling. 
  One of its responsibilities is to extract control plane traffic from incoming traffic and pass it upwards. As a single ASIC, it is finite and can only buffer
  so many packets before loss.  The buffers are only cleared when the packets are read out by the line card or central CPU.  IMHO, this is a very likely point of congestion.

All of these can be instrumented to determine whether they are congesting.  It seems very unlikely that the transmitter ever congests.  Bruno’s data as presented supports this: the transmitter can outstrip the receiver. Thus, we tend to focus on congestion at the receiver.

The other feedback mechanisms that we have available are the ones that we’ve discussed: PSNPs provide acknowledgment of received packets. From that, and their timing, we may be able to infer more useful data, but we are discussing changing their behavior to make things more useful.

In our draft, we are proposing other feedback mechanisms.