Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

"Jakob Heitz (jheitz)" <jheitz@cisco.com> Tue, 15 December 2020 23:40 UTC

Return-Path: <jheitz@cisco.com>
X-Original-To: idr@ietfa.amsl.com
Delivered-To: idr@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 228303A08D6 for <idr@ietfa.amsl.com>; Tue, 15 Dec 2020 15:40:02 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -9.597
X-Spam-Level:
X-Spam-Status: No, score=-9.597 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=cisco.com header.b=BcD+LxAO; dkim=pass (1024-bit key) header.d=cisco.onmicrosoft.com header.b=SzhC7CDI
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Pdp-LYRBLX6z for <idr@ietfa.amsl.com>; Tue, 15 Dec 2020 15:39:59 -0800 (PST)
Received: from alln-iport-7.cisco.com (alln-iport-7.cisco.com [173.37.142.94]) (using TLSv1.2 with cipher DHE-RSA-SEED-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id EC2453A08D4 for <idr@ietf.org>; Tue, 15 Dec 2020 15:39:58 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=27412; q=dns/txt; s=iport; t=1608075598; x=1609285198; h=from:to:cc:subject:date:message-id:references: in-reply-to:mime-version; bh=McDtf/x3Xg0pmUnfHgkhIEjXOUFzpA1j7srmFGGnzY8=; b=BcD+LxAOGsv+j/fv6+U4IH9ghenpqShvYR/vvdeZnbr5fwlzxNeV+iQ2 70bTvjT26DGV94w3S/N7HLTa6F62xPkcYA3L5JFmOVxz8YRyrtluub29f +Kysp75BUNkT5tB0E9xFMedocmBlRjzquIXnk3PNDePOXDCoEKSs1hOOn s=;
X-IPAS-Result: =?us-ascii?q?A0APAACMR9lfmIgNJK1bBxkBAQEBAQEBAQEBAQEBAQEBA?= =?us-ascii?q?QESAQEBAQEBAQEBAQEBgX4BAQEBAQELAYEiL1F8Wy8uhD+DSAONWwOPC4UAD?= =?us-ascii?q?oRxgUKBEQNUCwEBAQ0BASMKAgQBAYRKAheBWQIlNwYOAgMBAQEDAgMBAQEBB?= =?us-ascii?q?QEBAQIBBgQUAQEBAQEBAQGGCQcmDIVyAQEBBBIRChMBATcBDwIBCBEEAQEBI?= =?us-ascii?q?wQDAgICMBQJCAIEDgUIEweDBAGBflcDLgEOoUcCgTyIaXaBMoMEAQEFhRoYg?= =?us-ascii?q?hADBoE4AYJ0g3mGNiYbgUE/gRFDglY+gUmBFAQXgREBEgEIBBcVFgmCYTOCL?= =?us-ascii?q?IIFDy0vAwRIDhkSAgQTDA9CHzIHEgEFBCAOHAg3BI0pgWIYFYMhhyuDMoh6k?= =?us-ascii?q?TYKgnSJI4c4ixKDJoMim3WWB4kLkV8MhDACBAIEBQIOAQEFgWwiaXBwFTtag?= =?us-ascii?q?g8TPRcCDViKL4J/GxoJFIM6glmHfgF0NwIGAQkBAQMJfIc0BySBPF8BAQ?=
IronPort-PHdr: =?us-ascii?q?9a23=3AZz9yNxLPnr+3J04wTtmcpTVXNCE6p7X5OBIU4Z?= =?us-ascii?q?M7irVIN76u5InmIFeGv68/l1jDWp/d7OlYzezbr/OoVW8B5MOHt3YPONxJWg?= =?us-ascii?q?QegMob1wonHIaeCEL9IfKrCk5yHMlLWFJ/uX3uN09TFZXlYFfVuHu19iJUHB?= =?us-ascii?q?jjZkJ5I+3vEdvUiMK6n+m555zUZVBOgzywKbN/JRm7t0PfrM4T1IBjMa02jB?= =?us-ascii?q?DOpyhF?=
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-AV: E=Sophos;i="5.78,422,1599523200"; d="scan'208,217";a="615061697"
Received: from alln-core-3.cisco.com ([173.36.13.136]) by alln-iport-7.cisco.com with ESMTP/TLS/DHE-RSA-SEED-SHA; 15 Dec 2020 23:39:55 +0000
Received: from XCH-ALN-003.cisco.com (xch-aln-003.cisco.com [173.36.7.13]) by alln-core-3.cisco.com (8.15.2/8.15.2) with ESMTPS id 0BFNdtGC008681 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=FAIL); Tue, 15 Dec 2020 23:39:55 GMT
Received: from xhs-rcd-001.cisco.com (173.37.227.246) by XCH-ALN-003.cisco.com (173.36.7.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 15 Dec 2020 17:39:55 -0600
Received: from xhs-rcd-001.cisco.com (173.37.227.246) by xhs-rcd-001.cisco.com (173.37.227.246) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 15 Dec 2020 17:39:54 -0600
Received: from NAM04-BN3-obe.outbound.protection.outlook.com (72.163.14.9) by xhs-rcd-001.cisco.com (173.37.227.246) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Tue, 15 Dec 2020 17:39:54 -0600
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=au3wRtuOppzG/9PYFfemNEfHmd3Hho1TBW6wWC/VepVxjEQAjuMW5TDu4vxGc06vPb+fskQ32Q+fnNVe0xy94QLrtfXFb6B+nwfQ/LIwoFiSDIEZI+FuK8pnXkqI1JCF/7tSg2QlK9ZXl2h4UT6L2cjAVRzc2eH+V0asCfUWWhdhBOK7C7U2ibb21t4uGBNQU3AjgDSGbY1z0Fmhb+ibep1ZIL/+misXaw0pSPlGPXhu+RpvEv5a56I4OKgRtAAkI86+ZLmMO1w/YM25Jd+uMRkc3vUBiFZyioY2GZaX/v6vRcgrSnJjLyASCbS0poOzP7UD8r0eusaDqz35NBclMg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=McDtf/x3Xg0pmUnfHgkhIEjXOUFzpA1j7srmFGGnzY8=; b=GHOj6g5JtcZ3c6xAtZ8KAXhGZCj9KmdOzfTnqE8b4mtb6f6IxREsCYEpIQQ1DvXHixZFlYiu98/iy7t3NcRVZtkhUj9wAxSe4o13TU5lsBJ9+f9OioKGi5Zr3J5tpYfymlHQt0JnoFsj1AXxOm2aiITuDVXnzcDZhiIP7RuZ05p1qq8cjuhykpUQMT2hBe+22pOykqd4YLhfYuG8kSMa5yBbRzyxSw6aYFsvpdtk1+p4Zc+QVaYxfijd9pfCNtOFhKL+a4iZDjGdsdll2EUm0TJ6NCpBiNWMvk50F9KvwBlI67Wv1UD2h6qSzKAANlYEpgxSVPpwhAM5T7ngEUD+Rw==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cisco.com; dmarc=pass action=none header.from=cisco.com; dkim=pass header.d=cisco.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.onmicrosoft.com; s=selector2-cisco-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=McDtf/x3Xg0pmUnfHgkhIEjXOUFzpA1j7srmFGGnzY8=; b=SzhC7CDIID5XejkvIm+fywUMbT4NKt6PHVY07qlZr/pQGhXzBJ9mkgrD6ClYLmnovqjUJy235/s06o3l0ntC9Kf3Y4Bs9QgqOeWvAu1E/YHO2CGGGNGlJWaoJrad3b6THB70tpZIbiDhUinqXLCkyYb0SpRUB5fFvm9wM6MXbrw=
Received: from BYAPR11MB3207.namprd11.prod.outlook.com (2603:10b6:a03:7c::14) by BYAPR11MB2646.namprd11.prod.outlook.com (2603:10b6:a02:c6::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3654.24; Tue, 15 Dec 2020 23:39:53 +0000
Received: from BYAPR11MB3207.namprd11.prod.outlook.com ([fe80::2581:444d:50af:1701]) by BYAPR11MB3207.namprd11.prod.outlook.com ([fe80::2581:444d:50af:1701%4]) with mapi id 15.20.3654.025; Tue, 15 Dec 2020 23:39:53 +0000
From: "Jakob Heitz (jheitz)" <jheitz@cisco.com>
To: Job Snijders <job@sobornost.net>
CC: "idr@ietf.org" <idr@ietf.org>
Thread-Topic: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0
Thread-Index: AQHWz/PZ/nZ2Wy6ptUq1oN4xA4s39qnzMNSAgABM7oCAAAJ3QIAABHgAgAUVKwCAABAegIAAD7aAgAAXogCAAALoQA==
Date: Tue, 15 Dec 2020 23:39:52 +0000
Message-ID: <BYAPR11MB3207412804697588E4AA3F03C0C60@BYAPR11MB3207.namprd11.prod.outlook.com>
References: <X9PHRuGndvsFzQrG@bench.sobornost.net> <CAOj+MME4OHmoqJfzNQ4Tj6+wCd1kJVHPfJsDbk_+Xh8fh5G8Dg@mail.gmail.com> <6F7C5906-51A8-43C2-8AEC-3DB74CB9941F@tix.at> <1B4E7C9D-BBFE-4865-87F9-133ACE55D122@cisco.com> <22C381D0-2174-4828-A724-FD97B2FE0BCB@tix.at> <9D6268BD-C555-4B9A-A883-9B55EEB5D5DA@juniper.net> <91D9B9F7-0DBE-45E6-84D5-2E3D9F8C44A1@tix.at> <X9kweQ5EtTL7tOAM@bench.sobornost.net> <CAOj+MMFySPXpE8QxcO+7szKzQ78faQASYKnBUYg_h_aLd=P4Lg@mail.gmail.com>
In-Reply-To: <CAOj+MMFySPXpE8QxcO+7szKzQ78faQASYKnBUYg_h_aLd=P4Lg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
authentication-results: sobornost.net; dkim=none (message not signed) header.d=none;sobornost.net; dmarc=none action=none header.from=cisco.com;
x-originating-ip: [2601:647:5701:46e0:45ca:9d3c:5636:a4db]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 7e6d4d66-aaa6-4b3d-bf52-08d8a152b8b0
x-ms-traffictypediagnostic: BYAPR11MB2646:
x-microsoft-antispam-prvs: <BYAPR11MB2646A42745823CFEB4645C99C0C60@BYAPR11MB2646.namprd11.prod.outlook.com>
x-ms-oob-tlc-oobclassifiers: OLM:7219;
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: UwHzeRpy2VaufCOiKsrmmLtNwg3zDu2KTf3ORhT+rag9AGTE50r3AqxKXs6n5tYTUkQGUY4Qerd7425z/BLAB/ObokBNgjNn0woujvBl8w/Rh6jN2JcIXlVIo+HMrSznrEg7/zYyayvL0Hf73hN4JyA527kZGUE0p7k4fS3NPHoAMnu0nBgLJgX0MrTLN/o+8ff/ViQ3feqrpWT/6/dkxu+TDtM6p0Nfrd1eRjFo1f5sWuZ3nEXj7gZt9/fzgHCiqFTucDWZkM3TYh8BHHuyY7PviP0ZqegDnfHshRTglQDbLjpr7vYxf2rbq+5GVaeg2KbRxV1ZvFmCd72aSTZAK4hUMpmv8jiPTi7tarP5wD+Jz9cRjNnBHWzOq9sm5wr8HwgSPfk9VI1ZdC3GMfXft9fwOYbUGw7fEX1XgUImtMXrTEmKsg8gaoC5BTLhbFEeLKG737JPx1/aqeEF+wPSIg==
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR11MB3207.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(346002)(376002)(366004)(136003)(55016002)(66556008)(86362001)(9686003)(33656002)(966005)(8936002)(6916009)(76116006)(66476007)(186003)(66446008)(53546011)(52536014)(64756008)(6506007)(166002)(2906002)(5660300002)(71200400001)(83380400001)(66574015)(8676002)(508600001)(66946007)(7696005)(4326008)(6606295002); DIR:OUT; SFP:1101;
x-ms-exchange-antispam-messagedata: =?utf-8?B?Z3Q2L1ZNaldTTXdQL1lmT09NanhVK0NDRyt6YjViYURkSlQ4SFl3V2JnekE5?= =?utf-8?B?MjhMdDd1YnU2NEVJVzZEUjRocU1sVkh0MW9ULzJCSm9LaWo1UnljVWFoNWo2?= =?utf-8?B?V3lnaG9rRDMwaFMwdnVjYi8yTWM2ZWthdXRIOUlaL21nblU4WXN3ZE9jcnBa?= =?utf-8?B?bENzaUdSdjhkN3JiY3U2dlR2OHZFRVFsbDhaN3NSY2lJL1lNUWhtSjNya2F2?= =?utf-8?B?TXpheFN5YWxBUU1NVTA5dEc2cVZySU5RY0crQjNuT2s4dEprNDFVT2FibFM5?= =?utf-8?B?L3RKV3l5NFFJTmtXL0dqdVhIQWx0RnNLVkhJS1Fjb1lodStEUFRUNjQ4eitr?= =?utf-8?B?dHhvSUl1dzAvMEh4SGRDb3J0WTJ3QjduRG16R0lqUTVLVG53ZmRPclFYeXpN?= =?utf-8?B?VW56MWpUMWs3Yi9PZjhNeHdkRkt1blNhbE5MbUdwZ0JVMWdMUEZHZ1JFckk3?= =?utf-8?B?R3B2aWJDc2poYlBGaGZRNHcyb3ZuUVhPSFNEemVUODJnd0RrZFhGUGJ0NUds?= =?utf-8?B?SXBuRUowTzhWRTBFd1pFSklNWE1ya1VucnhxZ3BqRGdiRkVIaE9TRlNMbWh0?= =?utf-8?B?RGRGOW15bVVNUmtFVGlrUkkydkE0VWhybm0zbjd3a2ttSFU5R2ZIa2xPeGI4?= =?utf-8?B?bXRYcEhDWUU0VmFFencybnRkWk0zNWVBNDJZVXhTRTlkeU05WjZSK21Id0cz?= =?utf-8?B?STFtbVBEUmV2V0xvZnNVb2Fva1FlNUxiRlF5TGJJaGpwU2xEejdHZTJiYXo0?= =?utf-8?B?WWkvVGlCdVpQTWQycGdWalE5ZHU1SkZFYjFzS0V5WjA5Z0tVSFc2Vk5uSmhz?= =?utf-8?B?SG1tMzNkR0daNk82K0lPQlIzYTBNRWxLd00zUGphaTVPclBLYlBXeXRyeWdP?= =?utf-8?B?WlBIbk9iNndVc2lMTDlxRS9ZOGV6OHlRS1FlSEMrMDhybGVUTU5GNDRrbVFj?= =?utf-8?B?Zk9QZzRKb0NtT2V5ZzhUYTQrT0RxVmxFQzg0OXpzbkZLbnIrWkFuZEpUalc0?= =?utf-8?B?RjRldTZWUWx3V3Npam5DejBHUTVpb1hYRURPYTQveWttVjE5OGF3aGJFQmZW?= =?utf-8?B?eDFPMnY1UXQvM004SCtmLzFYRUI2SmhjdVk2M2pjYzdiNDJZdWxxazNnTjVL?= =?utf-8?B?VStJNDdtV1hUUTdRV3VRVnl5eVMyYUwwOFpSSUdWQlZGTDFUYWd5TzQzSXBz?= =?utf-8?B?YXlvZnNSa2dGUi9PYkpmN0k0VjFETDZxL1lFNU5GTUw3Wkk2dnZQeTF5SG9R?= =?utf-8?B?WFpieERld3pFMVZKVnFwSjhzMWhIcUVBZSt1ZUxieUQ4NmlkbGZTb2hQYmlZ?= =?utf-8?B?RXpvMm9qRy9TYkY3K09Ba1JEZGl2eFVnczcwWmpPMWtXekw4ajJtYjRiWjho?= =?utf-8?Q?iXgT82RtpZNOAoe/XOcjOUoNF/yjlPQs=3D?=
x-ms-exchange-transport-forked: True
Content-Type: multipart/alternative; boundary="_000_BYAPR11MB3207412804697588E4AA3F03C0C60BYAPR11MB3207namp_"
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3207.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 7e6d4d66-aaa6-4b3d-bf52-08d8a152b8b0
X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Dec 2020 23:39:52.9807 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 5ae1af62-9505-4097-a69a-c1553ef7840e
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: dpu1Zr7v2WeR3sF0lbAYG1owjgBVeXM19GkCD7SIvNxhdPfT4lb52PtH4twn83xyJ64dfOyO5DpkwzPDobBZiw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR11MB2646
X-OriginatorOrg: cisco.com
X-Outbound-SMTP-Client: 173.36.7.13, xch-aln-003.cisco.com
X-Outbound-Node: alln-core-3.cisco.com
Archived-At: <https://mailarchive.ietf.org/arch/msg/idr/311cGKccfLs4x9nkXvDE25OgcK8>
Subject: Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0
X-BeenThere: idr@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Inter-Domain Routing <idr.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/idr>, <mailto:idr-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/idr/>
List-Post: <mailto:idr@ietf.org>
List-Help: <mailto:idr-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/idr>, <mailto:idr-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 15 Dec 2020 23:40:02 -0000

If you tell the socket to shutdown and then close, it will attempt to
send everything in the queue with the FIN at the end.
Then wait for the FIN ACK and all manner of nonsense to bore rtr-B to tears.
So, to get on with it, send the RST.

Next question is what to do if GR is in effect.
rtr-A will dutifully retain all the routes from rtr-B and Job's beloved WITHDRAW
will still not happen.
The new session will come up (maybe), rtr-B will send all its routes again and
(if it doesn't get stuck again) will send its EOR. Only now can Job breathe easy.

Might we need a new bit in the GR capability in the OPEN message?
"WITHDRAW ALL MY ROUTES NOW"

Regards,
Jakob.

From: Idr <idr-bounces@ietf.org> On Behalf Of Robert Raszuk
Sent: Tuesday, December 15, 2020 3:19 PM
To: Job Snijders <job@sobornost.net>
Cc: idr@ietf.org
Subject: Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

Hi Job,

Putting all other concerns aside I have few questions ...

1. Is this BGP which should trigger the session RST or FIN or TCP ?

2. If this is BGP (TCP would not be aware of HOLD_SEND) how exactly do we know that peer's window is 0 for HOLD_SEND TIME ?

3. Which TCP socket option will return BGP an error that for the duration of X sec window for a given peer was 0 ? I presumed even if it jumped for 100 ms above 0 the timer would be reset indicating peer is still alive ?

From your bgpd example you are not checking anything other then BGP's ability to write to out queue. So is this the suggestion now forgetting all about TCP layer ? Simply if I can not write anything to a peer for over X sec RST the session ?

Hi John,

I think the suggestion is to add a second HOLD_SEND TIME different from normal HOLD TIME.

Also there could be lost of different type of peers so unless HOLD_SEND would be say 5 x HOLD putting all peers under same time value may be suboptimal.

Thx,
R.


On Tue, Dec 15, 2020 at 10:54 PM Job Snijders <job@sobornost.net<mailto:job@sobornost.net>> wrote:
On Tue, Dec 15, 2020 at 09:57:47PM +0100, Christoph Loibl wrote:
> Thanks for answering my question in more detail. Maybe I was unclear
> (but reading your email I think we are talking about the same).
> > On 15.12.2020, at 21:00, John Scudder <jgs@juniper.net<mailto:jgs@juniper.net>> wrote:
> >
> > I think you are talking about this scenario. I’ll copy the example
> > from Rob’s message cited above:
> >
> >   rtr-A                   rtr-B
> >   (congested c-p)         (uncongested c-p)
> >   send window: >0         send window: 0
> >   recv window: 0          recv window: >0
> >
> > In this case we expect:
> >  a) rtr-B does not send any BGP packet (KEEPALIVE/UPDATE/NOTIFICATION)
> > to rtr-A in normal operating circumstances.
> >  b) rtr-A does not expect any KEEPALIVE/UPDATE packets from rtr-B. The
> > session remains established even if no packet is received in the
> > holdtime.
> >  c) rtr-A continues to send KEEPALIVE packets to rtr-B.
>
> The part I have a problem to understand is b). It is clear that rtr-A
> will not receive any packets from rtr-B because rtr-B cannot send them
> (send window: 0). But does "rtr-A does not expect any KEEPALIVE/UPDATE
> packets from rtr-B” mean that rtr-A has essentially suspended its
> hold-timer until it is ready to receive new messages and opens up its
> recv window? If yes, why? I would expect timers to run independently
> of the transport protocol.

Yeah, I'd expect that too. We've seen congested BGP implementations
continue to send KEEPALIVEs but not accept (or send!) other BGP
messages. And rtr-B's attempts at KEEPALIVE just be TCP ACked with zero
window.

I'd argue in the above scenario rtr-A is simply broken and rtr-B MUST
proceed to close down the session towards rtr-A, rtr-B must cleanup and
generate WITHDRAWs for any routes pointing to rtr-A. By doing the
clean-up rtr-B does both itself and rtr-A a favor. If the issue was
transcient rtr-A and rtr-B will re-establish a few minutes later
(IdleHoldTimer, right?) and things will normalize.

Arguably and measurably, rtr-A is operating its Loc-RIB (forwarding)
based on stale routing information (assuming rtr-A is working at all!):
rtr-A has not received any WITHDRAWs, UPDATEs (or somewhat less
importantly KEEPALIVEs) from rtr-B.

Rtr-B is fully aware of this stale situation, because rtr-B was not able
to write these BGP messages to the network: the messages are still in
OutQ. Rtr-A didn't accept any KEEPALIVE (or UPDATE/WITHDRAW) from
rtr-B.

How to solve this? Claudio Jeker took a look at what it would take in
OpenBGPD and came up with the (tiny!) following patch, should be
readable to most: https://marc.info/?l=openbsd-tech&m=160796802508185&w=2

Ben Cox helped me create a 'EBGP peer from hell': a publicly accessible
EBGP multihop instance which can reliably produce the undesirable
TCP/BGP behavior we're discussing here. This 'peer from hell' will do
the OPEN exchange but then manipulates the TCP recvwindow towards zero.

All BGP implementations tested so far (5 famous ones) appear vulnerable
because they continue to consider the BGP session healthy & stable
(meanwhile OutQ keeps growing endlessly and zero BGP messages go across
the wire).

One network operator (with thousands of EBGP sessions in the DFZ)
reported to me the above stalled-TCP scenario is *not* a common case on
the Internet. On a normal day, a network operator will see no (zero)
sessions stuck this way, which leads me to believe 'recvwind=0' ...
*for the duration of the hold timer* is a very strong indicator for a
really broken situation which should be attempted to automatically
resolve.

I believe BGP implementations are not helping any known deployment
scenarios by *not* disconnecting a stuck peer, however on the other we
now know about various operational examples where honoring recvwind=0
for (hours, days) longer than $holdtimer led to global scale problems.

As the 'not-at-all progressing OutQ' situation seems somewhat rare in
the wild (yet continues to happen from time to time) I think it is worth
discussing & documenting how implementers can attempt to avoid this
state from happening. It might help make the Internet 1% more robust.

BGP implementers (or operators wanting to test their equipment) feel
free to contact me off-list if you'd like to set up an EBGP multihop
session towards the 'peer from hell' testbed. Testing potential
solutions this way is quite easy, the behavior can be triggered within a
few seconds.

Kind regards,

Job

ps. At this moment we have (1) an attempt at problem description, (2) a
demonstration BGP-4 implementation of a 'problem causer', and (3) a
different BGP-4 implementation with a 'solution'. This enables IDR to
test interopability & (potentially revised) protocol compliance,
hopefully moving the problem a bit from theoretical to practical
reality? :)