Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

"Jakob Heitz (jheitz)" <jheitz@cisco.com> Fri, 11 December 2020 22:08 UTC

Return-Path: <jheitz@cisco.com>
X-Original-To: idr@ietfa.amsl.com
Delivered-To: idr@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C525A3A0FC2 for <idr@ietfa.amsl.com>; Fri, 11 Dec 2020 14:08:05 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -9.6
X-Spam-Level:
X-Spam-Status: No, score=-9.6 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=cisco.com header.b=m+DDZSMf; dkim=pass (1024-bit key) header.d=cisco.onmicrosoft.com header.b=R3V+v7Zk
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 52OT4GztDGEG for <idr@ietfa.amsl.com>; Fri, 11 Dec 2020 14:08:02 -0800 (PST)
Received: from alln-iport-8.cisco.com (alln-iport-8.cisco.com [173.37.142.95]) (using TLSv1.2 with cipher DHE-RSA-SEED-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 1AB483A0FB8 for <idr@ietf.org>; Fri, 11 Dec 2020 14:08:02 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=39648; q=dns/txt; s=iport; t=1607724482; x=1608934082; h=from:to:subject:date:message-id:references:in-reply-to: mime-version; bh=NuLmoStcYOLq58WzTy36bzxJRPtdYGeA5t/v/gN0vWA=; b=m+DDZSMfUGC6VUvkQ/1tNtupWVKjR++2RnzUBGBb/JoNV9ye0IMFOboq qpMrIWvpbLCNtPDBP7ae5/AXvRX9BCQHVg4+7cVMPZgvtsbQ7Rvc00qbQ 3TW+Um6kBTzQz65zolkk58eCTx2PuRPEWO/FYCb4o/iC39mQTL0+Enq2L 8=;
X-IPAS-Result: =?us-ascii?q?A0A7AABC7NNfmIoNJK1iGgEBAQEBAQEBAQEDAQEBARIBA?= =?us-ascii?q?QEBAgIBAQEBgg+BIy8jBih8Wy8uhD+DSAONVQOPC4l/gUKBEQNPBQsBAQENA?= =?us-ascii?q?QEYAQUPAgQBAYRKAheBaAIlOBMCAwEBAQMCAwEBAQEFAQEBAgEGBBQBAQEBA?= =?us-ascii?q?QEBAYY2DIVyAQEBAwEBARAICQoTAQEsDAQLAgEIEQQBAQEgAwQDAgICJQsUC?= =?us-ascii?q?QgCBAESCBMHgjlLAYF+VwMOIAEOoXcCgTyIaXaBMoMEAQEFgTMBg2QYghADB?= =?us-ascii?q?oE4gnWDeYZZG4FBP4ERQ4JVPoJdAQECgR8EBQEBEQEjKwmCYTOCLIFPGlgGD?= =?us-ascii?q?lIEFCcICgIECQoMAg0sIAoLMggSCQEcLQEHNwKPIwKDNIcogzKIeJE1CoJ0i?= =?us-ascii?q?SKHN4sQgyWDIptzlAGCAokKkScoDxWEJwIEAgQFAg4BAQWBbSEsPXBwFTtag?= =?us-ascii?q?g9QFwINWI0mIw0NCRSDOoMGgVM7hUMBdAIBNAIGAQkBAQMJfIcPAQYhgT9fA?= =?us-ascii?q?QE?=
IronPort-PHdr: =?us-ascii?q?9a23=3AUIOqDBU3ubNDi3j1bDn0/AsaIwLV8LGuZFwc94?= =?us-ascii?q?YnhrRSc6+q45XlOgnF6O5wiEPSBN+BuelDge3Mtaf7RSoG7IrS+HwBcZkZUR?= =?us-ascii?q?gDhI1WmgE7G8eKBAX9K+KidC01GslOFToHt3G2OERYAoDyMlvVpHDh9jkYHQ?= =?us-ascii?q?/5MhFpYOL4Bt2ag8G+zevn/ZrVbk1Bjya8ZrUnKhKwoE3Ru8AajJEkJLw2z0?= =?us-ascii?q?7Co2BDfKJdwmY7KA=3D=3D?=
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-AV: E=Sophos;i="5.78,412,1599523200"; d="scan'208,217";a="632706278"
Received: from alln-core-5.cisco.com ([173.36.13.138]) by alln-iport-8.cisco.com with ESMTP/TLS/DHE-RSA-SEED-SHA; 11 Dec 2020 22:08:00 +0000
Received: from XCH-RCD-003.cisco.com (xch-rcd-003.cisco.com [173.37.102.13]) by alln-core-5.cisco.com (8.15.2/8.15.2) with ESMTPS id 0BBM80Jc028729 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=FAIL); Fri, 11 Dec 2020 22:08:00 GMT
Received: from xhs-rtp-003.cisco.com (64.101.210.230) by XCH-RCD-003.cisco.com (173.37.102.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 11 Dec 2020 16:07:59 -0600
Received: from xhs-rcd-003.cisco.com (173.37.227.248) by xhs-rtp-003.cisco.com (64.101.210.230) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 11 Dec 2020 17:07:59 -0500
Received: from NAM12-MW2-obe.outbound.protection.outlook.com (72.163.14.9) by xhs-rcd-003.cisco.com (173.37.227.248) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Fri, 11 Dec 2020 16:07:59 -0600
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cD6B/jMep5ifr3RPUBFdVbFRRbLtHou+4ava04KdbM9bRPkpbcGr7v12WcQ01kZmoQBB4F3ci1Qr/vKEiko69z2dFdjYYlbzp4DMNtxIbarpBc9vVPvveAGzn97/YoCAkDp12m6RRUNvuQ1gMlsL1jzGPbu4VWmFjZHxqnLA0+3s8xmWV/wWoFxzSK5hX0IV94Xpm2XaD3NXROARrxZAsNd++pk8wsNFAxDK3CLDKda40wU35CE5gKul1z2Nu2Fbij2/Sk75gcBsyCtpHhOeCCmWf+KXZL84VQk9VI+D0KqlFd9lPBpEr1O9DC8difwVM8s8NA9meG3oGAi9BtjZlQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NuLmoStcYOLq58WzTy36bzxJRPtdYGeA5t/v/gN0vWA=; b=DLJjWT308iP9MbfNe/+UCDJSJZ2gvv/wZ/pPiRTvcAjnfaQurCouqTlgHZfByoa2t2UesFr2GyiaA9ZxWFjKqtrFpi2F7KIsiIFY2d5KU6tQRLJh4T5Ujl3e+SZSsIMf3DBkFa3bguV6CahkgmRBxGFJTe9qKVgmyKDleVIjgJiDLR94VZklsOsp41qXLqM0K8IjdVBw6Ivwwv4pLP9a6RMiKkRr8rYRTEcxsOGT7dRKFPnjcdUhS/kVo53sgMZcoacKaW3Sl/OxujOHS/sJy+nmS7l40nDx+Kpr5Q9J6Pld1U9H4dfr3Q/Im8pNmRKkBsjZNTUrRe7eoqfroGNwxw==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cisco.com; dmarc=pass action=none header.from=cisco.com; dkim=pass header.d=cisco.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.onmicrosoft.com; s=selector2-cisco-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NuLmoStcYOLq58WzTy36bzxJRPtdYGeA5t/v/gN0vWA=; b=R3V+v7ZkHQ6TYlxJcLJqpDnaNNi1K1u0Bme7LyHzjtQICvfozzVTqOdrpMq5adCiq33EcKzMt2VwVFr+Y1Una7jLEd8Cg2+KyGAdxTtfNHN/hRXAankeeavZ859INXfc+Hn3AfI5XhjmBvUbYZ5c18YVo7oL6/BbyIOEMaUmXOQ=
Received: from BYAPR11MB3207.namprd11.prod.outlook.com (2603:10b6:a03:7c::14) by BYAPR11MB2741.namprd11.prod.outlook.com (2603:10b6:a02:bf::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3654.17; Fri, 11 Dec 2020 22:07:58 +0000
Received: from BYAPR11MB3207.namprd11.prod.outlook.com ([fe80::2581:444d:50af:1701]) by BYAPR11MB3207.namprd11.prod.outlook.com ([fe80::2581:444d:50af:1701%4]) with mapi id 15.20.3632.023; Fri, 11 Dec 2020 22:07:58 +0000
From: "Jakob Heitz (jheitz)" <jheitz@cisco.com>
To: Job Snijders <job@sobornost.net>, "idr@ietf.org" <idr@ietf.org>
Thread-Topic: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0
Thread-Index: AQHWz/PZ/nZ2Wy6ptUq1oN4xA4s39qnyUWWAgAAWDoCAAAizkA==
Date: Fri, 11 Dec 2020 22:07:57 +0000
Message-ID: <BYAPR11MB32072F8635A4EE22D89A606AC0CA0@BYAPR11MB3207.namprd11.prod.outlook.com>
References: <X9PHRuGndvsFzQrG@bench.sobornost.net> <FCB1ADB7-AD8C-447E-82FE-2EC15B8C3FB9@juniper.net> <CAOj+MMEGRLw9cRXJR4VgOYtoj+tRyeY4WhWsdkMuYktGh6THag@mail.gmail.com>
In-Reply-To: <CAOj+MMEGRLw9cRXJR4VgOYtoj+tRyeY4WhWsdkMuYktGh6THag@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
authentication-results: sobornost.net; dkim=none (message not signed) header.d=none;sobornost.net; dmarc=none action=none header.from=cisco.com;
x-originating-ip: [2601:647:5701:46e0:5d3:4c32:3bfa:306c]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 0952708b-e67d-41c3-20c5-08d89e2137ce
x-ms-traffictypediagnostic: BYAPR11MB2741:
x-microsoft-antispam-prvs: <BYAPR11MB27417ABACB1615EF4419CCF2C0CA0@BYAPR11MB2741.namprd11.prod.outlook.com>
x-ms-oob-tlc-oobclassifiers: OLM:10000;
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: +jdqVpjz3pBaXooyhS0+UxNVlvdwbh60epJqqDgehBroke4DdUPnm76eVYlevRz7FS0zXeXRdDcWnLven2dOg7GwlSbWJy3shmDoDn9ApidT0GF/xHuEOhrtl6VGpDMNlnyfFRIswqDfOJtWSBjiaJ+4SHZCXnnA6Xv7atwSnQJ50C+ObLcx3m7F9TardPBK/Z9Amwg/nS2dz90y6YGQwqd3NJ50mK+NTzPG79Kns9NFF2S0yAeN4Olm5I1888x6TAwnclS0s6m09TTRKWQF85LSwYvMZu5WFz5A8okR9Vi54nZr6/w1iy6q7ETxmgZ58XuISRl9+mjZUF1x/ZJYd1s/Ccu3Z9TIuTCGvz8QguBxccXudIYLUPzjh4C9ZH/zqHkjywjdenN++FuUBcS3zw==
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR11MB3207.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(136003)(376002)(346002)(52536014)(33656002)(53546011)(8936002)(508600001)(6506007)(966005)(7696005)(2906002)(71200400001)(9686003)(55016002)(66574015)(66946007)(76116006)(66556008)(5660300002)(186003)(64756008)(83380400001)(66446008)(86362001)(66476007)(110136005)(8676002)(166002); DIR:OUT; SFP:1101;
x-ms-exchange-antispam-messagedata: =?utf-8?B?bWFaalp2U09IMmFhU1dhYU4xNVBDREFuSENZbkcrZmx0RGhuRTJUMEU4aW02?= =?utf-8?B?bHFkTE5oWG03Rm0wdUV2NkV4azhKSEF3ZWk5WnZvZVVtVUxjRkpwUyt0YTBr?= =?utf-8?B?dldOQjBjOU16Mi8wdlBRN1NQWTZWTUl2VDZpYWpJQ3dNcFNJVnlrZU1GVUJX?= =?utf-8?B?L1dMVmVzRlI0UzVzVGc5UEx4UmtxVGhCK3VhNEhKU0lweXdUbjNzYlFxbFZq?= =?utf-8?B?VWlrWWdsdUg5ZTV2RjVBK1k5dEtvRXRMUk92OUVsd0Eyc1FXZmxTSCtVdWZD?= =?utf-8?B?N3VwTFgwb2ZVM1NsM09lUm9Mb1FTRitlVGNCaFN3TjIydVhLcjFqM3FUNHFF?= =?utf-8?B?VjI4Z0tmU1M1RlJpZTRXRFFPSEdvd2t4UC9mYlpkbjV1aDAxNEVaTVEvZXFp?= =?utf-8?B?NTlTYVZmaFNyU01UN0htbEIyb1RNUzlvWS9oWXBCMkU3Z2dJUER5VS9MZUI0?= =?utf-8?B?cGk0cUZvY01zeHIrTmNwU1hYT0VVYTdnSWk2bDJGeitsdEJEUWJvS1ZlcHUr?= =?utf-8?B?eisyWlMrclNyVmNQYURTQW83VEI5dlZIRVJLSDFWQkFDaGhQdGhQSGIzanhJ?= =?utf-8?B?UUMzWUhvVzh2UlFlNmdIdXlMZDJYUVIxbnRzV2Qwd3BxSThjUXN0Z2crVU5F?= =?utf-8?B?enVqQlFzVVk3cDFiSW1XM09RcUJyeEp4THppL3JRQTU4VGhGbEdKa2VOOGIw?= =?utf-8?B?Q3pCZTBZdGgzbGZ5dVRyU09yNGVKTUwxNTBGdFNaSTVxZ3c0MlJqb0l3WUhI?= =?utf-8?B?OHlTK2VJSGFMZlhUZVJDU0ZLdGx0RU9wck5WdmJ3NXh6Q1pxYTMrMHFXaXFT?= =?utf-8?B?U24wYkJCeXVmOHBEUlBPSmNaYytYbktrcDdmczE4b1hQWnpxek5jem9aVGJB?= =?utf-8?B?VGRnVHYzbHd5T2hWU2NBZWhFTVFpU3FWQk4yVzRMLzhSL2dmL1k2NDc1T3h1?= =?utf-8?B?dkcxQVp5bDlRUVUySmRkTDVUMVpLWjcrQXh5NEphMldHZ3VTOE5OekU5dEpo?= =?utf-8?B?WTdLSS9JeFg3eElNMXZJbTB2Y1FRNXh1VHA2c3grRjJGTVArMERRaFNuUFNr?= =?utf-8?B?WERLdWprOURTY3Jrdzc0NTNUZ3BsREh1Rmgxbk1xUzFTK1liZU94SXUvWjVt?= =?utf-8?B?M3lGMnNEWmhRTG5ZWEc5cUpFNEJVUlBud0M2MFhiQkcxbGlFamxiaUVWWG1x?= =?utf-8?B?bUI4ZzlDYW5Ia2dmZ1FLRHZPRUxTRVZGWGlBWFJkYi8wRXJxWFlSbDdVa1pP?= =?utf-8?B?SHVnelZSVlBvYWxteFN5ZTNiM2p0Skgwcjl2NzNBT1ZNQlhaVHRQS0h0RmQr?= =?utf-8?B?S0NoWUFoZzlZazdTclFpekg1RkFCanBKTzJXWEJwUEgwZUdvNTFhYkF3N00w?= =?utf-8?Q?UksNwnotIFfbZKR0QDnY2a9++FEIODBA=3D?=
x-ms-exchange-transport-forked: True
Content-Type: multipart/alternative; boundary="_000_BYAPR11MB32072F8635A4EE22D89A606AC0CA0BYAPR11MB3207namp_"
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3207.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0952708b-e67d-41c3-20c5-08d89e2137ce
X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Dec 2020 22:07:57.9410 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 5ae1af62-9505-4097-a69a-c1553ef7840e
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 7OBTyGaEU1+S2o43Mcfgz8SRxrlw0noTcmrTiw4T2Lu+so4kMYvSv33wE8C+jWa5cpjJirLLAKUvaF7IWWiSuw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR11MB2741
X-OriginatorOrg: cisco.com
X-Outbound-SMTP-Client: 173.37.102.13, xch-rcd-003.cisco.com
X-Outbound-Node: alln-core-5.cisco.com
Archived-At: <https://mailarchive.ietf.org/arch/msg/idr/H0MS_MvhsZda8hUfA9ogdFAnVHU>
Subject: Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0
X-BeenThere: idr@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Inter-Domain Routing <idr.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/idr>, <mailto:idr-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/idr/>
List-Post: <mailto:idr@ietf.org>
List-Help: <mailto:idr-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/idr>, <mailto:idr-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 11 Dec 2020 22:08:06 -0000

Perhaps we are focusing too much on TCP.
The real issue is that BGP can't send a single byte for the duration of the hold timer.
We could instead say if the socket is send blocked.

Now, I think BGP should close the socket ungracefully, with close() and no shutdown()
and no NOTIFICATION. It cannot send a NOTIFICATION if the socket is blocked.
The management plane issue can be fixed in any number of ways.

I advocate for TCP to send a RST rather than wait for the window to open up and
drain the queue with a FIN at the end.
Allowing TCP to wait only drags out the pain. It must die. And it must die NOW.

Holding up WITHDRAWs is serious, because it does not allow redundant paths
to take effect.

Also, a RST allows a recovery using graceful restart, whereas NOTIFICATION and FIN precludes GR.
Notwithstanding RFC8538.

Regards,
Jakob.

From: Idr <idr-bounces@ietf.org> On Behalf Of Robert Raszuk
Sent: Friday, December 11, 2020 1:22 PM
To: Job Snijders <job@sobornost.net>
Cc: idr@ietf.org
Subject: Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

Hi,

Couple of observations/questions:

* I do share some concerns that this will make BGP peering less stable.

* Is the "unable to send" only possible under Window = 0 ? What if there is a local NIC buffer full and we keep dropping it locally ? If we are going there perhaps we could say "unable to successfully send" meaning send and get an ACK for it ?

* The proposal is about reusing the HOLD TIME value to bring BGP down when you are still receiving keepalives however peer sent ACK for the last segment indicating zero window - is this right ?

* What happens if our side is stuck and not able to process subsequent ACKs which potentially increase the window on the peer ?

* Most deployments use BFD to make sure peer is reachable. As BFD is often offloaded from CPU this is not an indication of health of TCP or BGP path. But what if some deployments can not use BFD (say not supported by the peer) ? Then those typically reduce BGP HOLD TIME. Would it not be too fragile in some cases to bring a TCP down in such cases too fast ? Aren't we overloading HOLD TIME value here a bit ?

* Assume we bring TCP down ... when does it attempt to go up again ? Are we ok to bring it up only after manual/script action from such a state ?

* As proposed it seems that the change will affect all AFI/SAFIs using given single TCP session. Even if perhaps all are perfectly healthy and running on separate cores. If each SAFI would run on a separate TCP session this would not have such an impact.

* The change seems applicable to both iBGP and eBGP right ?

In summary the attempt here is to fix application issues by cutting the transport. Sure half broken transport for whatever reason may not be a good thing to keep in UP state. Especially when redundancy is in place. But my main concerns are that we are only trying to focus on a single low level trigger to detect it.

Wouldn't per AFI/SAFI heartbeat be a better option to detect if a peer's BGP stack is still up and running fine for all applications ?

Thx,
R.


On Fri, Dec 11, 2020 at 9:04 PM John Scudder <jgs=40juniper.net@dmarc.ietf.org<mailto:40juniper.net@dmarc.ietf.org>> wrote:
[all hats on]

Hi Job,

Thanks for bringing this up.

To take the liberty of summarizing your wall of text :-) you’re saying that you believe BGP should tear down its session if it’s unable to send a message for the duration of the hold time.

Given that the conversation last time was inconclusive I think this is a good thing for the WG to discuss again. If you want to, you (or someone) could turn the idea into a short draft that updates RFC 4271, and we could have a WG adoption discussion about it. It might help focus the discussion but it’s not mandatory.

I’ll point out a few things to start with —

- Making it mandatory to apply hold time to the sending of messages would potentially make BGP peerings less stable. It clearly can’t make them *more* stable. Of course one can argue that if you haven’t been able to send a message for the hold time, the session has failed its metric of usefulness anyway, so any veneer of stability at this point is a harmful sham.
- If I recall correctly, RST doesn’t work (or may not work) if you’re using the MD5 TCP option. Nothing much to be done, but be aware.
- There is nothing stopping an implementation from doing what you describe now. The formalism that keeps you within the letter of 4271 would be that the implementation supplies a configuration option, that you set to enable the behavior. Once you’ve done that, when the implementation notices that the hold time has been exceeded in the outbound direction, it generates a ManualStop event for the session.

Thanks,

—John

> On Dec 11, 2020, at 2:23 PM, Job Snijders <job@sobornost.net<mailto:job@sobornost.net>> wrote:
>
>
> Dear group,
>
> Not too long ago an incident [1] in one Autonomous System resulted in
> the global Internet being unusable in many parts of the world for
> multiple hours. Some have reported the root cause was a 'configuration
> error', however I believe much of the observed communication blackouts
> in the global routing system stemmed from a pre-existing condition: a
> specific implementation property present in multiple implementations
> currently in use in the default-free zone.
>
> Usually when an incident happens in one AS, affected parties can through
> unilateral action 'route around the problem', but the ability to 'route
> around problems' critically depends on the ability to distribute
> WITHDRAW or UPDATE messages. When messages are not processed, what
> generally was assumed to be a unilaterally solvable problem, now requires
> coordination between *all* neighbors of the suffering AS.
>
> The global routing system requires every participant to process BGP
> messages, because the alternative is intervention on thousands of BGP
> devices to manually shutdown thousands of BGP sessions disconnecting the
> AS suffering from an incident, to help the rest of the default-free
> zone. I speak from experience when saying that coordinating a disconnection
> of an AS at global scale is incredibly hard and slow, any many approval
> levels must be worked through. It takes *hours* of phone calls & email
> chains, a time window during which internet traffic is routed towards
> stale (now blackholing) locations.
>
> In the average ISP's network design using IBGP Route Reflectors, these
> blackout effects are aggravated when BGP sessions landing in such
> devices are not terminated when TCP causes the BGP session to stall.
>
> The problem of how TCP and BGP-4 can interact has been discussed before,
> but I'm not sure the working group followed up with any publication
> detailing the problem and the solution.
>
>    https://urldefense.com/v3/__https://mailarchive.ietf.org/arch/msg/idr/q0Sx5d3zZjfOmOQ4lO2OZAHh9Lc/__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkPhCc8cBA$<https://urldefense.com/v3/__https:/mailarchive.ietf.org/arch/msg/idr/q0Sx5d3zZjfOmOQ4lO2OZAHh9Lc/__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkPhCc8cBA$>
>
> Does everyone agree BGP-4 sessions MUST be terminated using a TCP RST
> (instead of a BGP-4 Cease NOTIFICATION) if the peer has indicated for
> the duration of the Hold Timer that the TCP receive window is zero?
> I'm fine with there being buttons to make this different, but the
> default for routers in the global Internet routing system should be to
> consider the remote peer to be 'a lost cause' when it won't accept new
> BGP messages for the duration of the hold timer.
>
> Perhaps RFC 4271 Section 6.5 should be amended as following:
>
> OLD:
>    If a system does not receive successive KEEPALIVE, UPDATE, and/or
>    NOTIFICATION messages within the period specified in the Hold Time
>    field of the OPEN message, then the NOTIFICATION message with the
>    Hold Timer Expired Error Code is sent and the BGP connection is
>    closed.
>
> NEW:
>    If a system does not receive (or is unable to send) successive
>    KEEPALIVE, UPDATE, and/or NOTIFICATION messages within the period
>    specified in the Hold Time field of the OPEN message, then the
>    NOTIFICATION message with the Hold Timer Expired Error Code is sent
>    and the BGP connection is closed. If the NOTIFICATION message cannot
>    be send the BGP connection is closed.
>
> This is an ongoing problem. I suspect the BGP Nyancat's discoloration at
> the left most eye might have been caused by an active TCP session
> keeping a stale BGP session alive. But also the observations from "BGP
> Zombies: an Analysis of Beacons Stuck Routes" [3] could be explained by
> the problematic interaction between TCP and BGP.
>
> I appreciate the work the IDR working group has done to *SOFTEN* the
> blow from implementation defects on global routing (RFC 7606 is a
> brilliant example of this), but I fear in this case there is no subtle
> way to say goodbye when the peer doesn't process messages in a timely
> fashion. It might be good to document this.
>
> Kind regards,
>
> Job
>
> [1]: https://urldefense.com/v3/__https://www.reuters.com/article/level-3-communi-outages-idUSL2N1CB00C__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMkF2w4cg$<https://urldefense.com/v3/__https:/www.reuters.com/article/level-3-communi-outages-idUSL2N1CB00C__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMkF2w4cg$>
> [2]: https://urldefense.com/v3/__https://labs.ripe.net/Members/cteusche/bgp-meets-cat__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMry7Ktyw$<https://urldefense.com/v3/__https:/labs.ripe.net/Members/cteusche/bgp-meets-cat__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMry7Ktyw$>
> [3]: https://urldefense.com/v3/__https://www.iij-ii.co.jp/en/members/romain/pdf/romain_pam2019.pdf__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkO8A78j8Q$<https://urldefense.com/v3/__https:/www.iij-ii.co.jp/en/members/romain/pdf/romain_pam2019.pdf__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkO8A78j8Q$>
>
> _______________________________________________
> Idr mailing list
> Idr@ietf.org<mailto:Idr@ietf.org>
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/idr__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMMXdwc-g$<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/idr__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMMXdwc-g$>

_______________________________________________
Idr mailing list
Idr@ietf.org<mailto:Idr@ietf.org>
https://www.ietf.org/mailman/listinfo/idr