Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

John Scudder <jgs@juniper.net> Fri, 11 December 2020 20:03 UTC

Return-Path: <jgs@juniper.net>
X-Original-To: idr@ietfa.amsl.com
Delivered-To: idr@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 930AF3A0F00 for <idr@ietfa.amsl.com>; Fri, 11 Dec 2020 12:03:19 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.1
X-Spam-Level:
X-Spam-Status: No, score=-2.1 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=juniper.net header.b=GJmZ6caR; dkim=pass (1024-bit key) header.d=juniper.net header.b=OQ7F7AIG
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xWdMmZ96LJkr for <idr@ietfa.amsl.com>; Fri, 11 Dec 2020 12:03:17 -0800 (PST)
Received: from mx0b-00273201.pphosted.com (mx0b-00273201.pphosted.com [67.231.152.164]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 62B9D3A0F41 for <idr@ietf.org>; Fri, 11 Dec 2020 12:03:16 -0800 (PST)
Received: from pps.filterd (m0108160.ppops.net [127.0.0.1]) by mx0b-00273201.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 0BBJxSNs013380; Fri, 11 Dec 2020 12:03:15 -0800
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=juniper.net; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-id : content-transfer-encoding : mime-version; s=PPS1017; bh=sIHabu48GKArVEt7hL4lWS6c4f6GvmfHRI1785K+w/g=; b=GJmZ6caRsEu5pP6f3W2KGzW6HysGL+UayFAo3MuEor95BkAtU+vrUPtnHQNnIYj27wC1 GmG7AowtYVtH/YHTqqkfzSmFqBoJn7DhmOxKqAPTaOdbxBshJ5KP9k3/B8sGEGx899ao fq4hTD61dFno3y8/9VQzCoNvHN66/PvUXBr3vHFjPz7z4XyH+eDqwuwldITXz9Mdu/gz ULtupE4rFy5iEAarpaSFXSCAF3eSdz9mzvrDKmT8+Q9cOVc3RvQSKUp7RdFQFEqRG77b 8adwKQ5so03E/f7QULAvQrfbgD7JFBq/+s54cTysRYXfOPb4OqNnsMJrJQsoZHmZP1Sg tA==
Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2104.outbound.protection.outlook.com [104.47.58.104]) by mx0b-00273201.pphosted.com with ESMTP id 35bw53suyb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 11 Dec 2020 12:03:15 -0800
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KOwLGSF+9MxzKSpWxnZWm28Cn+Upvb4okxuKH4D+9H241CkPMx3tKMaBjwuSgBKeZ7zfN1VhAWbr2jTPqNnPODLSmVQk9/OJTKYoPV77J7vgbFXagn/c6MMJ9KDu96Y3Qm7wWKgEdQo5LHm6wQuk0YJYT9tPS2Rv+q81xkKvlTyRaDoA75QUlfJuXPEbrKn+0ZjoGiGpzE6UcdLjPbP8WU9JaeeCbQDINcfCSuMpqNLo53zYH5qxXdr4QXaDKFhJgtXBtu/JwiRPdPB0aoinGZ1NdRtQ0sJN4JrfbHEY7mNFM/wcrPQ8Lv4QGkXQhx1j39BwrCG8rWiG5/X75bBPDQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sIHabu48GKArVEt7hL4lWS6c4f6GvmfHRI1785K+w/g=; b=R9BmiUC59B/r3c1cfXmM7jSoIZ6ymA+ZGCESeFHgENmf5N1TFKCtTAh5RMaYKWh0haQ3bXNf1SLPyqCykEIXfoitFHi14hlKDbLUqP3GKaPYncFIwnUkUeSiQ0E3imYvAt4GcFUD2wMraL+fHeu52qFTHM0h76H1foTNWoj6fedRnRKWQ90N/Bd0R1DxHryjrGC06BiVDxTZ7IR/3Rlh24yVO1u/0AEKKH4XxZEu3l1J/4eiYrj14UPrbXzGWb6yBUxvmPMRRnG/G/gJPOEnGcymRXcMSpH51qKWo7a1uqxdEcd2A8JbTNSkbgR7xMjrUMU9Rnz/zARvgc22fmk9KQ==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=juniper.net; dmarc=pass action=none header.from=juniper.net; dkim=pass header.d=juniper.net; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=juniper.net; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sIHabu48GKArVEt7hL4lWS6c4f6GvmfHRI1785K+w/g=; b=OQ7F7AIGaVe8Kmr9j6oAeezCrgBXtOQGf5WsuvBQ2elee8NE7SUyhRjAex3j/B8UesqtI2EAekmmTOKr5rmN+S/ldZPx3f+atbZ+agVol19M5KzaIaun5hcpmjDcTeQmrC2qPOUldlqqyQjVJnVnvGGTpEfxDeJFDwPIPKqdEJ4=
Received: from BN8PR05MB6098.namprd05.prod.outlook.com (2603:10b6:408:45::29) by BN7PR05MB4419.namprd05.prod.outlook.com (2603:10b6:406:f8::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3654.8; Fri, 11 Dec 2020 20:03:12 +0000
Received: from BN8PR05MB6098.namprd05.prod.outlook.com ([fe80::e1b6:748b:a75e:172c]) by BN8PR05MB6098.namprd05.prod.outlook.com ([fe80::e1b6:748b:a75e:172c%3]) with mapi id 15.20.3654.016; Fri, 11 Dec 2020 20:03:12 +0000
From: John Scudder <jgs@juniper.net>
To: Job Snijders <job@sobornost.net>
CC: "idr@ietf.org" <idr@ietf.org>
Thread-Topic: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0
Thread-Index: AQHWz/PUVnNdU7dhTEany/fuPhqhnKnyUWWA
Date: Fri, 11 Dec 2020 20:03:11 +0000
Message-ID: <FCB1ADB7-AD8C-447E-82FE-2EC15B8C3FB9@juniper.net>
References: <X9PHRuGndvsFzQrG@bench.sobornost.net>
In-Reply-To: <X9PHRuGndvsFzQrG@bench.sobornost.net>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-mailer: Apple Mail (2.3608.120.23.2.4)
authentication-results: sobornost.net; dkim=none (message not signed) header.d=none;sobornost.net; dmarc=none action=none header.from=juniper.net;
x-originating-ip: [162.225.191.192]
x-ms-publictraffictype: Email
x-ms-office365-filtering-ht: Tenant
x-ms-office365-filtering-correlation-id: c2be70d2-9fd9-4f04-5801-08d89e0fc9d8
x-ms-traffictypediagnostic: BN7PR05MB4419:
x-microsoft-antispam-prvs: <BN7PR05MB4419C67BE686E0A49AE4943EAACA0@BN7PR05MB4419.namprd05.prod.outlook.com>
x-ms-oob-tlc-oobclassifiers: OLM:10000;
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: OcXdjN8k2byHDDS0s+GyPVgatobk6lTSh9laxiG0GxRALJmnwcTrL5zIXcxBRbIpSOkHSEbUT4VV3KEuTvzaqruNcar+yiM4MminAEHimh+7ky87RL6KwI08zl7VOexFo9Pr03rtgCu9W5kxBiSNjS+fa75OMABs26QfFXhsgHp6imYutclXH+sMqw/fVzcx2o83Z3L76mFHNvP3MxYvgCPtxLcldmieDjh9pNISligXmOtqSiSIg7hXYCygmhgsfjAt6CeOIkVqy/awYYEOvPrlcuNO+pPzmu3b2A1fPD3LuF1VdRmsPw9Q4MOwohnhTkDp/2BNGCp0UPaeXxILXU70dKDb+Tqd/5RjjuVMdgu5RJd+bwAE7W5pYXp6ACoapeReCPPE1LnZyr7V/dODUAQSxYDMyDdkHqELDOb8ffAmJgcA6JlYAqRtvZs5qeiTiafGJuazAwIjL9UJfL7EsA==
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN8PR05MB6098.namprd05.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(136003)(376002)(366004)(346002)(36756003)(71200400001)(2616005)(966005)(8936002)(66476007)(66574015)(66556008)(66946007)(64756008)(91956017)(86362001)(2906002)(8676002)(76116006)(6512007)(6486002)(33656002)(53546011)(6506007)(508600001)(186003)(26005)(6916009)(4326008)(66446008)(5660300002)(83380400001)(45980500001); DIR:OUT; SFP:1102;
x-ms-exchange-antispam-messagedata: =?utf-8?B?TG9rRm41Qk1lQXE2aTB5WjNBaDRZdjI5OFc3ZUdGa2IxZXRaUkVDeUVPYzVW?= =?utf-8?B?SkZDTjFCL1pnVFkzN2V5bGNXTWc5TkIzaVR1UFVJaUpoNFFEZlpWVzQ0UEF1?= =?utf-8?B?Mko1MmV3R2Q2R280SUdHbmV0djloMDBJT1J0YUhQUm1TMnVFKytyanBHeGJK?= =?utf-8?B?allJRFpETDVhcSttSnRVQURHZXJHc28zZS9LelVscVUrdWtqejM2SnlZaXVQ?= =?utf-8?B?eUhxdjFpUFliWWFVZVh5RVhHNDFMU1ZOaTZDbmowQzZ0L0o5TlpCVEFFZUt4?= =?utf-8?B?OXI1YmR1am1CUnNIT1RrT1ZDdDFOSE5vbHBtQmZZcndqMHZhUnc1cVFNZmRY?= =?utf-8?B?YXV5ZlV6S2tyTzZya3ZBWmdtaTQ3TzF1NFAreFdaT0kxMXZVSXUzcmwrcWVC?= =?utf-8?B?M2p0Y2t2SGN5cHpQWWdiMXNYaCttZm95Mi9mdTVCR2JuQUMwdGxYMG8wL3lM?= =?utf-8?B?MnNjR09aRGRFRmFsNHNZODR5ZG4wdGtIai9Ga2NMSDZMNjJqT0JGb3hURnE1?= =?utf-8?B?blVKdHZBMm1KRXRJRzAxT2xzQUtld0gwaXZDYTlXMm5BM2ZTb2p1bFFQMjYv?= =?utf-8?B?M3ZweVh5UVVJVXVmanJwekk1MDZKdWhraWpaWm9TQ0l3RDIvUlJrUEpLdmZP?= =?utf-8?B?MlNzTWExK3hjYS9HS0lFOTAyT2dsYjZNVEFTU2V4allXSTc0Um9NMGR3Wm9U?= =?utf-8?B?S1BRRDVsWlRIR0NQRzljd3BhaDFaaGxqa1UrYUxkc21jYmhyNlJJelR4cWR0?= =?utf-8?B?S2M0WmxuOXF4dmFVZzhzckhOd3RzRVlOUmlGRG1DYTlHV0d3OUZUV0RGTEJ1?= =?utf-8?B?T2hhNnZQNWYvOHJQWEtxM05xV3FZWWxjRE1VMTJxS09JWHc0SjBabmVIWkZw?= =?utf-8?B?MTF6NEdDYyt1enUxSmpiWVFMZWRmUUN1dHhBOTNGVHVUa2N2UDljTng5THpZ?= =?utf-8?B?ZjAxRUdnZ2drcFJHcE9SaDNIYmJLcVo2Y3l1ZlpjOUsxSE9GaG0rbVg0T1Av?= =?utf-8?B?d0lPajZ4a3l4dW43UCtlc3dNU1Jxams1Wnk1Qk5xWDlwenhDbTJSZzNYM0VR?= =?utf-8?B?Smphd3UxODFKUEVwblNTdm5Iay9tWHRWbUFSQml0NDEvTUVHUzlnMHJXMUFN?= =?utf-8?B?M3RrYUZkOHU1WlM1VmltcCtrUFh5Yjg4dktCdWZOYTRCYVNVeU1RNEhKc3hw?= =?utf-8?B?WWxxNTRrZ3kwRFlWb2hEWHR6M1pvVW4wWjRzT2FBdUdvd21tMEh3M0x5bkZh?= =?utf-8?B?c2h1T1pzdVg3R25BckhWbERndEhDNjE2U2ZIZXRtS3JOb0o5UEsrUjdsbXB4?= =?utf-8?Q?jbsEJ8ipbz2fDKJMfzrxZKhRxXet+7enSk?=
x-ms-exchange-transport-forked: True
Content-Type: text/plain; charset="utf-8"
Content-ID: <7946247F1C4BF749850549B22D23625B@namprd05.prod.outlook.com>
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-OriginatorOrg: juniper.net
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN8PR05MB6098.namprd05.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c2be70d2-9fd9-4f04-5801-08d89e0fc9d8
X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Dec 2020 20:03:11.9522 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: bea78b3c-4cdb-4130-854a-1d193232e5f4
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: +ynIFH+HddIuIO8+NaY7EkXEcI2ZojaQI8cLCnERmofgicaMgMdDFCnaWqgKnVDu
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR05MB4419
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343, 18.0.737 definitions=2020-12-11_06:2020-12-11, 2020-12-11 signatures=0
X-Proofpoint-Spam-Details: rule=outbound_spam_notspam policy=outbound_spam score=0 suspectscore=0 bulkscore=0 lowpriorityscore=0 priorityscore=1501 clxscore=1011 adultscore=0 spamscore=0 mlxscore=0 impostorscore=0 malwarescore=0 phishscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2012110133
Archived-At: <https://mailarchive.ietf.org/arch/msg/idr/VX4D99Y0WXVVR4UsfVG0GeIpPSw>
Subject: Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0
X-BeenThere: idr@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Inter-Domain Routing <idr.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/idr>, <mailto:idr-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/idr/>
List-Post: <mailto:idr@ietf.org>
List-Help: <mailto:idr-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/idr>, <mailto:idr-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 11 Dec 2020 20:03:27 -0000

[all hats on]

Hi Job,

Thanks for bringing this up.

To take the liberty of summarizing your wall of text :-) you’re saying that you believe BGP should tear down its session if it’s unable to send a message for the duration of the hold time. 

Given that the conversation last time was inconclusive I think this is a good thing for the WG to discuss again. If you want to, you (or someone) could turn the idea into a short draft that updates RFC 4271, and we could have a WG adoption discussion about it. It might help focus the discussion but it’s not mandatory.

I’ll point out a few things to start with —

- Making it mandatory to apply hold time to the sending of messages would potentially make BGP peerings less stable. It clearly can’t make them *more* stable. Of course one can argue that if you haven’t been able to send a message for the hold time, the session has failed its metric of usefulness anyway, so any veneer of stability at this point is a harmful sham.
- If I recall correctly, RST doesn’t work (or may not work) if you’re using the MD5 TCP option. Nothing much to be done, but be aware.
- There is nothing stopping an implementation from doing what you describe now. The formalism that keeps you within the letter of 4271 would be that the implementation supplies a configuration option, that you set to enable the behavior. Once you’ve done that, when the implementation notices that the hold time has been exceeded in the outbound direction, it generates a ManualStop event for the session. 

Thanks,

—John

> On Dec 11, 2020, at 2:23 PM, Job Snijders <job@sobornost.net> wrote:
> 
> 
> Dear group,
> 
> Not too long ago an incident [1] in one Autonomous System resulted in
> the global Internet being unusable in many parts of the world for
> multiple hours. Some have reported the root cause was a 'configuration
> error', however I believe much of the observed communication blackouts
> in the global routing system stemmed from a pre-existing condition: a
> specific implementation property present in multiple implementations
> currently in use in the default-free zone.
> 
> Usually when an incident happens in one AS, affected parties can through
> unilateral action 'route around the problem', but the ability to 'route
> around problems' critically depends on the ability to distribute
> WITHDRAW or UPDATE messages. When messages are not processed, what
> generally was assumed to be a unilaterally solvable problem, now requires
> coordination between *all* neighbors of the suffering AS.
> 
> The global routing system requires every participant to process BGP
> messages, because the alternative is intervention on thousands of BGP
> devices to manually shutdown thousands of BGP sessions disconnecting the
> AS suffering from an incident, to help the rest of the default-free
> zone. I speak from experience when saying that coordinating a disconnection
> of an AS at global scale is incredibly hard and slow, any many approval
> levels must be worked through. It takes *hours* of phone calls & email
> chains, a time window during which internet traffic is routed towards
> stale (now blackholing) locations.
> 
> In the average ISP's network design using IBGP Route Reflectors, these
> blackout effects are aggravated when BGP sessions landing in such
> devices are not terminated when TCP causes the BGP session to stall.
> 
> The problem of how TCP and BGP-4 can interact has been discussed before,
> but I'm not sure the working group followed up with any publication
> detailing the problem and the solution.
> 
>    https://urldefense.com/v3/__https://mailarchive.ietf.org/arch/msg/idr/q0Sx5d3zZjfOmOQ4lO2OZAHh9Lc/__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkPhCc8cBA$
> 
> Does everyone agree BGP-4 sessions MUST be terminated using a TCP RST
> (instead of a BGP-4 Cease NOTIFICATION) if the peer has indicated for
> the duration of the Hold Timer that the TCP receive window is zero?
> I'm fine with there being buttons to make this different, but the
> default for routers in the global Internet routing system should be to
> consider the remote peer to be 'a lost cause' when it won't accept new
> BGP messages for the duration of the hold timer.
> 
> Perhaps RFC 4271 Section 6.5 should be amended as following:
> 
> OLD:
>    If a system does not receive successive KEEPALIVE, UPDATE, and/or
>    NOTIFICATION messages within the period specified in the Hold Time
>    field of the OPEN message, then the NOTIFICATION message with the
>    Hold Timer Expired Error Code is sent and the BGP connection is
>    closed.
> 
> NEW:
>    If a system does not receive (or is unable to send) successive
>    KEEPALIVE, UPDATE, and/or NOTIFICATION messages within the period
>    specified in the Hold Time field of the OPEN message, then the
>    NOTIFICATION message with the Hold Timer Expired Error Code is sent
>    and the BGP connection is closed. If the NOTIFICATION message cannot
>    be send the BGP connection is closed.
> 
> This is an ongoing problem. I suspect the BGP Nyancat's discoloration at
> the left most eye might have been caused by an active TCP session
> keeping a stale BGP session alive. But also the observations from "BGP
> Zombies: an Analysis of Beacons Stuck Routes" [3] could be explained by
> the problematic interaction between TCP and BGP.
> 
> I appreciate the work the IDR working group has done to *SOFTEN* the
> blow from implementation defects on global routing (RFC 7606 is a
> brilliant example of this), but I fear in this case there is no subtle
> way to say goodbye when the peer doesn't process messages in a timely
> fashion. It might be good to document this.
> 
> Kind regards,
> 
> Job
> 
> [1]: https://urldefense.com/v3/__https://www.reuters.com/article/level-3-communi-outages-idUSL2N1CB00C__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMkF2w4cg$
> [2]: https://urldefense.com/v3/__https://labs.ripe.net/Members/cteusche/bgp-meets-cat__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMry7Ktyw$
> [3]: https://urldefense.com/v3/__https://www.iij-ii.co.jp/en/members/romain/pdf/romain_pam2019.pdf__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkO8A78j8Q$
> 
> _______________________________________________
> Idr mailing list
> Idr@ietf.org
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/idr__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMMXdwc-g$