Re: [bmwg] bmwg Digest, Vol 183, Issue 1
Sudhin Jacob <sjacob@juniper.net> Tue, 03 December 2019 03:57 UTC
Return-Path: <sjacob@juniper.net>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 361FC120112 for <bmwg@ietfa.amsl.com>; Mon, 2 Dec 2019 19:57:18 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.701
X-Spam-Level:
X-Spam-Status: No, score=-2.701 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=juniper.net header.b=GFjfRBO4; dkim=pass (1024-bit key) header.d=juniper.net header.b=YnRU08Wo
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZZ5OIQzBfGGR for <bmwg@ietfa.amsl.com>; Mon, 2 Dec 2019 19:57:14 -0800 (PST)
Received: from mx0b-00273201.pphosted.com (mx0b-00273201.pphosted.com [67.231.152.164]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 81570120110 for <bmwg@ietf.org>; Mon, 2 Dec 2019 19:57:14 -0800 (PST)
Received: from pps.filterd (m0108161.ppops.net [127.0.0.1]) by mx0b-00273201.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id xB33vCul027132 for <bmwg@ietf.org>; Mon, 2 Dec 2019 19:57:13 -0800
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=juniper.net; h=from : to : subject : date : message-id : references : in-reply-to : content-type : content-transfer-encoding : mime-version; s=PPS1017; bh=AQnRLCzGN56Obivt2ZofA99DEFwCTx7Ig3eYu60RFvw=; b=GFjfRBO4hyrTaaLXLRQ80O/ksssclRjQhmE1GjE8pr2VnGTCLQDn2ow63hklAnsB1Lxp Yg3QYVrEQ4pyvglcj3daTFGl3NYCpNH7g78I7eXshNaXpL62NX3J/XE/rFQJgPxANpeJ DrVDRaC9AvQbs5x1a6iA2/KScc2BSqFTqhTRJYiD039PvX3Q2ypst+gAA/EGMNzzq6x8 +atAge607S8hvxmb62jYh82TvcTXfDsDWM6RuNRo+GACkw9AqYiMmkq717uyw0VozfS4 cWUWOcPfOiT9vuyFmocKcpuIu87SZcTYt43pGxrSD+uq9Dfe7HBgd6AuMfHwrfjo7ard /g==
Received: from nam01-sn1-obe.outbound.protection.outlook.com (mail-sn1nam01lp2057.outbound.protection.outlook.com [104.47.32.57]) by mx0b-00273201.pphosted.com with ESMTP id 2wn8qerpc5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for <bmwg@ietf.org>; Mon, 02 Dec 2019 19:57:13 -0800
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NrhzAiiwMykKdWUZo2E5t7EqGqVaoVmq95fRJ8j+FCQysjDASOv3E0a7FaBKEPQ+eJG/jwyoEU//61qP+bKqAW4tAAwM/G39gMo1/tsTMHF5HmNX1S8+AzQkibA7siqOy+jc7qypFBFSYGddGQtwca0xEz9aUD6h7lPifcNy6NIj/+Lr8TTbFN4k386v6QavevoTPCRFwcrh1Xy3ihBJEARsH4RuLkNwqR9IPoM4ppNK4tJmXEkH7QRhb0Uxc6gxKGGcEQDhW+M5SlYtY46pm0lb4AWZw9qtGoqBFD6AfG6SS0f9t0F2kx0wDvSO9/rnqrxVJBMEFq5u5GCy/OtGHw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AQnRLCzGN56Obivt2ZofA99DEFwCTx7Ig3eYu60RFvw=; b=EkN3PM5AE/mCpCQTLGqkI3mU2NBGvcMwhJbSElOuj99eKm8g9JeaVnqSl2vA7iSP77Cmz1EVyVSKeMEODCAuVOlET7LJou0uXcT1CX06FuGfhwS7UQHtHdYTZ143cs4uCh+5mF/p/eqezYla72HYljOaNNQtDvG9mhMok7bPDGbYKzupqMXBCYz/PrDtlrkRvMtd9WTxaPTqrbjoAW7Is/KLwrHsDADmEQcGDYyfkn9VunP6k4SrDYugCrUlmImw1rEixzkBiafoJIVsrgzFboZTYY2Sz9kguaFrChxH+XomRMc9Krt4FXS2Pc21vOmt3bklo3/EUec+OfSLKeNIfg==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=juniper.net; dmarc=pass action=none header.from=juniper.net; dkim=pass header.d=juniper.net; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=juniper.net; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AQnRLCzGN56Obivt2ZofA99DEFwCTx7Ig3eYu60RFvw=; b=YnRU08WoP2iVYlL21DbVzljzQvDOqgIKFUNawjG5AjekF1qwWWpmfI4lnRsl5HkWjcp0Qm5R+i8kjKJFkiLkl2TTlA0c/hsqCoexyUPOL6YYIlfMBoFFu/CX74CejgZ6PG64ZoqwBVPccN3RDZH/ec3RF2gbCKtSYnxpHBC15Lo=
Received: from MN2PR05MB6208.namprd05.prod.outlook.com (20.178.241.91) by MN2PR05MB7184.namprd05.prod.outlook.com (52.135.36.143) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2516.7; Tue, 3 Dec 2019 03:57:10 +0000
Received: from MN2PR05MB6208.namprd05.prod.outlook.com ([fe80::ed0d:9029:720c:1459]) by MN2PR05MB6208.namprd05.prod.outlook.com ([fe80::ed0d:9029:720c:1459%2]) with mapi id 15.20.2516.003; Tue, 3 Dec 2019 03:57:10 +0000
From: Sudhin Jacob <sjacob@juniper.net>
To: "bmwg@ietf.org" <bmwg@ietf.org>
Thread-Topic: bmwg Digest, Vol 183, Issue 1
Thread-Index: AQHVqWN5cSU1fgWHy0WbyP+cIKtJWKenx/eQ
Content-Class:
Date: Tue, 03 Dec 2019 03:57:10 +0000
Message-ID: <MN2PR05MB62080E941E385395831FBDF6C2420@MN2PR05MB6208.namprd05.prod.outlook.com>
References: <mailman.201.1575327260.6825.bmwg@ietf.org>
In-Reply-To: <mailman.201.1575327260.6825.bmwg@ietf.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
msip_labels: MSIP_Label_0633b888-ae0d-4341-a75f-06e04137d755_Enabled=True; MSIP_Label_0633b888-ae0d-4341-a75f-06e04137d755_SiteId=bea78b3c-4cdb-4130-854a-1d193232e5f4; MSIP_Label_0633b888-ae0d-4341-a75f-06e04137d755_Owner=sjacob@juniper.net; MSIP_Label_0633b888-ae0d-4341-a75f-06e04137d755_SetDate=2019-12-03T03:57:05.3595669Z; MSIP_Label_0633b888-ae0d-4341-a75f-06e04137d755_Name=Juniper Business Use Only; MSIP_Label_0633b888-ae0d-4341-a75f-06e04137d755_Application=Microsoft Azure Information Protection; MSIP_Label_0633b888-ae0d-4341-a75f-06e04137d755_ActionId=b75454c7-553f-4f27-ac1a-7e81967fc49e; MSIP_Label_0633b888-ae0d-4341-a75f-06e04137d755_Extended_MSFT_Method=Automatic
dlp-product: dlpe-windows
dlp-version: 11.3.2.8
dlp-reaction: no-action
x-originating-ip: [116.197.184.10]
x-ms-publictraffictype: Email
x-ms-office365-filtering-ht: Tenant
x-ms-office365-filtering-correlation-id: 47013402-4180-4067-65a5-08d777a4df9d
x-ms-traffictypediagnostic: MN2PR05MB7184:
x-microsoft-antispam-prvs: <MN2PR05MB7184C8A2C1DA508177356D8EC2420@MN2PR05MB7184.namprd05.prod.outlook.com>
x-ms-oob-tlc-oobclassifiers: OLM:9508;
x-forefront-prvs: 02408926C4
x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(366004)(396003)(376002)(346002)(39860400002)(136003)(10533003)(13464003)(53754006)(27574002)(199004)(189003)(18543002)(966005)(66066001)(53546011)(66574012)(6506007)(102836004)(76176011)(6246003)(7696005)(26005)(71190400001)(186003)(1730700003)(81156014)(81166006)(2906002)(30864003)(305945005)(8676002)(9686003)(25786009)(6306002)(5640700003)(55016002)(11346002)(478600001)(6436002)(446003)(7736002)(5660300002)(33656002)(8936002)(6116002)(99286004)(71200400001)(3846002)(66476007)(76116006)(52536014)(66556008)(64756008)(86362001)(66946007)(74316002)(2501003)(66446008)(2351001)(229853002)(14454004)(14444005)(6916009)(316002)(256004)(579004)(559001)(569006); DIR:OUT; SFP:1102; SCL:1; SRVR:MN2PR05MB7184; H:MN2PR05MB6208.namprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1;
received-spf: None (protection.outlook.com: juniper.net does not designate permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: XdpQcUYLR/3o/jsTsvwdrgs7mdfVaCCrP4W6tdxPAGHTI1xGkI+jZ3/a/zM/WhdNkluf46vFb1a/9YKfplk7feDV+e6uxLHnIc/bT489gEyvtTHMliCFBrbJUUqB/hiWSNvzaAB8lMsMdvLZuAbWSVAm5iYiYJ02W/avcON5h4y7/K2bAwrInMaPsEEmBj/FNXShUtMuDMxmuWih1zXmw4BH4XQvVXnPfs2Yj///PkDUVBiCliNJDgRClQkNsOl9G2dZFxNb+HpIhI84GC3yPr9TfvoHWI8am0azGJ3iLhM4mR3UymbjHfah6DVvqMx8ne/wn+bCvdK9UIZENKPfR9WldSO0oI9F8fKuDweRKW4dJWM5ct672DHxHASIXa2rwvouERmXOh+3VihSLI0jVUUgmxrwSRfiDgjtVjCYSqf5jW5dlmbVDXl9Dmkhbsvt6Kj6uN5ZQzEkJqYhes6iwjG7IDZgTp/zbskAfOm/QR8=
x-ms-exchange-transport-forked: True
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: juniper.net
X-MS-Exchange-CrossTenant-Network-Message-Id: 47013402-4180-4067-65a5-08d777a4df9d
X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Dec 2019 03:57:10.4871 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: bea78b3c-4cdb-4130-854a-1d193232e5f4
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: l5dWB9tBlun9hZz47feS5TOHEfi5ryqf+IeYtZqXvzefoSfWZNEEDfwnHizUh0GhYqYPVnhyKR80vcSmGU28bQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR05MB7184
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.95,18.0.572 definitions=2019-12-02_06:2019-11-29,2019-12-02 signatures=0
X-Proofpoint-Spam-Details: rule=outbound_spam_notspam policy=outbound_spam score=0 mlxlogscore=999 priorityscore=1501 impostorscore=0 mlxscore=0 lowpriorityscore=0 malwarescore=0 bulkscore=0 spamscore=0 phishscore=0 adultscore=0 suspectscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-1910280000 definitions=main-1912030034
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/vOeqltd19q9SAWyWlFjvcAVBycs>
Subject: Re: [bmwg] bmwg Digest, Vol 183, Issue 1
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 03 Dec 2019 03:57:18 -0000
Hi Warren, Appreciate your comments. We will fix it. Once it is done we will get back as early as possible. Regards, Sudhin Juniper Business Use Only -----Original Message----- From: bmwg <bmwg-bounces@ietf.org> On Behalf Of bmwg-request@ietf.org Sent: Tuesday, December 3, 2019 4:24 AM To: bmwg@ietf.org Subject: bmwg Digest, Vol 183, Issue 1 Send bmwg mailing list submissions to bmwg@ietf.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/bmwg__;!8WoA6RjC81c!UJm1S-C5cGIqsiU6lghEbZsWCL6Y-nyqkrkYpBbb_QrHnEHb3tdHYyz1u4wevoE$ or, via email, send a message with subject or body 'help' to bmwg-request@ietf.org You can reach the person managing the list at bmwg-owner@ietf.org When replying, please edit your Subject line so it is more specific than "Re: Contents of bmwg digest..." Today's Topics: 1. AD Review of draft-ietf-bmwg-evpntest (Warren Kumari) 2. Re: AD Review of draft-ietf-bmwg-evpntest (Sarah B) ---------------------------------------------------------------------- Message: 1 Date: Mon, 2 Dec 2019 17:42:43 -0500 From: Warren Kumari <warren@kumari.net> To: bmwg@ietf.org Subject: [bmwg] AD Review of draft-ietf-bmwg-evpntest Message-ID: <CAHw9_i+biry0-sH8Ez1NZKo3V5rBkPL2amTk74akXH8E27Bttg@mail.gmail.com> Content-Type: text/plain; charset="UTF-8" Hi all, Unfortunately, I do not think that this document is ready to be progressed -- there are a large number of areas where the document is *very* unclear, and / or incomplete. I usually break my reviews up into "substantive" and "Nits / Editorial"; but in this case I didn't, largely because many of the editorial-type issues make the text unclear; there are also a sufficient quantity that I do not feel comfortable asking all of the IETF to review, nor would I feel comfortable asking the directorates / IESG to review. I've annotated my comments with "WK: ", but many of the issues occur multiple times -- for example, there are many instances of "SA", which from context is "Source Address" but this could also be "SA" in the "Single-Active" sense. W ------------------- Abstract This document defines methodologies for benchmarking EVPN and PBB- EVPN performance. EVPN is defined in RFC 7432, and is being deployed in Service Provider networks. Specifically this document defines the WK (nit) : Specifically, this (missing comma) methodologies for benchmarking EVPN/PBB-EVPN convergence, data plane performance, and control plane performance. 1. Introduction EVPN is defined in RFC 7432, and describes BGP MPLS- based Ethernet VPNs (EVPN). PBB-EVPN is defined in RFC 7623, discusses how Ethernet Provider backbone Bridging can be combined with EVPNs to provide a new/combined solution. This draft defines methodologies that can be used to benchmark both RFC 7432 and RFC 7623 solutions. Further, this draft provides methodologies for benchmarking the performance of EVPN data and control planes, MAC learning, MAC flushing, MAC ageing, WK: s/ageing/aging (RFC index has 258 instances of 'aging' and only 4 'ageing') convergence, high availability, and scale. 1.1. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. WK: Replace RFC2119 with RFC8174 language. WK: Please fix formatting, and sort alphabetically. Also, this isn't "Terminology", it is mainly Abbreviations. 1.2. Terminologies MHPE Multi homed Provide Edge router. RR Route Reflector. P Provider Router. CE Customer Router/Devices/Switch. MHPE2 Multi homed Provider Edge router 2. MHPE1 Multi homed Provider Edge router 1. SHPE3 Single homed Provider Edge Router 3. AA EVPN Terminologies AA All-Active. SA EVPN Terminologies SA Single-Active. RT Router Tester. Sub Interface Each physical Interfaces is subdivided in to Logical units. WK: s/in to/into/ EVI EVPN Instances which will be running on sub interface or physical port of the provider Edge routers. DF Designated Forwarder. ESI Ethernet Segment Identifier. 2. Test Topology EVPN/PBB-EVPN Services running on SHPE3, MHPE1 and MHPE2 in Single Active Mode: | [Traffic Generator ] Router Tester traffic sender/receiver of layer 2 traffic with multiple vlan. WK: VLANs. Also, this doesn't really parse - is 'Router Tester' supposed to be in the brackets? +----------+ | | | SHPE3 | | | +----------+ | |Core link +----------+ | | | RR | | | Route Reflector/Core router WK: Core router or Provider router? (Core router isn't defined in the terminology section) +----------+-------------| | | | Core links | +----------+ +-----------+ | | | MHPE2 | | DUT | | | | MHPE1 | | | +----------+ +-----------+ | PE-CE link | +----------+------------ | | | CE | | layer2 | |bridge | +----------+------------ [Traffic Generator](Router Tester sender/reciever of layer 2 traffic with multiple vlan) WK: As above. Aslo, receiver is a typo WK: The formatting of this table is very poor, esp the header. +-----------------+---------------------+---------------------+---------------------+----------------------+-----------------------+ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Mode | | | |Receiver | | | | Test |Traffic Direction |Sender | | | | | | | | | | | | | | | | | | | | | | | | +----------------------------------------------------------------------------------------------------------------------------------+ | | | | | | | | | | | | SHPE3 | | |Single Active | Local Mac | |CE | |Layer 2 traffic | | | Learning | Uni | | | | | | | | | | multiple MAC | | | | | | | | +-----------------------------------------------------------------------------------------------------------------------------------+ | | | | | | | |Single Active | Remote MAC | | | CE |Layer 2 traffic | | | Learning | uni | SHPE3 | | | | | | | | |multiple MAC | | | | | | | ++ +----------------------------------------------------------------------------------------------------------------------------------+ | | | | | | | |Single Active | Scale Convergence | Bi | | CE/SHPE3 | | | | | | CE/SHPE3 | |Layer 2 traffic | | | Local& Remote | | | |multiple mac& vlans | | | Learning | | | | | +-----------------+---------------------+---------------------+--------------------------------------------+-----------------------+ | --- SNIP --- Test Setup Configurations: There are five routers in the Test setup. SHPE3, RR/P, MHPE1 and MHPE2 emulating a service provider network. CE is a customer device connected to MHPE1 and MHPE2, it is configured with bridge domains in multiple vlans. The router tester is connected to CE and SHPE3.The MHPE1 acts as DUT.The RT will be used as sender and receiver of traffic.The measurement will be taken in DUT. WK: What exactly is a "Router Tester"? This term doesn't seem to be defined anywhere -- is it just a traffic generator? Please also fix the formatting above (missing spaces after periods) - this is throughout the document. All routers except CE is configured with OSPF/IS-IS,LDP,MPLS,BGP with EVPN address family. WK: All routers except CE are configured with. "Configured with" doesn't really provide sufficient detail to allow repeatable testing. All routers except CE must have IBGP configured with RR acting as route reflector. WK: iBGP. Also, expand acronyms. MHPE1,MHPE2,SHPE3 must be configured with "N" EVPN/PBB-EVPN instances depends up on the cases. WK: s/MHPE1,MHPE2,SHPE3/MHPE1, MHPE2, SHPE3/ (and elsewhere) MHPE1 and MHEPE2 must be configured with ESI per vlan or ESI on IFD. WK: What is an IFD? MHPE1 and MHEPE2 are running Single Active mode of EVPN. CE is acting as bridge configured with vlans that is configured on MHPE1,MHPE2,SHPE3. Depends up on the test traffic will be flowing uni directional or bi directional depends on the test performed. WK: This is unparsable (and even if I could parse it I'm not sure what it is related to) The above configuration will be serving as the base configuration for all test cases. 3. Test Cases for EVPN Benchmarking 3.1. Local MAC Learning Objective: To Record the time taken to learn the MAC address locally in DUT. Topology : Topology 1 Procedure: The data plane MAC learning can be measured using the parameters defined in RFC 2889 section 5.8. Send "X" unicast frames from CE to MHPE1(DUT) working in SA mode with "X" different source and destination address from RT. The DUT must learn these "X" macs in data plane. WK: MACs (and elsewhere) Measurement : Measure the time taken to learn "X" MACs in DUT evpn mac table. The data plane measurement is taken by considering DUT as black box the range of X MAC is known from RT and the same must be learned in DUT, the time taken to learn "X" macs is measured. WK: This is also largely unparsable. Repeat these test and plot the data. The test is repeated for "N" times and the values are collected. The mac learning time is calculated by averaging the values obtained from "N" samples. Mac learning in sec = (T1+T2+..Tn/N) WK: Presumably (T1+T2+..Tn)/N ? 3.2. Remote MAC Learning Objective: To Record the time taken to learn the remote macs. Topology : Topology 1 Procedure: Send X frames with X different SA and DA to SHPE3 from RT. WK: Please define / expand SA / DA. Also, SA conflicts with "SA EVPN Terminologies SA Single-Active." SHPE3 will advertise these locally learned macs to MHPE1 and MHPE2 via control plane.Measure the time taken to learn these X MACs from remote peer in DUT EVPN MAC address table.The DUT and MHPE2 are running SA mode. Measurement : Measure the time taken by the DUT to learn the "X" MACs in the data plane.Repeat these test and plot the data.The test is repeated for WK: Repeat these tests... "N" times and the values are collected.The mac learning time is calculated by averaging the values obtained from "N" samples. Mac learning in sec = (T1+T2+..Tn/N) WK: As above. Also, what *exactly* do I report? This isn't (AFAICT) "Mac learning in sec", it is (presumably) MAC Learning Rate, and I'd *assume* that I report a line graph showing MACs learnt vs time? This (and the other tests) need to be *much* more explicit about what exactly is being reported. --- SNIP ---- 3.9. ARP/ND Scale These tests are conducted to Record the scaling parameter of ARP/ND of the DUT. Objective: To Record the ARP/ND scale of the DUT. Topology : Topology 1 Procedure: Send X arp/icmpv6 request from RT to DUT with different sender ip/ ipv6 address to the same target gateway ip address. Measure whether X MAC+IPv4 address/MAC+IPv6 address of the hosts are learned in DUT. WK: I don't think I can run this test -- how fast do I send these? What *exactly* is the packet I'm sending (icmpv6 is not a packet, it's a protocol) Where do I send it? (to DUT? to the same target gateway IP? What does that even mean?) The X MAC+IPv4 bit doesn't make sense. Measurement : The DUT must learn X MAC+IPV4/MAC+IPv6 and it must advertise the X MAC+IPV4/MAC+IPV6 to the remote router. 3.10. Scaling of Services Objective: To measure the scale limit of DUT for EVPN.This is to measure the performance of DUT in scaling to "X" EVPN instances. Topology : Topology 1 Procedure: The DUT,MHPE2 and SHPE3 are scaled to "N" EVI.Clear BGP neighbors of the DUT. Once adjacency is established in the DUT. WK: Fragment? Measure the routes received from MHPE2 and SHPE3 for "N" EVI in the DUT. Measurement : There should not be any loss of route types 1,2,3 and 4 in DUT. DUT must relearn all type 1,2,3 and 4 from remote routers. The DUT must be subjected to various values of N to find the optimal scale limit WK: Do the first 2 sentences mean the same thing? Presumable I *do* lose all the routes, but then I should recover them? Why is this being compared to what I had before clearing, and not just the maximum / number sent from the tester? What ratio should there be for types 1, 2, 3, 4? Is it important? 3.11. Scale Convergence Objective: To measure the convergence time of DUT when the DUT is scaled with EVPN instance along with traffic. Topology : Topology 1 Procedure: Scale N EVIs in DUT,SHPE3 and MHPE2.Send F frames to DUT from CE using traffic generator with X different SA and DA for N EVI's. Send F frames from traffic generator to SHPE3 with X different SA and DA. There will be 2X number of MAC address will be learned in DUT EVPN MAC table. WK: Doesn't parse. There is a bi directional traffic flow with F pps in each direction. Then clear the BGP neighbors in the DUT. Once the adjacency is restored in DUT. Measure the time taken to learn 2X MAC address in DUT MAC table. WK: Also doesn't parse. Measurement : The DUT must learn 2X MAC address. Measure the time taken to learn 2X MAC in DUT. Repeat these test and plot the data.The test is repeated for "N" times and the values are collected.The convergence time is calculated by averaging the values obtained by "N" samples. Convergence time in sec = (T1+T2+..Tn/N) 3.12. SOAK Test. Objective: This test is carried out to measure the stability of the DUT in a scaled environment with traffic over a period of time "T'". In each interval "t1" the DUT CPU usage, memory usage are measured. The DUT is checked for any crashes during this time period. Topology : Topology 1 Procedure: Scale N EVI's in DUT,SHPE3 and MHPE2.Send F frames to DUT from CE using traffic generator with different X SA and DA for N EVI's. Send F frames from traffic generator to SHPE3 with X different SA and DA. There will be 2X number of MAC address will be learned in DUT EVPN MAC table. There is a bi directional traffic flow with F pps in each direction. The DUT must run with traffic for 24 hours, every hour check for memory leak, CPU usage and crash. Measurement : Take the hourly reading of CPU, process memory. There should not be any leak, crashes, CPU spikes. WK: What all am I supposed to be reporting for this test? Can N be 1 and F be 1 and X be 1? I'm happy to do that and report no issues for 24 hours.... 4. Test Cases for PBB-EVPN Benchmarking WK: ----- Are these the same as for "Test Cases for EVPN Benchmarking"? If so, why are they repeated? 4.8. High Availability Objective: To record traffic loss during routing engine failover. Topology : Topology 1 Procedure: Send X frames to DUT with X different SA and DA from CE using the traffic generator. Send X frames from traffic generator to SHPE3 with X different SA and DA so that 2X MAC address will be Learned in DUT. There is a bi directional traffic flow with X pps in each direction. Then do a routing engine fail-over. Measurement : There should be 0 traffic loss which is the ideal case, No change in the DF role. DUT should not withdraw any routes.Repeat the test "N" times and plot the data.The packet loss is calculated by averaging the values obtained from "N" samples. Packet loss in sec = (T1+T2+..Tn/N) WK: What exactly am I reporting? "There should be 0 traffic loss which is the ideal case" - am I reporting the number of seconds with dropped packets? What if it *does* withdraw routes? What do I report if I don't have multiple REs? Presumably if X == 2 I can do this easily, but if X == 1e8 it's harder - but all I'm reporting is "Packet loss". --- SNIP --- 4.11. Soak Test Objective: To measure the stability of the DUT in a scaled environment with traffic. Topology : Topology 1 Procedure: Scale N PBB-EVI's in DUT,SHPE3 and MHPE2.Send F frames to DUT from CE using traffic generator with X different SA and DA for N EVI's. Send F frames from traffic generator to SHPE3 with X different SA and DA. There will be 2X number of MAC address will be learned in DUT PBB- EVPN MAC table. There is a bi directional traffic flow with F pps in Each direction. The DUT must run with traffic for 24 hours, every hour check the memory leak, crashes. Measurement : Take the hourly reading of CPU process, memory usages. There should not be any memory leak, crashes,CPU spikes. WK: "any memory leak, crashes,CPU spikes" is not at all clear - e.g: How are you defining a 'spike'? If the CPU jumps from 2% to 10% for 3 seconds, is that a spike? Many routers run a BGP reaper every N seconds / minutes, which looks like a spike - is this an issue? 7. Security Considerations There is no additional consideration from RFC 6192. WK: I thought that all BMWG documents included standard boilerplate for Security Considerations? ---------------- -- I don't think the execution is relevant when it was obviously a bad idea in the first place. This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants. ---maf ------------------------------ Message: 2 Date: Mon, 2 Dec 2019 14:54:13 -0800 From: Sarah B <sarah.banks@gmail.com> To: Warren Kumari <warren@kumari.net> Cc: bmwg@ietf.org Subject: Re: [bmwg] AD Review of draft-ietf-bmwg-evpntest Message-ID: <3FFADB16-2C79-48B5-A1BE-5C8CBC2E72D3@gmail.com> Content-Type: text/plain; charset=us-ascii Hi Warren, Thanks for your thorough review. The authors will review the document with your feedback and edit accordingly. While We (and I, specifically) have looked at this document a LOT, we missed some obvious things, and our authors are first time authors; I'll work with them to sort out the obvious catches (thanks for those). Cheers Sarah > On Dec 2, 2019, at 2:42 PM, Warren Kumari <warren@kumari.net> wrote: > > Hi all, > > Unfortunately, I do not think that this document is ready to be > progressed -- there are a large number of areas where the document is > *very* unclear, and / or incomplete. I usually break my reviews up > into "substantive" and "Nits / Editorial"; but in this case I didn't, > largely because many of the editorial-type issues make the text > unclear; there are also a sufficient quantity that I do not feel > comfortable asking all of the IETF to review, nor would I feel > comfortable asking the directorates / IESG to review. > > I've annotated my comments with "WK: ", but many of the issues occur > multiple times -- for example, there are many instances of "SA", which > from context is "Source Address" but this could also be "SA" in the > "Single-Active" sense. > > W > ------------------- > > > Abstract > > This document defines methodologies for benchmarking EVPN and PBB- > EVPN performance. EVPN is defined in RFC 7432, and is being deployed > in Service Provider networks. Specifically this document defines the > > WK (nit) : Specifically, this (missing comma) > > methodologies for benchmarking EVPN/PBB-EVPN convergence, data plane > performance, and control plane performance. > > > > 1. Introduction > > EVPN is defined in RFC 7432, and describes BGP MPLS- based Ethernet > VPNs (EVPN). PBB-EVPN is defined in RFC 7623, discusses how Ethernet > Provider backbone Bridging can be combined with EVPNs to provide a > new/combined solution. This draft defines methodologies that can be > used to benchmark both RFC 7432 and RFC 7623 solutions. Further, > this draft provides methodologies for benchmarking the performance of > EVPN data and control planes, MAC learning, MAC flushing, MAC ageing, > > WK: s/ageing/aging (RFC index has 258 instances of 'aging' and only 4 'ageing') > > convergence, high availability, and scale. > > 1.1. Requirements Language > > The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", > "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this > document are to be interpreted as described in RFC 2119 [RFC2119]. > > WK: Replace RFC2119 with RFC8174 language. > > > > WK: Please fix formatting, and sort alphabetically. Also, this isn't > "Terminology", it is mainly Abbreviations. > > 1.2. Terminologies > > MHPE Multi homed Provide Edge router. > > RR Route Reflector. > > P Provider Router. > > CE Customer Router/Devices/Switch. > > MHPE2 Multi homed Provider Edge router 2. > > MHPE1 Multi homed Provider Edge router 1. > > SHPE3 Single homed Provider Edge Router 3. > > AA EVPN Terminologies AA All-Active. > > SA EVPN Terminologies SA Single-Active. > > RT Router Tester. > > Sub Interface Each physical Interfaces is subdivided in to Logical > units. > WK: s/in to/into/ > > EVI EVPN Instances which will be running on sub interface or physical > port of the provider Edge routers. > > DF Designated Forwarder. > > ESI Ethernet Segment Identifier. > > 2. Test Topology > > EVPN/PBB-EVPN Services running on SHPE3, MHPE1 and MHPE2 in Single > Active Mode: > > > | [Traffic Generator ] Router Tester traffic sender/receiver > of layer 2 traffic with multiple vlan. > > WK: VLANs. Also, this doesn't really parse - is 'Router Tester' > supposed to be in the brackets? > > +----------+ > | | > | SHPE3 | > | | > +----------+ > | > |Core link > +----------+ > | | > | RR | > | | Route Reflector/Core router > > WK: Core router or Provider router? (Core router isn't defined in the > terminology section) > > +----------+-------------| > | | > | Core links | > +----------+ +-----------+ > | | | MHPE2 | > | DUT | | | > | MHPE1 | | | > +----------+ +-----------+ > | PE-CE link | > +----------+------------ > | | > | CE | > | layer2 | > |bridge | > +----------+------------ [Traffic Generator](Router Tester > sender/reciever of layer 2 traffic with multiple vlan) > > WK: As above. Aslo, receiver is a typo > > > WK: The formatting of this table is very poor, esp the header. > > > > +-----------------+---------------------+---------------------+---------------------+----------------------+-----------------------+ > | | | | > | | | > | | | | > | | | > | | | | > | | | > | | | | > | | | > | Mode | | | > |Receiver | | > | | Test |Traffic Direction |Sender > | | | > | | | | > | | | > | | | | > | | | > | | | | > | | | > +----------------------------------------------------------------------------------------------------------------------------------+ > | | | | > | | | > | | | | > | SHPE3 | | > |Single Active | Local Mac | |CE > | |Layer 2 traffic | > | | Learning | Uni | > | | | > | | | | > | | multiple MAC | > | | | | > | | | > +-----------------------------------------------------------------------------------------------------------------------------------+ > | | | | > | | | > |Single Active | Remote MAC | | > | CE |Layer 2 traffic | > | | Learning | uni | SHPE3 > | | | > | | | | > | |multiple MAC | > | | | | > | | ++ > +----------------------------------------------------------------------------------------------------------------------------------+ > | | | | > | | | > |Single Active | Scale Convergence | Bi | > | CE/SHPE3 | | > | | | | > CE/SHPE3 | |Layer 2 traffic | > | | Local& Remote | | > | |multiple mac& vlans | > | | Learning | | > | | | > +-----------------+---------------------+---------------------+--------------------------------------------+-----------------------+ > > | > --- SNIP --- > > Test Setup Configurations: > > There are five routers in the Test setup. SHPE3, RR/P, MHPE1 and > MHPE2 emulating a service provider network. CE is a customer device > connected to MHPE1 and MHPE2, it is configured with bridge domains in > multiple vlans. The router tester is connected to CE and SHPE3.The > MHPE1 acts as DUT.The RT will be used as sender and receiver of > traffic.The measurement will be taken in DUT. > > > WK: What exactly is a "Router Tester"? This term doesn't seem to be defined > anywhere -- is it just a traffic generator? > Please also fix the formatting above (missing spaces after periods) - > this is throughout the document. > > All routers except CE is configured with OSPF/IS-IS,LDP,MPLS,BGP with > EVPN address family. > > WK: All routers except CE are configured with. "Configured with" > doesn't really provide > sufficient detail to allow repeatable testing. > > > All routers except CE must have IBGP configured with RR acting as > route reflector. > > WK: iBGP. Also, expand acronyms. > > MHPE1,MHPE2,SHPE3 must be configured with "N" EVPN/PBB-EVPN instances > depends up on the cases. > > WK: s/MHPE1,MHPE2,SHPE3/MHPE1, MHPE2, SHPE3/ (and elsewhere) > > > MHPE1 and MHEPE2 must be configured with ESI per vlan or ESI on IFD. > > WK: What is an IFD? > > > MHPE1 and MHEPE2 are running Single Active mode of EVPN. > > CE is acting as bridge configured with vlans that is configured on > MHPE1,MHPE2,SHPE3. > > Depends up on the test traffic will be flowing uni directional or bi > directional depends on the test performed. > > WK: This is unparsable (and even if I could parse it I'm not sure what > it is related to) > > > The above configuration will be serving as the base configuration for > all test cases. > > 3. Test Cases for EVPN Benchmarking > > 3.1. Local MAC Learning > > Objective: > > To Record the time taken to learn the MAC address locally in DUT. > > Topology : Topology 1 > > Procedure: > > The data plane MAC learning can be measured using the parameters > defined in RFC 2889 section 5.8. Send "X" unicast frames from CE to > MHPE1(DUT) working in SA mode with "X" different source and > destination address from RT. The DUT must learn these "X" macs in > data plane. > > WK: MACs (and elsewhere) > > > Measurement : > > Measure the time taken to learn "X" MACs in DUT evpn mac table. The > data plane measurement is taken by considering DUT as black box the > range of X MAC is known from RT and the same must be learned in DUT, > the time taken to learn "X" macs is measured. > > WK: This is also largely unparsable. > > > > Repeat these test and plot the data. The test is repeated for "N" > times and the values are collected. The mac learning time is > calculated by averaging the values obtained from "N" samples. > > Mac learning in sec = (T1+T2+..Tn/N) > > WK: Presumably (T1+T2+..Tn)/N ? > > > 3.2. Remote MAC Learning > > Objective: > > To Record the time taken to learn the remote macs. > > Topology : Topology 1 > > Procedure: > > > Send X frames with X different SA and DA to SHPE3 from RT. > > WK: Please define / expand SA / DA. Also, SA conflicts with "SA EVPN > Terminologies SA Single-Active." > > SHPE3 > will advertise these locally learned macs to MHPE1 and MHPE2 via > control plane.Measure the time taken to learn these X MACs from > remote peer in DUT EVPN MAC address table.The DUT and MHPE2 are > running SA mode. > > Measurement : > > Measure the time taken by the DUT to learn the "X" MACs in the data > plane.Repeat these test and plot the data.The test is repeated for > > WK: Repeat these tests... > > "N" times and the values are collected.The mac learning time is > calculated by averaging the values obtained from "N" samples. > > Mac learning in sec = (T1+T2+..Tn/N) > > WK: As above. Also, what *exactly* do I report? This isn't (AFAICT) > "Mac learning in sec", it is (presumably) MAC Learning Rate, and I'd > *assume* that I report a line graph showing MACs learnt vs time? This > (and the other tests) need to be *much* more explicit about what > exactly is being reported. > > --- SNIP ---- > > 3.9. ARP/ND Scale > > These tests are conducted to Record the scaling parameter of ARP/ND > of the DUT. > > Objective: > > To Record the ARP/ND scale of the DUT. > > Topology : Topology 1 > > Procedure: > > Send X arp/icmpv6 request from RT to DUT with different sender ip/ > ipv6 address to the same target gateway ip address. Measure whether > X MAC+IPv4 address/MAC+IPv6 address of the hosts are learned in DUT. > > WK: I don't think I can run this test -- how fast do I send these? What > *exactly* is the packet I'm sending (icmpv6 is not a packet, it's a protocol) > Where do I send it? (to DUT? to the same target gateway IP? What > does that even mean?) > The X MAC+IPv4 bit doesn't make sense. > > > Measurement : > > The DUT must learn X MAC+IPV4/MAC+IPv6 and it must advertise the X > MAC+IPV4/MAC+IPV6 to the remote router. > > 3.10. Scaling of Services > > Objective: > > To measure the scale limit of DUT for EVPN.This is to measure the > performance of DUT in scaling to "X" EVPN instances. > > Topology : Topology 1 > > Procedure: > The DUT,MHPE2 and SHPE3 are scaled to "N" EVI.Clear BGP neighbors of > the DUT. Once adjacency is established in the DUT. > > WK: Fragment? > > Measure the > routes received from MHPE2 and SHPE3 for "N" EVI in the DUT. > > Measurement : > > There should not be any loss of route types 1,2,3 and 4 in DUT. DUT > must relearn all type 1,2,3 and 4 from remote routers. The DUT must > be subjected to various values of N to find the optimal scale limit > > WK: Do the first 2 sentences mean the same thing? Presumable I *do* > lose all the routes, but then I should recover them? Why is this being > compared to what I had before clearing, and not just the maximum / > number sent from the tester? What ratio should there be for types 1, > 2, 3, 4? Is it important? > > > 3.11. Scale Convergence > > Objective: > > To measure the convergence time of DUT when the DUT is scaled with > EVPN instance along with traffic. > > Topology : Topology 1 > > Procedure: > > > Scale N EVIs in DUT,SHPE3 and MHPE2.Send F frames to DUT from CE > using traffic generator with X different SA and DA for N EVI's. Send > F frames from traffic generator to SHPE3 with X different SA and DA. > There will be 2X number of MAC address will be learned in DUT EVPN > MAC table. > > WK: Doesn't parse. > > There is a bi directional traffic flow with F pps in each > direction. Then clear the BGP neighbors in the DUT. Once the > adjacency is restored in DUT. Measure the time taken to learn 2X MAC > address in DUT MAC table. > > WK: Also doesn't parse. > > > Measurement : > > The DUT must learn 2X MAC address. Measure the time taken to learn > 2X MAC in DUT. Repeat these test and plot the data.The test is > repeated for "N" times and the values are collected.The convergence > time is calculated by averaging the values obtained by "N" samples. > > Convergence time in sec = (T1+T2+..Tn/N) > > 3.12. SOAK Test. > > Objective: > > This test is carried out to measure the stability of the DUT in a > scaled environment with traffic over a period of time "T'". In each > interval "t1" the DUT CPU usage, memory usage are measured. The DUT > is checked for any crashes during this time period. > > Topology : Topology 1 > > Procedure: > > > Scale N EVI's in DUT,SHPE3 and MHPE2.Send F frames to DUT from CE > using traffic generator with different X SA and DA for N EVI's. Send > F frames from traffic generator to SHPE3 with X different SA and DA. > There will be 2X number of MAC address will be learned in DUT EVPN > MAC table. There is a bi directional traffic flow with F pps in each > direction. The DUT must run with traffic for 24 hours, every hour > check for memory leak, CPU usage and crash. > > Measurement : > > Take the hourly reading of CPU, process memory. There should not be > any leak, crashes, CPU spikes. > > WK: What all am I supposed to be reporting for this test? Can N be 1 > and F be 1 and X be 1? I'm happy to do that and report no issues for > 24 hours.... > > > 4. Test Cases for PBB-EVPN Benchmarking > > WK: ----- Are these the same as for "Test Cases for EVPN > Benchmarking"? If so, why are they repeated? > > 4.8. High Availability > > Objective: > > To record traffic loss during routing engine failover. > > Topology : Topology 1 > > Procedure: > > > Send X frames to DUT with X different SA and DA from CE using the > traffic generator. Send X frames from traffic generator to SHPE3 > with X different SA and DA so that 2X MAC address will be Learned in > DUT. There is a bi directional traffic flow with X pps in each > direction. Then do a routing engine fail-over. > > Measurement : > > There should be 0 traffic loss which is the ideal case, No change in > the DF role. DUT should not withdraw any routes.Repeat the test "N" > times and plot the data.The packet loss is calculated by averaging > the values obtained from "N" samples. > > Packet loss in sec = (T1+T2+..Tn/N) > > WK: What exactly am I reporting? "There should be 0 traffic loss which > is the ideal case" - am I reporting > the number of seconds with dropped packets? What if it *does* withdraw > routes? What do I report if I don't have multiple REs? Presumably if X > == 2 I can do this easily, but if X == 1e8 it's harder - but all I'm > reporting is "Packet loss". > > > > --- SNIP --- > > 4.11. Soak Test > > Objective: > > To measure the stability of the DUT in a scaled environment with > traffic. > > Topology : Topology 1 > > Procedure: > > Scale N PBB-EVI's in DUT,SHPE3 and MHPE2.Send F frames to DUT from CE > using traffic generator with X different SA and DA for N EVI's. Send > F frames from traffic generator to SHPE3 with X different SA and DA. > There will be 2X number of MAC address will be learned in DUT PBB- > EVPN MAC table. There is a bi directional traffic flow with F pps in > Each direction. The DUT must run with traffic for 24 hours, every > hour check the memory leak, crashes. > > Measurement : > > Take the hourly reading of CPU process, memory usages. There should > not be any memory leak, crashes,CPU spikes. > > WK: "any memory leak, crashes,CPU spikes" is not at all clear - e.g: > How are you defining a 'spike'? If the CPU jumps from 2% to 10% for 3 > seconds, is that a spike? Many routers run a BGP reaper every N > seconds / minutes, which looks like a spike - is this an issue? > > 7. Security Considerations > > There is no additional consideration from RFC 6192. > > WK: I thought that all BMWG documents included standard boilerplate > for Security Considerations? > > ---------------- > > -- > I don't think the execution is relevant when it was obviously a bad > idea in the first place. > This is like putting rabid weasels in your pants, and later expressing > regret at having chosen those particular rabid weasels and that pair > of pants. > ---maf > > _______________________________________________ > bmwg mailing list > bmwg@ietf.org > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/bmwg__;!8WoA6RjC81c!UJm1S-C5cGIqsiU6lghEbZsWCL6Y-nyqkrkYpBbb_QrHnEHb3tdHYyz1u4wevoE$ ------------------------------ Subject: Digest Footer _______________________________________________ bmwg mailing list bmwg@ietf.org https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/bmwg__;!8WoA6RjC81c!UJm1S-C5cGIqsiU6lghEbZsWCL6Y-nyqkrkYpBbb_QrHnEHb3tdHYyz1u4wevoE$ ------------------------------ End of bmwg Digest, Vol 183, Issue 1 ************************************
- Re: [bmwg] bmwg Digest, Vol 183, Issue 1 Sudhin Jacob