Re: [bmwg] AD Review of draft-ietf-bmwg-evpntest

Hi Warren,
	Thanks for your thorough review. The authors will review the document with your feedback and edit accordingly. While We (and I, specifically) have looked at this document a LOT, we missed some obvious things, and our authors are first time authors; I'll work with them to sort out the obvious catches (thanks for those).

Cheers
Sarah

> On Dec 2, 2019, at 2:42 PM, Warren Kumari <warren@kumari.net> wrote:
> 
> Hi all,
> 
> Unfortunately, I do not think that this document is ready to be
> progressed -- there are a large number of areas where the document is
> *very* unclear, and / or incomplete. I usually break my reviews up
> into "substantive" and "Nits / Editorial"; but in this case I didn't,
> largely because many of the editorial-type issues make the text
> unclear; there are also a sufficient quantity that I do not feel
> comfortable asking all of the IETF to review, nor would I feel
> comfortable asking the directorates / IESG to review.
> 
> I've annotated my comments with "WK: ", but many of the issues occur
> multiple times -- for example, there are many instances of "SA", which
> from context is "Source Address" but this could also be "SA" in the
> "Single-Active" sense.
> 
> W
> -------------------
> 
> 
> Abstract
> 
>   This document defines methodologies for benchmarking EVPN and PBB-
>   EVPN performance.  EVPN is defined in RFC 7432, and is being deployed
>   in Service Provider networks.  Specifically this document defines the
> 
> WK (nit) : Specifically, this (missing comma)
> 
>   methodologies for benchmarking EVPN/PBB-EVPN convergence, data plane
>   performance, and control plane performance.
> 
> 
> 
> 1.  Introduction
> 
>   EVPN is defined in RFC 7432, and describes BGP MPLS- based Ethernet
>   VPNs (EVPN).  PBB-EVPN is defined in RFC 7623, discusses how Ethernet
>   Provider backbone Bridging can be combined with EVPNs to provide a
>   new/combined solution.  This draft defines methodologies that can be
>   used to benchmark both RFC 7432 and RFC 7623 solutions.  Further,
>   this draft provides methodologies for benchmarking the performance of
>   EVPN data and control planes, MAC learning, MAC flushing, MAC ageing,
> 
> WK: s/ageing/aging  (RFC index has 258 instances of 'aging' and only 4 'ageing')
> 
>   convergence, high availability, and scale.
> 
> 1.1.  Requirements Language
> 
>   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
>   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
>   document are to be interpreted as described in RFC 2119 [RFC2119].
> 
> WK: Replace RFC2119 with RFC8174 language.
> 
> 
> 
> WK: Please fix formatting, and sort alphabetically. Also, this isn't
> "Terminology", it is mainly Abbreviations.
> 
> 1.2.  Terminologies
> 
>   MHPE Multi homed Provide Edge router.
> 
>   RR Route Reflector.
> 
>   P Provider Router.
> 
>   CE Customer Router/Devices/Switch.
> 
>   MHPE2 Multi homed Provider Edge router 2.
> 
>   MHPE1 Multi homed Provider Edge router 1.
> 
>   SHPE3 Single homed Provider Edge Router 3.
> 
>   AA EVPN Terminologies AA All-Active.
> 
>   SA EVPN Terminologies SA Single-Active.
> 
>   RT Router Tester.
> 
>   Sub Interface Each physical Interfaces is subdivided in to Logical
>   units.
> WK: s/in to/into/
> 
>   EVI EVPN Instances which will be running on sub interface or physical
>   port of the provider Edge routers.
> 
>   DF Designated Forwarder.
> 
>   ESI Ethernet Segment Identifier.
> 
> 2.  Test Topology
> 
>   EVPN/PBB-EVPN Services running on SHPE3, MHPE1 and MHPE2 in Single
>   Active Mode:
> 
> 
>         | [Traffic Generator ] Router Tester traffic sender/receiver
> of layer 2 traffic with multiple vlan.
> 
> WK: VLANs. Also, this doesn't really parse - is 'Router Tester'
> supposed to be in the brackets?
> 
> +----------+
> |          |
> |  SHPE3   |
> |          |
> +----------+
>    |
>    |Core link
> +----------+
> |          |
> |  RR      |
> |          | Route Reflector/Core router
> 
> WK: Core router or Provider router? (Core router isn't defined in the
> terminology section)
> 
> +----------+-------------|
>   |                     |
>   |     Core links      |
> +----------+       +-----------+
> |          |       |    MHPE2  |
> |   DUT    |       |           |
> |  MHPE1   |       |           |
> +----------+       +-----------+
>     |    PE-CE link    |
> +----------+------------
> |          |
> |  CE      |
> |  layer2  |
> |bridge    |
> +----------+------------ [Traffic Generator](Router Tester
> sender/reciever of layer 2 traffic with multiple vlan)
> 
> WK: As above. Aslo, receiver is a typo
> 
> 
> WK: The formatting of this table is very poor, esp the header.
> 
> 
> 
> +-----------------+---------------------+---------------------+---------------------+----------------------+-----------------------+
> |                 |                     |                     |
>             |                      |                       |
> |                 |                     |                     |
>             |                      |                       |
> |                 |                     |                     |
>             |                      |                       |
> |                 |                     |                     |
>             |                      |                       |
> | Mode            |                     |                     |
>             |Receiver              |                       |
> |                 |  Test               |Traffic Direction    |Sender
>             |                      |                       |
> |                 |                     |                     |
>             |                      |                       |
> |                 |                     |                     |
>             |                      |                       |
> |                 |                     |                     |
>             |                      |                       |
> +----------------------------------------------------------------------------------------------------------------------------------+
> |                 |                     |                     |
>             |                      |                       |
> |                 |                     |                     |
>             |      SHPE3           |                       |
> |Single Active    |  Local Mac          |                     |CE
>             |                      |Layer 2 traffic        |
> |                 | Learning            | Uni                 |
>             |                      |                       |
> |                 |                     |                     |
>             |                      | multiple MAC          |
> |                 |                     |                     |
>             |                      |                       |
> +-----------------------------------------------------------------------------------------------------------------------------------+
> |                 |                     |                     |
>             |                      |                       |
> |Single Active    | Remote MAC          |                     |
>             |         CE           |Layer 2 traffic        |
> |                 | Learning            | uni                 | SHPE3
>             |                      |                       |
> |                 |                     |                     |
>             |                      |multiple MAC           |
> |                 |                     |                     |
>             |                      |                      ++
> +----------------------------------------------------------------------------------------------------------------------------------+
> |                 |                     |                     |
>             |                      |                       |
> |Single Active    | Scale Convergence   |   Bi                |
>             |  CE/SHPE3            |                       |
> |                 |                     |                     |
> CE/SHPE3          |                      |Layer 2 traffic        |
> |                 | Local& Remote       |                     |
>             |                      |multiple mac& vlans    |
> |                 | Learning            |                     |
>             |                      |                       |
> +-----------------+---------------------+---------------------+--------------------------------------------+-----------------------+
> 
>             |
> --- SNIP ---
> 
>   Test Setup Configurations:
> 
>   There are five routers in the Test setup.  SHPE3, RR/P, MHPE1 and
>   MHPE2 emulating a service provider network.  CE is a customer device
>   connected to MHPE1 and MHPE2, it is configured with bridge domains in
>   multiple vlans.  The router tester is connected to CE and SHPE3.The
>   MHPE1 acts as DUT.The RT will be used as sender and receiver of
>   traffic.The measurement will be taken in DUT.
> 
> 
> WK: What exactly is a "Router Tester"? This term doesn't seem to be defined
> anywhere -- is it just a traffic generator?
> Please also fix the formatting above (missing spaces after periods) -
> this is throughout the document.
> 
>   All routers except CE is configured with OSPF/IS-IS,LDP,MPLS,BGP with
>   EVPN address family.
> 
> WK: All routers except CE are configured with. "Configured with"
> doesn't really provide
>  sufficient detail to allow repeatable testing.
> 
> 
>   All routers except CE must have IBGP configured with RR acting as
>   route reflector.
> 
> WK: iBGP. Also, expand acronyms.
> 
>   MHPE1,MHPE2,SHPE3 must be configured with "N" EVPN/PBB-EVPN instances
>   depends up on the cases.
> 
> WK: s/MHPE1,MHPE2,SHPE3/MHPE1, MHPE2, SHPE3/ (and elsewhere)
> 
> 
>   MHPE1 and MHEPE2 must be configured with ESI per vlan or ESI on IFD.
> 
> WK: What is an IFD?
> 
> 
>   MHPE1 and MHEPE2 are running Single Active mode of EVPN.
> 
>   CE is acting as bridge configured with vlans that is configured on
>   MHPE1,MHPE2,SHPE3.
> 
>   Depends up on the test traffic will be flowing uni directional or bi
>   directional depends on the test performed.
> 
> WK: This is unparsable (and even if I could parse it I'm not sure what
> it is related to)
> 
> 
>   The above configuration will be serving as the base configuration for
>   all test cases.
> 
> 3.  Test Cases for EVPN Benchmarking
> 
> 3.1.  Local MAC Learning
> 
>   Objective:
> 
>   To Record the time taken to learn the MAC address locally in DUT.
> 
>   Topology : Topology 1
> 
>   Procedure:
> 
>   The data plane MAC learning can be measured using the parameters
>   defined in RFC 2889 section 5.8.  Send "X" unicast frames from CE to
>   MHPE1(DUT) working in SA mode with "X" different source and
>   destination address from RT.  The DUT must learn these "X" macs in
>   data plane.
> 
> WK: MACs (and elsewhere)
> 
> 
>   Measurement :
> 
>   Measure the time taken to learn "X" MACs in DUT evpn mac table.  The
>   data plane measurement is taken by considering DUT as black box the
>   range of X MAC is known from RT and the same must be learned in DUT,
>   the time taken to learn "X" macs is measured.
> 
> WK: This is also largely unparsable.
> 
> 
> 
>   Repeat these test and plot the data.  The test is repeated for "N"
>   times and the values are collected.  The mac learning time is
>   calculated by averaging the values obtained from "N" samples.
> 
>   Mac learning in sec = (T1+T2+..Tn/N)
> 
> WK: Presumably (T1+T2+..Tn)/N ?
> 
> 
> 3.2.  Remote MAC Learning
> 
>   Objective:
> 
>   To Record the time taken to learn the remote macs.
> 
>   Topology : Topology 1
> 
>   Procedure:
> 
> 
>   Send X frames with X different SA and DA to SHPE3 from RT.
> 
> WK: Please define / expand SA / DA. Also, SA conflicts with "SA EVPN
> Terminologies SA Single-Active."
> 
>   SHPE3
>   will advertise these locally learned macs to MHPE1 and MHPE2 via
>   control plane.Measure the time taken to learn these X MACs from
>   remote peer in DUT EVPN MAC address table.The DUT and MHPE2 are
>   running SA mode.
> 
>   Measurement :
> 
>   Measure the time taken by the DUT to learn the "X" MACs in the data
>   plane.Repeat these test and plot the data.The test is repeated for
> 
> WK: Repeat these tests...
> 
>   "N" times and the values are collected.The mac learning time is
>   calculated by averaging the values obtained from "N" samples.
> 
>   Mac learning in sec = (T1+T2+..Tn/N)
> 
> WK: As above.  Also, what *exactly* do I report? This isn't (AFAICT)
> "Mac learning in sec", it is (presumably) MAC Learning Rate, and I'd
> *assume* that I report a line graph showing MACs learnt vs time? This
> (and the other tests) need to be *much* more explicit about what
> exactly is being reported.
> 
> --- SNIP ----
> 
> 3.9.  ARP/ND Scale
> 
>   These tests are conducted to Record the scaling parameter of ARP/ND
>   of the DUT.
> 
>   Objective:
> 
>   To Record the ARP/ND scale of the DUT.
> 
>   Topology : Topology 1
> 
>   Procedure:
> 
>   Send X arp/icmpv6 request from RT to DUT with different sender ip/
>   ipv6 address to the same target gateway ip address.  Measure whether
>   X MAC+IPv4 address/MAC+IPv6 address of the hosts are learned in DUT.
> 
> WK: I don't think I can run this test -- how fast do I send these? What
>  *exactly* is the packet I'm sending (icmpv6 is not a packet, it's a protocol)
>  Where do I send it? (to DUT? to the same target gateway IP? What
> does that even mean?)
>  The X MAC+IPv4 bit doesn't make sense.
> 
> 
>   Measurement :
> 
>   The DUT must learn X MAC+IPV4/MAC+IPv6 and it must advertise the X
>   MAC+IPV4/MAC+IPV6 to the remote router.
> 
> 3.10.  Scaling of Services
> 
>   Objective:
> 
>   To measure the scale limit of DUT for EVPN.This is to measure the
>   performance of DUT in scaling to "X" EVPN instances.
> 
>   Topology : Topology 1
> 
>   Procedure:
>   The DUT,MHPE2 and SHPE3 are scaled to "N" EVI.Clear BGP neighbors of
>   the DUT.  Once adjacency is established in the DUT.
> 
> WK: Fragment?
> 
>    Measure the
>   routes received from MHPE2 and SHPE3 for "N" EVI in the DUT.
> 
>   Measurement :
> 
>   There should not be any loss of route types 1,2,3 and 4 in DUT.  DUT
>   must relearn all type 1,2,3 and 4 from remote routers.  The DUT must
>   be subjected to various values of N to find the optimal scale limit
> 
> WK: Do the first 2 sentences mean the same thing? Presumable I *do*
> lose all the routes, but then I should recover them? Why is this being
> compared to what I had before clearing, and not just the maximum /
> number sent from the tester? What ratio should there be for types 1,
> 2, 3, 4? Is it important?
> 
> 
> 3.11.  Scale Convergence
> 
>   Objective:
> 
>   To measure the convergence time of DUT when the DUT is scaled with
>   EVPN instance along with traffic.
> 
>   Topology : Topology 1
> 
>   Procedure:
> 
> 
>   Scale N EVIs in DUT,SHPE3 and MHPE2.Send F frames to DUT from CE
>   using traffic generator with X different SA and DA for N EVI's.  Send
>   F frames from traffic generator to SHPE3 with X different SA and DA.
>   There will be 2X number of MAC address will be learned in DUT EVPN
>   MAC table.
> 
> WK: Doesn't parse.
> 
>   There is a bi directional traffic flow with F pps in each
>   direction.  Then clear the BGP neighbors in the DUT.  Once the
>   adjacency is restored in DUT.  Measure the time taken to learn 2X MAC
>   address in DUT MAC table.
> 
> WK: Also doesn't parse.
> 
> 
>   Measurement :
> 
>   The DUT must learn 2X MAC address.  Measure the time taken to learn
>   2X MAC in DUT.  Repeat these test and plot the data.The test is
>   repeated for "N" times and the values are collected.The convergence
>   time is calculated by averaging the values obtained by "N" samples.
> 
>   Convergence time in sec = (T1+T2+..Tn/N)
> 
> 3.12.  SOAK Test.
> 
>   Objective:
> 
>   This test is carried out to measure the stability of the DUT in a
>   scaled environment with traffic over a period of time "T'".  In each
>   interval "t1" the DUT CPU usage, memory usage are measured.  The DUT
>   is checked for any crashes during this time period.
> 
>   Topology : Topology 1
> 
>   Procedure:
> 
> 
>   Scale N EVI's in DUT,SHPE3 and MHPE2.Send F frames to DUT from CE
>   using traffic generator with different X SA and DA for N EVI's.  Send
>   F frames from traffic generator to SHPE3 with X different SA and DA.
>   There will be 2X number of MAC address will be learned in DUT EVPN
>   MAC table.  There is a bi directional traffic flow with F pps in each
>   direction.  The DUT must run with traffic for 24 hours, every hour
>   check for memory leak, CPU usage and crash.
> 
>   Measurement :
> 
>   Take the hourly reading of CPU, process memory.  There should not be
>   any leak, crashes, CPU spikes.
> 
> WK: What all am I supposed to be reporting for this test? Can N be 1
> and F be 1 and X be 1? I'm happy to do that and report no issues for
> 24 hours....
> 
> 
> 4.  Test Cases for PBB-EVPN Benchmarking
> 
> WK: ----- Are these the same as for "Test Cases for EVPN
> Benchmarking"? If so, why are they repeated?
> 
> 4.8.  High Availability
> 
>   Objective:
> 
>   To record traffic loss during routing engine failover.
> 
>   Topology : Topology 1
> 
>   Procedure:
> 
> 
>   Send X frames to DUT with X different SA and DA from CE using the
>   traffic generator.  Send X frames from traffic generator to SHPE3
>   with X different SA and DA so that 2X MAC address will be Learned in
>   DUT.  There is a bi directional traffic flow with X pps in each
>   direction.  Then do a routing engine fail-over.
> 
>   Measurement :
> 
>   There should be 0 traffic loss which is the ideal case, No change in
>   the DF role.  DUT should not withdraw any routes.Repeat the test "N"
>   times and plot the data.The packet loss is calculated by averaging
>   the values obtained from "N" samples.
> 
>   Packet loss in sec = (T1+T2+..Tn/N)
> 
> WK: What exactly am I reporting? "There should be 0 traffic loss which
> is the ideal case" - am I reporting
> the number of seconds with dropped packets? What if it *does* withdraw
> routes? What do I report if I don't have multiple REs? Presumably if X
> == 2 I can do this easily, but if X == 1e8 it's harder - but all I'm
> reporting is "Packet loss".
> 
> 
> 
> --- SNIP ---
> 
> 4.11.  Soak Test
> 
>   Objective:
> 
>   To measure the stability of the DUT in a scaled environment with
>   traffic.
> 
>   Topology : Topology 1
> 
>   Procedure:
> 
>   Scale N PBB-EVI's in DUT,SHPE3 and MHPE2.Send F frames to DUT from CE
>   using traffic generator with X different SA and DA for N EVI's.  Send
>   F frames from traffic generator to SHPE3 with X different SA and DA.
>   There will be 2X number of MAC address will be learned in DUT PBB-
>   EVPN MAC table.  There is a bi directional traffic flow with F pps in
>   Each direction.  The DUT must run with traffic for 24 hours, every
>   hour check the memory leak, crashes.
> 
>   Measurement :
> 
>   Take the hourly reading of CPU process, memory usages.  There should
>   not be any memory leak, crashes,CPU spikes.
> 
> WK: "any memory leak, crashes,CPU spikes" is not at all clear - e.g:
> How are you defining a 'spike'? If the CPU jumps from 2% to 10% for 3
> seconds, is that a spike? Many routers run a BGP reaper every N
> seconds / minutes, which looks like a spike - is this an issue?
> 
> 7.  Security Considerations
> 
>   There is no additional consideration from RFC 6192.
> 
> WK: I thought that all BMWG documents included standard boilerplate
> for Security Considerations?
> 
> ----------------
> 
> -- 
> I don't think the execution is relevant when it was obviously a bad
> idea in the first place.
> This is like putting rabid weasels in your pants, and later expressing
> regret at having chosen those particular rabid weasels and that pair
> of pants.
>   ---maf
> 
> _______________________________________________
> bmwg mailing list
> bmwg@ietf.org
> https://www.ietf.org/mailman/listinfo/bmwg