Re: [bmwg] REG: https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch-03

"Maciek Konstantynowicz (mkonstan)" <mkonstan@cisco.com> Tue, 17 November 2020 21:10 UTC

Return-Path: <mkonstan@cisco.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E1C593A0A2E for <bmwg@ietfa.amsl.com>; Tue, 17 Nov 2020 13:10:23 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -9.6
X-Spam-Level:
X-Spam-Status: No, score=-9.6 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=cisco.com header.b=N4l619lB; dkim=pass (1024-bit key) header.d=cisco.onmicrosoft.com header.b=XL1Qw7UF
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8nNO6hSyWwnX for <bmwg@ietfa.amsl.com>; Tue, 17 Nov 2020 13:10:21 -0800 (PST)
Received: from alln-iport-6.cisco.com (alln-iport-6.cisco.com [173.37.142.93]) (using TLSv1.2 with cipher DHE-RSA-SEED-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id E6D083A0EA9 for <bmwg@ietf.org>; Tue, 17 Nov 2020 13:10:00 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=27017; q=dns/txt; s=iport; t=1605647401; x=1606857001; h=from:to:cc:subject:date:message-id:references: in-reply-to:mime-version; bh=NWXzDw2GbPpevExP1Hoyq/yHdxKlQI4fDKLk5zl5bXc=; b=N4l619lBxWO4Etn+fnGEQ/gDZyK63dOQPNIyVJOFwJ2BEVvUz8R/pS6S ipLG9C0hVETSJP1v2O2+QjwQ1tUACmoBjP2N466XXLIInUmRSjcEhXaho CAuP3dQbpICYmGq/+UcKGnCXKVGsmciLgkN/DMZv/DJgs+RLBi4A15YG/ o=;
X-IPAS-Result: A0DgCADsObRffYkNJK1iHgEBCxIMgzIvUXtZLy4KhDKDSQONNpRohEKCUwNUAQoBAQENAQEYAQwIAgQBAYFVgnUCF4ILAiU4EwIDAQEBAwIDAQEBAQUBAQECAQYEFAEBhjwMhXMCBAEBEBEdAQEsCwEPAgEIOAcDAgICJQsUEQIEDgUZCYMEAYF+VwMuAQ6jUAKBPIhodoEygwQBAQWBNwIBDUFEgj4YghAJgTiCc4N2hlcbgUE/gREnDBCCTz6CXQEBAgEBgVouEYJZM4IskEqBFoIKhx6LK2SQJ3kKgm2JEZIIAx+DGYVGhFCUSoYfmDORc4NkAgQCBAUCDgEBBYFrIYFZcBU7KgGCPgk1EhcCDY4fNxiDIoJkgjCFRHQCNQIGAQkBAQMJfIw7AYEQAQE
IronPort-PHdr: 9a23:xfzashFCbkAm8QXnhyuizp1GYnJ96bzpIg4Y7IYmgLtSc6Oluo7vJ1Hb+e401gObUYDS8fkCiufKvebnQ2NTqZqCsXVXdptKWldFjMgNhAUvDYaDDlGzN//laSE2XaEgHF9o9n22Kw5ZTcD5YVCBrni79zVUGxjjO0xyPOumUoLXht68gua1/ZCbag5UhT27NLV1Khj+rQjYusQMx4V4LaNkwRrSqXwOcONTlm4=
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-AV: E=Sophos;i="5.77,486,1596499200"; d="scan'208,217";a="629252882"
Received: from alln-core-4.cisco.com ([173.36.13.137]) by alln-iport-6.cisco.com with ESMTP/TLS/DHE-RSA-SEED-SHA; 17 Nov 2020 21:09:59 +0000
Received: from XCH-RCD-005.cisco.com (xch-rcd-005.cisco.com [173.37.102.15]) by alln-core-4.cisco.com (8.15.2/8.15.2) with ESMTPS id 0AHL9xcH007588 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=FAIL); Tue, 17 Nov 2020 21:09:59 GMT
Received: from xhs-rcd-001.cisco.com (173.37.227.246) by XCH-RCD-005.cisco.com (173.37.102.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 17 Nov 2020 15:09:59 -0600
Received: from xhs-rtp-003.cisco.com (64.101.210.230) by xhs-rcd-001.cisco.com (173.37.227.246) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 17 Nov 2020 15:09:58 -0600
Received: from NAM02-SN1-obe.outbound.protection.outlook.com (64.101.32.56) by xhs-rtp-003.cisco.com (64.101.210.230) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Tue, 17 Nov 2020 16:09:57 -0500
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fVlBjZcP/REfqKvGV+7bagMtjLO3Q6usYUNMQlKjLOhXMEd9vqjyR0tbS/BBc4Go0hkqihBu3+SgvBKx+njxNdYdVtcukCSFK5UibFCxEyDfMwsLTm0z+yjKmjeumVA1WV9bxGX1wSWm6p7CqajTzWKvykTdDOo8CQyC6rDB+y8suOMzy0cWpSHoI2FPhamRMkPgK3UaNfYGCRoCsNZGbfBKoAJxcA/w3CxJtz6nGyE4QFKhjJddXapWY9cJPrPYFBlvvTJMze+L87tgfUx7i9Q/7MqGgouvTZkGPlevhfOWL5zd8SRdmfRHMuDAwvlbwMxKdIWYemXEoKma8MTqWw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NWXzDw2GbPpevExP1Hoyq/yHdxKlQI4fDKLk5zl5bXc=; b=dhMUjwfsP0ppecWIJ+uHNhAl7ZoYa3QhO5txdCssDjv2eFfvGmPJpImbdHN4RWEf3hiimgUZlf4FtUbiOekyijPejUhtyWUTkXuP9fUZpT4i6RMwya5LT4WLWRYNSoiLgBfCL1w9krWtTweb6Zdc9uHVKamOTEPipzw/Zj6zrn1SC5vuzJSrXl3nTGUjqudVFZxaL3KHDcQnznXGF4oaj2g6qm9BrUfxD0xeiLVlUkTYNRd1pQJbDGNCX2CG0msZioXCq/HCSMFVAZd9eeFvDFAnar5cXqRfC6knldrL1xnXDdUt4iGDQ7mfJ3+O+5c/ENd6s5O1t7TtV2sclbNJjg==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cisco.com; dmarc=pass action=none header.from=cisco.com; dkim=pass header.d=cisco.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.onmicrosoft.com; s=selector2-cisco-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NWXzDw2GbPpevExP1Hoyq/yHdxKlQI4fDKLk5zl5bXc=; b=XL1Qw7UFn2C2mIAfxm6In27oUit0/KbgEf1hqNJ2ornxr3SPDGpF7OuS56oL0r70/TG8HuHAEZrjmbjs/Aqqol9OuNrXqM6/TMNMjBDLaCV9NX3LzS8AnksQibJNbwhMGMsDUANlBvG1B/bjYdeD4689qrCrGwOFdWx1HQb8g7Y=
Received: from DM6PR11MB2924.namprd11.prod.outlook.com (2603:10b6:5:66::10) by DM5PR1101MB2090.namprd11.prod.outlook.com (2603:10b6:4:51::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3589.20; Tue, 17 Nov 2020 21:09:56 +0000
Received: from DM6PR11MB2924.namprd11.prod.outlook.com ([fe80::c3e:bf86:cca1:4c42]) by DM6PR11MB2924.namprd11.prod.outlook.com ([fe80::c3e:bf86:cca1:4c42%2]) with mapi id 15.20.3564.025; Tue, 17 Nov 2020 21:09:56 +0000
From: "Maciek Konstantynowicz (mkonstan)" <mkonstan@cisco.com>
To: Sudhin Jacob <sjacob=40juniper.net@dmarc.ietf.org>
CC: "bmwg@ietf.org" <bmwg@ietf.org>
Thread-Topic: [bmwg] REG: https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch-03
Thread-Index: AQHWvSX/aeqNITC5CkabjR0qq0piJQ==
Date: Tue, 17 Nov 2020 21:09:56 +0000
Message-ID: <67B6B113-8260-4C70-BBC1-40CF6B970FC6@cisco.com>
References: <BYAPR05MB5494C6FA0B43E0FBB0C3DA7CC26B0@BYAPR05MB5494.namprd05.prod.outlook.com>
In-Reply-To: <BYAPR05MB5494C6FA0B43E0FBB0C3DA7CC26B0@BYAPR05MB5494.namprd05.prod.outlook.com>
Accept-Language: en-GB, en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-mailer: Apple Mail (2.3608.120.23.2.4)
authentication-results: dmarc.ietf.org; dkim=none (message not signed) header.d=none;dmarc.ietf.org; dmarc=none action=none header.from=cisco.com;
x-originating-ip: [173.38.220.56]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: c6692ae2-a0cb-418a-4834-08d88b3d22bd
x-ms-traffictypediagnostic: DM5PR1101MB2090:
x-microsoft-antispam-prvs: <DM5PR1101MB2090AE2BCCFF9357681E0DDFCFE20@DM5PR1101MB2090.namprd11.prod.outlook.com>
x-ms-oob-tlc-oobclassifiers: OLM:8273;
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: dg8yePCMZSPPxSRLjQ2Dtb5Fur0X767RycnXI+/WRHtWaY9LpkIqflnf9Ma8O3ZBLuCu6JIvoiYQ1gni/HFlaVDtuEgYdB/nBaPmwpKlaqt7DnGTuHZNTTEv5vDqlae996PbBDbiF1wwejJzhA70uEQiDACtnA48QjGKXS5ud+lh1yaJdehCgIjHKzOIWWzlJ229hoPnlX+MS4ZMVSICsOfBxqL9DmifGv7tkxiEe3d6bAdUbM6/fWynCtVWr8UZcZeSITKW17O9TF0nX03vaTS4R/io60xzUyt/TySaU341ShVZIjUdT/cjygbs3Nh9e8Pf6Rt13+Y+lqaKDz20fPjkJpobW+dWL1TVeUZyAb9A/nuq4OVkpAQyhh7fDoCfCOyAh9OhzJoVcqnKJbUh9Q==
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM6PR11MB2924.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(136003)(39860400002)(376002)(396003)(346002)(366004)(86362001)(966005)(316002)(91956017)(33656002)(36756003)(6506007)(66446008)(66476007)(66556008)(64756008)(66946007)(166002)(5660300002)(478600001)(6486002)(2616005)(8676002)(2906002)(53546011)(4326008)(6512007)(83380400001)(71200400001)(76116006)(8936002)(186003)(26005); DIR:OUT; SFP:1101;
x-ms-exchange-antispam-messagedata: krqd9Reum+24F64ejLyTfrXDo/P9dlxy2458nizxvWs/fiE/rTgDdaqDPKSjmzE9kkidK3/rJp1Krl8ZqCohrtFvSw99AC5uQq+yMIomAYaJkhIAjrnAshDwszgY0zhDicPBL098mZHq/OgfPQrHft6yp6Gzon86mFP/KtOwAvfmPEgm3JxFmuz+xQsic2sS3jx+LA2B3acHHjIr0QyVzb/owCD5tjv/t4XzucksrcQ5htgjGeTIDOiVgFnHg9SToQkJ/Vte4vHqNOWRBiW0YpkZZgwW1NlPBA4X0m9MEUEaJHwJO3CL29Mncb16Q7OuQS93pLO3eT1vln51OHwhJiilqExhIy6AfOtOkt0mmIcv3NbXPk4ODXPkzG20eVCfxEzcEr7CQw3DZOIfUQ/IAdnDvuJH21ZgB0rSeN0IMVD4VXTjZ3taGwPGP+Lh8rJP64KS5ztpkcuF1IQB3MX/ags7+7Y2eZ8mjdJlrVjKIDoXz1AdSaOWCzQB9lQ6Ot/Ph6zvQh2ihgYk1QLoBa2Gv6NR+MhtG792Vtj3yqUx6m+XSsCdUbfCqQ3pycTEJ830cKRjfeDir0+vXExoKZgDdW7uqjB3+iYJ6gmSFo8sXH4hCfSjbJWSSj39JkKw+pQWp9oRZSVnblqcJnhgCCD5u9dWGuiELu4hI9Ve4i/e3bLzAaT+bxkeZg4ylyFELf3UV5YCcm4tQJ33CHi4QtbsrPDzBuA6ODNKi7wNgUfrZ3DiaE5KFR9YgGPTo6XtYbCHzbY937VjPJ75xc22SlzpZxPijMR6jzIjGEupAUURiYlUPRpIQrnDW74tyw13+N0e9xju5QILOO3deMZdslPBW4w9ygPZbh0PsiMg4veeb8efBALOK9HZjLVIqHtOaW0d4lYqBsT/BX1zoctDQQjB7g==
x-ms-exchange-transport-forked: True
Content-Type: multipart/alternative; boundary="_000_67B6B11382604C70BBC140CF6B970FC6ciscocom_"
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: DM6PR11MB2924.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c6692ae2-a0cb-418a-4834-08d88b3d22bd
X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Nov 2020 21:09:56.3082 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 5ae1af62-9505-4097-a69a-c1553ef7840e
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: uLsdak7isOHBlEikdclsU1HLDBuUKjAo7skdgOXDkZyEauFm3f+bGsAUcZ+pBtExwZg2C0IPm88HcDi2AvwsNA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR1101MB2090
X-OriginatorOrg: cisco.com
X-Outbound-SMTP-Client: 173.37.102.15, xch-rcd-005.cisco.com
X-Outbound-Node: alln-core-4.cisco.com
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/0Qk3YYvCwHz0a3smMVxjNeYEEo8>
Subject: Re: [bmwg] REG: https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch-03
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 17 Nov 2020 21:10:24 -0000

Hi Sudhin,

Apologies for late reply, life and pandemic augmented reality got in the way.
See inline.

On 4 Jul 2020, at 05:08, Sudhin Jacob <sjacob=40juniper.net@dmarc.ietf.org<mailto:sjacob=40juniper.net@dmarc.ietf.org>> wrote:


Hi Maciek,

I   had gone through the draft. Could you please clarify the below points. Appreciate it.


Packet Throughput Rate: maximum packet offered load DUT/SUT forwards within the specified Packet Loss Ratio (PLR).

Normally it varies with the frame size isn’t it?

This is correct. Packet throughput is measured per frame size.


Could  you please explain NDR and PDR?

From section-1 of the draft [1]:


   o  Non Drop Rate (NDR): maximum packet/bandwith throughput rate
      sustained by DUT/SUT at PLR equal zero (zero packet loss) specific
      to tested frame size(s).  MUST be quoted with specific packet size
      as received by DUT/SUT during the measurement.  Packet NDR
      measured in packets-per-second (or fps), bandwidth NDR expressed
      in bits-per-second (bps).



   o  Partial Drop Rate (PDR): maximum packet/bandwith throughput rate
      sustained by DUT/SUT at PLR greater than zero (non-zero packet
      loss) specific to tested frame size(s).  MUST be quoted with
      specific packet size as received by DUT/SUT during the
      measurement.  Packet PDR measured in packets-per-second (or fps),
      bandwidth PDR expressed in bits-per-second (bps).

Both are measured per frame size of interest.


Could you please let me know whether this algorithm is used only for NFV, can it be used for Physical network devices?

Any network devices or network device topologies, including VNFs and CNFs or topologies / service chains thereof.


Could you please capture  the state machines of the algorithm in the draft..

I believe we have a good description of the MLRsearch algorithm workings in the draft and have illustrated sample runs in the previous IETF BMWG meeting, see interim post IETF-107 [2].

Let me know if this is sufficient, and/or if you have any further questions.


Could you please explain how this algorithm can perform better than binary search. Kindly clarify the loading condition, what about in 100G,10G?

In line with the goal outlined in the abstract: “MLRsearch is minimizing the total test duration and searching for multiple packet throughput rates (each with a corresponding PLR) concurrently, instead of doing it sequentially.”

MLRsearch is used in LFN FD.io<http://FD.io> CSIT for discovering NDR and PDR rates with equipment connected over 10GE, 25GE and 100GE links.


Could you please explain section 5.2 , the device which the test was conducted on what frame size, boundary conditions.

The test illustrated in section 5.2 is from the FD.io<http://FD.io> CSIT (open-source project in Linux Foundation Networking), and has been done with 64B Ethernet untagged frames. DUT was FD.io<http://FD.io> VPP (open-source software data-plane). For PDR packet-loss-ratio was set to 0.5% (default setting in FD.io<http://FD.io> CSIT for PDR tests).

We discussed it in IETF-107 BMWG meeting, but can do it again this week in IETF-109 BMWG meeting, if there is interest and time permitting.

Cheers,
Maciek

[1] https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch-03#section-1
[2] https://datatracker.ietf.org/doc/slides-interim-2020-bmwg-01-sessa-multiple-loss-ratio-search/


Regards,
Sudhin









Juniper Business Use Only

_______________________________________________
bmwg mailing list
bmwg@ietf.org<mailto:bmwg@ietf.org>
https://www.ietf.org/mailman/listinfo/bmwg