Re: [bmwg] I-D Action: draft-ietf-bmwg-b2b-frame-00.txt

"Vratko Polak -X (vrpolak - PANTHEON TECH SRO at Cisco)" <vrpolak@cisco.com> Fri, 15 May 2020 17:29 UTC

Return-Path: <vrpolak@cisco.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E318A3A1004 for <bmwg@ietfa.amsl.com>; Fri, 15 May 2020 10:29:14 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -9.598
X-Spam-Level:
X-Spam-Status: No, score=-9.598 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=cisco.com header.b=hJy0uMXZ; dkim=pass (1024-bit key) header.d=cisco.onmicrosoft.com header.b=dh6RkRgr
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Gmw8yf-78saj for <bmwg@ietfa.amsl.com>; Fri, 15 May 2020 10:29:11 -0700 (PDT)
Received: from rcdn-iport-7.cisco.com (rcdn-iport-7.cisco.com [173.37.86.78]) (using TLSv1.2 with cipher DHE-RSA-SEED-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id E88763A0FD0 for <bmwg@ietf.org>; Fri, 15 May 2020 10:28:54 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=22296; q=dns/txt; s=iport; t=1589563734; x=1590773334; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=1prkWsqwyBZ47QU782KflRh4gI4iUzN/atfvokUl7bo=; b=hJy0uMXZ8IvfyWl8mfQTr9RTXm0TP/4CjETDjX13KCOlG7siOaQGdxNy Ihg8+vWqgZAoxJmbenqJaMfiRqqEuedw/lqy0eogQZKg8Nkvpc/jNWdxM gRxVZqAxrNc1Sp0A9ioL82wVBPxMfpJ3suidP0n3UWcT78fGWdnZlARbW M=;
IronPort-PHdr: =?us-ascii?q?9a23=3ATvfwqha6RC5yzsT7dYIypFn/LSx94ef9IxIV55?= =?us-ascii?q?w7irlHbqWk+dH4MVfC4el21QWZD5/W8P9NkeHW9an8RSoL5pPS+HwBcZkZUR?= =?us-ascii?q?gDhI1WmgE7G8eKBAX9K+KidC01GslOFToHt3G2OERYAoDyMlvVpHDh9zMJFx?= =?us-ascii?q?zlPgozL/jpXIXVipf/2+W74ZaGZQJOiXK0aq9zKxPjqwLXu4EWjIJuJ7x3xA?= =?us-ascii?q?HOpy5Dev9dwiVjIlfAkg=3D=3D?=
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: =?us-ascii?q?A0AlCAA90L5e/5ldJa1fBx0BAQEBCQE?= =?us-ascii?q?SAQUFAUCBR4FUKSgHb1gvLAqEG4NGA41CgQGXOoFCgRADVAsBAQEMAQEYDQg?= =?us-ascii?q?CBAEBgVCCdAIXggEkOBMCAwEBCwEBBQEBAQIBBQRthVYMhXEBAQEBAgEBARA?= =?us-ascii?q?REQwBASwLAQQHBAIBCBEEAQEDAhESAwICAiULFAEICAIEDgUIGoMFgksDDiA?= =?us-ascii?q?BAwumGQKBOYhhdoEygwEBAQWBNgKEBhiCDgMGgQ4qgmOJXxqCAIERQ4FPSTU?= =?us-ascii?q?+gmcBAQIBGYECLQUXPYJVM4ItilGDXB8EECCCXIZGilKPGgZ3Ch6CMIgihgm?= =?us-ascii?q?EXIVYmFiEeI8qiniTZAIEAgQFAg4BAQWBaSKBVnAVO4JpUBgNkEAMF4EDAQe?= =?us-ascii?q?CRIUUhUJ0NwIGCAEBAwl8i3IrAYEJAS8BXwEB?=
X-IronPort-AV: E=Sophos;i="5.73,396,1583193600"; d="scan'208";a="758747375"
Received: from rcdn-core-2.cisco.com ([173.37.93.153]) by rcdn-iport-7.cisco.com with ESMTP/TLS/DHE-RSA-SEED-SHA; 15 May 2020 17:28:53 +0000
Received: from XCH-ALN-004.cisco.com (xch-aln-004.cisco.com [173.36.7.14]) by rcdn-core-2.cisco.com (8.15.2/8.15.2) with ESMTPS id 04FHSraQ031778 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=FAIL); Fri, 15 May 2020 17:28:53 GMT
Received: from xhs-rcd-002.cisco.com (173.37.227.247) by XCH-ALN-004.cisco.com (173.36.7.14) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 15 May 2020 12:28:53 -0500
Received: from xhs-rtp-003.cisco.com (64.101.210.230) by xhs-rcd-002.cisco.com (173.37.227.247) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 15 May 2020 12:28:52 -0500
Received: from NAM02-BL2-obe.outbound.protection.outlook.com (64.101.32.56) by xhs-rtp-003.cisco.com (64.101.210.230) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Fri, 15 May 2020 13:28:52 -0400
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Gc18YY4pAhrqSq0wglHZmdyqjYNYBw4w/5L9jIc2WD2/gV5NqaVfgbQWI0nxNKBmIMCojejEhNYzcEkJ21qSpEpG1ETFlZzONFykh/zbDIOvYAghiODOdYmXIcl5H8NGv8XawFWYOFwREBsVQ02Mryqu/kpgZWrARujrWQa8aPi+QP5+46kSJV/OdTxjd9iKu0Fk+Kgo6P/yH5Z8DMuiPO4WYT0/FakT09Nm5HtiWmr+tfeA6R4BPcY4Ovby6QG7kXMzvHCIrGqy2/WFQ8INkyJBvGGlYh/Sg7hba44M9HyeNHS6dLHBACTbVmSip5eMlH3VA7v7P1mfdswb0pk2DA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1prkWsqwyBZ47QU782KflRh4gI4iUzN/atfvokUl7bo=; b=PQUbO1WYDQeTdP61eN1tNphsQlA8WENfvtoIPeP7AHKclsE9okvwE9yKgJ3+zbtT6teqkD5lGPvHe0xgtnzFTZ57z5GTCP9kzZWfbBpxeE5EW2oZSgA7yQevUTDzmpfiyPWu0qewts3nKeVx7BK8K83CSGaY37+XCs00psrhkQY6bc34v1vhTbzgZ9a9+AcLMmNU9WKluWuu2uvNIyIr3WVixeHWw0Zl42bTbfiJNgqbnGs3Scm6X6XfXheD54kZQZemRE3lndvRZliA4l7k8mU+sbQ6G0tF5tZ7CeWk7hTB6nVQ0HHoZ3c36SDdvowSoPB/PiywNGNpEPZiXAhi7g==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cisco.com; dmarc=pass action=none header.from=cisco.com; dkim=pass header.d=cisco.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.onmicrosoft.com; s=selector2-cisco-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1prkWsqwyBZ47QU782KflRh4gI4iUzN/atfvokUl7bo=; b=dh6RkRgrMrGRwHn6HQa29/jY19/d6F8/gvj+ZrgSOjf0H9KHQk9uU7D18tzCnxkqX/kcTWcr7c5iS9inn5JssJjL9b1NEUTMkAaNUMlFYvrUTYfUuM1Myik6bR89v7LFBP9IR6SSxwvyZ21ztm83AzEpfHBiZNuF5rCWMSw3oHA=
Received: from CH2PR11MB4343.namprd11.prod.outlook.com (2603:10b6:610:3e::25) by CH2PR11MB4470.namprd11.prod.outlook.com (2603:10b6:610:4a::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3000.20; Fri, 15 May 2020 17:28:51 +0000
Received: from CH2PR11MB4343.namprd11.prod.outlook.com ([fe80::976:3bf0:83fd:9973]) by CH2PR11MB4343.namprd11.prod.outlook.com ([fe80::976:3bf0:83fd:9973%5]) with mapi id 15.20.3000.016; Fri, 15 May 2020 17:28:51 +0000
From: "Vratko Polak -X (vrpolak - PANTHEON TECH SRO at Cisco)" <vrpolak@cisco.com>
To: "MORTON, ALFRED C (AL)" <acm@research.att.com>
CC: "bmwg@ietf.org" <bmwg@ietf.org>
Thread-Topic: [bmwg] I-D Action: draft-ietf-bmwg-b2b-frame-00.txt
Thread-Index: AQHVMn/lX7eUmdKEBkqb5xm1LswHN6a6/bsAgMSEnRCAFWm8kIEWZxiw
Date: Fri, 15 May 2020 17:28:51 +0000
Message-ID: <CH2PR11MB43430D21D6C8F38C699FBA14BDBD0@CH2PR11MB4343.namprd11.prod.outlook.com>
References: <156225530325.12259.9465843928150563377@ietfa.amsl.com> <8da32ddedf32446698dea5a982d3b8c7@XCH-RCD-014.cisco.com> <4D7F4AD313D3FC43A053B309F97543CFA0B6F3D4@njmtexg5.research.att.com> <BN6PR11MB195671908EC41BF4376698F4BD4F0@BN6PR11MB1956.namprd11.prod.outlook.com>
In-Reply-To: <BN6PR11MB195671908EC41BF4376698F4BD4F0@BN6PR11MB1956.namprd11.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
authentication-results: research.att.com; dkim=none (message not signed) header.d=none; research.att.com; dmarc=none action=none header.from=cisco.com;
x-originating-ip: [195.91.6.15]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 99255653-8c5a-4389-f267-08d7f8f56f78
x-ms-traffictypediagnostic: CH2PR11MB4470:
x-microsoft-antispam-prvs: <CH2PR11MB447026E61D6FFE5A3BCA38F0BDBD0@CH2PR11MB4470.namprd11.prod.outlook.com>
x-ms-oob-tlc-oobclassifiers: OLM:6430;
x-forefront-prvs: 04041A2886
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: buTqmWOp2fliF1mA+4dtxqssbwUEWWWuyWSoV3gC9XqoCfmve9w9GZe4TSRDiG0SWWhR+NOGTsw+o1LVJ4bUMGpp2jGkeQ3/FxDWVH0gPDk6ZT9PynlsesPPu+OC7Vxa4yRK7o2WmdaCcIj3PBhgosRC8ghnDViTsj5wQeikeYafuimvhl6sa4Ueh7UnEI9HxDDIr71H0WsA3ixOKwjISjvOz1f6Qbgnz+JJQ9Cb/fT2HKBf+SynMsekh/J3BwTeXRQv0xhjZRsHDYxJXrBOlz5BwM49Cwt9At9+YD9HvQ496/2KObAuNAT7PBg0/qJfzqGp1Mc3qUM8SI2gdvkk3wv1BS5pa3dkXjcN+4VUogaAXL874zOYzlvRpglVHV7fIK4of7mhayAgu+ivUa5BZ7DL0EJ6NGojCVF2xUUHzlMHavWiqhmE8BJNKO2RFj/hg5VywK4FVeWn9ZSB30/ksco/oYSQaJFiQtgBXMZWC/u2a9PAQXYo8fwMec1MAYEHPbeXvDBQP83OQAY13E0w6zjIA96nvAYHHwKXdMC9HnQ30WSyKWWQwN18b32cxqLM
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR11MB4343.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(4636009)(366004)(346002)(396003)(376002)(136003)(39860400002)(33656002)(8676002)(76116006)(52536014)(53546011)(6506007)(66946007)(8936002)(66446008)(9686003)(7696005)(478600001)(66556008)(64756008)(55016002)(66476007)(26005)(86362001)(186003)(966005)(4326008)(6916009)(71200400001)(2906002)(5660300002)(316002)(30864003)(34154002); DIR:OUT; SFP:1101;
x-ms-exchange-antispam-messagedata: BcWGmp3c93gJflj3vuKPs8hTUR5vsgxfk/oDduMLX+a2Gl5Bb1gXgpUbvB3ueketyjRmjjDD2Qtl1wDoVgwvRhyc5VoiuhUouQADys6MzUjnxPveC2addBNWdbwRGPdJCC4a5vz512sUD7pnzlzlYI8HbfLLcoto2r9rNTdrNy7tpRYoos1BMfvoQWQS3KEc+du7+GBbGuA2ry577bsA/TXGRdCU3WLuJH86IMmorXXGXVpEZYLtQSmMimTcKw0VcmLKSZfKPo9jrGj9j48eEpftHmnCQ+F+3it6Nxb0zg3H3QDqk1K30NqPzESywjXl4Pr2GTLlVyIGcc7UGqjVvgIoE5nBsYoxXk4naWAv8Gz8SlBSpqc8HNLmvTX8dMiGRVOlcdl9qoMgCJoUnS8bt8T0o21cJ5PdVr3H/mbRn4cgU1748jbYwpTs+QaMgcFlDH0dNkw/kOPXK/LpuCNsRKSurvV+1gkJq+cS+nBvH6k=
x-ms-exchange-transport-forked: True
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 99255653-8c5a-4389-f267-08d7f8f56f78
X-MS-Exchange-CrossTenant-originalarrivaltime: 15 May 2020 17:28:51.5838 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 5ae1af62-9505-4097-a69a-c1553ef7840e
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: /R+J25OYkytO4eK8YSCQo0U3H8K4nuOzJHwG64va/oeBIhLrD8PoEQXqltVta92RDrzRIhITj4r7ZoFi34AJbg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR11MB4470
X-OriginatorOrg: cisco.com
X-Outbound-SMTP-Client: 173.36.7.14, xch-aln-004.cisco.com
X-Outbound-Node: rcdn-core-2.cisco.com
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/FNOWgx5FjFgubYNBI3bOwOciDh0>
Subject: Re: [bmwg] I-D Action: draft-ietf-bmwg-b2b-frame-00.txt
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 15 May 2020 17:29:20 -0000

Hi Al.

(Note this is commenting on -01 version of the draft.)

As mentioned in the meeting, I had one more idea.
We could focus more on the maximum number of successful b2b frames.
The motivation is similar to why we avoid other sources of packet loss
when determining the throughput (as an input to the buffer time calculation).
The other sources can lower the average number of b2b frames,
but the maximum number should better match the input throughput.

I think the average and standard deviation can be still useful
for some users, but it is less clear how that matches the input throughput.
Maybe average of throughput measurements (including those affected by other sources)
should be used as an input to calculate "average" buffer time?

Anyway, my earlier comments still apply, see below.

Vratko.

-----Original Message-----
From: bmwg <bmwg-bounces@ietf.org> On Behalf Of Vratko Polak -X (vrpolak - PANTHEON TECH SRO at Cisco)
Sent: Wednesday, 2019-November-20 18:22
To: MORTON, ALFRED C (AL) <acm@research.att.com>
Cc: bmwg@ietf.org
Subject: Re: [bmwg] I-D Action: draft-ietf-bmwg-b2b-frame-00.txt

I see a new draft version is published,
so this is a good time for another round of review.
(Should the e-mail subject be changed?)

Brace for another long e-mail,
as I feel particularly nitpicky today.

Firstly, comments on new content,
found by looking at the difference [2].

* In section 3, the first list, item 4
(talking about buffer time estimates):

> (measured according to Section 26.4 of [RFC2544])

Neither RFC 2544 nor RFC 1242 mention buffer time (nor size) estimation.
They only mention measurement of the number of back-to-back frames,
which is only an input to the buffer time estimate calculation.
Inserting "based on" somewhere into the sentence will fix that.

> tends to increase the "implied" estimate

I agree with the word implied being in quotes.
It is a colloquial term used by group doing the analysis.

The draft mentions (indirectly) two models.
One neglecting header processing,
and one assuming the header processing speed is constant
(thus equal to the measured Throughput).
As the naming for the two estimates is also under review,
I will call the two models "the first order model"
and "the second order model". The second order model
is the one taking the processing speed into account;
the first order model works only when the processing speed
is negligible compared to the offered load.

Now it is clearer what "increase" is being described,
it is the difference between the two model predictions,
namely the time correction term.
Still the sentence is wrong.
The "implied" estimate does not increase.
Merely, our best estimate is increased by switching to a better model.
It would be better to talk about "correcting" an estimate
(instead of increasing it).

* Section 2, few lines before the second list:

> The simplified model

It is the second order model.
We know it is not realistic enough (the draft mentions
unpredictable processing interrupts, not modeled),
but the word "simplified" means "created from a more convoluted model
by applying some simplifying assumption".
The second order model has not been simplified
from any other model.
I suggest to pick two names for the two models
and stick to them (instead of varying adjectives).

* Section 3, second list, item 2:

> The packet header processing function (HeaderProc) operates at	
> approximately the "Measured Throughput"

I think it would be better to split this into two assumptions.
One assumption is related to the model (header processing speed is constant).
Second assumption is related to parameter estimation
(Measured Throughput to be used as the value for the processing speed).

The "approximately" word would then not appear in model definition
nor the test description.
It would just describe the fact the model is not 100% realistic.
That is why the average processing speed can differ from Measured Throughput,
we just have to assume it is still the best approximation we have.

Secondly, general comments.

It occurred to me, that the term "buffer time" is used
without there being an explicit definition for it.
An implicit definition is along the lines of
"the time between traffic starts and first packet drops",
but that depends not only on buffer size,
but also on the offered load and the processing speed.

Section 3 (just after the second list) mentions the combination
of processing speed zero (interrupted) and presumably
offered load at Max Theoretical Frame Rate.
The Actual Buffer Time (just before section 7)
then presumably means buffer time for zero processing speed
and Actual Frame Rate being offered.
But other combinations are also useful.
For example, Implied DUT Buffer Time
already happens to be the correct buffer time for the combination
of offered load at Max Theoretical Frame Rate
and processing speed at 100% of Measured Throughput.
People can be interested in other combinations as well,
for example offered load at 110% of Measured Throughput
(while processing speed stays at 100% of Measured Throughput);
or processing speed at 90% of Measured Throughput (e.g. noisy neighbor)
while the offered load stays at 100% of Measured Throughput.

The draft could include one formula for all the combinations.
As the buffer time depends on intended load,
but buffer size (in number of frames) does not,
it would be nice if Report required the buffer size estimate,
with some buffer times optional.

Another general comment: Some explanations
will be nicer if we used a term Buffer Filling Speed.
In the first order model, buffer filling speed is equal to the offered load,
in the second order model it is the offered load
minus the processing speed.
In both models, the buffer time is the buffer size
divided by the buffer filling speed for the particular load.

Final general comment: I still do not like
the name Implied (when used for buffer times).
The quantities are only implied by the first order model.
People using the second order model (with non-zero processing speed)
are not implying that values.
Maybe it is a good idea to start with naming the two buffer times
(both for Max Theoretical Frame Rate offered,
with processing speed at either 0% or 100% of Measured Throughput),
and only then decide the names for the corresponding models.

Thirdly, comments on points from previous e-mails:

>>> Corrected DUT Buffer Time
>> 
>> This is a big one.
>> The correct name for this quantity could be "DUT Buffer Time 
>> Correction".
>
> If we were talking about a 'correction factor' I would agree,
> but we have two DUT buffer times distinguished by adjectives,
> which seems more straightforward to me.

No, I was not talking about any factor.
I was talking about a difference.

Let me use some artificial numbers.
Say we have a system with Measured Throughput of 10 fps,
Max Theoretical Frame Rate of 100 fps,
and buffer size of 1000 frames.
According to the second order model, the buffer filling speed
during the b2b measurement is 90 fps, so the buffer gets full
after 11.1111 seconds. Ideally, we would measure
1111.11 b2b frames before loss happens.
The formula for Implied DUT Buffer Time would give us back
the 11.1111 seconds value.
But plugging that to Corrected DUT Buffer Time formula
would give us just 1.11111 seconds.
No reasonable load gives this short a buffer time.
But, subtracting 1.1111 s from 11.1111 s gives us 10.0 s,
which is the buffer time with max offered load and blocked processor.
Thus, 1.1111 second is the correction to be subtracted,
not the final corrected result.

Vratko.

[2] https://tools.ietf.org//rfcdiff?url1=draft-ietf-bmwg-b2b-frame-00&url2=draft-ietf-bmwg-b2b-frame-01

-----Original Message-----
From: MORTON, ALFRED C (AL) <acm@research.att.com> 
Sent: Thursday, November 7, 2019 1:58 AM
To: Vratko Polak -X (vrpolak - PANTHEON TECH SRO at Cisco) <vrpolak@cisco.com>
Cc: bmwg@ietf.org
Subject: RE: [bmwg] I-D Action: draft-ietf-bmwg-b2b-frame-00.txt

Hi Vratko, Thanks for your review!

I just now re-discovered your comments, and have addressed them in the working version of the draft (where I had implemented changes to address Maciek's comments!). I will submit the new draft when submission opens-up again - sorry for the omission!

Al
(as a participant/author)


> -----Original Message-----
> From: Vratko Polak -X (vrpolak - PANTHEON TECHNOLOGIES at Cisco) 
> [mailto:vrpolak@cisco.com]
> Sent: Thursday, July 4, 2019 1:42 PM
> To: MORTON, ALFRED C (AL) <acm@research.att.com>
> Cc: bmwg@ietf.org
> Subject: RE: [bmwg] I-D Action: draft-ietf-bmwg-b2b-frame-00.txt
> 
> Hi Al.
> 
> Sorry for being late with the review.
> I wanted to think some things through, and then did not find big 
> enough chunk of time to write everything down.
> 
> First, narrow comments.
> 
> * Second line of 1. Introduction:
> 
> > in[RFC2544], supported by the terms and definitions in [RFC1242], 
> > and
> 
> Missing space after the first "in".
> 
> * Point 4 in chapter 3 seem to be referring to two different estimates:
> 
> > It was found that the actual buffer time in the DUT could be 
> > estimated using results from the Throughput tests conducted 
> > according to Section 26.1 of [RFC2544], because it appears that the 
> > DUT's frame processing rate may tend to increase the estimate.
> 
> Which estimate is increased, relative to what?
> Is it "implied estimate" tending to be larger than "corrected estimate"?
> 
[acm]
Thanks for pointing out this ambiguity - I have re-worded this long sentence for clarity.

> * Within 5.4:
> 
> > (assuming a simple model of the DUT
> > composed of a buffer and a forwarding function)
> 
> A reasonable simplification, but could be mentioned in Motivation, so 
> 5.4 does not need a sentence in parentheses.
[acm]
I used this as an opportunity to remind the reader with a reference to Section 3.

> 
> * Still within 5.4:
> 
> > Corrected DUT Buffer Time
> 
> This is a big one.
> The correct name for this quantity could be "DUT Buffer Time 
> Correction".
[acm]
If we were talking about a 'correction factor' I would agree, but we have two DUT buffer times distinguished by adjectives, which seems more straightforward to me.

> 
> * still 5.4:
> 
> > and the Buffer size is more accurately estimated by excluding them.
> 
> Can be turned into a formula, where
> "Corrected DUT Buffer Time" can be used in a more natural meaning.
> 
>   Corrected DUT Buffer Time = Implied DUT Buffer Time - DUT Buffer 
> Time Correction
[acm]
You are defining a new term here (?), DUT Buffer Time Correction (factor), which could be the unit-less term:

    Measured Throughput
  --------------------------
  Max Theoretical Frame Rate

But we only use this term once, and it contains terms we've defined using quantities easily understood, so that the meaning and limitations of this factor are clear (when numerator and denominator are equal, there's no correction needed).


> 
> * In 6. table:
> 
> > Min,Max,StdDev
> 
> Not sure if it is clear these refer to B2B length statistics.
> Include information that these quantities are to be expressed in frames.
[acm]
Ok

> 
> 
> And now some broader comments.
> 
> * Suspended processor in production:
> 
> > useful to estimate whether frame losses will occur if DUT forwarding 
> > is temporarily suspended in a production deployment
> 
> This is the main benefit, and Introduction already mentions 
> "compensating for disruptions in the software packet processor".
> 
> But I think it would be good to classify different scenarios leading 
> to frame loss, that could explain both two buffer times (implied and 
> corrected) and how to use them outside back-to-back load.
> 
> In back-to-back test we have processor processing, while the buffer is 
> being filled by maximal traffic.
> The time it takes to fill is the "implied buffer time".
> 
> In a hypothetic scenario where processor is suspended and buffer is 
> being filled by maximal traffic, the time to fill the buffer is 
> shorter, the "corrected buffer time".
> 
> A scenario occuring in practice has suspended processor, but the 
> buffer is being filled more slowly, say at throughput rate.
> Deployers wishing to predict the time for the buffer to fill up can 
> use this formula:
>   Real Buffer Time = Corrected Buffer Time * B2B Frame Rate / Real 
> Frame Rate That is why reporting corrected (instead of implied) buffer 
> time is useful.
[acm]
Yes, vCPU operation can be suspended, and processors can appear suspended while they handle higher priority processes that interrupt the data plane operations for a significant amount of time.
But orchestrating these suspensions/interrupts for measurements isn't practical.  

Nevertheless, the calculation above is useful, I think, so I added it below the table of results, but I think the correction should be based on the Measured Throughput we used before (B2B would be = Max Theoretical).
We have the corrected time using Measured Throughput and add time to that value.

> 
> Also, it would be nice to name the scenarios and rename the buffer times.
> For example, "running buffer time" and "suspended buffer time"
> (instead of "implied" and "corrected" respectively).
[acm]
I see what you are going, but the term "suspended buffer time"
seems anti-intuitive. The packet forwarding is running or suspended, not the buffer, so we need a longer term like "suspended forwarding buffer time" which if we measured correctly, is just the accurate estimate of "buffer time".

> 
> * B2B processor rate:
> 
> For the computation of the corrected buffer time to be correct, real 
> processor frame processing rate (average during B2B test) should be 
> used instead Measured Throughput in the 5.4 formula.
> 
> The process rate is not easy to measure directly, especially if the 
> immediate rate varies over the duration of B2B traffic.
> I agree that Throughput is a reasonable approximation, but there may 
> be other quantities (e.g. FRMOL [1]), that are either a better 
> approximation, or at least easier to measure.
> 
> Not sure how much attention other such quantities should get in the 
> draft, as Throughput has the advantage of avoiding some frame sizes.
[acm]
Right, that's a big advantage.
I have some sympathy for using other approximations/measurements of the "real" forwarding rate. But FRMOL may not be the best alternative, because overload behavior could be degenerative.
The definition of FRMOL goes on to say:

   Discussion:

      Forwarding rate at maximum offered load may be less than the
      maximum rate at which a device can be observed to successfully
      forward traffic.  This will be the case when the ability of a
      device to forward frames degenerates when offered traffic at
      maximum load.

> 
> * DUT vs SUT:
> 
> This is related to final items of 4. Prerequisities.
> 
> > Therefore, sources of packet loss
> > that are un-related to consistent evaluation of buffer size SHOULD 
> > be identified and removed or mitigated.
> 
> Do we have a separate document discussing differences between testing 
> DUT and SUT? We should have.
> 
> Usually I prefer testing SUT (meaning no extra mitigations), but in 
> this case, for the analysis of the three aforementioned scenarios to 
> work correctly, we need to make reasonably sure the processor is not 
> going to get suspended during B2B test.
[acm]
In BMWG's literature, a SUT is composed of multiple DUTs in an expected arrangement, or typical of a planned deployment.

> 
> Also, I agree that an average result of Binary Search with Loss 
> Verification gives a more realistic process rate estimate than an 
> average result without loss verification.
[acm]
Good.  Thanks again for your comments!
> 
> Vratko.
> 
> [1] https://tools.ietf.org/html/rfc2285#page-16
_______________________________________________
bmwg mailing list
bmwg@ietf.org
https://www.ietf.org/mailman/listinfo/bmwg