Re: [bmwg] Tsvart last call review of draft-ietf-bmwg-b2b-frame-03

"Black, David" <David.Black@dell.com> Wed, 25 November 2020 21:27 UTC

Return-Path: <David.Black@dell.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3884D3A1E95; Wed, 25 Nov 2020 13:27:51 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.1
X-Spam-Level:
X-Spam-Status: No, score=-2.1 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=dell.com header.b=l26uPsYY; dkim=pass (1024-bit key) header.d=dell.onmicrosoft.com header.b=gXuUjjDN
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2bqqWV6Yv-TP; Wed, 25 Nov 2020 13:27:47 -0800 (PST)
Received: from mx0a-00154904.pphosted.com (mx0a-00154904.pphosted.com [148.163.133.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7B71B3A1E94; Wed, 25 Nov 2020 13:27:46 -0800 (PST)
Received: from pps.filterd (m0170393.ppops.net [127.0.0.1]) by mx0a-00154904.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 0APLM7SG011155; Wed, 25 Nov 2020 16:27:46 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dell.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-transfer-encoding : mime-version; s=smtpout1; bh=IeF/ui37wsBkNWDcGYIF4nAObcvcee1cNcAoFAPe1Ps=; b=l26uPsYY9u9UGRqlEIHQsc4oWQKMT6zSrPjiC90ka0lqK1GJEERutamLyhbfgOWNodLr dB+LLJMoBtUcUuM81GT44kX5R3tSY6LbV9iT99VUXl8EoxcNNTA9p14RnA1FvTiIuJYB SNzPnriXLgoMSy5GPRFjoR8WODAConJkcYaOJ6LEGHwmdOYbJ/XldKq53OIad5V6CHDq N0Ys6YV8aZmlLUTE3BadlEQwGl7k4tqEeUkjJys+vjYrwVLDtBcX4JGCXOW96XyzENjl ucI/5i0aWeCzPuuH/p50hVU4tdfKkekabgMtf3bOmbYR7EMfmoret+MZta6fmAch9OLE RA==
Received: from mx0b-00154901.pphosted.com (mx0b-00154901.pphosted.com [67.231.157.37]) by mx0a-00154904.pphosted.com with ESMTP id 34xy3r02u0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 25 Nov 2020 16:27:45 -0500
Received: from pps.filterd (m0089483.ppops.net [127.0.0.1]) by mx0b-00154901.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 0APLOZfM053919; Wed, 25 Nov 2020 16:27:44 -0500
Received: from nam02-sn1-obe.outbound.protection.outlook.com (mail-sn1nam02lp2057.outbound.protection.outlook.com [104.47.36.57]) by mx0b-00154901.pphosted.com with ESMTP id 351udgbuay-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 25 Nov 2020 16:27:44 -0500
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dWgdHiphAZF49QgEf9HdEBxwPaqsByx18dUdhmUXHkwyhk5lhUl4TD9NdgK0mFgMSr2BZwfRJxxB8ONvn7AFyojFDQmIyREam+CLFEK1m3Fx5L3heWlQc33dCaPrg8rO53BRbPE+HbUlkJtS7s+BD0tqlQoiUips2rjUko0FUbNP/sGn6RxFOgbjHl2cUK1xEe4ndMzrJMEnvVXffmqYZws4fFBW6LncGVuARfvzR3+4h/FtiWQSxbfT7IB8yzlMMzM6avOOx3rF0cg5uLxRzR8+XJbrD08cARq/DvS2E0+JbJOWKQ5MoSqIwaWn1VnntOaUXKysXdazsN7FFMgXlQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IeF/ui37wsBkNWDcGYIF4nAObcvcee1cNcAoFAPe1Ps=; b=kA7bMZcWLJRB+jVPibPoQbZR79/fu+fnl21gTsrIbF9SNxJDNJ2Z+nHMOIAd0IXW0KkZYL/HJVpi20R9Or8vue4HWRMsN+NgpkLbiNfSeh8P8KTAWlZiZGUoGMah5wPs4x0yBtRvAGDLjYirXmGFBIpVmx9Gr/WFAdbRHzIzsTsNiiRoU63KsHtd6X9tqoK0TuN806ifLflvKnvPAR8vvpSwO5xVgeQYDaS5Tx2SVh+dWGUtQmX/bUK+7Z1gv62h7WzFQR5YNP9BPd7WAoupdZqBfG7sDwyw3JMuFwdr4L0s9taHsZeEfhAJevyeKEn8QpDwD8UPeBPWL+/GAaAxBA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=dell.com; dmarc=pass action=none header.from=dell.com; dkim=pass header.d=dell.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Dell.onmicrosoft.com; s=selector1-Dell-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IeF/ui37wsBkNWDcGYIF4nAObcvcee1cNcAoFAPe1Ps=; b=gXuUjjDN8hxESZu/uQRf1a6z32qseJ8E4fW5Gwd9ZYX6o0NEo63Y7CfurMhk+tnj3ZMwggynL9jXVy6O0nFEsokEcNloomGWgmNRM+S05mYcvpvbtLrcQD8GlMFJvVHUPZypR+pw1fL69qDdTMRpcWSH7z+Cn1S/faJ5bD18A9g=
Received: from MN2PR19MB4045.namprd19.prod.outlook.com (2603:10b6:208:1e4::9) by MN2PR19MB4128.namprd19.prod.outlook.com (2603:10b6:208:1e1::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3611.23; Wed, 25 Nov 2020 21:27:42 +0000
Received: from MN2PR19MB4045.namprd19.prod.outlook.com ([fe80::2853:5ccc:b023:dce4]) by MN2PR19MB4045.namprd19.prod.outlook.com ([fe80::2853:5ccc:b023:dce4%3]) with mapi id 15.20.3611.020; Wed, 25 Nov 2020 21:27:42 +0000
From: "Black, David" <David.Black@dell.com>
To: "MORTON, ALFRED C (AL)" <acm@research.att.com>, "tsv-art@ietf.org" <tsv-art@ietf.org>
CC: "bmwg@ietf.org" <bmwg@ietf.org>, "draft-ietf-bmwg-b2b-frame.all@ietf.org" <draft-ietf-bmwg-b2b-frame.all@ietf.org>, "last-call@ietf.org" <last-call@ietf.org>, "Black, David" <David.Black@dell.com>
Thread-Topic: Tsvart last call review of draft-ietf-bmwg-b2b-frame-03
Thread-Index: AQHWwnx0aANa5zmswk+xVn2oQ5//DqnXqHMQgACiSQCAAJwM4IAAQ18AgAAx80A=
Date: Wed, 25 Nov 2020 21:27:42 +0000
Message-ID: <MN2PR19MB404571612B0EE75E4040ECEF83FA0@MN2PR19MB4045.namprd19.prod.outlook.com>
References: <160623159725.20249.9390987464844223889@ietfa.amsl.com> <4D7F4AD313D3FC43A053B309F97543CF014764C4F2@njmtexg5.research.att.com> <MN2PR19MB4045E52E2E321662721891C683FB0@MN2PR19MB4045.namprd19.prod.outlook.com> <4D7F4AD313D3FC43A053B309F97543CF014764EAA2@njmtexg5.research.att.com> <MN2PR19MB4045F0FF2FC651C799B61C4C83FA0@MN2PR19MB4045.namprd19.prod.outlook.com> <4D7F4AD313D3FC43A053B309F97543CF014764F08F@njmtexg5.research.att.com>
In-Reply-To: <4D7F4AD313D3FC43A053B309F97543CF014764F08F@njmtexg5.research.att.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
msip_labels: MSIP_Label_17cb76b2-10b8-4fe1-93d4-2202842406cd_Enabled=True; MSIP_Label_17cb76b2-10b8-4fe1-93d4-2202842406cd_SiteId=945c199a-83a2-4e80-9f8c-5a91be5752dd; MSIP_Label_17cb76b2-10b8-4fe1-93d4-2202842406cd_Owner=david.black@emc.com; MSIP_Label_17cb76b2-10b8-4fe1-93d4-2202842406cd_SetDate=2020-11-25T21:18:49.9795796Z; MSIP_Label_17cb76b2-10b8-4fe1-93d4-2202842406cd_Name=External Public; MSIP_Label_17cb76b2-10b8-4fe1-93d4-2202842406cd_Application=Microsoft Azure Information Protection; MSIP_Label_17cb76b2-10b8-4fe1-93d4-2202842406cd_ActionId=898c9047-08e9-4e3f-bdc0-26a7009c2359; MSIP_Label_17cb76b2-10b8-4fe1-93d4-2202842406cd_Extended_MSFT_Method=Manual
authentication-results: research.att.com; dkim=none (message not signed) header.d=none;research.att.com; dmarc=none action=none header.from=dell.com;
x-originating-ip: [72.74.71.221]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: c8cb6ab3-8f73-4e61-9a3c-08d89188f184
x-ms-traffictypediagnostic: MN2PR19MB4128:
x-ms-exchange-transport-forked: True
x-microsoft-antispam-prvs: <MN2PR19MB4128CD34F968C1F2070C378383FA0@MN2PR19MB4128.namprd19.prod.outlook.com>
x-exotenant: 2khUwGVqB6N9v58KS13ncyUmMJd8q4
x-ms-oob-tlc-oobclassifiers: OLM:10000;
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: eAb+cPw71qdBSedM14bzcnlDpxICn7veS19Q75VZz31oydebRnJXFeMD0naOiEKPyFtxBSb1an32T/4STzzmvhyX+7vmogfQaeQoQkyCbYzDgxIX7keFUggyoJZMHBBOe5Ika2OpUfLfqca1vnnMZaKlGHi2UdDlQ8tUUa2oF0NrPA3MljAFKJGCem9YPRD2WC8Z8RhS7VFtCRpIhnjSoEhGhJ5ZXObxdqmcrUVS/0cP9PVeAOJnAPOEGTJ4IH71w5tPwxSR5YWCNdo2ZaDrCRD2mD0Vzknu0b/tEELPBBoe4k/Axaclhmm2kuK4t2hmWB7vs7B8NqQTJj9MAfrhOq8R4nEYb6Lhpl4+HSolvb2THoVPOrC5R/mRPDH/5h4tppWqEgB5Z1l2s+r4yl8+tw==
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:MN2PR19MB4045.namprd19.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(136003)(39860400002)(366004)(376002)(396003)(346002)(7696005)(9686003)(71200400001)(5660300002)(107886003)(86362001)(4326008)(52536014)(53546011)(6506007)(8936002)(66946007)(66446008)(64756008)(316002)(186003)(76116006)(66556008)(66476007)(26005)(2906002)(8676002)(966005)(786003)(33656002)(83380400001)(30864003)(54906003)(478600001)(110136005)(55016002); DIR:OUT; SFP:1101;
x-ms-exchange-antispam-messagedata: WbTRLuOhn9bvqAa5eqko2gWpN5QKyX/o7ZVzFRGZOQV37fAQZAuWNG3s0OZ05eKKY39xGR+2HAPdK7sYAnKqm2cAsmhNM69g1sV4soqgrSeIvhajLfV7Frh+JMeu7u514n6SkLhgap3DVQrUu87QayaU/3Ch5TuSvkCeKdkM0KcDVSvrk8/dMafhzDcgIp2lfcuTdO/LH1XTmjNmhJCFf+yXl4wo+hxYPqy1suDXdoI15U+cYY00+jmT2Z0Q9OsboJKYmr1Hk/VMZQad9UifWVaeNDN+nKMGlUp4/Do/GhP//qnV61ilOXeSR1WPDaIgRE4Ua/NPf8vlYJoxWD94ntLPWYpdpOCf8VgnaPN8CdwMsRwFV5nhB6SlTxnVKDRCbkAiSR2KnURFiOvnnhZasDOSbAYLzVj81lOvBvNT2cb8On1cSECHZcb38gG8lkByIkR8WWEJfSetlxD33jTncXSELKG6certEvrhTIrTZbE+dWTw63sWQ6HZxnUuBqHfOmr1DwlaOdCARkAPBcsFnJq6ULwojgllzFFWRezaLAQcRgCgFE7OvT5neAsxwFRan+eTlDz19vOG+tqsKN0rWYdkMqvGAPWWFXpfP3t0vL2UFxEvrPiOLczeparz3BVLAOuo/GrgTZxXltEKCCJXtJHbpWQ0TZ3O7WM0jiVvy6nibRDlJNHXcTcWNe4NXQrrnn4d/R6LELen6O2Q2zSON/SKUP3yJeLYFHpIUbA95tnIEZylDPbsvc7MIBFGO0jX66zrtQRehuFIQtMUJAbkE0fEYVAZnkiIuBwkH3v9tnqcTlbk6g/FsyK6y426kA84hPOQF3EPkugTz84F8dakQcCHMLo1h2/7S3I2catuIlwkLHTNeihTyrZB1fSgDVbUQXea5rXxaGK48uqTBBANvTA1LdUA5jk6sxhd1I4E4fBfzgWNUBHEv/EkgZiZ926Irisc+1Kmx68LvnKmr/LD3O6ehftLxySFmAlLAmYiryE=
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-OriginatorOrg: Dell.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: MN2PR19MB4045.namprd19.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c8cb6ab3-8f73-4e61-9a3c-08d89188f184
X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Nov 2020 21:27:42.5717 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 945c199a-83a2-4e80-9f8c-5a91be5752dd
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: xKLR9s09IebGAXSrPkiL0IuPXwYRFTOBVMPZsA6rdRvRiHgSqWNNrXkYt4TctL1B8mvfkAH7yC+G+Fe0+XA0bA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR19MB4128
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.312, 18.0.737 definitions=2020-11-25_13:2020-11-25, 2020-11-25 signatures=0
X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 clxscore=1015 mlxscore=0 adultscore=0 impostorscore=0 bulkscore=0 priorityscore=1501 suspectscore=0 spamscore=0 phishscore=0 mlxlogscore=999 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2011250131
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 bulkscore=0 adultscore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2011250131
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/6HzRgcUtePSted2jbnZZnfECeys>
Subject: Re: [bmwg] Tsvart last call review of draft-ietf-bmwg-b2b-frame-03
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 25 Nov 2020 21:27:51 -0000

Al,

> I clarified:
> The duration of the trial MUST include least 2 seconds in addition to the time
> required to send and receive each burst of frames, to ensure that DUT buffers to
> deplete.
> 
> and I'll add:
> The upper search limit for the time to send each burst MUST be configurable as
> high as 30 seconds (buffer time results reported at the configured upper limit are
> likely invalid, and the test MUST be repeated with a higher search limit).

I think this will work - please make sure to address the scenario in which the search algorithm hits the upper search limit without causing any frame loss (e.g., upper search limit configured to 2 seconds for a bizarre DUT that has 10 seconds of buffering).

Thanks, --David

> -----Original Message-----
> From: MORTON, ALFRED C (AL) <acm@research.att.com>
> Sent: Wednesday, November 25, 2020 1:20 PM
> To: Black, David; tsv-art@ietf.org
> Cc: bmwg@ietf.org; draft-ietf-bmwg-b2b-frame.all@ietf.org; last-call@ietf.org
> Subject: RE: Tsvart last call review of draft-ietf-bmwg-b2b-frame-03
> 
> 
> [EXTERNAL EMAIL]
> 
> David,
> 
> Allow me to  points more clearly:
> 
> - I'm relying on the Binary Search aspect of the procedure to measure unexpected
> large buffer times; searches are an integral part of the procedure.
> 
> - These are benchmarking measurements in an isolated test environment, where
> we will control every source of variability possible to foster repeatability.
> 
> - The trial duration has two components: time to conduct sending + receiving, plus
> additional waiting time for buffer depletion. The test equipment orchestrates this
> easily, because it is a single device.
> 
> I clarified:
> The duration of the trial MUST include least 2 seconds in addition to the time
> required to send and receive each burst of frames, to ensure that DUT buffers to
> deplete.
> 
> and I'll add:
> The upper search limit for the time to send each burst MUST be configurable as
> high as 30 seconds (buffer time results reported at the configured upper limit are
> likely invalid, and the test MUST be repeated with a higher search limit).
> 
> Al
> 
> 
> > -----Original Message-----
> > From: Black, David [mailto:David.Black@dell.com]
> > Sent: Wednesday, November 25, 2020 9:35 AM
> > To: MORTON, ALFRED C (AL) <acm@research.att.com>; tsv-art@ietf.org
> > Cc: bmwg@ietf.org; draft-ietf-bmwg-b2b-frame.all@ietf.org; last-
> > call@ietf.org; Black, David <David.Black@dell.com>
> > Subject: RE: Tsvart last call review of draft-ietf-bmwg-b2b-frame-03
> >
> > Al,
> >
> > We appear to not be communicating on a key aspect of the concern:
> >
> > > A burst of 1.7 seconds falls well within the coverage of a 2 second
> > > trial to count the frames and determine that loss has occurred. We
> > > don't need a big margin time- wise. Also, the trial duration
> > > increases with the search parameters: IOW, if the tester is still
> > > sending frames after 2 seconds, then obviously we can't allow the trial to
> terminate prematurely.
> >
> > The 1.6 seconds value is *not* an upper bound - it was measured in
> > practice on an operational conference network as part of what appears
> > to have been a just-in-time slide preparation exercise.  Hence that
> > 1.6 seconds of buffering ought to be viewed as being somewhere in the
> > midst of the expected distribution of buffer times.  For that reason,
> > the upper bound on test duration needs to be set with a safety factor
> > to ensure that even if a bizarre DUT has 10 seconds of buffering, the
> > B2B test depletes that buffering so that the B2B test is testing the
> > forwarding path as intended, not the buffering capacity.
> >
> > > I want to avoid over-reacting to the buffer bloat case. We all agree
> > > that bloated buffers are bad, and knowing a DUT has buffering >=1
> > > second may be enough. I certainly do not think that trial durations
> > > for the B2B Frame benchmark need to be as long as you originally
> > > suggested, where the overwhelming number of test conditions can use a <<1
> second burst and 2 second buffer depletion time:
> >
> > I agree with this approach provided that there is a test procedure (in
> > the B2B test and/or its prerequisites) that will discover the behavior
> > of the bizarre DUT with 10 seconds of buffering and adjust the B2B
> > test time to ensure that this excessive buffering is depleted.
> >
> > > I hope the approach of the single sentence above works for you.
> >
> > Based on this discussion, in addition to that sentence (which I think
> > has to contain a "MUST" for situations in which excessive buffering is
> > found), I would like to see a brief explanation of how the procedures
> > in the draft, along with additional test procedures pulled in from RFC
> > 2544 or elsewhere, avoid a 2 second test duration on a DUT with 10
> > seconds of buffering.
> >
> > Thanks, --David
> >
> > > -----Original Message-----
> > > From: MORTON, ALFRED C (AL) <acm@research.att.com>
> > > Sent: Wednesday, November 25, 2020 12:00 AM
> > > To: Black, David; tsv-art@ietf.org
> > > Cc: bmwg@ietf.org; draft-ietf-bmwg-b2b-frame.all@ietf.org; last-
> > call@ietf.org
> > > Subject: RE: Tsvart last call review of draft-ietf-bmwg-b2b-frame-03
> > >
> > >
> > > [EXTERNAL EMAIL]
> > >
> > > Hi David, some more clarity below.
> > > Al
> > >
> > >
> > > > -----Original Message-----
> > > > From: Black, David [mailto:David.Black@dell.com]
> > > > Sent: Tuesday, November 24, 2020 3:02 PM
> > > > To: MORTON, ALFRED C (AL) <acm@research.att.com>; tsv-art@ietf.org
> > > > Cc: bmwg@ietf.org; draft-ietf-bmwg-b2b-frame.all@ietf.org; last-
> > > > call@ietf.org; Black, David <David.Black@dell.com>
> > > > Subject: RE: Tsvart last call review of
> > > > draft-ietf-bmwg-b2b-frame-03
> > > >
> > > > Al,
> > > >
> > > > > [acm] I agree with your observation that there are cases where
> > > > > trial
> > > > duration
> > > > > should be increased to accommodate the encountered in the DUT,
> > > > > but not
> > > > as a
> > > > > mandate for all testing. I have four factors in mind:
> > > > > 1. Some of the virtual network DUTs we are testing now have very
> > > > > small
> > > > buffers,
> > > > > and the B2B stream of frames is quite short -- less than 2000
> > > > frames@10GE in
> > > > > some cases -- so 2 seconds fully sufficient.
> > > > > 2. The trial duration is a factor in total test duration, where
> > > > > each
> > > > trial is one step in
> > > > > the Binary Search. We need to manage the tension between the
> > > > > time needed
> > > > to
> > > > > reach a search result and confidence that we have depleted the
> > queues.
> > > > > 3. The RFC 2544 Latency benchmark will tell us if bufferbloat is
> > > > present.
> > > > > 4. The current text says "at least 2 seconds".
> > > > >
> > > > > So I suggest adding the following text:
> > > > >
> > > > >     The duration of the trial MUST be at least 2 seconds, to
> > > > > allow
> > DUT
> > > > >     buffers to deplete. When RFC2544 Latency measurements
> > > > > indicate
> > that
> > > > >     large buffers are present in the DUT, the trial duration
> > > > > SHOULD
> > be
> > > > >     increased to ensure that buffer depletion takes place,
> > > > > without
> > unduly
> > > > >     extending the total test time.
> > > >
> > > > The overall approach of collecting evidence that there is a
> > > > problem before increasing the 2 second minimum duration is fine,
> > > > but the details appear to need more attention:
> > > >
> > > > 1. Obtaining the RFC 2544 Latency measurements would need to be
> > > > added to Section 4 (Prerequisites) of this draft to ensure that
> > > > the buffer size information is available.
> > > [acm]
> > > Section 26.2 Latency is a very primitive procedure by today's standards.
> > It is
> > > common for testers to measure delay on every packet, and report a
> > > true
> > average.
> > > This is enough to alert the operator to the presence of bufferbloat,
> > > but
> > there's no
> > > assurance that the Latency benchmark measures the extent of the
> > > buffer
> > time.
> > >
> > > >
> > > > 2. I did not see any requirements in the RFC 2544 Latency test
> > > > (Section
> > > > 26.2) to deplete buffers.  Did I miss something?
> > > [acm]
> > > It's an overall requirement in Section 23, after running the trial:
> > >
> > >    d) Wait for two seconds for any residual frames to be received.
> > >
> > > >
> > > > 3. I would think that the trial duration MUST be increased, not
> > > > just SHOULD be increased if there is evidence of large buffer
> > > > size, as buffer depletion appears to be a necessary characteristic
> > > > of this B2B measurement.
> > > [acm]
> > >
> > > We need to keep in mind that the sentence(s) in this discussion
> > > pertain
> > to the trials
> > > of the B2B Frame Benchmark testing, in the section that begins:
> > >
> > >   Each trial in the test requires the tester to send a burst of
> > > frames
> > (after idle time)
> > > with the minimum inter-frame gap, and to count the corresponding
> > > frames forwarded by the DUT.
> > >
> > > Let's say we have a DUT that offers 1.6 seconds of buffering. The
> > > Binary
> > Search will
> > > increase the length of the burst until the buffer drops frames, and
> > > then
> > try to find
> > > the longest burst where frame loss is zero. This simple procedure
> > > always
> > waits the
> > > minimum trial duration for frames to exit the DUT.
> > >
> > > A burst of 1.7 seconds falls well within the coverage of a 2 second
> > trial to count the
> > > frames and determine that loss has occurred. We don't need a big
> > > margin
> > time-
> > > wise. Also, the trial duration increases with the search parameters:
> > IOW, if the
> > > tester is still sending frames after 2 seconds, then obviously we
> > > can't
> > allow the trial
> > > to terminate prematurely.
> > >
> > > So, I suggest we use this wording instead:
> > >
> > >        The duration of the trial MUST be at least 2 seconds in
> > > addition
> > to
> > >        the time to send each burst of frames, to allow DUT buffers
> > > to
> > deplete.
> > >
> > > and we establish an adaptive waiting time for extreme cases,
> > > consistent
> > with
> > > RFC2544 section 23 item d). For a DUT with 1.6 second buffers, the
> > > Trial
> > duration
> > > would be 3.6 seconds.
> > >
> > > BTW, one tester issue we are trying to fix here is the case when the
> > frame header
> > > processing rate is sufficiently high that no buffer limit is
> > > measureable
> > at the large
> > > frame sizes, and we implore people not to waste time with the B2B
> > > Frame Benchmark at those frame sizes. One commercial tester sent
> > > bursts up to
> > 30
> > > seconds in length, and then happily reported the impossible result
> > > of 30
> > second
> > > buffers.
> > >
> > > I want to avoid over-reacting to the buffer bloat case. We all agree
> > that bloated
> > > buffers are bad, and knowing a DUT has buffering >=1 second may be
> > enough. I
> > > certainly do not think that trial durations for the B2B Frame
> > > benchmark
> > need to be
> > > as long as you originally suggested, where the overwhelming number
> > > of
> > test
> > > conditions can use a <<1 second burst and 2 second buffer depletion
> > time:
> > >
> > > > > > Hence, the 2 second minimum duration ought to be increased by
> > > > > > at least a factor of 10.  I'd suggest changing it to 30
> > > > > > seconds or 60
> > seconds...
> > >
> > > I hope the approach of the single sentence above works for you.
> > >
> > >
> > >
> > > >
> > > > It also looks like the link to the entire slide deck didn't make
> > > > it into my original review correctly - that slide deck is at:
> > > > https://urldefense.com/v3/__http://www.taht.net/*d/lca_tcp3.odp__;fg!!
> > > > BhdT !0oWabwvUrtVlGQY7ZKnFnMa-
> > > un_e95tfCJcRKIWG7wECiJkVISb4sjd9Salwehw$
> > > > .  In addition to slide 6 from this slide deck (Figure 1 in the
> > > > APNIC blog), slide 14 is also relevant to this discussion.
> > > >
> > > > Thanks, --David
> > > >
> > > > > -----Original Message-----
> > > > > From: MORTON, ALFRED C (AL) <acm@research.att.com>
> > > > > Sent: Tuesday, November 24, 2020 11:11 AM
> > > > > To: Black, David; tsv-art@ietf.org
> > > > > Cc: bmwg@ietf.org; draft-ietf-bmwg-b2b-frame.all@ietf.org; last-
> > > > call@ietf.org
> > > > > Subject: RE: Tsvart last call review of
> > > > > draft-ietf-bmwg-b2b-frame-03
> > > > >
> > > > >
> > > > > [EXTERNAL EMAIL]
> > > > >
> > > > > Hi David, Thanks for your review and comment!
> > > > >
> > > > > Please see a proposed resolution below, [acm] Al
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: David Black via Datatracker [mailto:noreply@ietf.org]
> > > > > > Sent: Tuesday, November 24, 2020 10:27 AM
> > > > > > To: tsv-art@ietf.org
> > > > > > Cc: bmwg@ietf.org; draft-ietf-bmwg-b2b-frame.all@ietf.org;
> > > > > > last- call@ietf.org
> > > > > > Subject: Tsvart last call review of
> > > > > > draft-ietf-bmwg-b2b-frame-03
> > > > > >
> > > > > > Reviewer: David Black
> > > > > > Review result: Ready with Issues
> > > > > >
> > > > > ...
> > > > > >
> > > > > > This draft updates the back-to-back frame testing procedure in
> > > > > > RFC
> > > > > > 2544 to take account of experience.
> > > > > >
> > > > > > The draft is in good shape, with one notable exception in
> > > > > > Section
> > 5.2:
> > > > > >
> > > > > >    The duration of the trial MUST be at least 2 seconds, to
> > > > > > allow
> > DUT
> > > > > >    buffers to deplete.
> > > > > >
> > > > > > That duration of 2 seconds has been carried forward from RFC
> > > > > > 2544 without change.  A 2 second duration may have been
> > > > > > sufficient to deplete buffers in 1999, but that is no longer reliably the case.
> > > > > > For example, on-site measurement of the network for the 2020
> > > > > > Linux Conference in Australia indicated a at least 1.6 seconds
> > > > > > of buffering, as indicated by Figure 1 at
> > > > > >
> > > > https://urldefense.com/v3/__https://blog.apnic.net/2020/01/22/buff
> > > > erbl
> > > > oat-
> > > > may-be-solved-but-its-not-over-
> > > > yet/__;!!BhdT!wOuQE0NajXs4dT7tdIMVQU5FFpb0JiU0-
> > > > yK2DOVVn0ecoYjf7mFEABLmlwDk$ ,
> > > > > > which is slide 6 in from the complete slide deck at:
> > > > > >
> > > > https://urldefense.com/v3/__https://blog.apnic.net/2020/01/22/buff
> > > > erbl
> > > > oat-
> > > > may-be-solved-but-its-not-over-
> > > > yet/__;!!BhdT!wOuQE0NajXs4dT7tdIMVQU5FFpb0JiU0-
> > > > yK2DOVVn0ecoYjf7mFEABLmlwDk$
> > > > > > .  Experience with bufferbloat suggests that one network
> > > > > > device was primarily responsible.  Also, see slide 14 in that slide deck.
> > > > > >
> > > > > > That 1.6 seconds measured on an actual network is entirely too
> > > > > > close to 2 seconds for confidence that buffers will be
> > > > > > depleted in any
> > > > tested device.
> > > > > > Hence, the 2 second minimum duration ought to be increased by
> > > > > > at least a factor of 10.  I'd suggest changing it to 30
> > > > > > seconds or 60 seconds as convenient round numbers, and
> > > > > > providing the rationale that increased buffering in WiFi
> > > > > > devices, e.g., home "routers," as indicated by experience with
> > > > > > bufferbloat measurements, is the reason
> > > > for the
> > > > > increased duration.
> > > > > >
> > > > > [acm]
> > > > > [acm] I agree with your observation that there are cases where
> > > > > trial
> > > > duration
> > > > > should be increased to accommodate the encountered in the DUT,
> > > > > but not
> > > > as a
> > > > > mandate for all testing. I have four factors in mind:
> > > > > 1. Some of the virtual network DUTs we are testing now have very
> > > > > small
> > > > buffers,
> > > > > and the B2B stream of frames is quite short -- less than 2000
> > > > frames@10GE in
> > > > > some cases -- so 2 seconds fully sufficient.
> > > > > 2. The trial duration is a factor in total test duration, where
> > > > > each
> > > > trial is one step in
> > > > > the Binary Search. We need to manage the tension between the
> > > > > time needed
> > > > to
> > > > > reach a search result and confidence that we have depleted the
> > queues.
> > > > > 3. The RFC 2544 Latency benchmark will tell us if bufferbloat is
> > > > present.
> > > > > 4. The current text says "at least 2 seconds".
> > > > >
> > > > > So I suggest adding the following text:
> > > > >
> > > > >     The duration of the trial MUST be at least 2 seconds, to
> > > > > allow
> > DUT
> > > > >     buffers to deplete. When RFC2544 Latency measurements
> > > > > indicate
> > that
> > > > >     large buffers are present in the DUT, the trial duration
> > > > > SHOULD
> > be
> > > > >     increased to ensure that buffer depletion takes place,
> > > > > without
> > > > unduly
> > > > >     extending the total test time.
> > > > >
> > > > > I hope this suggestion resolves your issue; thanks for
> > > > > highlighting
> > it!
> > > > > Al