Re: [bmwg] Eric Rescorla's No Objection on draft-ietf-bmwg-sdn-controller-benchmark-meth-08: (with COMMENT)

"MORTON, ALFRED C (AL)" <acm@research.att.com> Tue, 17 April 2018 22:20 UTC

Return-Path: <acm@research.att.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E0AE3124B0A; Tue, 17 Apr 2018 15:20:39 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.6
X-Spam-Level:
X-Spam-Status: No, score=-2.6 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4xyYaAR-MH2K; Tue, 17 Apr 2018 15:20:37 -0700 (PDT)
Received: from mx0a-00191d01.pphosted.com (mx0b-00191d01.pphosted.com [67.231.157.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 48144120713; Tue, 17 Apr 2018 15:20:37 -0700 (PDT)
Received: from pps.filterd (m0049462.ppops.net [127.0.0.1]) by m0049462.ppops.net-00191d01. (8.16.0.21/8.16.0.21) with SMTP id w3HMFRDF046586; Tue, 17 Apr 2018 18:20:36 -0400
Received: from tlpd255.enaf.dadc.sbc.com (sbcsmtp3.sbc.com [144.160.112.28]) by m0049462.ppops.net-00191d01. with ESMTP id 2hdsba0bv1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 17 Apr 2018 18:20:36 -0400
Received: from enaf.dadc.sbc.com (localhost [127.0.0.1]) by tlpd255.enaf.dadc.sbc.com (8.14.5/8.14.5) with ESMTP id w3HMKZoW069553; Tue, 17 Apr 2018 17:20:35 -0500
Received: from zlp30497.vci.att.com (zlp30497.vci.att.com [135.46.181.156]) by tlpd255.enaf.dadc.sbc.com (8.14.5/8.14.5) with ESMTP id w3HMKT44069455; Tue, 17 Apr 2018 17:20:29 -0500
Received: from zlp30497.vci.att.com (zlp30497.vci.att.com [127.0.0.1]) by zlp30497.vci.att.com (Service) with ESMTP id 2D0B540003A0; Tue, 17 Apr 2018 22:20:29 +0000 (GMT)
Received: from clpi183.sldc.sbc.com (unknown [135.41.1.46]) by zlp30497.vci.att.com (Service) with ESMTP id 073FB4000384; Tue, 17 Apr 2018 22:20:29 +0000 (GMT)
Received: from sldc.sbc.com (localhost [127.0.0.1]) by clpi183.sldc.sbc.com (8.14.5/8.14.5) with ESMTP id w3HMKS5c016317; Tue, 17 Apr 2018 17:20:28 -0500
Received: from mail-green.research.att.com (mail-green.research.att.com [135.207.255.15]) by clpi183.sldc.sbc.com (8.14.5/8.14.5) with ESMTP id w3HMKMRm015819; Tue, 17 Apr 2018 17:20:23 -0500
Received: from exchange.research.att.com (njbdcas1.research.att.com [135.197.255.61]) by mail-green.research.att.com (Postfix) with ESMTP id 24752E14E5; Tue, 17 Apr 2018 18:20:01 -0400 (EDT)
Received: from njmtexg5.research.att.com ([fe80::b09c:ff13:4487:78b6]) by njbdcas1.research.att.com ([fe80::8c6b:4b77:618f:9a01%11]) with mapi id 14.03.0389.001; Tue, 17 Apr 2018 18:20:21 -0400
From: "MORTON, ALFRED C (AL)" <acm@research.att.com>
To: Eric Rescorla <ekr@rtfm.com>, The IESG <iesg@ietf.org>
CC: "draft-ietf-bmwg-sdn-controller-benchmark-meth@ietf.org" <draft-ietf-bmwg-sdn-controller-benchmark-meth@ietf.org>, "bmwg-chairs@ietf.org" <bmwg-chairs@ietf.org>, "bmwg@ietf.org" <bmwg@ietf.org>
Thread-Topic: [bmwg] Eric Rescorla's No Objection on draft-ietf-bmwg-sdn-controller-benchmark-meth-08: (with COMMENT)
Thread-Index: AQHT1pET11rNHtOot0+5803ZoQhgJaQFhLvg
Date: Tue, 17 Apr 2018 22:20:18 +0000
Message-ID: <4D7F4AD313D3FC43A053B309F97543CF4A8E9A19@njmtexg5.research.att.com>
References: <152399965258.11535.11874306299818806488.idtracker@ietfa.amsl.com>
In-Reply-To: <152399965258.11535.11874306299818806488.idtracker@ietfa.amsl.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [135.207.230.84]
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-04-17_11:, , signatures=0
X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1011 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=763 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1711220000 definitions=main-1804170193
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/9hvxTSHQzaMKRhHCbIJrfogNOrw>
Subject: Re: [bmwg] Eric Rescorla's No Objection on draft-ietf-bmwg-sdn-controller-benchmark-meth-08: (with COMMENT)
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 17 Apr 2018 22:20:40 -0000

Eric, a quick reply on your last comment, below.
Al
doc shepherd

...
> >
> >   Measurement:
> >
> >                                              (R1-T1) + (R2-T2)..(Rn-Tn)
> >      Asynchronous Message Processing Time Tr1 = -----------------------
> >                                                          Nrx
> 
> Incidentally, this formula is the same as \sum_i{R_i} - \sum_i{T_i}
> 
> 
> >      messages transmitted to the controller.
> >
> >      If this test is repeated with varying number of nodes with same
> >      topology, the results SHOULD be reported in the form of a graph. The
> >      X coordinate SHOULD be the Number of nodes (N), the Y coordinate
> >      SHOULD be the average Asynchronous Message Processing Time.
> 
> This is an odd metric because an implementation which handled overload
> by dropping every other message would look better than one which
> handled overload by queuing.
[acm] 
If processing time were the only number reported, you're right.

Although the early generation of controller benchmarking tools
overlooked the important combinations of metrics,
the Reporting Format adds the success/loss message performance:

   The report should capture the following information in addition to
   the configuration parameters captured in section 5.

   - Successful messages exchanged (Nrx)

   - Percentage of unsuccessful messages exchanged, computed using the
   formula (1 - Nrx/Ntx) * 100), Where Ntx is the total number of
   messages transmitted to the controller.

BUT, it would be better if SHOULD or RECOMMENDED terms were used,
to cover the case you identified.


> 
> 
> _______________________________________________
> bmwg mailing list
> bmwg@ietf.org
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_mailman_listinfo_bmwg&d=DwICAg&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=8vh-
> jwFdXG0pU1mgRW3_HbkfU1msplWbXuJZesLmozs&s=mfmCArWKOyMJgNPTRNNf467DrHNCKxkp
> QVKXcOCQKQA&e=