Re: [bmwg] Ignas Bagdonas' No Objection on draft-ietf-bmwg-sdn-controller-benchmark-meth-08: (with COMMENT)

"MORTON, ALFRED C (AL)" <acm@research.att.com> Thu, 19 April 2018 14:30 UTC

Return-Path: <acm@research.att.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BD661124D68; Thu, 19 Apr 2018 07:30:57 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.601
X-Spam-Level:
X-Spam-Status: No, score=-2.601 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sSrDBmVWdtPH; Thu, 19 Apr 2018 07:30:54 -0700 (PDT)
Received: from mx0a-00191d01.pphosted.com (mx0b-00191d01.pphosted.com [67.231.157.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C8D9B12741D; Thu, 19 Apr 2018 07:30:54 -0700 (PDT)
Received: from pps.filterd (m0083689.ppops.net [127.0.0.1]) by m0083689.ppops.net-00191d01. (8.16.0.21/8.16.0.21) with SMTP id w3JERdLm001759; Thu, 19 Apr 2018 10:30:52 -0400
Received: from tlpd255.enaf.dadc.sbc.com (sbcsmtp3.sbc.com [144.160.112.28]) by m0083689.ppops.net-00191d01. with ESMTP id 2hes1qmqsx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 19 Apr 2018 10:30:51 -0400
Received: from enaf.dadc.sbc.com (localhost [127.0.0.1]) by tlpd255.enaf.dadc.sbc.com (8.14.5/8.14.5) with ESMTP id w3JEUok6042374; Thu, 19 Apr 2018 09:30:50 -0500
Received: from zlp30497.vci.att.com (zlp30497.vci.att.com [135.46.181.156]) by tlpd255.enaf.dadc.sbc.com (8.14.5/8.14.5) with ESMTP id w3JEUk8d042318; Thu, 19 Apr 2018 09:30:46 -0500
Received: from zlp30497.vci.att.com (zlp30497.vci.att.com [127.0.0.1]) by zlp30497.vci.att.com (Service) with ESMTP id 6FE6940003B7; Thu, 19 Apr 2018 14:30:46 +0000 (GMT)
Received: from tlpd252.dadc.sbc.com (unknown [135.31.184.157]) by zlp30497.vci.att.com (Service) with ESMTP id 5C68440003A0; Thu, 19 Apr 2018 14:30:46 +0000 (GMT)
Received: from dadc.sbc.com (localhost [127.0.0.1]) by tlpd252.dadc.sbc.com (8.14.5/8.14.5) with ESMTP id w3JEUkEk022322; Thu, 19 Apr 2018 09:30:46 -0500
Received: from mail-green.research.att.com (mail-green.research.att.com [135.207.255.15]) by tlpd252.dadc.sbc.com (8.14.5/8.14.5) with ESMTP id w3JEUZPq021861; Thu, 19 Apr 2018 09:30:35 -0500
Received: from exchange.research.att.com (njbdcas1.research.att.com [135.197.255.61]) by mail-green.research.att.com (Postfix) with ESMTP id 420CDE1B31; Thu, 19 Apr 2018 10:30:12 -0400 (EDT)
Received: from njmtexg5.research.att.com ([fe80::b09c:ff13:4487:78b6]) by njbdcas1.research.att.com ([fe80::8c6b:4b77:618f:9a01%11]) with mapi id 14.03.0389.001; Thu, 19 Apr 2018 10:30:34 -0400
From: "MORTON, ALFRED C (AL)" <acm@research.att.com>
To: Ignas Bagdonas <ibagdona@gmail.com>, The IESG <iesg@ietf.org>, "Suresh Krishnan" <suresh@kaloom.com>
CC: "draft-ietf-bmwg-sdn-controller-benchmark-meth@ietf.org" <draft-ietf-bmwg-sdn-controller-benchmark-meth@ietf.org>, "bmwg-chairs@ietf.org" <bmwg-chairs@ietf.org>, "bmwg@ietf.org" <bmwg@ietf.org>
Thread-Topic: Ignas Bagdonas' No Objection on draft-ietf-bmwg-sdn-controller-benchmark-meth-08: (with COMMENT)
Thread-Index: AQHT1+VxKdv58OYXMUCGtT0xa8j/HKQII0Ng
Date: Thu, 19 Apr 2018 14:30:30 +0000
Message-ID: <4D7F4AD313D3FC43A053B309F97543CF4A8EAE22@njmtexg5.research.att.com>
References: <152414584922.28684.709859147342761564.idtracker@ietfa.amsl.com>
In-Reply-To: <152414584922.28684.709859147342761564.idtracker@ietfa.amsl.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [135.16.251.237]
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-04-19_05:, , signatures=0
X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1011 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1711220000 definitions=main-1804190128
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/K8QJaIFym_-9LXuVaJOuQkQu77Y>
Subject: Re: [bmwg] Ignas Bagdonas' No Objection on draft-ietf-bmwg-sdn-controller-benchmark-meth-08: (with COMMENT)
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 19 Apr 2018 14:30:58 -0000

Hi Ignas and Suresh (who supported this comment),

> -----Original Message-----
> From: Ignas Bagdonas [mailto:ibagdona@gmail.com]
> Sent: Thursday, April 19, 2018 9:51 AM
...
> ----------------------------------------------------------------------
> COMMENT:
> ----------------------------------------------------------------------
> 
> The document seems to assume the OpenFlow dataplane abstraction model – which
> is one of the possible models; the practical applicability of such model to
> anything beyond experimental deployments is a completely separate question
> outside of the scope of this document. The methodology tends to apply to a
> broader set of central control based systems, and not only to the data plane
> operations – therefore the document seems to be setting at least something
> practically usable for benchmarking of such central control systems. Possibly
> the document could mention such assumptions made about the overall model where
> the methodology defined applies to.
[acm] 
That's a good wording suggestion, it certainly captures the working group
consensus to prepare the methods independent from OpenFlow, for 
wider applicability.

However, we also received feedback from Stuart Bryant seeking *more* 
message specificity (appended below, see * ), which is not really possible 
for the general methods. Please try to strike a balance between 
these comments in discussion today, if possible!

Al
doc shepherd

-=-=-=-=- Stuart's comment-=-=-=-=-=-
2.3.1.3. Asynchronous Message Processing Rate

Definition:
   The number responses to asynchronous messages (such as new flow
SB> That should be the number of responses per second.

Discussion:
   As SDN assures flexible network and agile provisioning, it is
   important to measure how many network events the controller can
   handle at a time. This benchmark is obtained by sending asynchronous
   messages from every connected Network Device at the rate that the
   controller processes (without dropping them).

SB> So what you are testing here is the control network and the
SB> controller. This is perhaps the only practical way to run the
SB> test, but it seems a pity that you do not deconvolve these
SB> two aspects of the test.
SB>
SB> I suppose this is really network Async Msg Proc rate rather than
SB> controller Async proc rate.
SB>
*SB> We may get to this in the companion document, but doesn't there
*SB> need to be some standardization of the event message to compare
*SB> apple with apples over time?



> 
> A nit: s/Khasanov Boris/Boris Khasanov, unless Boris himself would insist
> otherwise.
>