Re: [bmwg] Alissa Cooper's Discuss on draft-ietf-bmwg-vswitch-opnfv-03: (with DISCUSS and COMMENT)

Alissa Cooper <alissa@cooperw.in> Thu, 08 June 2017 11:36 UTC

Return-Path: <alissa@cooperw.in>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id ED7BF12DFDB; Thu, 8 Jun 2017 04:36:56 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.721
X-Spam-Level:
X-Spam-Status: No, score=-2.721 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=cooperw.in header.b=VIcRgWkX; dkim=pass (2048-bit key) header.d=messagingengine.com header.b=Go61TxXV
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NE75EpkAgwEv; Thu, 8 Jun 2017 04:36:55 -0700 (PDT)
Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id DAF4912EAAA; Thu, 8 Jun 2017 04:36:54 -0700 (PDT)
Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id 5296620AAC; Thu, 8 Jun 2017 07:36:54 -0400 (EDT)
Received: from frontend2 ([10.202.2.161]) by compute7.internal (MEProxy); Thu, 08 Jun 2017 07:36:54 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cooperw.in; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-sender :x-me-sender:x-sasl-enc:x-sasl-enc; s=fm1; bh=bemEUIC52Fq76tEXmF 4c05OJTiF9A2+PpepSF35kY6Y=; b=VIcRgWkXBX9WdsvPIeuzTXann5W6iHL+VH 46A47wu5RBweYbik1Mj3n96uN9GcSWZH/bHSbGh9OlVPVDvYEExlT3qwRgr1QiOK rqMrRtpBNdWD9kO/PGj/t9lfmyrUGSTIL823GH1LbvDlrd6Uvzl7hasfAU9H1SsT TuG49jNWlnzfL0j8uQ8u65QUo2BBiS+IS5BRKdYNebYb75QPLNTnXHR2Xh4U768E 6NmUoKI+vYNPg0lIW1JBShal9NPgKEWoofnLZWx8ItrZp5NynIA7iBiBCswLMG+T JpOVO3hB4dJOph2o09hihpTXSqFb4A3Hk9S8vaunYXXTUu/YzwVA==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-sender:x-me-sender:x-sasl-enc:x-sasl-enc; s= fm1; bh=bemEUIC52Fq76tEXmF4c05OJTiF9A2+PpepSF35kY6Y=; b=Go61TxXV BU6cFoPKjh8yx1oDvhJJYAhJ40kvpdVTSMw8GhPuyHkC22v5cR7+yR769LE+p4uP 64/CIu3weO9vOoPIf5TTI0rLAgOoc38UK3IePqeKnDtchN/8rySvFL0yi05nyZCC 54VRzDRJewX1hLWer+difviR9FQYBZbGqQ+3OI495f+NRH32+cHroQGawhPpHXR+ ydAIALcza9hNBjLdKqxbWLHS9O7MlJZmzb2rxjbzQhLyFHjx3COfw8Luq3CEPTna RolyKQ+7kq4se8ucivjcb9MpP/SFNr7I0/nbEFOJZDbLdGY7AZuIE5ZMQHeivYaa AjGZboDgnlBjeg==
X-ME-Sender: <xms:1jY5WUMGVMUc4wA8C_QmxoqAMW3no2BXORuP8bsVIyrS9EyzwMSr0w>
X-Sasl-enc: LjSEAQwNklozHCNu6SX2PaPWBK48xrxQVQ74W1GtRN69 1496921813
Received: from [10.24.81.116] (unknown [128.107.241.183]) by mail.messagingengine.com (Postfix) with ESMTPA id 0328E248B4; Thu, 8 Jun 2017 07:36:52 -0400 (EDT)
Content-Type: text/plain; charset="utf-8"
Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\))
From: Alissa Cooper <alissa@cooperw.in>
In-Reply-To: <4D7F4AD313D3FC43A053B309F97543CF25FD6E1B@njmtexg5.research.att.com>
Date: Thu, 08 Jun 2017 07:36:50 -0400
Cc: IESG <iesg@ietf.org>, "draft-ietf-bmwg-vswitch-opnfv@ietf.org" <draft-ietf-bmwg-vswitch-opnfv@ietf.org>, Sarah Banks <sbanks@encrypted.net>, "bmwg-chairs@ietf.org" <bmwg-chairs@ietf.org>, "bmwg@ietf.org" <bmwg@ietf.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <7CEC6054-B033-4818-B0EE-7A7A1137A2F7@cooperw.in>
References: <149687236248.2644.15874239410315728328.idtracker@ietfa.amsl.com> <4D7F4AD313D3FC43A053B309F97543CF25FD6E1B@njmtexg5.research.att.com>
To: "MORTON, ALFRED C (AL)" <acmorton@att.com>
X-Mailer: Apple Mail (2.3124)
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/RfkuYJzrTTN_PbSL4pLcRW_Sptk>
Subject: Re: [bmwg] Alissa Cooper's Discuss on draft-ietf-bmwg-vswitch-opnfv-03: (with DISCUSS and COMMENT)
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 08 Jun 2017 11:36:57 -0000

Hi Al,

> On Jun 7, 2017, at 6:57 PM, MORTON, ALFRED C (AL) <acmorton@att.com> wrote:
> 
> Hi Alissa,
> Thanks for your review, please see reply below.
> Al (for the co-authors)
> 
>> -----Original Message-----
>> From: Alissa Cooper [mailto:alissa@cooperw.in]
>> Sent: Wednesday, June 07, 2017 5:53 PM
>> To: The IESG
>> Cc: draft-ietf-bmwg-vswitch-opnfv@ietf.org; Sarah Banks; bmwg-
>> chairs@ietf.org; sbanks@encrypted.net; bmwg@ietf.org
>> Subject: Alissa Cooper's Discuss on draft-ietf-bmwg-vswitch-opnfv-03:
>> (with DISCUSS and COMMENT)
> ...
>> 
>> ----------------------------------------------------------------------
>> DISCUSS:
>> ----------------------------------------------------------------------
>> 
>> Picking up on the thread that started with the Gen-ART review, I'm not clear on
>> the position of this document when it comes to repeatability. If all of the
>> parameters listed in Section 3.3 (assuming they apply) are configured and
>> documented, is it assumed that the benchmarks will be repeatable and comparable
>> with hardware implementation benchmarks?
> [ACM] 
>>>> 
> Of course, it has been the goal of the project to produce 
> repeatable results, and a large set of the parameters
> believed to be critical was provided so that the benchmarking
> community could better appreciate the increase in configuration
> complexity inherent in this work. So, yes, this set was assumed 
> sufficient for the infrastructure in use by the VSPERF project
> to obtain repeatable results from test-to-test. 
>>>> We could add summary above as a new paragraph in section 3.3.
> 
> Benchmark Comparability between virtual and physical/hardware 
> implementations of equivalent functions will likely place more
> detailed and exact requirements on the *testing systems*
> (in terms of stream generation, algorithms to search 
> for max values, and their configurations of course). 
> This is another area for standardization to appreciate,
> now that we have a few years testing experience.
> However, the is a topic for a future draft.
> 
>> Or is the position of the document
>> that the benchmarks are not likely to be repeatable/comparable in some (many?)
>> cases, given the increase in complexity? Or that more work needs to be done
>> (outside the specification process) to achieve repeatability?
> [ACM] 
> Thanks to some server re-arrangements in the lab that hosts
> the VSPERF project, we are able to evaluate our long-term
> result repeatability and repeatability across two similar
> hardware platforms in different buildings. I'm still processing
> some of the results we will present next week at the OPNFV Summit,
> but the cross hardware platform results look quite consistent,
> even with a benchmark where Scott Bradner experienced some
> repeatability problems in his lab back in the 90’s.
> 
>> I think this needs to be more clear in the document.
> [ACM] 
> If the >>>paragraph<<< accomplishes that for you,
> with some minor tweaks to word it less like a reply, let me know.

I think both of your first two paragraphs above make useful points for inclusion in the draft, so if you’re able to write up some text that incorporates them, that would resolve my issue.

Thanks,
Alissa


> 
>> 
>> 
>> ----------------------------------------------------------------------
>> COMMENT:
>> ----------------------------------------------------------------------
>> 
>> I understand the debate about the publication of this document as an IETF
>> consensus RFC. To me it seems valuable to publish and that this item is within
>> the WG's charter. I think adding the updates necessary to make the document
>> up-to-date with the Danube 3.0 release as suggested in the thread with Alvaro
>> would be valuable.
>> 
> [ACM] 
> Great, I've located the list of tests added in Colorado and Danube,
> so I'll add them in section 5, and update the release info accordingly.
> 
>