Re: [bmwg] Benjamin Kaduk's No Objection on draft-ietf-bmwg-sdn-controller-benchmark-meth-08: (with COMMENT)
Spencer Dawkins at IETF <spencerdawkins.ietf@gmail.com> Thu, 19 April 2018 17:02 UTC
Return-Path: <spencerdawkins.ietf@gmail.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CAD4A126B72; Thu, 19 Apr 2018 10:02:49 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.699
X-Spam-Level:
X-Spam-Status: No, score=-2.699 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id syPXHZ6t-7v9; Thu, 19 Apr 2018 10:02:47 -0700 (PDT)
Received: from mail-yw0-x22d.google.com (mail-yw0-x22d.google.com [IPv6:2607:f8b0:4002:c05::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 8C1351200C5; Thu, 19 Apr 2018 10:02:47 -0700 (PDT)
Received: by mail-yw0-x22d.google.com with SMTP id 199-v6so1882455ywd.9; Thu, 19 Apr 2018 10:02:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=/1M6vlGSUrX4NO91MiC+LsqrnyZRaAudy/n22v8r2zM=; b=mR49aT1lh0i/HmRFaojzjd22LWmgLW3Ne0XJU2YLwJ0iVTeW5rC/vT9v1UVVIvjnjF zKcu+HqqC01LUHQty/hbnyIuSVu1WXz1sWqPAi/2WX3w3poHkRQhrsUs18NnbDRqwBRj Pt8VzGKqAAn7U/5+etmuVj1vEDYqY0wTWNVuQSFERl3/9ffAjjz6WOF3Shv/EyIfFOme 1blCgnO9R1JkG+Bn3VbW9OvI2t5iuHK7npmzf4PhQTcXwnCpRBIWzW9KSFHm88+XWW2k 3n0xiVv2Y0UW28tetLpBGbhxKJKp4V2pk2T6x6W3WYI3c6PeRuzWwWwVsElGqVfRrmSj +/CA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=/1M6vlGSUrX4NO91MiC+LsqrnyZRaAudy/n22v8r2zM=; b=sbu8+/tVOEB9JQX+4N7Zm8yc+yCU0074tfqhQbk+hwRuyc+UJzZxcFOWNlpP9htsq1 PXVVMcbmjcHUFLTaTll1xnawDoCEcJ9FYG7bO6FbJNfGggsxNt7kwZ2wPJQB9OUnqh0z w6QQ1h8P1NcT/EiZMT+mU+zzY5mG1KxE0rD2hapy6n5oHS5JEwZgPoFFKG7RxhNT/XB0 axVwx/iEPIjYmkWbbxoQ6L0CS4qmr19LMrG6uVjR8mWoQkB2YyNU1QdBAHBP+mION9Z5 sBpe+QrK/xmAWBFBes0jaWelFEYuweE/x+yw+i6TYGUTzG/b2dhiThfJdmLC4DfcBiyN /lhA==
X-Gm-Message-State: ALQs6tDAubpHDOHLhfNX1aRCvA6EjCqI112f0ZVI1WgJLqmZ0nQwBtzM MqfXqf0395Y5zp0Nz/jbO51hnsKdHb72b+MAh/M=
X-Google-Smtp-Source: AIpwx4/zReUG6iYujWuETbHafXqBIHlWn0LqHobUMQPMSWi3kvljbYFpcxYcCl7MdHAaSAerAaUjhvhwRLt4pIOibN8=
X-Received: by 2002:a81:ec14:: with SMTP id j20-v6mr3991428ywm.311.1524157366535; Thu, 19 Apr 2018 10:02:46 -0700 (PDT)
MIME-Version: 1.0
Received: by 2002:a5b:192:0:0:0:0:0 with HTTP; Thu, 19 Apr 2018 10:02:45 -0700 (PDT)
In-Reply-To: <20180419164151.GT54168@kduck.kaduk.org>
References: <152414053109.28837.9925559089834201998.idtracker@ietfa.amsl.com> <82A6B56C-348F-4369-BB68-3A700F942146@encrypted.net> <20180419164151.GT54168@kduck.kaduk.org>
From: Spencer Dawkins at IETF <spencerdawkins.ietf@gmail.com>
Date: Thu, 19 Apr 2018 12:02:45 -0500
Message-ID: <CAKKJt-djJ0+cXu-TbEVm2dscGiydaCmfZt=NHE-rrgft3UnfHg@mail.gmail.com>
To: Benjamin Kaduk <kaduk@mit.edu>
Cc: Sarah B <sbanks@encrypted.net>, draft-ietf-bmwg-sdn-controller-benchmark-meth@ietf.org, bmwg-chairs@ietf.org, ALFRED MORTON <acmorton@att.com>, "Bhuvan (Veryx Technologies)" <bhuvaneswaran.vengainathan@veryxtech.com>, The IESG <iesg@ietf.org>, bmwg@ietf.org
Content-Type: multipart/alternative; boundary="0000000000006a0d1e056a368ae0"
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/rPE8TxBBI9X9rGFjQAf0lRlB0sk>
Subject: Re: [bmwg] Benjamin Kaduk's No Objection on draft-ietf-bmwg-sdn-controller-benchmark-meth-08: (with COMMENT)
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 19 Apr 2018 17:02:50 -0000
This is Benjamin's ballot thread, but ... On Thu, Apr 19, 2018 at 11:41 AM, Benjamin Kaduk <kaduk@mit.edu> wrote: > On Thu, Apr 19, 2018 at 09:32:38AM -0700, Sarah B wrote: > > Hi Benjamin, > > Thanks for your review. A few comments inline below with SB// > > > > Thanks > > Sarah > > > > > > ---------------------------------------------------------------------- > > > COMMENT: > > > ---------------------------------------------------------------------- > > > > > > In the Abstract: > > > > > > This document defines the methodologies for benchmarking control > > > plane performance of SDN controllers. > > > > > > Why "the" methodologies? That seems more authoritative than is > > > appropriate in an Informational document. > > > > > > > Sure, we can remove this. > > (I probably should have explicitly said that "a methodology" is > fine.) > > > > > > > Thank you for adding consideration to key distribution in Section > > > 4.4, as noted by the secdir review. But insisting on having key > > > distribution done prior to testing gives the impression that keys > > > are distributed once and updated never, which has questionable > > > security properties. Perhaps there is value in doing some testing > > > while rekeyeing is in progress? > > > > We'll discuss internally.. the goal here wasn't to test rekeying, which > is why the test doesn't call for anything other than doing the distribution > prior to the test, and remember, this is lab testing; this isn't what > happens on a live network, but it's an interesting point, and we'll discuss. > > Okay. I'm just approaching this from a "social norms" perspective > (regarding expectations that re-keying will be done), but I > definitely understand that there is less need for benchmarking > rekeying performance than the cases already covered. > > > > > > > I agree with others that the statistical methodology is not clearly > > > justified, such as the sample size of 10 in Section 4.7 (with no > > > consideration for sample relative variance), use of sample vs. > > > population veriance, etc. > > > > As a draft, we aren't saying you only have to do 10; it's been a point > of discussion in BMWG for some time, what number do you put? We haven't > always found a happy medium to this. We're suggesting you perform the test > 10 times, but there's nothing that stops a tester from making that <n> > times; for example, 100 times. It's important to note how many times the > test was performed and move on. We can clarify this text to reflect that, > if it helps. However, note, BMWG is about repeatability, and that was the > goal here. > > Well, there's a huge amount of literature (and, e.g., textbooks) on > statistical anlaysis and how to decide that a series of measurements > has converged on a stable result. So it might be better to just say > that benchmarkers must ensure that the data has converged and note > that this will require running many repeated trials, but not > necessarily list a specific value. FWIW, that's the kind of thing I was asking about in my own ballot. If 10 repetitions has been a good heuristic in testing so far, that could be worth saying, but if people are going to have to look at results and figure out whether they've repeated a test often enough, you might as well put it in those terms. IMO, of course. Spencer
- [bmwg] Benjamin Kaduk's No Objection on draft-iet… Benjamin Kaduk
- Re: [bmwg] Benjamin Kaduk's No Objection on draft… Sarah B
- Re: [bmwg] Benjamin Kaduk's No Objection on draft… Benjamin Kaduk
- Re: [bmwg] Benjamin Kaduk's No Objection on draft… Spencer Dawkins at IETF