Re: [secdir] [bmwg] Secdir telechat review of draft-ietf-bmwg-b2b-frame-03

Al,

I don't have a strong opinion on using the term "honesty" here. How about this phrasing, just before the last paragraph in Security Considerations:

The DUT developers are commonly independent from the personnel and institutions conducting the benchmarking.
The DUT developers might have incentives to alter the performance of the DUT if the test conditions are detected.
Procedures described in this document are not designed to detect such activity.
Additional testing, outside of the scope of this document, is needed and has been successfully used in the past to discover such malpractices.

Mališa

On 15/12/2020 20:22, "MORTON, ALFRED C (AL)" <acm@research.att.com> wrote:

    Hi Mališa,
    please see below...

    > -----Original Message-----
    > From: Mališa Vučinić [mailto:malisa.vucinic@inria.fr]
    > Sent: Tuesday, December 15, 2020 9:21 AM
    > To: MORTON, ALFRED C (AL) <acm@research.att.com>; secdir@ietf.org
    > Cc: last-call@ietf.org; bmwg@ietf.org; draft-ietf-bmwg-b2b-
    > frame.all@ietf.org
    > Subject: Re: [bmwg] Secdir telechat review of draft-ietf-bmwg-b2b-frame-03
    > 
    > Hi Al,
    > 
    > Thanks, that is clear. I think that discussing the assumption of honesty
    > among the parties involved in benchmarking  would be a useful addition to
    > the Security Considerations section in the draft.
    [acm] 

    I don't mind explaining the requirement using the term "honesty", but I can only imagine raised eyebrows and subsequent DISCUSS/comments if we try to assert a need for/assumption of honesty anywhere in the memo.

    Do you have suggested wording?

    Do others have opinions whether or not this is needed?

    thanks,
    Al

    > 
    > Mališa
    > 
    > On 15/12/2020 14:45, "MORTON, ALFRED C (AL)" <acm@research.att.com> wrote:
    > 
    >     Hi Mališa,
    >     thanks for your review, please see below for one reply to your
    > question (acm].
    >     Al
    > 
    >     > -----Original Message-----
    >     > From: bmwg [mailto:bmwg-bounces@ietf.org] On Behalf Of Mališa
    > Vucinic via
    >     > Datatracker
    >     > Sent: Tuesday, December 15, 2020 6:30 AM
    >     > To: secdir@ietf.org
    >     > Cc: last-call@ietf.org; bmwg@ietf.org; draft-ietf-bmwg-b2b-
    >     > frame.all@ietf.org
    >     > Subject: [bmwg] Secdir telechat review of draft-ietf-bmwg-b2b-frame-
    > 03
    >     >
    >     > Reviewer: Mališa Vučinić
    >     > Review result: Ready
    >     >
    >     > I reviewed this document as part of the Security Directorate's
    > ongoing
    >     > effort
    >     > to review all IETF documents being processed by the IESG. These
    > comments
    >     > were
    >     > written primarily for the benefit of the Security Area Directors.
    > Document
    >     > authors, document editors, and WG chairs should treat these comments
    > just
    >     > like
    >     > any other IETF Last Call comments.
    >     >
    >     > Thank you for this well-written document, it was a pleasure to read
    > and I
    >     > think
    >     > it is ready to proceed. Since the document updates RFC2544
    > benchmarking
    >     > procedure for estimating the buffer time of a Device Under Test
    > (DUT), it
    >     > does
    >     > not raise any security issues. Security Considerations section is
    > quite
    >     > clear
    >     > and it stresses that these tests are performed in a lab environment.
    >     >
    >     > I do have a question regarding the last paragraph of the Security
    >     > Considerations on special capabilities of DUTs for benchmarking
    > purposes.
    >     > Currently, the sentence reads: "Special capabilities SHOULD NOT
    > exist in
    >     > the
    >     > DUT/SUT specifically for benchmarking purposes." Why is this a
    > SHOULD NOT
    >     > and
    >     > not a MUST NOT? Could you give an example when such special
    > capabilities
    >     > in a
    >     > DUT are appropriate?
    >     [acm]
    >     We can only make a strong recommendation in this area. As
    > testers/benchmarkers are often independent from the DUT developers and
    > conduct testing external to the DUT, we assume honesty among other parties
    > but we cannot require it. If someone constructed a DUT that recognized
    > test conditions and operated differently to perform better somehow, our
    > tests would measure the intended "better" performance. It takes a
    > special/additional test effort to prove that a DUT has "designed to the
    > test" (consider Volkswagen and fuel efficiency testing [0]).
    > 
    >     We simply do not have any authority in this matter, but we can let all
    > parties know that gaming the test can be discovered and reported (albeit
    > with more testing that we do not describe).
    > 
    >     [0] https://urldefense.com/v3/__https://www.consumerreports.org/fuel-
    > economy-efficiency/volkswagen-used-special-software-to-exaggerate-fuel-
    > economy/__;!!BhdT!0KS_VCF5ZQfIGkVyPLoJXuAxdcoS3-
    > xJTE0LoKZPWuSiHjQZM1u0H9M36YXByCk$
    > 
    >     >
    >     >
    >     >
    >     > _______________________________________________
    >     > bmwg mailing list
    >     > bmwg@ietf.org
    >     >
    > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/bmwg__;!
    >     > !BhdT!1JFeLsENzMU-ew89jxmJKxfp4wj5Zo3AZ6V8iULU3hWAentH1dymqJmDOvw7$
    > 
    > 