Re: [Pearg] I-D Action: draft-irtf-pearg-safe-internet-measurement-00.txt

Eric Rescorla <ekr@rtfm.com> Mon, 08 July 2019 14:48 UTC

Return-Path: <ekr@rtfm.com>
X-Original-To: pearg@ietfa.amsl.com
Delivered-To: pearg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D8EEA120276 for <pearg@ietfa.amsl.com>; Mon, 8 Jul 2019 07:48:28 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.601
X-Spam-Level:
X-Spam-Status: No, score=-0.601 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, PDS_NO_HELO_DNS=1.295, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=rtfm-com.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vvtpw7jBSl5W for <pearg@ietfa.amsl.com>; Mon, 8 Jul 2019 07:48:27 -0700 (PDT)
Received: from mail-lj1-x22e.google.com (mail-lj1-x22e.google.com [IPv6:2a00:1450:4864:20::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 91544120219 for <pearg@irtf.org>; Mon, 8 Jul 2019 07:48:16 -0700 (PDT)
Received: by mail-lj1-x22e.google.com with SMTP id v24so16199275ljg.13 for <pearg@irtf.org>; Mon, 08 Jul 2019 07:48:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rtfm-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=3SnVUTQWcaD+nOK0R26yb4hvwMY+MGYoy2Bp1noiMdM=; b=RCALRxUgCazhRM+dC6jHSk3ZllLGgLgS6qCbZiZTAHTGO0CcpkqSPMn2jK1B5h3MDV /xjyH/U+hwXkxkZiIIo2r8S8J/My3AeN+p1W1QgWiaJtyCLBEPp51rX57pByJdQNvJWw m0zFcfu8trmoNboAlWgOKRXSzGEpuxqpDYuhsSwSllpkXAJXH4hC+LjLMw8r94Oz4SbO ar8BJRTfQIoIK14nFN83yFOJwQkeUb9AOoUo4lfhxHRpaMvQVjXZxTlII2Zx7tktpsDt wyKtlztsVxjkjb4InFPk4LOhW543RYG2A5dm9yhLNtRDc8i48qqB+U4LG6OipWlMvSOa jo7w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=3SnVUTQWcaD+nOK0R26yb4hvwMY+MGYoy2Bp1noiMdM=; b=RK/MKGnnsRmUAQeH0rKgJzqEhg2pnrxHfMuTiD3/XhR/e1rK6E2A0V9vMPs2uehEjm v1KdEdzg6Xqj+nufz7KeZcbj5X49JMvWrkho3dwsV9mNkRsZZ1eFUxkoUYshvy10Dc+S i/F1NPSB7yiy0U6/dROzpARcGmyebyKUN0j8JneDYErkzWsdx7ph+EXTSkv0mRclUEyS fRP9NR1CVyoPKmXI60pAgc7/S48Pee65kI5uv498Wez5jOZktDHV7pglbSU8zXIF93NH pGIjezFv4qngZ1CqBfq6mAAXEK8A/adP6eWYabQglbdVLSvs6SmvrZGkKL5WE9M3BeAm c2kg==
X-Gm-Message-State: APjAAAXKNSMJSt++PfGC7xAv+e6XASK+73CvN5rGXknpdIJxaWvJyPYT qYaYKkZpbNsgUMy1asP9mDkxTHrpKwMkyKiQzgkMUw==
X-Google-Smtp-Source: APXvYqzB3gcXQNJ5omSz6u+suDoYo8DCIG8f4+L5TBE3vdops8ZHf5jN0AxCeW3iGOu7OMfhO6dF2c+zRdkjXu5XFic=
X-Received: by 2002:a2e:3008:: with SMTP id w8mr10918358ljw.13.1562597294816; Mon, 08 Jul 2019 07:48:14 -0700 (PDT)
MIME-Version: 1.0
References: <156254420044.4995.7471139515518776754@ietfa.amsl.com> <240d826f-1d7a-834a-919a-f4d5aa9fed58@torproject.org> <CABcZeBMUyXVyAQZkzHc+uCD8AS-_apihjop9QwQxkFOGz4KrZg@mail.gmail.com> <279a7516-a08d-12a6-1693-b49c94c3c2e5@torproject.org> <CABcZeBOJvzPdPy49_8aQ6w5GiJF2fqFbUbSGGLnQokj4Bo5_XA@mail.gmail.com> <62cbf8ac-8ed2-913b-07b6-cafaf8250468@digitaldissidents.org>
In-Reply-To: <62cbf8ac-8ed2-913b-07b6-cafaf8250468@digitaldissidents.org>
From: Eric Rescorla <ekr@rtfm.com>
Date: Mon, 8 Jul 2019 07:47:38 -0700
Message-ID: <CABcZeBOM7rXU1Jt8Cbyp3c4n1FrWd6NS98wU5-3Cv-4r-t674Q@mail.gmail.com>
To: Niels ten Oever <lists@digitaldissidents.org>
Cc: pearg@irtf.org
Content-Type: multipart/alternative; boundary="000000000000af4e78058d2c88da"
Archived-At: <https://mailarchive.ietf.org/arch/msg/pearg/t5QS_qRnS0hF3krNYCKYDGxDqIg>
Subject: Re: [Pearg] I-D Action: draft-irtf-pearg-safe-internet-measurement-00.txt
X-BeenThere: pearg@irtf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Privacy Enhancements and Assessment Proposed RG <pearg.irtf.org>
List-Unsubscribe: <https://www.irtf.org/mailman/options/pearg>, <mailto:pearg-request@irtf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/pearg/>
List-Post: <mailto:pearg@irtf.org>
List-Help: <mailto:pearg-request@irtf.org?subject=help>
List-Subscribe: <https://www.irtf.org/mailman/listinfo/pearg>, <mailto:pearg-request@irtf.org?subject=subscribe>
X-List-Received-Date: Mon, 08 Jul 2019 14:48:39 -0000

On Mon, Jul 8, 2019 at 7:42 AM Niels ten Oever <lists@digitaldissidents.org>
wrote:

>
>
> On 7/8/19 4:15 PM, Eric Rescorla wrote:
> >
> >
> > On Mon, Jul 8, 2019 at 6:47 AM Iain Learmonth <irl@torproject.org
> <mailto:irl@torproject.org>> wrote:
> >
> >     Hi Eric,
> >
> >     On 08/07/2019 14:30, Eric Rescorla wrote:
> >     > After a quick look, I see....
> >     >
> >     > Document: draft-irtf-pearg-safe-internet-measurement-01.txt
> >     >
> >     >    The reduced impact should not be used as an excuse for pushing
> higher
> >     >    risk updates, only updates that could be considered appropriate
> to
> >     >    push to all users should be A/B tested.
> >     >
> >     > This may just be wordsmithing, but as written, this text is
> entirely
> >     > unrealistic. One of the major reasons that one does A/B testing is
> >     > that you are concerned about risk in the Treatment group (e.g.,
> that
> >     > there will be a higher risk of failures, crashes, etc.) and the
> >     > reason you are doing an A/B test is to mitigate that risk.
> >
> >     The point that I'd like to put across here is that it is not an
> excuse
> >     to be reckless or careless. A/B testing can mitigate risk to
> reputation
> >     perhaps, and sure it can reduce the risk that any individual user is
> >     affected by a bad update, but it doesn't mitigate the impact for the
> >     users that are affected.
> >
> >
> > This seems like the kind of product question that is well out of scope
> for PEARG. Software vendors have a wide variety of processes for
> determining whether a given piece of code is suitable for shipping to their
> users, ranging (at least) from "some developer thought it was good" to
> "multiple detailed code reviews". I think we can all agree that defining
> that is out of scope, but without that,
>
> Why could we not document best practices for this? Seems quite useful to
> me.
>

The question isn't useful, but rather what this group is chartered to do.
Do you have some charter text that would support such an activity?

-Ekr

> defining what you have to do before you ship to a fraction of your users
> is largely irrelevant.
> >
>
> Why ?
>
> Niels
>
> > -Ekr
> >
> >
>
> --
> Niels ten Oever
> Researcher and PhD Candidate
> Datactive Research Group
> University of Amsterdam
>
> PGP fingerprint    2458 0B70 5C4A FD8A 9488
>                    643A 0ED8 3F3A 468A C8B3
>
> --
> Pearg mailing list
> Pearg@irtf.org
> https://www.irtf.org/mailman/listinfo/pearg
>