Re: [Pearg] I-D Action: draft-irtf-pearg-safe-internet-measurement-00.txt

Eric Rescorla <ekr@rtfm.com> Mon, 08 July 2019 14:16 UTC

Return-Path: <ekr@rtfm.com>
X-Original-To: pearg@ietfa.amsl.com
Delivered-To: pearg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C6F35120279 for <pearg@ietfa.amsl.com>; Mon, 8 Jul 2019 07:16:18 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.601
X-Spam-Level:
X-Spam-Status: No, score=-0.601 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, PDS_NO_HELO_DNS=1.295, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=rtfm-com.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ElncGIgvYhfg for <pearg@ietfa.amsl.com>; Mon, 8 Jul 2019 07:16:17 -0700 (PDT)
Received: from mail-lf1-x12a.google.com (mail-lf1-x12a.google.com [IPv6:2a00:1450:4864:20::12a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 0B1EF120273 for <pearg@irtf.org>; Mon, 8 Jul 2019 07:16:17 -0700 (PDT)
Received: by mail-lf1-x12a.google.com with SMTP id q26so11054986lfc.3 for <pearg@irtf.org>; Mon, 08 Jul 2019 07:16:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rtfm-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=6NuEk1AjKGLGlP8OvQdmXQ5PUj0OKKNvmpfUCZso8zg=; b=u4Tz7MxaruI4cQ/0Nibi3G153OOkN3za5sq1P6CoBShM6ttZBhOQy0oKim/0+rPOLo wM2LkbkdsOJfF+2vNymK0y5vOHTylhkv3+N+Dl1uX+o79cmw9NsbamadbGUCrJrvz9sb muKaQ3ogc6stjOGpuImFk5CM5I2m/v0D37spU3ZUo3CXwp/gYieIaulbU9b9dWzx8y3J O5tYj57VbUKG2ZjTjeZ3yq8orvvUNkhG6vq2Z6UurtWlNZFyGjohptFJz456PPivD8so zdB183hR5JKX2lgz7iLCinHX3bEEVJrXbTaXPZm3d1Owv4/PywOjoBG6ieV84Z1Yh8Pw GipA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=6NuEk1AjKGLGlP8OvQdmXQ5PUj0OKKNvmpfUCZso8zg=; b=dnrSFzRFIMiW9g3kXdwIOnrllvHBiTTRc8bf4sEjHreWrPB3ECsYgHclmErFdRWQw/ V5DOCZ+Q01VMzsbmcjc3J9dGkGlMxO5dDb9x5sapEx7ha4mPlTil5AIex1txRSveladd GSsFhNBzKqKw5oBxVitZJRxD9pSfu78ugHktbg/YObEDziQO9wEPuXK5dc0dWxHBmW/p voLXr9F8YIkHkODnpXWOSZe70LstChj4JISFTMN78qtHFVcKN/jlWcDjzINqsasxAufj UYwcnC6kOcL5dVuUOsrvlTETzEptVTVDU1DRirmbL8YCRzWi+JuukFoQHgmXtS0QlAR7 +jEw==
X-Gm-Message-State: APjAAAWsdLjs6fu/OgATZOU01m01EonMlvabC0wZEhCvEqt7SdQc+Y/E 4k/uuV3SyFDxlKJOIKZGDM4jAMKfxg+HSW2wFmyNZw==
X-Google-Smtp-Source: APXvYqzEMTogcB+6YL+LkNNL/YZrSpGZBPNPejmWNh8uJ6591Bb6PbDpQoQddwCdM3hhiOVohBN4iF8iVQof2aMJC4c=
X-Received: by 2002:a19:f24e:: with SMTP id d14mr9315095lfk.184.1562595375170; Mon, 08 Jul 2019 07:16:15 -0700 (PDT)
MIME-Version: 1.0
References: <156254420044.4995.7471139515518776754@ietfa.amsl.com> <240d826f-1d7a-834a-919a-f4d5aa9fed58@torproject.org> <CABcZeBMUyXVyAQZkzHc+uCD8AS-_apihjop9QwQxkFOGz4KrZg@mail.gmail.com> <279a7516-a08d-12a6-1693-b49c94c3c2e5@torproject.org>
In-Reply-To: <279a7516-a08d-12a6-1693-b49c94c3c2e5@torproject.org>
From: Eric Rescorla <ekr@rtfm.com>
Date: Mon, 8 Jul 2019 07:15:39 -0700
Message-ID: <CABcZeBOJvzPdPy49_8aQ6w5GiJF2fqFbUbSGGLnQokj4Bo5_XA@mail.gmail.com>
To: Iain Learmonth <irl@torproject.org>
Cc: pearg@irtf.org
Content-Type: multipart/alternative; boundary="00000000000043dc16058d2c1632"
Archived-At: <https://mailarchive.ietf.org/arch/msg/pearg/vnuLuAH88Kt_DrPhudEzk2KwE84>
Subject: Re: [Pearg] I-D Action: draft-irtf-pearg-safe-internet-measurement-00.txt
X-BeenThere: pearg@irtf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Privacy Enhancements and Assessment Proposed RG <pearg.irtf.org>
List-Unsubscribe: <https://www.irtf.org/mailman/options/pearg>, <mailto:pearg-request@irtf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/pearg/>
List-Post: <mailto:pearg@irtf.org>
List-Help: <mailto:pearg-request@irtf.org?subject=help>
List-Subscribe: <https://www.irtf.org/mailman/listinfo/pearg>, <mailto:pearg-request@irtf.org?subject=subscribe>
X-List-Received-Date: Mon, 08 Jul 2019 14:16:28 -0000

On Mon, Jul 8, 2019 at 6:47 AM Iain Learmonth <irl@torproject.org> wrote:

> Hi Eric,
>
> On 08/07/2019 14:30, Eric Rescorla wrote:
> > After a quick look, I see....
> >
> > Document: draft-irtf-pearg-safe-internet-measurement-01.txt
> >
> >    The reduced impact should not be used as an excuse for pushing higher
> >    risk updates, only updates that could be considered appropriate to
> >    push to all users should be A/B tested.
> >
> > This may just be wordsmithing, but as written, this text is entirely
> > unrealistic. One of the major reasons that one does A/B testing is
> > that you are concerned about risk in the Treatment group (e.g., that
> > there will be a higher risk of failures, crashes, etc.) and the
> > reason you are doing an A/B test is to mitigate that risk.
>
> The point that I'd like to put across here is that it is not an excuse
> to be reckless or careless. A/B testing can mitigate risk to reputation
> perhaps, and sure it can reduce the risk that any individual user is
> affected by a bad update, but it doesn't mitigate the impact for the
> users that are affected.
>

This seems like the kind of product question that is well out of scope for
PEARG. Software vendors have a wide variety of processes for determining
whether a given piece of code is suitable for shipping to their users,
ranging (at least) from "some developer thought it was good" to "multiple
detailed code reviews". I think we can all agree that defining that is out
of scope, but without that, defining what you have to do before you ship to
a fraction of your users is largely irrelevant.

-Ekr