Re: [Pearg] I-D Action: draft-irtf-pearg-safe-internet-measurement-00.txt

Eric Rescorla <ekr@rtfm.com> Mon, 08 July 2019 15:27 UTC

Return-Path: <ekr@rtfm.com>
X-Original-To: pearg@ietfa.amsl.com
Delivered-To: pearg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2DF9A1202D3 for <pearg@ietfa.amsl.com>; Mon, 8 Jul 2019 08:27:01 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.601
X-Spam-Level:
X-Spam-Status: No, score=-0.601 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, PDS_NO_HELO_DNS=1.295, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=rtfm-com.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rTqXePE6CJJE for <pearg@ietfa.amsl.com>; Mon, 8 Jul 2019 08:26:59 -0700 (PDT)
Received: from mail-lj1-x22c.google.com (mail-lj1-x22c.google.com [IPv6:2a00:1450:4864:20::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 020211202C5 for <pearg@irtf.org>; Mon, 8 Jul 2019 08:25:41 -0700 (PDT)
Received: by mail-lj1-x22c.google.com with SMTP id v24so16338453ljg.13 for <pearg@irtf.org>; Mon, 08 Jul 2019 08:25:40 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rtfm-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=SW31GyEynzeWG9cuo24ePuZS60LXg97ymZHt3ZGwyAI=; b=YEyaSUYhu3h4f0nW/jFnQhmA9zFYfhPgmMPLSMocFEZgAiRVyo8CdUTDO0w57gHc3n xPLAijT2TOeeOqVkyFy6IqNufr2gDzHBnHXVmVUuZPUFnF/UpmVuO6ulUJy8nz0ZZjql IuAqZQciJYql8chxMRYrdmsPKzG6K55vy+uRdy2SmYybdvKJQNWKaeI2MylIkHqqHPgJ PyU5lbTl70/GRNRMoxNZwLPDjK52CKOWiOHSeVJgce9sWF9ClszVHMojecLMW7UHhb0v w20yiAHYINUOapmOccA8eNx3ekO+TAH+/+ruf3zFzgmrY3CPY6zIUS9WPrd3dNlcf44u Wrng==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=SW31GyEynzeWG9cuo24ePuZS60LXg97ymZHt3ZGwyAI=; b=kEesedKhDjTtbhWRVuUf5bK49Mf4QIVsqhTA/VhYYbUbnl6icAtVKrWbkNm+Qdz/Td JAFO+ldgXgLZ6EporbhLMsIS6cPb2p3ssPWLWlo7XYtbf03q9UfsDOvx6BlWLWwwjLh8 U7mK2o8HjBhwIrWLg/H0CYO5/Z11hGrZe9MRU+x5Gr4auKuKsJDHByNzRq58qXiCYFJo DCmBexiepvggi1OGaEEFBlPK9hhZwEgCRqREDe8MKM4WmoimPufs14KbfFmcJSUxt1BL cMXOht9/qGPbUvP5vUQM9v9LsTOvxudPEpnVoAGPMasC916VkPmWQ4EzeMNbqrSCjN90 qtcQ==
X-Gm-Message-State: APjAAAWWajUZZbhAisUmcGYjUEZ5lV9n4Jp1TNRGvKYYmHuayDxnRPNT mpR1vpKMj/v9QFJuxlmpsM4eCg2xuQXb4WDzPJslAg==
X-Google-Smtp-Source: APXvYqxWS8yngRFLiLNmr8KxbLpTGXK0HD7+Pa9pPsPTfuF3poWKVYQpCKYH8dLE5ol/hsQHxKv1ncqb9fLiRR9NPSI=
X-Received: by 2002:a2e:96d0:: with SMTP id d16mr10856877ljj.14.1562599539096; Mon, 08 Jul 2019 08:25:39 -0700 (PDT)
MIME-Version: 1.0
References: <156254420044.4995.7471139515518776754@ietfa.amsl.com> <240d826f-1d7a-834a-919a-f4d5aa9fed58@torproject.org> <CABcZeBMUyXVyAQZkzHc+uCD8AS-_apihjop9QwQxkFOGz4KrZg@mail.gmail.com> <279a7516-a08d-12a6-1693-b49c94c3c2e5@torproject.org> <CABcZeBOJvzPdPy49_8aQ6w5GiJF2fqFbUbSGGLnQokj4Bo5_XA@mail.gmail.com> <4c853f4c-2103-527a-e1c2-b5b8c92884cd@torproject.org>
In-Reply-To: <4c853f4c-2103-527a-e1c2-b5b8c92884cd@torproject.org>
From: Eric Rescorla <ekr@rtfm.com>
Date: Mon, 8 Jul 2019 08:25:02 -0700
Message-ID: <CABcZeBMSb9aCM4=WZ53rZFHmdN-KjHwMz-bLgJUjhaUrm-bg2w@mail.gmail.com>
To: Iain Learmonth <irl@torproject.org>
Cc: pearg@irtf.org
Content-Type: multipart/alternative; boundary="000000000000745df4058d2d0e10"
Archived-At: <https://mailarchive.ietf.org/arch/msg/pearg/YiO0MBfXlFFySJ19f-GZdUXTOK8>
Subject: Re: [Pearg] I-D Action: draft-irtf-pearg-safe-internet-measurement-00.txt
X-BeenThere: pearg@irtf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Privacy Enhancements and Assessment Proposed RG <pearg.irtf.org>
List-Unsubscribe: <https://www.irtf.org/mailman/options/pearg>, <mailto:pearg-request@irtf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/pearg/>
List-Post: <mailto:pearg@irtf.org>
List-Help: <mailto:pearg-request@irtf.org?subject=help>
List-Subscribe: <https://www.irtf.org/mailman/listinfo/pearg>, <mailto:pearg-request@irtf.org?subject=subscribe>
X-List-Received-Date: Mon, 08 Jul 2019 15:27:09 -0000

On Mon, Jul 8, 2019 at 8:00 AM Iain Learmonth <irl@torproject.org> wrote:

> Hi Eric,
>
> On 08/07/2019 15:15, Eric Rescorla wrote:
> > This seems like the kind of product question that is well out of scope
> > for PEARG. Software vendors have a wide variety of processes for
> > determining whether a given piece of code is suitable for shipping to
> > their users, ranging (at least) from "some developer thought it was
> > good" to "multiple detailed code reviews".
>
> The point is not so much to define what make code suitable for shipping,
> but to say that you should not use A/B testing as an excuse to ship code
> that you would not have otherwise shipped.
>

But again, that's precisely the point of much A/B testing: to verify that
the
code is good enough to ship. It's *part* of the shipping process.


Your criteria for shipping to some users should be the same as the
> criteria for shipping to all users. The code quality should be the same,
> it should have undergone the same pre-shipping testing, etc. The only
> difference should be the feature that is enabled/disabled.
>
> This is also needed to make the experiment into good science. If you've
> got other variables like relaxed review processes then you've introduced
> other factors that don't make it a fair comparison between the A/B groups.
>
> This all comes back to implied consent, and when it is appropriate to
> presume that users will be expecting something. If you typically have
> stringent review processes, users will expect this from future updates.
> You violate the user's expectations, and also their trust, when you push
> a shoddy update to them. This is true even if you've only pushed it to a
> tiny fraction of your users.
>

As I said to Niels, the question of precisely what code to ship to users
seems well outside the charter of this group, which is focused strictly
on privacy. Can you please point to some charter text which would
authorize the group to work on this?

-Ekr






> Thanks,
> Iain.
>
>