Re: [Pearg] I-D Action: draft-irtf-pearg-safe-internet-measurement-00.txt

"Christopher Wood" <caw@heapingbits.net> Wed, 10 July 2019 16:56 UTC

Return-Path: <caw@heapingbits.net>
X-Original-To: pearg@ietfa.amsl.com
Delivered-To: pearg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7E3F51203E3 for <pearg@ietfa.amsl.com>; Wed, 10 Jul 2019 09:56:18 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.7
X-Spam-Level:
X-Spam-Status: No, score=-2.7 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=heapingbits.net header.b=WUHITZTy; dkim=pass (2048-bit key) header.d=messagingengine.com header.b=kBjVdHYq
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id boU_qwMSybgs for <pearg@ietfa.amsl.com>; Wed, 10 Jul 2019 09:56:15 -0700 (PDT)
Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 9CC231202BF for <pearg@irtf.org>; Wed, 10 Jul 2019 09:55:19 -0700 (PDT)
Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 9913421F41 for <pearg@irtf.org>; Wed, 10 Jul 2019 12:55:18 -0400 (EDT)
Received: from imap4 ([10.202.2.54]) by compute6.internal (MEProxy); Wed, 10 Jul 2019 12:55:18 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heapingbits.net; h=mime-version:message-id:in-reply-to:references:date:from:to :subject:content-type:content-transfer-encoding; s=fm1; bh=ENE1A 5viB4dyaPNHacXpx6rQze8MLtQPqmSGH62yj9E=; b=WUHITZTyenCwtFUfSwhSs t/bveeSexbb0XBj/jsgWLcgVF8NxW2hPDmwOIAEL6eSktdxqIWhS1e0ZYKj/9FpF ln0cWc/Pu/ud8FBlalAJqm7fUniDKMVt6+4n292oNF382X/SOI4G7+BkNdCu/iC9 wrddRqi0AJOt3KPiWL1FBfIAo4x2EuDi9orljGouniN3q+k2A6NQ8Cd44E/n9od1 g8GT08zDHzc6EBZ17X07FqzFGRetf3z5DoWKDQ5sj7/17seFWioNnUHBd5HCGeAu wKWfpUtT0tn81cLoGs6iw6Rg8qlyPlMRT+jP7kT08nGZnWotYU1GuVv0u+E8Y8CQ g==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=ENE1A5viB4dyaPNHacXpx6rQze8MLtQPqmSGH62yj 9E=; b=kBjVdHYqsrSyvmxvm+Y1yEXH96HEbYQdYmPCa5Ur3hcBw87BlqPaz69Yg OY7uAvPp/lofwA59ypRn/ksRC635+tRQpPnRSv7uewwJETnDW3Za19uUgG2Opcbx RwFDJRlNbhUAJIXeGFOFeI2ZRXwJ9U7HYhfkPYy1gm0TQZUOqW+m8y96ufLNuoR+ yoj7p7dRIME/qjq3CjYaElfzGrZGtIKeET+bEIoRX+QB3EPejK10pM7YUJDomS0e OWgvLLfCLP7PuSfcB/W3zOAXciohsVtWmwWSKkOaLAviTPRS/DVOeVydctGDC3zm rKAyAkztaqasx69AArREKxjoe9EOA==
X-ME-Sender: <xms:dhgmXcrRsNiYSfKvP174953OOC0zdgDhGviJZaD7TBVosgK0KPyigQ>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduvddrgeeigddutdekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfgjfhffhffvufgtgfesth hqredtreerjeenucfhrhhomhepfdevhhhrihhsthhophhhvghrucghohhougdfuceotggr fieshhgvrghpihhnghgsihhtshdrnhgvtheqnecuffhomhgrihhnpehirhhtfhdrohhrgh enucfrrghrrghmpehmrghilhhfrhhomheptggrfieshhgvrghpihhnghgsihhtshdrnhgv thenucevlhhushhtvghrufhiiigvpedt
X-ME-Proxy: <xmx:dhgmXYYUI50M7C7LR3kZj3W4NHYlgIdryCFgjNtvz6zVOi0bojrXyw> <xmx:dhgmXYDCFgEKfXnIzUHQCnLBHrTXvjZyUn0tjiWHVuz9r9U07K3tcw> <xmx:dhgmXbFnAPOhqiu8YCVBBhjvCEvbgCbyPdx_hUddyLm1sp_KBGiGPw> <xmx:dhgmXQqm-rhQtfbidnFFaVAgHysGDkTTCRuZxCKJHbIoZfpPvJJr4Q>
Received: by mailuser.nyi.internal (Postfix, from userid 501) id 0B11B3C00A1; Wed, 10 Jul 2019 12:55:18 -0400 (EDT)
X-Mailer: MessagingEngine.com Webmail Interface
User-Agent: Cyrus-JMAP/3.1.6-731-g19d3b16-fmstable-20190627v1
Mime-Version: 1.0
Message-Id: <63de9f42-96c2-4ef9-bd7d-e1a9c62a3c6e@www.fastmail.com>
In-Reply-To: <CABcZeBMSb9aCM4=WZ53rZFHmdN-KjHwMz-bLgJUjhaUrm-bg2w@mail.gmail.com>
References: <156254420044.4995.7471139515518776754@ietfa.amsl.com> <240d826f-1d7a-834a-919a-f4d5aa9fed58@torproject.org> <CABcZeBMUyXVyAQZkzHc+uCD8AS-_apihjop9QwQxkFOGz4KrZg@mail.gmail.com> <279a7516-a08d-12a6-1693-b49c94c3c2e5@torproject.org> <CABcZeBOJvzPdPy49_8aQ6w5GiJF2fqFbUbSGGLnQokj4Bo5_XA@mail.gmail.com> <4c853f4c-2103-527a-e1c2-b5b8c92884cd@torproject.org> <CABcZeBMSb9aCM4=WZ53rZFHmdN-KjHwMz-bLgJUjhaUrm-bg2w@mail.gmail.com>
Date: Wed, 10 Jul 2019 09:55:17 -0700
From: "Christopher Wood" <caw@heapingbits.net>
To: pearg@irtf.org
Content-Type: text/plain;charset=utf-8
Content-Transfer-Encoding: quoted-printable
Archived-At: <https://mailarchive.ietf.org/arch/msg/pearg/7ng5Bxk5RTn3M-3YAwt54z_10ss>
Subject: Re: [Pearg] =?utf-8?q?I-D_Action=3A_draft-irtf-pearg-safe-internet-m?= =?utf-8?q?easurement-00=2Etxt?=
X-BeenThere: pearg@irtf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Privacy Enhancements and Assessment Proposed RG <pearg.irtf.org>
List-Unsubscribe: <https://www.irtf.org/mailman/options/pearg>, <mailto:pearg-request@irtf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/pearg/>
List-Post: <mailto:pearg@irtf.org>
List-Help: <mailto:pearg-request@irtf.org?subject=help>
List-Subscribe: <https://www.irtf.org/mailman/listinfo/pearg>, <mailto:pearg-request@irtf.org?subject=subscribe>
X-List-Received-Date: Wed, 10 Jul 2019 16:56:23 -0000

Hi Ian,

As Eric points out, criteria for shipping software is well outside the scope of this RG. Thus, it should not be used as a basis for specifying measurement practices. More generally, as a document describing safe Internet measurements in a privacy-oriented RG, issues such as code quality are out of scope. While it’s true that software bugs can lead to privacy issues, criteria for preventing bugs is not a topic this group is chartered to work on. That said, being mindful of privacy when conducting A/B testing is good (though not very actionable) advice.

Also, for what it’s worth, I think it’s prudent to consider the advice and recommendations from folks who have done these experiments at scale. 

Best,
Chris, on behalf of the chairs 

On Mon, Jul 8, 2019, at 8:27 AM, Eric Rescorla wrote:
> 
> 
> On Mon, Jul 8, 2019 at 8:00 AM Iain Learmonth <irl@torproject.org> wrote:
> > Hi Eric,
> > 
> >  On 08/07/2019 15:15, Eric Rescorla wrote:
> >  > This seems like the kind of product question that is well out of scope
> >  > for PEARG. Software vendors have a wide variety of processes for
> >  > determining whether a given piece of code is suitable for shipping to
> >  > their users, ranging (at least) from "some developer thought it was
> >  > good" to "multiple detailed code reviews".
> > 
> >  The point is not so much to define what make code suitable for shipping,
> >  but to say that you should not use A/B testing as an excuse to ship code
> >  that you would not have otherwise shipped.
> 
> But again, that's precisely the point of much A/B testing: to verify that the
> code is good enough to ship. It's *part* of the shipping process.
> 
> 
> >  Your criteria for shipping to some users should be the same as the
> >  criteria for shipping to all users. The code quality should be the same,
> >  it should have undergone the same pre-shipping testing, etc. The only
> >  difference should be the feature that is enabled/disabled.
> > 
> >  This is also needed to make the experiment into good science. If you've
> >  got other variables like relaxed review processes then you've introduced
> >  other factors that don't make it a fair comparison between the A/B groups.
> > 
> >  This all comes back to implied consent, and when it is appropriate to
> >  presume that users will be expecting something. If you typically have
> >  stringent review processes, users will expect this from future updates.
> >  You violate the user's expectations, and also their trust, when you push
> >  a shoddy update to them. This is true even if you've only pushed it to a
> >  tiny fraction of your users.
> 
> As I said to Niels, the question of precisely what code to ship to users
> seems well outside the charter of this group, which is focused strictly
> on privacy. Can you please point to some charter text which would
> authorize the group to work on this?
> 
> -Ekr
> 
> 
> 
> 
> 
> > 
> >  Thanks,
> >  Iain.
> > 
> -- 
> Pearg mailing list
> Pearg@irtf.org
> https://www.irtf.org/mailman/listinfo/pearg
>