Re: [Pearg] I-D Action: draft-irtf-pearg-safe-internet-measurement-00.txt

Iain Learmonth <irl@torproject.org> Mon, 08 July 2019 15:00 UTC

Return-Path: <irl@torproject.org>
X-Original-To: pearg@ietfa.amsl.com
Delivered-To: pearg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CCF03120294 for <pearg@ietfa.amsl.com>; Mon, 8 Jul 2019 08:00:50 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level:
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_NONE=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=messagingengine.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WAsRgx5ISZxG for <pearg@ietfa.amsl.com>; Mon, 8 Jul 2019 08:00:48 -0700 (PDT)
Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A61A4120250 for <pearg@irtf.org>; Mon, 8 Jul 2019 08:00:48 -0700 (PDT)
Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id D80F2220AA; Mon, 8 Jul 2019 11:00:47 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Mon, 08 Jul 2019 11:00:47 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=hV11R0 lSsLxszFkz+z6lB7lvT/dTuWa7K2aoP1UDXOM=; b=TOeTm1iLdIdL8fS69z/+KY HW65JvPXJnPdmZ89rsJ1Lpe0ReDWxXjEUI/MLGlcYHtloBbCbYdrtD5K7M+0L5IV qxswhx+h6NrIUieODriDPhO2OIjLgVnMfqF44oFAQYfo0EzuUcABTTlFQcoEcnUR 6S3rKHRCeBDmLT70E2YzeMisU1icGzgyEnkQPJg31Z+72U1ahbljNF+BbmWFj8lo 8G1s14fc/Y9pqM2Yf7/EkIe82TTm6HGCn0NkSURtOzpG5+AZhu+kaLkmQ2HpKue8 zjMD4dhVP5oMyqFUZlUlJqOnEY+Nr4FsMZKuRGJzskEYhyVF1sCmWcQsLrrRIUQw ==
X-ME-Sender: <xms:n1ojXXW8XkU_I-Haa_Zv904TIUqdUZ5kl_WO2FEeo_8N6U7EKId2KQ>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduvddrgedtgdekiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepvfhfhfhouffkffgfgggjtgesghdtre fotdefjeenucfhrhhomhepkfgrihhnucfnvggrrhhmohhnthhhuceoihhrlhesthhorhhp rhhojhgvtghtrdhorhhgqeenucfkphepkeegrdelvddrgedurddvtddunecurfgrrhgrmh epmhgrihhlfhhrohhmpehirhhlsehtohhrphhrohhjvggtthdrohhrghenucevlhhushht vghrufhiiigvpedt
X-ME-Proxy: <xmx:n1ojXT1RqHnNzVwzVOyElktKnzLgsk3yJ-M_9b8bUQiDW2i-YNWvTw> <xmx:n1ojXeZRjpHQ1Z42FW4wb3cvzNcI4S8SgTiptaNjx0CGBiXokrL-bg> <xmx:n1ojXWqMNNq2Zz5wnm51rfPh0OZs9y-Za0DarsFc0o8n0QrbufI6BA> <xmx:n1ojXYcZF2sdGU8Ygq4CvsGj7Ihk6YPYZXkVd5-GpZzKfmFrGRBuHw>
Received: from [172.22.152.150] (shiftout.plus.com [84.92.41.201]) by mail.messagingengine.com (Postfix) with ESMTPA id C96D380069; Mon, 8 Jul 2019 11:00:46 -0400 (EDT)
To: Eric Rescorla <ekr@rtfm.com>
Cc: pearg@irtf.org
References: <156254420044.4995.7471139515518776754@ietfa.amsl.com> <240d826f-1d7a-834a-919a-f4d5aa9fed58@torproject.org> <CABcZeBMUyXVyAQZkzHc+uCD8AS-_apihjop9QwQxkFOGz4KrZg@mail.gmail.com> <279a7516-a08d-12a6-1693-b49c94c3c2e5@torproject.org> <CABcZeBOJvzPdPy49_8aQ6w5GiJF2fqFbUbSGGLnQokj4Bo5_XA@mail.gmail.com>
From: Iain Learmonth <irl@torproject.org>
Openpgp: preference=signencrypt
Autocrypt: addr=irl@torproject.org; prefer-encrypt=mutual; keydata= mQINBFZp8zEBEACxOYriD+tEuc3Wpnbh+GGnyiaLEMABBrfn6JlDQphbBq/YTz9M9OPkttjx hLL/yrxlM1nD69XbGKQ9gIL3LEgOz9+OdivPbN+Q5iNMqk/WCQUqd3bCFbbsn1yvoTumFy9S 9kYX45Db3jRJoN/Nye6Stf7KKPxHxot14iY+PUR/5Gx5KbeWVKfDtQejGnhxQD73KjrX4wds BAaxnQ7KbjQyUf+IxE+8qSDcyTP+pPqxspVzx+eFqsW5+kK1eJMHxJmY/KsAs6IsGf5lvyDJ JECc2iE0mFS6vc14lGcD7BAYMPRnvlK3OcDlbdJS3ZU0LQu3/AplM7cNcesq2Btm06OUTsbj 10ZiyLi7Q0WZRuUbn7t3jOQVyOlNfjUpJhKPMMobBL2R0KzcptJbUrKc08wZD/TPaXuHKWAE JuA6kFMXtHhV8Qhxz5/d2KUA8ex+zpVd2xSR6q4llcYu1w8zHZtLN+YKSmjjKs+AjiTrCMYs OYxt4cwxuaIIhBNvCC9WqZOxHX7YHmpVcSV6K9Wwhk9mVIU3Ii0G2HWs6OQ0vIueCDGMEdVk ig/a7cVlfXNz7WuaXuhOJmHz6d6Yk4dFn5mLbEY9cZhBxf5hjCwtp9b6v+ueuptfcnOd+38G 9KH6NyHKZyS4jcd3E6Dp0+9Isbl/EohjPCujevoW3/DlT08OKQARAQABtDRJYWluIFIuIExl YXJtb250aCAoVG9yIFByb2plY3QpIDxpcmxAdG9ycHJvamVjdC5vcmc+iQJUBBMBCgA+AhsD BQsJCAcDBRUKCQgLBRYCAwEAAh4BAheAFiEEqPe6UEHhMzOcuhaWdtWAk/VAq80FAlxe8QQF CQbiS9IACgkQdtWAk/VAq80isw//QGWLLb6wjPW9K/seio9dSVM3gDk6jU7/FhlUuz7oq/6H gPrl7jrxH9PY5jAlkWCK5iPFWpFczUKC+AzzuZQt8VXQp5iYllm1KRrZA6mxpx085Yo3cSLp H9B55IHENo5RFdXMgXZ2aNc6ENG31vB6IZDHXjw9fUxtqyB7fCbIhFtO9pCqwnH/WipISdrl P5/gllUJKsvZiQj8vFfOe2rEHfldhqXxN0ZFvQtQMy0SipqfY2eM47HoDLZ2L5U1iK3iG1fa UWN059Y92zcc9B/AFqHK9xFMObUlWEvqhCGKgullDp4mVF4/OPAC/l02IS85BeFxWbV/HxTY AeoEzO9qrWVitCPTuFC4TQxJq4TbVr/jriFugmbvlE3IIXz5OjqTpyJZw+A8jJ3lwF+V5z21 LdJpEbh+fBBz6tl99Hnn7rjN9WhhsH+BvSC9fCnxs21h+oiGw9zom6blFsLW1yP7mNX2ZRJR DmQs+7kPL4ZS8XekNN82UHXE2pd9M2fSEi5dHK0R9xKJmMSNi/cnEZp1SRx+e+FFDZwzxiFO R30sDEEf+Z1MB/7o5uQXtw59W5Q7p0KycJMfacnb3dulOhFF4Qu1w1K3Iqefq8cQpnlmIGV0 dYT6juPyDlRW19LQVQSTejkLAbkBuXVE0T6dAYVyudJ2lVa3aDvQV5PqL1Y5tdK5Ag0EWmt/ jQEQANhwgR5fq1Slk0T0Sh7Tkc5LjanQH1VuNTxOE7wzcYXrgva2ic4zdhLc2QHP9G4kz0AY /oLxtw5Sj/IMtdzBHKDBXgiqBmvkrz7mOZSQQh6K/JBcKau4MorzB80Z/Z6XQcfFKQh61+8e Im/I7AnJFUELxDe8CYmkJYKVJS9b+i9RNFvXAsamOkClcyXqPYBB9bBI8QlZHweTWDsXJqul xjHLjCWOQyJxfl9xFmlSJtskrLEmprw/PaOtXglrz/2vadn2lL3ack9V1ux9ALa5q8Oc6dnx vA/W3palFpdBoz75FckhRfliYNfdCpgo86w00bJvbJ1f6XfBIBYvsAvrIRWIkMYEmhTYm4xG gEDWz5CvGuuzkF7kJwfPdj1RgVe80JHr9OJc8ZrafBb/p35ldLxmhDi16j2VOj58CKwpi3En nMOMPteDxmCZrWeYKhfwgVAP4/zc4+9vmTp5Wpze00DnQ9MEinmJ2bxf6TyQP36Bf/8zgJVs rpqMOaumbwwmcUh85Q1PXneVWX2ryEv1I7xVMpW4APZDeOMiiHb+EpXPLTPBvXTerd/Pwzb7 WLKNpZI4tl/vh88INdrs3xZd174skDJNaNSnqzUP2kNYcxWZSH/FP4AH1IXvIxwIr9g0RpvY oX55ALOSQOdZ9ioCjf9x9mvnBsehcoEPrgozljzpABEBAAGJBHIEGAEKACYCGwIWIQSo97pQ QeEzM5y6FpZ21YCT9UCrzQUCXF7xLQUJAuC/nwJAwXQgBBkBCgAdFiEE/ps4MHN0fw3t6Aud F4mIfdjVvF0FAlprf40ACgkQF4mIfdjVvF0NJBAAuAVw1aY3IUtp8ZRI5xuhieVFDbye7z86 gP5KW8cAMKTpuowD7E3Rm31R2Pb5dX+vZlunSTBPuzRVnVNd6Kj7CR8BR426L48ogSUfIh4V IWiYxXn4DqqNGow+saPI8VwJ/y80NSb2v+qUdv7us052vs9UBBnI/zl8XrnUNix+B0g+jekT ENEVD9fNJ+9YE9cn80wLGKBx047id1IqgOeJShjE13WRj7sa+LhB8FtoOUeXduQ1sJBZTVJq XW9v0e2PNc37cc8VATrA+3HNSQJY1PETDxjJ1TW8gALpZQrfR+FDu/d2vALdULw/djCeIpl0 dr79isd6mINBKTkEcR8XPQjDUnrnO1U18YETo71N5Z6HpvaV5/XWLUl81wovddpVRvwbwuFP bJE506UiQJrthPKi6s670VBV6u5I+5+h6D+CHp1xp0Bq8PLz0EwqjZwOiLAByBT/ryD2ZY8+ 2rP9gphnEBcCtyp6Fz6Q0KyutpCPv94b4OU0eUTvl3Nroq2GWGGHAV+8r1uXpqdNSHxYmQgy b3H1dvObmE37L3BWrVNzeFb0USNiV04up/55QUrqrz7kLdBO0ZVMVLU3j7e48KcbJG2yzNHZ dnQa+ZMMIRFtm8vzSt+jEIwodi6nOeBgEiqODF8TZPbTc6183ErFoYc/FuhkUh0aR0MLZ92Q bPEJEHbVgJP1QKvNZ94QAK9KrF8hpdlSLTdlQptQ+pfKH2AyC1zyyrSxoxh56inKMgETXLVJ WmEYc1OCLWktUU0qQkCzo6umR7TNVZjx86L6jKKlFTptulQyFVaeaOU8DgFuiqnDXWn5E1cu bp1KCVU9TpBqIC7hN4ifJeY1lVoRGR+x+PCDeYgT+birtIanK5mqKzx74pbLeTTQZALqD8Hr UouxO41NyOGMv4BgZrr9z2ATIEIHZDLihKMEbF6LMjC6oCOgf51x6gV8doVw4ySNc3i/mk6h ztLQBhEc1q5SWfNgXcPDnzzoYsVJMSmV6yMCw3OEwtfZ0BYPQ6hihShOBKVQEHVnGbMIT4md V1S/axXDsLPFLkq1CxZ1L7N8Y5n2lUH00iSo+yWs7lhvt9WKl7sTewjXtibgrAC89uKvIyHe bA0GO9wojlwGAQboCMxZkkADgpHkDNvY675QLWaEs7+JJq4iWwojFh6QrDuOZmR3e9a5xZDb nQScTuPWvhZUtq7ZNZxmjPDdSUomSHe/ahBd+FHNKbgumN3PuXitFTg7LozIZV05oVv9arMs 9KyKGOS6rbim5C3Nx+MitMdr2Vh3tRfHugy0iWFSX5ITkG/RH4fMOzJMgMz/SftZMwMOU4J/ RBiPRyeaMBMqwomg/7o+NjulWHrUlAX+DAsyVtV+K9OLkglikNQtkG0MuQINBFprf6sBEADO ANf22so7uoGcvok2TM/T8BHI5+TqHEc4hVe+JGGJ1ZnWlgtGmpOs0fOQj3WAgGI0ZmTqMuoz KF/K9ljbjaMXsLD+JIBTD4rINy60VX2zHhmWhNaOcJvq+wbuHx0tMbhqsTStGnSkvRhH61nc MqVqlTTTLVQQSxKl9D2l7ZGwEPLHRFlydTOOix+F+Y1ehxYLVaPkaycs8wvgjYsDLo3T8Tmu OL+rcEfvxJ6lT2V5I51xqievqoBazAfXvA8FW/0G+Z9LUJmViOVluWg3xjP8okKYgOkOeX00 vMBCVaiEA08oaxY0ebS7uBEgppjWSwn+WAhB+6spd67d4W+DmAnM262lxFMhVYhXpfeV9zyg ULQOofdE6xtFkaxr/y8xQ4Bf7zX8ko6X9aFQFB/vc+zUtjzjg4VaQvWrThjaHlbEKR55MDxJ u2T9S7g5bR4zxZNv36gwlIdmx33a1AeR1nGcWa/7OtoS53+lUwyFVWLOnucqKh71Y38AAMd5 L9Fsb+ArQem71knEUTC+HvBGkPb2Y2PzGnnzhZyC8zgE8AjVD0wB+RMDNI3+fIW6biKAHDqr S6ZCVkzJ1R9nOjXMHRYZ5qlG+rCOeu6Jp4yNwp46z4PqiiLJ9NtmdNttLCEn5PDVF3g9g811 JcadvFVH1ZELoDGWMg3Q+QOHQBFYj7cj2QARAQABiQI8BBgBCgAmAhsMFiEEqPe6UEHhMzOc uhaWdtWAk/VAq80FAlxe8S0FCQLgv4EACgkQdtWAk/VAq81y0Q//YAbM50BQSjWAyVk52AMv PDEalSOau9jd1zllgurs1+BD4eWkNOJtN9BKaR5pd5VsjYPhL6ypUv/lRsyDPM3h7knuCZmI XYSBkloDqutZ020kR1jNhIZLL3Nma1xqh+oMsB7M+1AcT+Nlez74WgYjNyb7uuFWr99pUBuJ KbVjBq772Jjz3U0q7sa35wQvLaMC+AG/8L+e/fos6jgJhvIq2QeVLuVyeKVlEEO6tpe07Q+e mdCNu4l5eNbWAuvOgvtbVCLz5C4nLNZzr27u0thXzVhZ8ovAzdQTx6d6X4YI3fwBhPxXpqkg eTo3+B8D5lKUgkSZ9xTo+JP9yjCWwytezTL3oElL5LBVlYxdnpDWwOx8rCIgi5OlfVwU03/P O7IrEkqU5jvhZYT4c5/ktCx99kJcYwPUbT4wuhI1JnXyILrtmfC9a0vK8hpIUP4HU9GJTnVH Bl/XZZ6OJWEmSlJWm8KbageOcfLewc3BdWFnY6k5TmrzsFbamaBOgZmJgkNgylEyTjxVnLTk i5wciAp+N9K+tOP5FvgR943RSESZrxHo+XAV+BAK6K6Oae/jlrzGzNLAyJKWjefhyGL3AOru EBhSQaRemp/IUmyKREfowz3f8Lw2NFyJuWGzDgo9/1fmqmZ1JegtfWw0uPHB/rooajODBaol obxU65Qt1SOk2Ws=
Organization: Tor Project
Message-ID: <4c853f4c-2103-527a-e1c2-b5b8c92884cd@torproject.org>
Date: Mon, 8 Jul 2019 16:00:45 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.7.2
MIME-Version: 1.0
In-Reply-To: <CABcZeBOJvzPdPy49_8aQ6w5GiJF2fqFbUbSGGLnQokj4Bo5_XA@mail.gmail.com>
Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="m4X1ZkShPljuPIBtc8LYaOYWhTy17Swa5"
Archived-At: <https://mailarchive.ietf.org/arch/msg/pearg/la_1RsS9aJItDOvrBmRWh0-Dgfs>
Subject: Re: [Pearg] I-D Action: draft-irtf-pearg-safe-internet-measurement-00.txt
X-BeenThere: pearg@irtf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Privacy Enhancements and Assessment Proposed RG <pearg.irtf.org>
List-Unsubscribe: <https://www.irtf.org/mailman/options/pearg>, <mailto:pearg-request@irtf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/pearg/>
List-Post: <mailto:pearg@irtf.org>
List-Help: <mailto:pearg-request@irtf.org?subject=help>
List-Subscribe: <https://www.irtf.org/mailman/listinfo/pearg>, <mailto:pearg-request@irtf.org?subject=subscribe>
X-List-Received-Date: Mon, 08 Jul 2019 15:00:59 -0000

Hi Eric,

On 08/07/2019 15:15, Eric Rescorla wrote:
> This seems like the kind of product question that is well out of scope
> for PEARG. Software vendors have a wide variety of processes for
> determining whether a given piece of code is suitable for shipping to
> their users, ranging (at least) from "some developer thought it was
> good" to "multiple detailed code reviews".

The point is not so much to define what make code suitable for shipping,
but to say that you should not use A/B testing as an excuse to ship code
that you would not have otherwise shipped.

Your criteria for shipping to some users should be the same as the
criteria for shipping to all users. The code quality should be the same,
it should have undergone the same pre-shipping testing, etc. The only
difference should be the feature that is enabled/disabled.

This is also needed to make the experiment into good science. If you've
got other variables like relaxed review processes then you've introduced
other factors that don't make it a fair comparison between the A/B groups.

This all comes back to implied consent, and when it is appropriate to
presume that users will be expecting something. If you typically have
stringent review processes, users will expect this from future updates.
You violate the user's expectations, and also their trust, when you push
a shoddy update to them. This is true even if you've only pushed it to a
tiny fraction of your users.

Thanks,
Iain.