Re: [Pearg] I-D Action: draft-irtf-pearg-safe-internet-measurement-00.txt

Niels ten Oever <lists@digitaldissidents.org> Mon, 08 July 2019 14:41 UTC

Return-Path: <lists@digitaldissidents.org>
X-Original-To: pearg@ietfa.amsl.com
Delivered-To: pearg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0EA3D12018A for <pearg@ietfa.amsl.com>; Mon, 8 Jul 2019 07:41:58 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.9
X-Spam-Level:
X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id njbrYRYEqM6f for <pearg@ietfa.amsl.com>; Mon, 8 Jul 2019 07:41:55 -0700 (PDT)
Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 700B2120199 for <pearg@irtf.org>; Mon, 8 Jul 2019 07:41:54 -0700 (PDT)
Received: from smtp.greenhost.nl ([213.108.110.112]) by smarthost1.greenhost.nl with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from <lists@digitaldissidents.org>) id 1hkUpV-0006bE-CE for pearg@irtf.org; Mon, 08 Jul 2019 16:41:51 +0200
To: pearg@irtf.org
References: <156254420044.4995.7471139515518776754@ietfa.amsl.com> <240d826f-1d7a-834a-919a-f4d5aa9fed58@torproject.org> <CABcZeBMUyXVyAQZkzHc+uCD8AS-_apihjop9QwQxkFOGz4KrZg@mail.gmail.com> <279a7516-a08d-12a6-1693-b49c94c3c2e5@torproject.org> <CABcZeBOJvzPdPy49_8aQ6w5GiJF2fqFbUbSGGLnQokj4Bo5_XA@mail.gmail.com>
From: Niels ten Oever <lists@digitaldissidents.org>
Openpgp: preference=signencrypt
Autocrypt: addr=lists@digitaldissidents.org; prefer-encrypt=mutual; keydata= mQINBFgpcR0BEACnfvNwTMlN+pyZT0AFYhWqxG3N4AoPIeNfbxLQH7dk8ZL7Ls05xtORfnu9 ovoaRrZpDufkMviUFidNYePbQNdgf63vWVgwpQR7utluwWraetcmZOu6tayJuyBK2b6d2Z23 MJAQxfa2/GMlN3QkvobaoyKtgbc8rOCgNla7WwkgtiVJ89xbAUHXPFpKWZluVRjaFh4p5C5r 7E5OvUiEGLQ5Cn2ir2PGIyIVqjB+hLTyaI6dIGCz2jtL0RATjmsmYUX7UkU/pz8MPPC2BJ5P KU9pdXMRBhAStxcph8vCo2ze9xSi3+1/5A2ULVtvO4s0hZ+exbTfMxMg3H5CCRFEEJXlQEXa Cd0ZHvqcv5xq8n9w/Ccd0CqYWATIwyP8Jlzd+BY3QGTWnWlgoAbs3Guh/pFYhEFNuuAF5Jk1 k5OlNGsRE/LQJmbT5SE7AtLJLbWewcHlEyIH+K6J8uVa4ExLXmRy+eRkFaxjGy3fLlUpy1Ee 1kU7VsQ/TZ8g8ujsMzxqsdB6y0TD/kVlWaDqPL6F+b+pm3lAuCBGWM1YZROTG58R6pD7sNVm i0ift4dIttAsg+2KoShm9A8kQ3tACXZDgNPC0l7VOqnVayjnF0RmjGeiX7PjOcLQCZ9a5wAH 5mrXMaKvfszqAVkP9HSrk1QVZOipF6vEimL43Czy7Rp1aUaUwwARAQABtC1OaWVscyB0ZW4g T2V2ZXIgPGxpc3RzQGRpZ2l0YWxkaXNzaWRlbnRzLm9yZz6JAj8EEwEIACkFAlgvB3YCGyMF CQlmAYAHCwkIBwMCAQYVCAIJCgsEFgIDAQIeAQIXgAAKCRAO2D86RorIs56yD/44BSJvKnjH ex0nhPDI9nIJlzlnypa4qsniy0obG5GRbVRikT1E1xaz7VBoPs39hCywoIWd6p0hs1PG1Tcj WV0GwNKRt90PPEh6iNJSGjV2Aq3IlME/aUViD9008yfbRSqfsnPXLW1kpCoZNaOSNzpURoM9 OkVU/z4LSLD61SfFFByBne/GkJKt96/fcspBif1GPC//63ZKFrDqQ9JFR6dECAmsKv7baayz MTv3wrTcqpuHcqJIv4vTm8IPx1QiGgEvrMwsPZz/vx8bMdxxxHWgCcbrt+0b3tRzq9ATZwG3 xDiwnJgKd+ioZOC/b5sY69721sqwBmWYyXWVqtqt01xIgNZjr/wixam+l1bTGUgj0rwPWJWx +7Whe25ff+mNNW/UQeCBjZlxoxAJWSr1Pp3n+SQKQ4TLs8wIwHZtcVCffepfHd47CEbnR8Kc Tjm3tlKzSWq4zcUy6BaxHfgn9+HaAM7fwLqx9/WAtSfdmLXJTN+Swy0w/slakD75jl2o7U3e ETjoYQWt+306X2Uly/0ge7VEQ4ySmmbru6U5ainGE95gjsc++s+hvKmMuGYL3h4ijE1RSe/k wgM6/Z1B6JosssdX+KRuuk2A4FHGbcee8LUIJ3C36qyI7s6PJBXi6SjIPN0wpx30P/DUf/Lr o5lmHF03qQ5eeqI8lDwIobWlJbkCDQRYKXEdARAAxYOE3/AFmEfQ0SVVFujYFhZKX+BGXolY ytC2a1soZogVYTIIlypxkRtN+ljteFAY3xX/El7cx5Fxj+uXvLKAm9xQRI/DCug7/NGULMk9 bDK5bzSGw817cyiL5Kb+0RkWj2Y5ArOAK6XPGBZWZTHwyIawsSCN9AhDXZQWVRqkR1QXcq3I YKl+OHWMO7+1VfixCSakNf7T/Kiq46rQEPW8Eghk6CVOBR8xUCBbyk5aRW4VSGO6pUD3H21u r+5fTLsVyan1NHhxNNiXfnEJKr+JI5dXSkj7WqA5n8ITaNdFSAttkdT56wAQpxE2h8zaOmBa FUWQ4D8SdXDVymP5QMtLG+ItMMiNV6kXgsRFugAKM5yZtPP9gIX+ic8QO5iuct37bRXJU/rm rH54Ab0kyAeeRE7oSsfTZPKvgtUh7VLAUEw/wy6TORJHE8JMaX0yYT6h4PGRS3mNM4bka8hj dfcrexI0zSqFOl2I22zQlG3YqSzIvVh98W67hxfAIaCVaTfJLFPEru3drxNwi6ogdkRmcLGK qqTgeYItrvITyFvzqbrcO2exp0KKEK3cDIZypqHHUf4+uPlDtuExehLsNOMpjP8qhZpFtyLe DS07qunbvstcyvR30wOJ3DyAbHGzq739UyDcO9Jt5jwODyVwk3MK5Em4pJ0+IAJx+F6gta0B k2MAEQEAAYkCJQQYAQgADwUCWClxHQIbDAUJCWYBgAAKCRAO2D86RorIs0ykD/4t151SZG9M beKRVKbs9Ecjady9bO0L3oBos4rhqY12ha8smFlsUzvbgB4CtkBuXQlq+plOBWv+rFEThOzy 3bezgEDjlxycoO1W2wJD6E7Fo9fkHT6UOm9fQBkuKRqK83OGnfM02qP1Ky8d7EoZz+nTSMf/ DJgWw1YRKrXkMHBwKD83lCENsmePWE5AjMqk8cojPv9Oy1wWy6fHjwx3r+wQSokBNfxgQyAF onmgBbhlic/pZUYRSIcldyUlaomrjFfr4egzmNE7aWDvLwOUYKevBIeJJcqTyfAn3TtJbPCE HOC2+lP6EcmPFyhQdiia+RqOClumqbWOPeQ2VM8j7NWvKKmBNBB5OJ/rmHogbNU+wWPJ723q MBoOp1jIwFNkQhx01W6v55VMwLr+IuBKY1ggJ2BhwQiGpWv4tMc5oB/qVh3my1VO65ErcJ3S 9blpwJdDj5/YDOU7BKEmpRUP+xkaryNzH2x7FzrOOHzJBX6jeYZabGvnTicQlBAzfGpblFqV 3YN6EhCF2AHmGLTZ/DrjGYToIsW8cXlEMqN4u8ODEUY0OhbnytnopKJKk99bwMoCqDkfQvT3 LKDWtZj9NzFndfuoKXsVpwAitrG0mau0/16DKDyVWdtJ9DYmtE40zO6g70VVxUj+dKt2hbJT y/KQTb7Ijhw7wZrGp/P7nhbVyA==
Message-ID: <62cbf8ac-8ed2-913b-07b6-cafaf8250468@digitaldissidents.org>
Date: Mon, 8 Jul 2019 16:41:19 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.7.2
MIME-Version: 1.0
In-Reply-To: <CABcZeBOJvzPdPy49_8aQ6w5GiJF2fqFbUbSGGLnQokj4Bo5_XA@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-Authenticated-As-Hash: 29cc722430e8f1f6ed904119444c0d49b0f3ee91
X-Virus-Scanned: by clamav at smarthost1.samage.net
X-Scan-Signature: 4c3c8b3d32e7d0cfaf2d58264fc1daa3
Archived-At: <https://mailarchive.ietf.org/arch/msg/pearg/Qn0weHdD-VqRjYRlIWiiQ096ays>
Subject: Re: [Pearg] I-D Action: draft-irtf-pearg-safe-internet-measurement-00.txt
X-BeenThere: pearg@irtf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Privacy Enhancements and Assessment Proposed RG <pearg.irtf.org>
List-Unsubscribe: <https://www.irtf.org/mailman/options/pearg>, <mailto:pearg-request@irtf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/pearg/>
List-Post: <mailto:pearg@irtf.org>
List-Help: <mailto:pearg-request@irtf.org?subject=help>
List-Subscribe: <https://www.irtf.org/mailman/listinfo/pearg>, <mailto:pearg-request@irtf.org?subject=subscribe>
X-List-Received-Date: Mon, 08 Jul 2019 14:41:58 -0000


On 7/8/19 4:15 PM, Eric Rescorla wrote:
> 
> 
> On Mon, Jul 8, 2019 at 6:47 AM Iain Learmonth <irl@torproject.org <mailto:irl@torproject.org>> wrote:
> 
>     Hi Eric,
> 
>     On 08/07/2019 14:30, Eric Rescorla wrote:
>     > After a quick look, I see....
>     >
>     > Document: draft-irtf-pearg-safe-internet-measurement-01.txt
>     >
>     >    The reduced impact should not be used as an excuse for pushing higher
>     >    risk updates, only updates that could be considered appropriate to
>     >    push to all users should be A/B tested.
>     >
>     > This may just be wordsmithing, but as written, this text is entirely
>     > unrealistic. One of the major reasons that one does A/B testing is
>     > that you are concerned about risk in the Treatment group (e.g., that
>     > there will be a higher risk of failures, crashes, etc.) and the
>     > reason you are doing an A/B test is to mitigate that risk.
> 
>     The point that I'd like to put across here is that it is not an excuse
>     to be reckless or careless. A/B testing can mitigate risk to reputation
>     perhaps, and sure it can reduce the risk that any individual user is
>     affected by a bad update, but it doesn't mitigate the impact for the
>     users that are affected.
> 
> 
> This seems like the kind of product question that is well out of scope for PEARG. Software vendors have a wide variety of processes for determining whether a given piece of code is suitable for shipping to their users, ranging (at least) from "some developer thought it was good" to "multiple detailed code reviews". I think we can all agree that defining that is out of scope, but without that, 

Why could we not document best practices for this? Seems quite useful to me. 

> defining what you have to do before you ship to a fraction of your users is largely irrelevant.
> 

Why ?

Niels

> -Ekr
> 
> 

-- 
Niels ten Oever
Researcher and PhD Candidate
Datactive Research Group
University of Amsterdam

PGP fingerprint	   2458 0B70 5C4A FD8A 9488  
                   643A 0ED8 3F3A 468A C8B3