Re: [Pearg] I-D Action: draft-irtf-pearg-safe-internet-measurement-00.txt

Eric Rescorla <> Mon, 08 July 2019 14:16 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id C6F35120279 for <>; Mon, 8 Jul 2019 07:16:18 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -0.601
X-Spam-Status: No, score=-0.601 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, PDS_NO_HELO_DNS=1.295, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id ElncGIgvYhfg for <>; Mon, 8 Jul 2019 07:16:17 -0700 (PDT)
Received: from ( [IPv6:2a00:1450:4864:20::12a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 0B1EF120273 for <>; Mon, 8 Jul 2019 07:16:17 -0700 (PDT)
Received: by with SMTP id q26so11054986lfc.3 for <>; Mon, 08 Jul 2019 07:16:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=6NuEk1AjKGLGlP8OvQdmXQ5PUj0OKKNvmpfUCZso8zg=; b=u4Tz7MxaruI4cQ/0Nibi3G153OOkN3za5sq1P6CoBShM6ttZBhOQy0oKim/0+rPOLo wM2LkbkdsOJfF+2vNymK0y5vOHTylhkv3+N+Dl1uX+o79cmw9NsbamadbGUCrJrvz9sb muKaQ3ogc6stjOGpuImFk5CM5I2m/v0D37spU3ZUo3CXwp/gYieIaulbU9b9dWzx8y3J O5tYj57VbUKG2ZjTjeZ3yq8orvvUNkhG6vq2Z6UurtWlNZFyGjohptFJz456PPivD8so zdB183hR5JKX2lgz7iLCinHX3bEEVJrXbTaXPZm3d1Owv4/PywOjoBG6ieV84Z1Yh8Pw GipA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=6NuEk1AjKGLGlP8OvQdmXQ5PUj0OKKNvmpfUCZso8zg=; b=dnrSFzRFIMiW9g3kXdwIOnrllvHBiTTRc8bf4sEjHreWrPB3ECsYgHclmErFdRWQw/ V5DOCZ+Q01VMzsbmcjc3J9dGkGlMxO5dDb9x5sapEx7ha4mPlTil5AIex1txRSveladd GSsFhNBzKqKw5oBxVitZJRxD9pSfu78ugHktbg/YObEDziQO9wEPuXK5dc0dWxHBmW/p voLXr9F8YIkHkODnpXWOSZe70LstChj4JISFTMN78qtHFVcKN/jlWcDjzINqsasxAufj UYwcnC6kOcL5dVuUOsrvlTETzEptVTVDU1DRirmbL8YCRzWi+JuukFoQHgmXtS0QlAR7 +jEw==
X-Gm-Message-State: APjAAAWsdLjs6fu/OgATZOU01m01EonMlvabC0wZEhCvEqt7SdQc+Y/E 4k/uuV3SyFDxlKJOIKZGDM4jAMKfxg+HSW2wFmyNZw==
X-Google-Smtp-Source: APXvYqzEMTogcB+6YL+LkNNL/YZrSpGZBPNPejmWNh8uJ6591Bb6PbDpQoQddwCdM3hhiOVohBN4iF8iVQof2aMJC4c=
X-Received: by 2002:a19:f24e:: with SMTP id d14mr9315095lfk.184.1562595375170; Mon, 08 Jul 2019 07:16:15 -0700 (PDT)
MIME-Version: 1.0
References: <> <> <> <>
In-Reply-To: <>
From: Eric Rescorla <>
Date: Mon, 8 Jul 2019 07:15:39 -0700
Message-ID: <>
To: Iain Learmonth <>
Content-Type: multipart/alternative; boundary="00000000000043dc16058d2c1632"
Archived-At: <>
Subject: Re: [Pearg] I-D Action: draft-irtf-pearg-safe-internet-measurement-00.txt
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Privacy Enhancements and Assessment Proposed RG <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Mon, 08 Jul 2019 14:16:28 -0000

On Mon, Jul 8, 2019 at 6:47 AM Iain Learmonth <> wrote:

> Hi Eric,
> On 08/07/2019 14:30, Eric Rescorla wrote:
> > After a quick look, I see....
> >
> > Document: draft-irtf-pearg-safe-internet-measurement-01.txt
> >
> >    The reduced impact should not be used as an excuse for pushing higher
> >    risk updates, only updates that could be considered appropriate to
> >    push to all users should be A/B tested.
> >
> > This may just be wordsmithing, but as written, this text is entirely
> > unrealistic. One of the major reasons that one does A/B testing is
> > that you are concerned about risk in the Treatment group (e.g., that
> > there will be a higher risk of failures, crashes, etc.) and the
> > reason you are doing an A/B test is to mitigate that risk.
> The point that I'd like to put across here is that it is not an excuse
> to be reckless or careless. A/B testing can mitigate risk to reputation
> perhaps, and sure it can reduce the risk that any individual user is
> affected by a bad update, but it doesn't mitigate the impact for the
> users that are affected.

This seems like the kind of product question that is well out of scope for
PEARG. Software vendors have a wide variety of processes for determining
whether a given piece of code is suitable for shipping to their users,
ranging (at least) from "some developer thought it was good" to "multiple
detailed code reviews". I think we can all agree that defining that is out
of scope, but without that, defining what you have to do before you ship to
a fraction of your users is largely irrelevant.