Re: [Pearg] I-D Action: draft-irtf-pearg-safe-internet-measurement-00.txt

[Eric Rescorla <[e-mail]>]

On Mon, Jul 8, 2019 at 6:47 AM Iain Learmonth <[e-mail]> wrote:

> Hi Eric,
>
> On 08/07/2019 14:30, Eric Rescorla wrote:
> > After a quick look, I see....
> >
> > Document: draft-irtf-pearg-safe-internet-measurement-01.txt
> >
> >    The reduced impact should not be used as an excuse for pushing higher
> >    risk updates, only updates that could be considered appropriate to
> >    push to all users should be A/B tested.
> >
> > This may just be wordsmithing, but as written, this text is entirely
> > unrealistic. One of the major reasons that one does A/B testing is
> > that you are concerned about risk in the Treatment group (e.g., that
> > there will be a higher risk of failures, crashes, etc.) and the
> > reason you are doing an A/B test is to mitigate that risk.
>
> The point that I'd like to put across here is that it is not an excuse
> to be reckless or careless. A/B testing can mitigate risk to reputation
> perhaps, and sure it can reduce the risk that any individual user is
> affected by a bad update, but it doesn't mitigate the impact for the
> users that are affected.
>

This seems like the kind of product question that is well out of scope for
PEARG. Software vendors have a wide variety of processes for determining
whether a given piece of code is suitable for shipping to their users,
ranging (at least) from "some developer thought it was good" to "multiple
detailed code reviews". I think we can all agree that defining that is out
of scope, but without that, defining what you have to do before you ship to
a fraction of your users is largely irrelevant.

-Ekr