Re: [Pearg] I-D Action: draft-irtf-pearg-safe-internet-measurement-00.txt

On Mon, Jul 8, 2019 at 7:42 AM Niels ten Oever <lists@digitaldissidents.org>
wrote:

>
>
> On 7/8/19 4:15 PM, Eric Rescorla wrote:
> >
> >
> > On Mon, Jul 8, 2019 at 6:47 AM Iain Learmonth <irl@torproject.org
> <mailto:irl@torproject.org>> wrote:
> >
> >     Hi Eric,
> >
> >     On 08/07/2019 14:30, Eric Rescorla wrote:
> >     > After a quick look, I see....
> >     >
> >     > Document: draft-irtf-pearg-safe-internet-measurement-01.txt
> >     >
> >     >    The reduced impact should not be used as an excuse for pushing
> higher
> >     >    risk updates, only updates that could be considered appropriate
> to
> >     >    push to all users should be A/B tested.
> >     >
> >     > This may just be wordsmithing, but as written, this text is
> entirely
> >     > unrealistic. One of the major reasons that one does A/B testing is
> >     > that you are concerned about risk in the Treatment group (e.g.,
> that
> >     > there will be a higher risk of failures, crashes, etc.) and the
> >     > reason you are doing an A/B test is to mitigate that risk.
> >
> >     The point that I'd like to put across here is that it is not an
> excuse
> >     to be reckless or careless. A/B testing can mitigate risk to
> reputation
> >     perhaps, and sure it can reduce the risk that any individual user is
> >     affected by a bad update, but it doesn't mitigate the impact for the
> >     users that are affected.
> >
> >
> > This seems like the kind of product question that is well out of scope
> for PEARG. Software vendors have a wide variety of processes for
> determining whether a given piece of code is suitable for shipping to their
> users, ranging (at least) from "some developer thought it was good" to
> "multiple detailed code reviews". I think we can all agree that defining
> that is out of scope, but without that,
>
> Why could we not document best practices for this? Seems quite useful to
> me.
>

The question isn't useful, but rather what this group is chartered to do.
Do you have some charter text that would support such an activity?

-Ekr

> defining what you have to do before you ship to a fraction of your users
> is largely irrelevant.
> >
>
> Why ?
>
> Niels
>
> > -Ekr
> >
> >
>
> --
> Niels ten Oever
> Researcher and PhD Candidate
> Datactive Research Group
> University of Amsterdam
>
> PGP fingerprint    2458 0B70 5C4A FD8A 9488
>                    643A 0ED8 3F3A 468A C8B3
>
> --
> Pearg mailing list
> Pearg@irtf.org
> https://www.irtf.org/mailman/listinfo/pearg
>