Re: [Pearg] I-D Action: draft-irtf-pearg-safe-internet-measurement-00.txt

[Iain Learmonth <[e-mail]>]

Hi Eric,

On 08/07/2019 14:30, Eric Rescorla wrote:
> After a quick look, I see....
> 
> Document: draft-irtf-pearg-safe-internet-measurement-01.txt
> 
>    The reduced impact should not be used as an excuse for pushing higher
>    risk updates, only updates that could be considered appropriate to
>    push to all users should be A/B tested.
> 
> This may just be wordsmithing, but as written, this text is entirely
> unrealistic. One of the major reasons that one does A/B testing is
> that you are concerned about risk in the Treatment group (e.g., that
> there will be a higher risk of failures, crashes, etc.) and the
> reason you are doing an A/B test is to mitigate that risk.

The point that I'd like to put across here is that it is not an excuse
to be reckless or careless. A/B testing can mitigate risk to reputation
perhaps, and sure it can reduce the risk that any individual user is
affected by a bad update, but it doesn't mitigate the impact for the
users that are affected.

> I'm not sure how to phrase this (internally, we usually talk about
> the code having been appropriately reviewed and landed according
> to policy), but the current text doesn't work.
How about:

  The reduced impact should not be used as an excuse for pushing updates
without the same careful consideration and review that would be applied
to any software update pushed to all users.

and perhaps also adding:

  In cases which may have high impact for any individual user,
simulations or emulations should be carried out in a testing environment
before pushing the updates to users via automated channels.

Thanks,
Iain.