Re: [Pearg] I-D Action: draft-irtf-pearg-safe-internet-measurement-00.txt

[Iain Learmonth <[e-mail]>]

Hi Eric,

On 08/07/2019 15:15, Eric Rescorla wrote:
> This seems like the kind of product question that is well out of scope
> for PEARG. Software vendors have a wide variety of processes for
> determining whether a given piece of code is suitable for shipping to
> their users, ranging (at least) from "some developer thought it was
> good" to "multiple detailed code reviews".

The point is not so much to define what make code suitable for shipping,
but to say that you should not use A/B testing as an excuse to ship code
that you would not have otherwise shipped.

Your criteria for shipping to some users should be the same as the
criteria for shipping to all users. The code quality should be the same,
it should have undergone the same pre-shipping testing, etc. The only
difference should be the feature that is enabled/disabled.

This is also needed to make the experiment into good science. If you've
got other variables like relaxed review processes then you've introduced
other factors that don't make it a fair comparison between the A/B groups.

This all comes back to implied consent, and when it is appropriate to
presume that users will be expecting something. If you typically have
stringent review processes, users will expect this from future updates.
You violate the user's expectations, and also their trust, when you push
a shoddy update to them. This is true even if you've only pushed it to a
tiny fraction of your users.

Thanks,
Iain.