Re: [Pearg] Call for adoption: draft-learmonth-pearg-safe-internet-measurement-02.txt

[Iain Learmonth <[e-mail]>]

Hi Eric,

On 27/05/2019 14:34, Eric Rescorla wrote:
> I have reviewed this document and while I think some of the advice
> here is potentially useful, I don't think the recommendations really
> match what's current practice or what's practical. As such, I don't
> think it should be adopted without quite a bit more work.

In my opinion, a lot of current practice is not safe. This document does
not aim to set out current practice. It aims to raise the bar on user
safety when it comes to performing Internet measurement.

> As background, here is a partial list of the kinds of studies that
> are commonly done:

Thanks for these examples. I see no reason why each of these could not
be done safely. I have refrained so far from relating to specific
examples to try to keep the guidelines general but an "applied" section
may be useful to readers to better understand how to use the guidelines.

> I don't really want to get into a long debate about whether any
> particular study type is appropriate. Rather, these are common study
> types and so if the advice in this document is to be useful, then it
> needs to reasonably match what people do -- or at least have a much
> stronger argument that people should change what they do than is
> offered here.

Not necessarily. If it turns out that upon analysis, a lot of studies
are dangerous for users, this document should not weaken its guidelines
to allow those studies to continue. That would be silly.

You can do more dangerous things as long as either you have the consent
of the users, or you believe that what you are doing would be expected
by the users anyway. Neither of these things removes risk, but ensures
that the user is taking an informed risk.

> S 2. CONSENT
> The text in this draft leans pretty heavily on getting consent,
> either direct consent (including all users of the shared network)
> or "proxy consent".

This version contains a first draft of this section. Perhaps the
"implied consent" portion could do with more work.

>     The experiment uses an online advertisement campaign to deliver
>     the test code to end systems. When the end system is passed an ad
>     that is carrying the experiment the system runs embedded Adobe
>     Flash code. The code is executed when the ad is passed to the
>     user, and does not rely on a user "click" or any other user
>     trigger action. The active code interrogates one of two experiment
>     controllers by performing a URL fetch. The contents of the fetched
>     experiment control URL are a dynamically generated sequence of
>     four URLs. These four URLs are the substance of the test setup.

This is great until you run on a user machine in a country which has
some censorship/monitoring infrastructure in place that has
misinterpreted the URL as some proscribed content, landing the user in
trouble.

At the very least you may have generated costs for the user's bandwidth
there, which are uncompensated.

> It's worth noting at this point that the Web is a platform for running
> remote code, and by browsing you're opting into that, and ad studies
> just leverage that behavior.

Tell that to the 1,543,235 users that have installed NoScript from
Firefox Add-ons. Now you could say that they have opted out and the code
won't be run, but the way you've phrased this makes me think that
actually you just haven't understood the wider range of Internet users
that exist.

> As another example, Mozilla's Shield Studies system
> (https://wiki.mozilla.org/Firefox/Shield/Shield_Studies),is generally
> opt-out, with a specific opt-in for when the study collects
> more sensitive data:

Exactly, when a study would collect more data than a user might expect
would be collected, you ask for informed consent. When a study is
looking at non-personal information then it is OK to go with implied
consent.

> There seem to be two core issues here:
> - The effects of various changes on the user or the network they are
>   on.
> - The data collection inherent in doing the study.

This is a good split that I had not considered before.

> WRT to the first point, as a general matter, modern browsers
> auto-update, so the user has generally opted into regularly getting
> whatever new code the vendor thinks makes the best browser.

Say a mobile phone vendor wanted to test out how its camera was doing.
It ships you an auto-update that sends back every photo you've taken to
work out things like light levels, noise, and what might need tuned.
This is for the purpose of improving the camera.

> Similarly, on the topic of data collection, browsers report back quite
> a bit of technical data about their behavior. In both Chrome and
> Firefox, this is on by default, though you can turn it off. As
> suggested by the quote above on Shield studies, many studies just
> gather this kind of data (and often data that the browser would
> already report back) and Mozilla, at least, has a pretty
> well-developed framework for determining what kind of data requires
> what level of consent
> (https://wiki.mozilla.org/Firefox/Data_Collection).

The question is not about whether or not you can report back data. The
question is about what safeguards you can take to protect your users
from harm. You can collect data and report it back safely. We have
techniques for doing this in ways that respect users. The only reason to
not use them is if you don't actually care about protecting your users.

This draft is still in its early stages and there's a lot to flesh out
still. This feedback is useful and I'll be incorporating some of the
topics, but I feel that you've maybe missed the point of it.

Thanks,
Iain.