Re: [Pearg] More comments on draft-irtf-pearg-safe-internet-measurement-08

[Mallory Knodel <[e-mail]>]

Thanks for the review Marwan,

Please share elsewhere outside IRTF for reviews just ask folks to send them
with the list on CC or in GitHub issues.

I have shared with the Tor research review board as per our last meeting.

I top posted to make clear that under no circumstances do the authors of
this draft consider measurement bad. In fact it is based on the Tor network
safe measurement guidelines— so, from the community of measurement and
internet research.

This isn’t about making hard trade offs, it’s accepting responsibility to
do the right thing— all researchers in every field do this— and documenting
what that right thing is.

Kind regards,
-Mallory

On Fri, 27 Oct 2023 at 13:01, Marwan Fayed <marwan=
[e-mail]> wrote:

> Opening with thanks for the reminders to comment on this draft! Perhaps a
> most useful suggestion is to solicit feedback from beyond the IRTF/IETF
> community since so much of our best practice and guidance has been
> developed, and is tracked, outside of the IRTF/IETF. I'm happy to help in
> this regard.
>
> As a member of the measurement community, I can echo earlier sentiments
> that this draft is useful. I also see gaps, but have additional concerns
> about the way it positions measurement, overall, in ways that (at a
> minimum) could adversely bias institutional review or other forms of
> scrutiny by panels and people unfamiliar with the measurement domain.
>
> First, and most importantly, my reading of the draft's set-up in its
> current form is that it (unintentionally no doubt) presupposes that
> measurement is bad and harmful. Specifically, the draft explicitly equates
> many if not all forms of measurement each as being a type of attack. Some
> examples:
>
>  - [1.2] "Passive measurements use surveillance..." -- the Oxford
> definition of surveillance is "[noun] close observation, especially of a
> suspected spy or criminal."
>
>  - [1.3] encapsulates all measurements as being sources of attack. There
> are 12 explicit definitions, 9 of which start by "An attack".
>
> It's important to recognise that a draft like this would be beneficial to
> a wide audience. At the same time I am unable to see how an outside
> observer or reviewer, e.g. institutional review, using guidelines of this
> form (and likely have to consider liability along with safe and responsible
> practice), wouldn't automatically be forced into a starting position of
> defensiveness against measurement. (Aside: one additional suggestion may be
> to use 'responsible' alongside 'safe', which is a common convention in the
> research community.)
>
> The 'risk' in this current draft is that its current set-up reverses a lot
> of the incredibly hard and tireless efforts from the measurement community
> to define, account for, report, and review best practices, as well as call
> out violations.
>
> Second, there are a few subtleties missing from some 'Guidelines' that
> cannot be overlooked. Their omission could lull a reader into false senses
> of security that they've checked certain boxes. In particular,
>
>  - [2.1] suggests that informed consent is sufficient, when in fact we
> know that even fully informed consent in some circumstances may not be
> enough. For example, the value and merit of consent to statements like "by
> helping this measurement you could go to jail" is questionable, at best.
> Alongside, the seeker of consent has to be taken into consideration; for
> example, there is no way to know the effect of a consent-seeker on a
> consent-giver who may be more or less inclined to trust a multinational
> corporation, foreign government, civil liberty group, or a university lab
> -- these are just not the same, and an individual may trust more or less
> depending.
>
>  - [2.3] This is a good one! If it's helpful, even the seemingly obviously
> 'ok' can have caveats. Speedtests are a great example. It might be
> reasonable to assume that any speedtest server is happy to receive a test
> request, and any subscriber who pays for access is entitled to request the
> test -- but what about the transit provider in-between?
>
>  - [2.4] Also great, and easy to overlook. Here, too, however, I wonder if
> all requests are equal? As an example, what does it mean for Google to
> request do-not-scan? (I honestly don't know!) One possible addition may be
> to suggest that "self-identify" is good practice, especially when releasing
> measurement tools into the world. For example, the popular zmap tool self
> identifies itself by placing "54321" in the IP-ID field.
>
> One last suggestion: It's hard to know which are the most useful
> references in this space, so a draft like this would be a great place to
> have a comprehensive list of pointers to best and current practice in this
> space (e.g. the CACM written by Partridge and Allman), and docs that might
> include case studies. Even better might be to have one or two sentences for
> each to describe why the reference is useful, or what it offers.
> (Acknowledging in advance that distilling down from multiple inputs is a
> challenge in itself, and would be a great feat!)
>
> Hope some of this helps.
>
> --marwan
>
>
>
>
>
> --
> Pearg mailing list
> [e-mail]
> https://mailman.irtf.org/mailman/listinfo/pearg
>