Re: [Pearg] More comments on draft-irtf-pearg-safe-internet-measurement-08

[Mallory Knodel <[e-mail]>]

Hi Marwan,

Top post on terminology. IRTF documents cannot use rfc2119 terms with
precision, because this is not a standards track doc.

And, the use of attacker is the appropriate and precise term for designing
systems with a threat model, which is what this document does. I’ll look
for ways to bring the tone down a notch, though this will add more words to
a doc that was meant to be lightweight. Worth it to get it right, however.

Many thanks for the ongoing discussion and offer to bring in more reviews.
I haven’t got a new version out with these changes but I will do soon.

Let’s keep talking,
-Mallory


On Tue, 7 Nov 2023 at 09:01, Marwan Fayed <marwan=
[e-mail]> wrote:

> Thanks Mallory and Tara. I shall keep my promise and share with the wider
> measurement community at SIGCOMM and others.
>
> In the meantime, a constructive thought: Perhaps it would be useful to
> reconstruct or reframe the contents of this draft using IETF terminology?
> (For non-IETFers, last I knew, the vocabulary is
> https://datatracker.ietf.org/doc/rfc2119 .) Measurement practice has many
> subtleties, some very clear do/do-nots. It occurs to me that many of the
> subtleties in a draft like this would pop out when presented as MUST or
> SHOULD NOT statements (if they can be, at all). It may help to read public
> statements from open research, in which measurement studies are now always
> accompanied by documentation explaining choices, and more. For example, see
> 'Ethics' in https://conferences.sigcomm.org/imc/2023/cfp/
>
> The suggestion to user IETF terminology is motivated by a simple question,
> the likes of which IETF exists to answer: What does "interop" look like for
> measurement practice?
>
> Setting details and subtleties aside, I must reiterate that the tone in
> this draft placing measurement on par with attacks is likely to present as
> problematic for people who know this domain well, many whose established
> expertise is measurement practice. It also could be misleading for
> non-experts who are reviewing measurement plans or practice -- in other
> words, the risks may outweigh the benefits in current form.
>
> It has taken the measurement community years to get to where it is now;
> even so our knowledge and practice continues to evolve. In this wider
> context this draft is no small endeavour, and the effort is commendable!
> (Thank you, Contributors!) No matter the outcome of this draft, it brings
> visibility to an important subject.
>
> Best,
> --marwan
>
>
>
>
> On Sun, 5 Nov 2023 at 10:27, Tara Tarakiyee <[e-mail]> wrote:
>
>> Dear Marwan,
>>
>> Thank you for sharing your review. I've read the draft with the
>> perspective of someone who was a measurement volunteer and who organised
>> measurement volunteers in contexts very similar to the example under 2.1
>> and I can say I didn't find the positioning of measurement in this document
>> to be bad or biased, considering that it's a safety considerations
>> document. In fact, an alternative framing that sugarcoats risks involved
>> with measurement would counter the goals of the document.
>>
>> That said, I agree with your point on 2.1 about taking into account the
>> power dynamic between the seeker of consent and the user. Furthermore, I
>> think 2.1 should be closely tied to 2.7, it's not enough that the user
>> considers risk to themselves, but the measurer should also take into
>> consideration the overall benefits vs risks involved to their measurement
>> participants even if they have informed consent. Another consideration to
>> add to informed consent is the capabilities of the user. A volunteer
>> working alone might be inclined to accept the risk because they want to
>> help, but their understanding of the risk could be improved by connecting
>> them with a local technologist, or exposure to the risk could be minimized
>> by connecting them to local legal resources.
>>
>> Best wishes,
>> Tara Tarakiyee
>>
>>
>>
>>
>> On Fri, Oct 27, 2023 at 7:01 PM Marwan Fayed <marwan=
>> [e-mail]> wrote:
>>
>>> Opening with thanks for the reminders to comment on this draft! Perhaps
>>> a most useful suggestion is to solicit feedback from beyond the IRTF/IETF
>>> community since so much of our best practice and guidance has been
>>> developed, and is tracked, outside of the IRTF/IETF. I'm happy to help in
>>> this regard.
>>>
>>> As a member of the measurement community, I can echo earlier sentiments
>>> that this draft is useful. I also see gaps, but have additional concerns
>>> about the way it positions measurement, overall, in ways that (at a
>>> minimum) could adversely bias institutional review or other forms of
>>> scrutiny by panels and people unfamiliar with the measurement domain.
>>>
>>> First, and most importantly, my reading of the draft's set-up in its
>>> current form is that it (unintentionally no doubt) presupposes that
>>> measurement is bad and harmful. Specifically, the draft explicitly equates
>>> many if not all forms of measurement each as being a type of attack. Some
>>> examples:
>>>
>>>  - [1.2] "Passive measurements use surveillance..." -- the Oxford
>>> definition of surveillance is "[noun] close observation, especially of
>>> a suspected spy or criminal."
>>>
>>>  - [1.3] encapsulates all measurements as being sources of attack. There
>>> are 12 explicit definitions, 9 of which start by "An attack".
>>>
>>> It's important to recognise that a draft like this would be beneficial
>>> to a wide audience. At the same time I am unable to see how an outside
>>> observer or reviewer, e.g. institutional review, using guidelines of this
>>> form (and likely have to consider liability along with safe and responsible
>>> practice), wouldn't automatically be forced into a starting position of
>>> defensiveness against measurement. (Aside: one additional suggestion may be
>>> to use 'responsible' alongside 'safe', which is a common convention in the
>>> research community.)
>>>
>>> The 'risk' in this current draft is that its current set-up reverses a
>>> lot of the incredibly hard and tireless efforts from the measurement
>>> community to define, account for, report, and review best practices, as
>>> well as call out violations.
>>>
>>> Second, there are a few subtleties missing from some 'Guidelines' that
>>> cannot be overlooked. Their omission could lull a reader into false senses
>>> of security that they've checked certain boxes. In particular,
>>>
>>>  - [2.1] suggests that informed consent is sufficient, when in fact we
>>> know that even fully informed consent in some circumstances may not be
>>> enough. For example, the value and merit of consent to statements like "by
>>> helping this measurement you could go to jail" is questionable, at best.
>>> Alongside, the seeker of consent has to be taken into consideration; for
>>> example, there is no way to know the effect of a consent-seeker on a
>>> consent-giver who may be more or less inclined to trust a multinational
>>> corporation, foreign government, civil liberty group, or a university lab
>>> -- these are just not the same, and an individual may trust more or less
>>> depending.
>>>
>>>  - [2.3] This is a good one! If it's helpful, even the seemingly
>>> obviously 'ok' can have caveats. Speedtests are a great example. It might
>>> be reasonable to assume that any speedtest server is happy to receive a
>>> test request, and any subscriber who pays for access is entitled to request
>>> the test -- but what about the transit provider in-between?
>>>
>>>  - [2.4] Also great, and easy to overlook. Here, too, however, I wonder
>>> if all requests are equal? As an example, what does it mean for Google to
>>> request do-not-scan? (I honestly don't know!) One possible addition may be
>>> to suggest that "self-identify" is good practice, especially when releasing
>>> measurement tools into the world. For example, the popular zmap tool self
>>> identifies itself by placing "54321" in the IP-ID field.
>>>
>>> One last suggestion: It's hard to know which are the most useful
>>> references in this space, so a draft like this would be a great place to
>>> have a comprehensive list of pointers to best and current practice in this
>>> space (e.g. the CACM written by Partridge and Allman), and docs that might
>>> include case studies. Even better might be to have one or two sentences for
>>> each to describe why the reference is useful, or what it offers.
>>> (Acknowledging in advance that distilling down from multiple inputs is a
>>> challenge in itself, and would be a great feat!)
>>>
>>> Hope some of this helps.
>>>
>>> --marwan
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Pearg mailing list
>>> [e-mail]
>>> https://mailman.irtf.org/mailman/listinfo/pearg
>>>
>> --
>> Pearg mailing list
>> [e-mail]
>> https://mailman.irtf.org/mailman/listinfo/pearg
>>
> --
> Pearg mailing list
> [e-mail]
> https://mailman.irtf.org/mailman/listinfo/pearg
>