Re: [Pearg] More comments on draft-irtf-pearg-safe-internet-measurement-08

[Tobias Fiebig <[e-mail]>]

Moin,
I just was stumbled over this draft and just gave it a read. At the
moment, i think that the draft has crucial limitations that may cause
significant harm. In general, my core comments are relatively aligned
with Marwan's (very spot-on comments). However, I also believe that
there are some more fundamental issues that must be resolved.

# Authoritativeness of documents
Marwan noted that the document in its current form may cause harm if it
is read by a non-domain expert (say: Medicine Professor) when
evaluating a request for ethical clearance of a network measurement
study. And yes, this _does_ happen.

Reading the reply by Mallory creates the impression in me that there is
some form of assumption that what is written in a document will not be
read as authoritative if the authors only (outside the document or
somewhere in another part of the document the concerned medicine prof
just didn't scroll over) phrase things more in terms of their
intentions.

This is not really the case; For the domain where the document will be
relevant and useful, those nuances will be lost. It is already hard to
have security researchers understand the difference between a draft and
and RFC, let alone all the other nuances there after. I would hence
caution significant care with how the document can be perceived,
especially in context for the point below.

# Positionality / Perspective

The draft has a strong perspective from the privacy measurement angle,
also highlighted by the statement that some points "[are] based on the
Tor network safe measurement guidelines— so, from the community of
measurement and internet research."; The Tor community is a very
specific (and small) subset of network measurement work, and I would
argue that they are more rooted in the PETS community (also looking at
the members of the board).

This community, in terms of type of measurements and also in terms of
volume, is very different from the network measurement community (ACM
SIGCOMM-Bobble), which _again_ is different from the Network Security
bobble (IEEE S&P/NDSS/USENIX SEC/ACM CCS).

These communities have very different requirements and perspectives on
network measurements (and potential harm they may cause).

Hence, i'd argue that the draft needs a lot more engagement with these
different communities, and should include the experiences of
researchers doing that kind of work as well.

# Superficial Nature of Sections 2.4ff
>From Section 2.4 the draft gets extremely thin and ignores practical
requirements of network measurements by taking a strong data
minimization perspective (which is imperative for censorship
measurements, but often _harmful_ for large scale measurements; If you
do a large scale measurement and stripped a bit too much, redoing the
measurement might be unnecessary harm; The same as having done an
effectively useless measurement.).

Similarly, the--somewhat simon-says-y--headlines thereafter just keep
introducing shorter and shorter sections that hand wave away _critical_
aspects that necessitate _very_ careful consideration and
instruction... and--as said before--sometimes these imperative things
may not even make sense.

2.6 and 2.7 are then close to dangerously short. A document claiming to
provide guidelines should provide, for these critical mechanics, actual
guidelines on how to, e.g., execute a harm benefit analysis. With the
current form, the document essentially provides a blanket for
researchers being like "ah, well, thought about it, best we could...
uhm... ups, failed after following best practices, nothing we could
have done". See also my paper from anrw at 117.

# Summary
In summary, as those speaking earlier, I am deeply convinced that the
topic of the draft is direly needed; However, such a document should
take a lot of context (communities etc.) into account which is not in -
08 (and i would argue what is missing is a multiple of what is already
there). It must also provide guidelines on _how_ to do some of the
things suggested, instead of just throwing in words, hoping that it is
feasible.

Otherwise, this document might cause harm by creating unnecessary red-
tape, which will result in losing a lot of the progress that was made
in researchers' consideration of ethical concerns and adherence to
ethical principles; There will be frustration and 'trying to get
around'.

With best regards,
Tobias

On Tue, 2023-11-07 at 13:56 +0100, Mallory Knodel wrote:
> On Tue, 7 Nov 2023 at 13:45, Eric Rescorla <[e-mail]> wrote:
> 
> > 
> > 
> > On Tue, Nov 7, 2023 at 12:17 AM Mallory Knodel <mknodel=
> > [e-mail]> wrote:
> > 
> > > Hi Marwan,
> > > 
> > > Top post on terminology. IRTF documents cannot use rfc2119 terms
> > > with
> > > precision, because this is not a standards track doc.
> > > 
> > > And, the use of attacker is the appropriate and precise term for
> > > designing systems with a threat model, which is what this
> > > document does.
> > > I’ll look for ways to bring the tone down a notch, though this
> > > will add
> > > more words to a doc that was meant to be lightweight. Worth it to
> > > get it
> > > right, however.
> > > 
> > 
> > Hi Mallory,
> > 
> > I concur with Marwan that the use of "attack" here is not really
> > quite
> > right.
> > 
> > It's certainly true that measurements can have negative impacts on
> > users,
> > but when we use the term "attack" in the security context, we
> > import
> > assumptions about the motivations of the attackers. In cases where
> > there
> > are incidental negative impacts of behaviors -- e.g., where NATs
> > are
> > misdesigned and make it difficult to deploy protocols like QUIC --
> > we don't
> > typically refer to that as an attack. It seems to me that a number
> > of the
> > impacts you discuss in 1.3 fall into this category, so I think you
> > should
> > avoid the term "attack".
> > 
> 
> Convinced. Suggested replacement terms are welcomed!
> 
> -M
> 
> 
> > -Ekr
> > 
> > 
> > > 
> > > Many thanks for the ongoing discussion and offer to bring in more
> > > reviews. I haven’t got a new version out with these changes but I
> > > will do
> > > soon.
> > > 
> > > Let’s keep talking,
> > > -Mallory
> > > 
> > > 
> > > On Tue, 7 Nov 2023 at 09:01, Marwan Fayed <marwan=
> > > [e-mail]> wrote:
> > > 
> > > > Thanks Mallory and Tara. I shall keep my promise and share with
> > > > the
> > > > wider measurement community at SIGCOMM and others.
> > > > 
> > > > In the meantime, a constructive thought: Perhaps it would be
> > > > useful to
> > > > reconstruct or reframe the contents of this draft using IETF
> > > > terminology?
> > > > (For non-IETFers, last I knew, the vocabulary is
> > > > https://datatracker.ietf.org/doc/rfc2119 .) Measurement
> > > > practice has
> > > > many subtleties, some very clear do/do-nots. It occurs to me
> > > > that many of
> > > > the subtleties in a draft like this would pop out when
> > > > presented as MUST or
> > > > SHOULD NOT statements (if they can be, at all). It may help to
> > > > read public
> > > > statements from open research, in which measurement studies are
> > > > now always
> > > > accompanied by documentation explaining choices, and more. For
> > > > example, see
> > > > 'Ethics' in https://conferences.sigcomm.org/imc/2023/cfp/
> > > > 
> > > > The suggestion to user IETF terminology is motivated by a
> > > > simple
> > > > question, the likes of which IETF exists to answer: What does
> > > > "interop"
> > > > look like for measurement practice?
> > > > 
> > > > Setting details and subtleties aside, I must reiterate that the
> > > > tone in
> > > > this draft placing measurement on par with attacks is likely to
> > > > present as
> > > > problematic for people who know this domain well, many whose
> > > > established
> > > > expertise is measurement practice. It also could be misleading
> > > > for
> > > > non-experts who are reviewing measurement plans or practice --
> > > > in other
> > > > words, the risks may outweigh the benefits in current form.
> > > > 
> > > > It has taken the measurement community years to get to where it
> > > > is now;
> > > > even so our knowledge and practice continues to evolve. In this
> > > > wider
> > > > context this draft is no small endeavour, and the effort is
> > > > commendable!
> > > > (Thank you, Contributors!) No matter the outcome of this draft,
> > > > it brings
> > > > visibility to an important subject.
> > > > 
> > > > Best,
> > > > --marwan
> > > > 
> > > > 
> > > > 
> > > > 
> > > > On Sun, 5 Nov 2023 at 10:27, Tara Tarakiyee <[e-mail]>
> > > > wrote:
> > > > 
> > > > > Dear Marwan,
> > > > > 
> > > > > Thank you for sharing your review. I've read the draft with
> > > > > the
> > > > > perspective of someone who was a measurement volunteer and
> > > > > who organised
> > > > > measurement volunteers in contexts very similar to the
> > > > > example under 2.1
> > > > > and I can say I didn't find the positioning of measurement in
> > > > > this document
> > > > > to be bad or biased, considering that it's a safety
> > > > > considerations
> > > > > document. In fact, an alternative framing that sugarcoats
> > > > > risks involved
> > > > > with measurement would counter the goals of the document.
> > > > > 
> > > > > That said, I agree with your point on 2.1 about taking into
> > > > > account the
> > > > > power dynamic between the seeker of consent and the user.
> > > > > Furthermore, I
> > > > > think 2.1 should be closely tied to 2.7, it's not enough that
> > > > > the user
> > > > > considers risk to themselves, but the measurer should also
> > > > > take into
> > > > > consideration the overall benefits vs risks involved to their
> > > > > measurement
> > > > > participants even if they have informed consent. Another
> > > > > consideration to
> > > > > add to informed consent is the capabilities of the user. A
> > > > > volunteer
> > > > > working alone might be inclined to accept the risk because
> > > > > they want to
> > > > > help, but their understanding of the risk could be improved
> > > > > by connecting
> > > > > them with a local technologist, or exposure to the risk could
> > > > > be minimized
> > > > > by connecting them to local legal resources.
> > > > > 
> > > > > Best wishes,
> > > > > Tara Tarakiyee
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > On Fri, Oct 27, 2023 at 7:01 PM Marwan Fayed <marwan=
> > > > > [e-mail]> wrote:
> > > > > 
> > > > > > Opening with thanks for the reminders to comment on this
> > > > > > draft!
> > > > > > Perhaps a most useful suggestion is to solicit feedback
> > > > > > from beyond the
> > > > > > IRTF/IETF community since so much of our best practice and
> > > > > > guidance has
> > > > > > been developed, and is tracked, outside of the IRTF/IETF.
> > > > > > I'm happy to help
> > > > > > in this regard.
> > > > > > 
> > > > > > As a member of the measurement community, I can echo
> > > > > > earlier
> > > > > > sentiments that this draft is useful. I also see gaps, but
> > > > > > have additional
> > > > > > concerns about the way it positions measurement, overall,
> > > > > > in ways that (at
> > > > > > a minimum) could adversely bias institutional review or
> > > > > > other forms of
> > > > > > scrutiny by panels and people unfamiliar with the
> > > > > > measurement domain.
> > > > > > 
> > > > > > First, and most importantly, my reading of the draft's set-
> > > > > > up in its
> > > > > > current form is that it (unintentionally no doubt)
> > > > > > presupposes that
> > > > > > measurement is bad and harmful. Specifically, the draft
> > > > > > explicitly equates
> > > > > > many if not all forms of measurement each as being a type
> > > > > > of attack. Some
> > > > > > examples:
> > > > > > 
> > > > > >  - [1.2] "Passive measurements use surveillance..." -- the
> > > > > > Oxford
> > > > > > definition of surveillance is "[noun] close observation,
> > > > > > especially
> > > > > > of a suspected spy or criminal."
> > > > > > 
> > > > > >  - [1.3] encapsulates all measurements as being sources of
> > > > > > attack.
> > > > > > There are 12 explicit definitions, 9 of which start by "An
> > > > > > attack".
> > > > > > 
> > > > > > It's important to recognise that a draft like this would be
> > > > > > beneficial
> > > > > > to a wide audience. At the same time I am unable to see how
> > > > > > an outside
> > > > > > observer or reviewer, e.g. institutional review, using
> > > > > > guidelines of this
> > > > > > form (and likely have to consider liability along with safe
> > > > > > and responsible
> > > > > > practice), wouldn't automatically be forced into a starting
> > > > > > position of
> > > > > > defensiveness against measurement. (Aside: one additional
> > > > > > suggestion may be
> > > > > > to use 'responsible' alongside 'safe', which is a common
> > > > > > convention in the
> > > > > > research community.)
> > > > > > 
> > > > > > The 'risk' in this current draft is that its current set-up
> > > > > > reverses a
> > > > > > lot of the incredibly hard and tireless efforts from the
> > > > > > measurement
> > > > > > community to define, account for, report, and review best
> > > > > > practices, as
> > > > > > well as call out violations.
> > > > > > 
> > > > > > Second, there are a few subtleties missing from some
> > > > > > 'Guidelines' that
> > > > > > cannot be overlooked. Their omission could lull a reader
> > > > > > into false senses
> > > > > > of security that they've checked certain boxes. In
> > > > > > particular,
> > > > > > 
> > > > > >  - [2.1] suggests that informed consent is sufficient, when
> > > > > > in fact we
> > > > > > know that even fully informed consent in some circumstances
> > > > > > may not be
> > > > > > enough. For example, the value and merit of consent to
> > > > > > statements like "by
> > > > > > helping this measurement you could go to jail" is
> > > > > > questionable, at best.
> > > > > > Alongside, the seeker of consent has to be taken into
> > > > > > consideration; for
> > > > > > example, there is no way to know the effect of a consent-
> > > > > > seeker on a
> > > > > > consent-giver who may be more or less inclined to trust a
> > > > > > multinational
> > > > > > corporation, foreign government, civil liberty group, or a
> > > > > > university lab
> > > > > > -- these are just not the same, and an individual may trust
> > > > > > more or less
> > > > > > depending.
> > > > > > 
> > > > > >  - [2.3] This is a good one! If it's helpful, even the
> > > > > > seemingly
> > > > > > obviously 'ok' can have caveats. Speedtests are a great
> > > > > > example. It might
> > > > > > be reasonable to assume that any speedtest server is happy
> > > > > > to receive a
> > > > > > test request, and any subscriber who pays for access is
> > > > > > entitled to request
> > > > > > the test -- but what about the transit provider in-between?
> > > > > > 
> > > > > >  - [2.4] Also great, and easy to overlook. Here, too,
> > > > > > however, I
> > > > > > wonder if all requests are equal? As an example, what does
> > > > > > it mean for
> > > > > > Google to request do-not-scan? (I honestly don't know!) One
> > > > > > possible
> > > > > > addition may be to suggest that "self-identify" is good
> > > > > > practice,
> > > > > > especially when releasing measurement tools into the world.
> > > > > > For example,
> > > > > > the popular zmap tool self identifies itself by placing
> > > > > > "54321" in the
> > > > > > IP-ID field.
> > > > > > 
> > > > > > One last suggestion: It's hard to know which are the most
> > > > > > useful
> > > > > > references in this space, so a draft like this would be a
> > > > > > great place to
> > > > > > have a comprehensive list of pointers to best and current
> > > > > > practice in this
> > > > > > space (e.g. the CACM written by Partridge and Allman), and
> > > > > > docs that might
> > > > > > include case studies. Even better might be to have one or
> > > > > > two sentences for
> > > > > > each to describe why the reference is useful, or what it
> > > > > > offers.
> > > > > > (Acknowledging in advance that distilling down from
> > > > > > multiple inputs is a
> > > > > > challenge in itself, and would be a great feat!)
> > > > > > 
> > > > > > Hope some of this helps.
> > > > > > 
> > > > > > --marwan
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > --
> > > > > > Pearg mailing list
> > > > > > [e-mail]
> > > > > > https://mailman.irtf.org/mailman/listinfo/pearg
> > > > > > 
> > > > > --
> > > > > Pearg mailing list
> > > > > [e-mail]
> > > > > https://mailman.irtf.org/mailman/listinfo/pearg
> > > > > 
> > > > --
> > > > Pearg mailing list
> > > > [e-mail]
> > > > https://mailman.irtf.org/mailman/listinfo/pearg
> > > > 
> > > --
> > > Pearg mailing list
> > > [e-mail]
> > > https://mailman.irtf.org/mailman/listinfo/pearg
> > > 
> > 

-- 
Dr.-Ing. Tobias Fiebig
T [phone]
M [e-mail]