Re: [Pearg] More comments on draft-irtf-pearg-safe-internet-measurement-08

[Christian Huitema <[e-mail]>]

After reading the draft and the comments, I think we mostly have an 
issue of scope. The draft seems very focused on "privacy related" 
measurements, such as detecting various means or censorship, and to a 
lesser degree on "usage related" documents, such as measuring how people 
actually use the Internet.

Previous comments have suggested expanding the scope of the draft to 
encompass other types of measurements, such as low level metrics, etc. I 
am concerned that broadening the scope implies a lot of work. It would 
also result in loss of focus, which would weaken the draft. My 
suggestion instead is to scope the draft, stating clearly in the title 
and abstract that this is intended for some specific types of measurements.

I perform quite a few measurements myself, working with ICANN to publish 
and update a set of "internet identifiers" metrics at 
ithi.research.icann.org. I was looking at the parallels between what we 
do and what the draft describes, checking whether we should improve our 
practice. I am not sure that I got much information. We do periodic 
series of DNS queries. Does that fall under the "scanning" 
recommendation? We publish aggregated statistics, some grouped by 
address prefixes or by names of service providers. Is there an 
expectation of privacy for service providers? Does that depend on their 
size? When we do active queries, does that fall under the scanning 
recommendations?

These are very specialized questions, each requiring an analysis of 
specific tradeoffs. Mallory's draft cannot possibly answer them all. But 
then, it would be nice if the draft acknowledged that, and focused on 
the parts that are clear. I really like the analysis of consent, what it 
implies for users to consent to add a measurement application on their 
devices. Maybe focus the draft on exactly that?

-- Christian Huitema


On 11/7/2023 5:56 AM, Mallory Knodel wrote:
> Hi Tobias,
> 
> Thanks for your comments. This is a research group document and so any
> suggestions for concrete text will help this direly needed text evolve into
> something useful and not harmful.
> 
> I’ll capture your comments in an issue and with another version try to
> address them. A PR or draft text to the list will expedite closing this
> issue.
> 
> Thanks!
> -Mallory
> 
> On Tue, 7 Nov 2023 at 14:43, Tobias Fiebig <tobias=
> [e-mail]> wrote:
> 
>> Moin,
>> I just was stumbled over this draft and just gave it a read. At the
>> moment, i think that the draft has crucial limitations that may cause
>> significant harm. In general, my core comments are relatively aligned
>> with Marwan's (very spot-on comments). However, I also believe that
>> there are some more fundamental issues that must be resolved.
>>
>> # Authoritativeness of documents
>> Marwan noted that the document in its current form may cause harm if it
>> is read by a non-domain expert (say: Medicine Professor) when
>> evaluating a request for ethical clearance of a network measurement
>> study. And yes, this _does_ happen.
>>
>> Reading the reply by Mallory creates the impression in me that there is
>> some form of assumption that what is written in a document will not be
>> read as authoritative if the authors only (outside the document or
>> somewhere in another part of the document the concerned medicine prof
>> just didn't scroll over) phrase things more in terms of their
>> intentions.
>>
>> This is not really the case; For the domain where the document will be
>> relevant and useful, those nuances will be lost. It is already hard to
>> have security researchers understand the difference between a draft and
>> and RFC, let alone all the other nuances there after. I would hence
>> caution significant care with how the document can be perceived,
>> especially in context for the point below.
>>
>> # Positionality / Perspective
>>
>> The draft has a strong perspective from the privacy measurement angle,
>> also highlighted by the statement that some points "[are] based on the
>> Tor network safe measurement guidelines— so, from the community of
>> measurement and internet research."; The Tor community is a very
>> specific (and small) subset of network measurement work, and I would
>> argue that they are more rooted in the PETS community (also looking at
>> the members of the board).
>>
>> This community, in terms of type of measurements and also in terms of
>> volume, is very different from the network measurement community (ACM
>> SIGCOMM-Bobble), which _again_ is different from the Network Security
>> bobble (IEEE S&P/NDSS/USENIX SEC/ACM CCS).
>>
>> These communities have very different requirements and perspectives on
>> network measurements (and potential harm they may cause).
>>
>> Hence, i'd argue that the draft needs a lot more engagement with these
>> different communities, and should include the experiences of
>> researchers doing that kind of work as well.
>>
>> # Superficial Nature of Sections 2.4ff
>> >From Section 2.4 the draft gets extremely thin and ignores practical
>> requirements of network measurements by taking a strong data
>> minimization perspective (which is imperative for censorship
>> measurements, but often _harmful_ for large scale measurements; If you
>> do a large scale measurement and stripped a bit too much, redoing the
>> measurement might be unnecessary harm; The same as having done an
>> effectively useless measurement.).
>>
>> Similarly, the--somewhat simon-says-y--headlines thereafter just keep
>> introducing shorter and shorter sections that hand wave away _critical_
>> aspects that necessitate _very_ careful consideration and
>> instruction... and--as said before--sometimes these imperative things
>> may not even make sense.
>>
>> 2.6 and 2.7 are then close to dangerously short. A document claiming to
>> provide guidelines should provide, for these critical mechanics, actual
>> guidelines on how to, e.g., execute a harm benefit analysis. With the
>> current form, the document essentially provides a blanket for
>> researchers being like "ah, well, thought about it, best we could...
>> uhm... ups, failed after following best practices, nothing we could
>> have done". See also my paper from anrw at 117.
>>
>> # Summary
>> In summary, as those speaking earlier, I am deeply convinced that the
>> topic of the draft is direly needed; However, such a document should
>> take a lot of context (communities etc.) into account which is not in -
>> 08 (and i would argue what is missing is a multiple of what is already
>> there). It must also provide guidelines on _how_ to do some of the
>> things suggested, instead of just throwing in words, hoping that it is
>> feasible.
>>
>> Otherwise, this document might cause harm by creating unnecessary red-
>> tape, which will result in losing a lot of the progress that was made
>> in researchers' consideration of ethical concerns and adherence to
>> ethical principles; There will be frustration and 'trying to get
>> around'.
>>
>> With best regards,
>> Tobias
>>
>> On Tue, 2023-11-07 at 13:56 +0100, Mallory Knodel wrote:
>>> On Tue, 7 Nov 2023 at 13:45, Eric Rescorla <[e-mail]> wrote:
>>>
>>>>
>>>>
>>>> On Tue, Nov 7, 2023 at 12:17 AM Mallory Knodel <mknodel=
>>>> [e-mail]> wrote:
>>>>
>>>>> Hi Marwan,
>>>>>
>>>>> Top post on terminology. IRTF documents cannot use rfc2119 terms
>>>>> with
>>>>> precision, because this is not a standards track doc.
>>>>>
>>>>> And, the use of attacker is the appropriate and precise term for
>>>>> designing systems with a threat model, which is what this
>>>>> document does.
>>>>> I’ll look for ways to bring the tone down a notch, though this
>>>>> will add
>>>>> more words to a doc that was meant to be lightweight. Worth it to
>>>>> get it
>>>>> right, however.
>>>>>
>>>>
>>>> Hi Mallory,
>>>>
>>>> I concur with Marwan that the use of "attack" here is not really
>>>> quite
>>>> right.
>>>>
>>>> It's certainly true that measurements can have negative impacts on
>>>> users,
>>>> but when we use the term "attack" in the security context, we
>>>> import
>>>> assumptions about the motivations of the attackers. In cases where
>>>> there
>>>> are incidental negative impacts of behaviors -- e.g., where NATs
>>>> are
>>>> misdesigned and make it difficult to deploy protocols like QUIC --
>>>> we don't
>>>> typically refer to that as an attack. It seems to me that a number
>>>> of the
>>>> impacts you discuss in 1.3 fall into this category, so I think you
>>>> should
>>>> avoid the term "attack".
>>>>
>>>
>>> Convinced. Suggested replacement terms are welcomed!
>>>
>>> -M
>>>
>>>
>>>> -Ekr
>>>>
>>>>
>>>>>
>>>>> Many thanks for the ongoing discussion and offer to bring in more
>>>>> reviews. I haven’t got a new version out with these changes but I
>>>>> will do
>>>>> soon.
>>>>>
>>>>> Let’s keep talking,
>>>>> -Mallory
>>>>>
>>>>>
>>>>> On Tue, 7 Nov 2023 at 09:01, Marwan Fayed <marwan=
>>>>> [e-mail]> wrote:
>>>>>
>>>>>> Thanks Mallory and Tara. I shall keep my promise and share with
>>>>>> the
>>>>>> wider measurement community at SIGCOMM and others.
>>>>>>
>>>>>> In the meantime, a constructive thought: Perhaps it would be
>>>>>> useful to
>>>>>> reconstruct or reframe the contents of this draft using IETF
>>>>>> terminology?
>>>>>> (For non-IETFers, last I knew, the vocabulary is
>>>>>> https://datatracker.ietf.org/doc/rfc2119 .) Measurement
>>>>>> practice has
>>>>>> many subtleties, some very clear do/do-nots. It occurs to me
>>>>>> that many of
>>>>>> the subtleties in a draft like this would pop out when
>>>>>> presented as MUST or
>>>>>> SHOULD NOT statements (if they can be, at all). It may help to
>>>>>> read public
>>>>>> statements from open research, in which measurement studies are
>>>>>> now always
>>>>>> accompanied by documentation explaining choices, and more. For
>>>>>> example, see
>>>>>> 'Ethics' in https://conferences.sigcomm.org/imc/2023/cfp/
>>>>>>
>>>>>> The suggestion to user IETF terminology is motivated by a
>>>>>> simple
>>>>>> question, the likes of which IETF exists to answer: What does
>>>>>> "interop"
>>>>>> look like for measurement practice?
>>>>>>
>>>>>> Setting details and subtleties aside, I must reiterate that the
>>>>>> tone in
>>>>>> this draft placing measurement on par with attacks is likely to
>>>>>> present as
>>>>>> problematic for people who know this domain well, many whose
>>>>>> established
>>>>>> expertise is measurement practice. It also could be misleading
>>>>>> for
>>>>>> non-experts who are reviewing measurement plans or practice --
>>>>>> in other
>>>>>> words, the risks may outweigh the benefits in current form.
>>>>>>
>>>>>> It has taken the measurement community years to get to where it
>>>>>> is now;
>>>>>> even so our knowledge and practice continues to evolve. In this
>>>>>> wider
>>>>>> context this draft is no small endeavour, and the effort is
>>>>>> commendable!
>>>>>> (Thank you, Contributors!) No matter the outcome of this draft,
>>>>>> it brings
>>>>>> visibility to an important subject.
>>>>>>
>>>>>> Best,
>>>>>> --marwan
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, 5 Nov 2023 at 10:27, Tara Tarakiyee <[e-mail]>
>>>>>> wrote:
>>>>>>
>>>>>>> Dear Marwan,
>>>>>>>
>>>>>>> Thank you for sharing your review. I've read the draft with
>>>>>>> the
>>>>>>> perspective of someone who was a measurement volunteer and
>>>>>>> who organised
>>>>>>> measurement volunteers in contexts very similar to the
>>>>>>> example under 2.1
>>>>>>> and I can say I didn't find the positioning of measurement in
>>>>>>> this document
>>>>>>> to be bad or biased, considering that it's a safety
>>>>>>> considerations
>>>>>>> document. In fact, an alternative framing that sugarcoats
>>>>>>> risks involved
>>>>>>> with measurement would counter the goals of the document.
>>>>>>>
>>>>>>> That said, I agree with your point on 2.1 about taking into
>>>>>>> account the
>>>>>>> power dynamic between the seeker of consent and the user.
>>>>>>> Furthermore, I
>>>>>>> think 2.1 should be closely tied to 2.7, it's not enough that
>>>>>>> the user
>>>>>>> considers risk to themselves, but the measurer should also
>>>>>>> take into
>>>>>>> consideration the overall benefits vs risks involved to their
>>>>>>> measurement
>>>>>>> participants even if they have informed consent. Another
>>>>>>> consideration to
>>>>>>> add to informed consent is the capabilities of the user. A
>>>>>>> volunteer
>>>>>>> working alone might be inclined to accept the risk because
>>>>>>> they want to
>>>>>>> help, but their understanding of the risk could be improved
>>>>>>> by connecting
>>>>>>> them with a local technologist, or exposure to the risk could
>>>>>>> be minimized
>>>>>>> by connecting them to local legal resources.
>>>>>>>
>>>>>>> Best wishes,
>>>>>>> Tara Tarakiyee
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Oct 27, 2023 at 7:01 PM Marwan Fayed <marwan=
>>>>>>> [e-mail]> wrote:
>>>>>>>
>>>>>>>> Opening with thanks for the reminders to comment on this
>>>>>>>> draft!
>>>>>>>> Perhaps a most useful suggestion is to solicit feedback
>>>>>>>> from beyond the
>>>>>>>> IRTF/IETF community since so much of our best practice and
>>>>>>>> guidance has
>>>>>>>> been developed, and is tracked, outside of the IRTF/IETF.
>>>>>>>> I'm happy to help
>>>>>>>> in this regard.
>>>>>>>>
>>>>>>>> As a member of the measurement community, I can echo
>>>>>>>> earlier
>>>>>>>> sentiments that this draft is useful. I also see gaps, but
>>>>>>>> have additional
>>>>>>>> concerns about the way it positions measurement, overall,
>>>>>>>> in ways that (at
>>>>>>>> a minimum) could adversely bias institutional review or
>>>>>>>> other forms of
>>>>>>>> scrutiny by panels and people unfamiliar with the
>>>>>>>> measurement domain.
>>>>>>>>
>>>>>>>> First, and most importantly, my reading of the draft's set-
>>>>>>>> up in its
>>>>>>>> current form is that it (unintentionally no doubt)
>>>>>>>> presupposes that
>>>>>>>> measurement is bad and harmful. Specifically, the draft
>>>>>>>> explicitly equates
>>>>>>>> many if not all forms of measurement each as being a type
>>>>>>>> of attack. Some
>>>>>>>> examples:
>>>>>>>>
>>>>>>>>   - [1.2] "Passive measurements use surveillance..." -- the
>>>>>>>> Oxford
>>>>>>>> definition of surveillance is "[noun] close observation,
>>>>>>>> especially
>>>>>>>> of a suspected spy or criminal."
>>>>>>>>
>>>>>>>>   - [1.3] encapsulates all measurements as being sources of
>>>>>>>> attack.
>>>>>>>> There are 12 explicit definitions, 9 of which start by "An
>>>>>>>> attack".
>>>>>>>>
>>>>>>>> It's important to recognise that a draft like this would be
>>>>>>>> beneficial
>>>>>>>> to a wide audience. At the same time I am unable to see how
>>>>>>>> an outside
>>>>>>>> observer or reviewer, e.g. institutional review, using
>>>>>>>> guidelines of this
>>>>>>>> form (and likely have to consider liability along with safe
>>>>>>>> and responsible
>>>>>>>> practice), wouldn't automatically be forced into a starting
>>>>>>>> position of
>>>>>>>> defensiveness against measurement. (Aside: one additional
>>>>>>>> suggestion may be
>>>>>>>> to use 'responsible' alongside 'safe', which is a common
>>>>>>>> convention in the
>>>>>>>> research community.)
>>>>>>>>
>>>>>>>> The 'risk' in this current draft is that its current set-up
>>>>>>>> reverses a
>>>>>>>> lot of the incredibly hard and tireless efforts from the
>>>>>>>> measurement
>>>>>>>> community to define, account for, report, and review best
>>>>>>>> practices, as
>>>>>>>> well as call out violations.
>>>>>>>>
>>>>>>>> Second, there are a few subtleties missing from some
>>>>>>>> 'Guidelines' that
>>>>>>>> cannot be overlooked. Their omission could lull a reader
>>>>>>>> into false senses
>>>>>>>> of security that they've checked certain boxes. In
>>>>>>>> particular,
>>>>>>>>
>>>>>>>>   - [2.1] suggests that informed consent is sufficient, when
>>>>>>>> in fact we
>>>>>>>> know that even fully informed consent in some circumstances
>>>>>>>> may not be
>>>>>>>> enough. For example, the value and merit of consent to
>>>>>>>> statements like "by
>>>>>>>> helping this measurement you could go to jail" is
>>>>>>>> questionable, at best.
>>>>>>>> Alongside, the seeker of consent has to be taken into
>>>>>>>> consideration; for
>>>>>>>> example, there is no way to know the effect of a consent-
>>>>>>>> seeker on a
>>>>>>>> consent-giver who may be more or less inclined to trust a
>>>>>>>> multinational
>>>>>>>> corporation, foreign government, civil liberty group, or a
>>>>>>>> university lab
>>>>>>>> -- these are just not the same, and an individual may trust
>>>>>>>> more or less
>>>>>>>> depending.
>>>>>>>>
>>>>>>>>   - [2.3] This is a good one! If it's helpful, even the
>>>>>>>> seemingly
>>>>>>>> obviously 'ok' can have caveats. Speedtests are a great
>>>>>>>> example. It might
>>>>>>>> be reasonable to assume that any speedtest server is happy
>>>>>>>> to receive a
>>>>>>>> test request, and any subscriber who pays for access is
>>>>>>>> entitled to request
>>>>>>>> the test -- but what about the transit provider in-between?
>>>>>>>>
>>>>>>>>   - [2.4] Also great, and easy to overlook. Here, too,
>>>>>>>> however, I
>>>>>>>> wonder if all requests are equal? As an example, what does
>>>>>>>> it mean for
>>>>>>>> Google to request do-not-scan? (I honestly don't know!) One
>>>>>>>> possible
>>>>>>>> addition may be to suggest that "self-identify" is good
>>>>>>>> practice,
>>>>>>>> especially when releasing measurement tools into the world.
>>>>>>>> For example,
>>>>>>>> the popular zmap tool self identifies itself by placing
>>>>>>>> "54321" in the
>>>>>>>> IP-ID field.
>>>>>>>>
>>>>>>>> One last suggestion: It's hard to know which are the most
>>>>>>>> useful
>>>>>>>> references in this space, so a draft like this would be a
>>>>>>>> great place to
>>>>>>>> have a comprehensive list of pointers to best and current
>>>>>>>> practice in this
>>>>>>>> space (e.g. the CACM written by Partridge and Allman), and
>>>>>>>> docs that might
>>>>>>>> include case studies. Even better might be to have one or
>>>>>>>> two sentences for
>>>>>>>> each to describe why the reference is useful, or what it
>>>>>>>> offers.
>>>>>>>> (Acknowledging in advance that distilling down from
>>>>>>>> multiple inputs is a
>>>>>>>> challenge in itself, and would be a great feat!)
>>>>>>>>
>>>>>>>> Hope some of this helps.
>>>>>>>>
>>>>>>>> --marwan
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Pearg mailing list
>>>>>>>> [e-mail]
>>>>>>>> https://mailman.irtf.org/mailman/listinfo/pearg
>>>>>>>>
>>>>>>> --
>>>>>>> Pearg mailing list
>>>>>>> [e-mail]
>>>>>>> https://mailman.irtf.org/mailman/listinfo/pearg
>>>>>>>
>>>>>> --
>>>>>> Pearg mailing list
>>>>>> [e-mail]
>>>>>> https://mailman.irtf.org/mailman/listinfo/pearg
>>>>>>
>>>>> --
>>>>> Pearg mailing list
>>>>> [e-mail]
>>>>> https://mailman.irtf.org/mailman/listinfo/pearg
>>>>>
>>>>
>>
>> --
>> Dr.-Ing. Tobias Fiebig
>> T [phone]
>> M [e-mail]
>>
>> --
>> Pearg mailing list
>> [e-mail]
>> https://mailman.irtf.org/mailman/listinfo/pearg
>>
> 
>