Hum theatre

[Dave Crocker <[e-mail]>]

Folks,

An IETF hum is a method of surveying a group for its views. 
Unfortunately the hums that were taken at the end of this week's IAB 
plenary do not permit any meaningful interpretation.


Here's why...

Surveys are extremely sensitive to the phrasing of the questions, the 
phrasing and range of the response choices, the sequencing of the 
questions, and the context of the asking.  Get any of these wrong and 
you can get the wrong information, or even just the appearance of 
information -- that is, misunderstandings -- but nothing actually useful.

A common response to such a concern is "well, at least we'll get some 
answers", but that's like saying "well, at least we'll get some noise." 
  The fact that the noise is misunderstood to be signal does not 
actually make it signal.



The different phrasings of a question can produce very different 
understandings by responders.  The challenge is to formulate a question 
that is likely to be interpreted similarly amongst responders (and the 
person asking.)  It's also a challenge to ask a question that captures 
something that is actually meaningful (and was intended) rather than 
merely sounding good.

The offered response choices can bias the responses.  A set of choices 
like (Good, Excellent) obviously leaves out (Bad, Don't Care, Don't 
Know.)  Or they can have bias in their phrasing by making some choices 
more or less appealing (Could be better, Excellent), rather than 
equivalent vocabulary in tone (Bad, Good).  So it's a challenge to make 
sure that choices cover the proper range and with equanimity to the 
alternative choices.

A sequence of questions also needs to be carefully orchestrated.  For 
example today's questions took as a given that surveillance is an 
attack.  Due diligence might expect establishing that relationship 
explicitly.  And yes, it is possible that some IETF attendees do not see 
it as an attack.  Another example of sequencing is dealing with 
subtleties and complexities.  For example some anti-surveillance 
mechanisms are certain to defeat popular operational management 
mechanisms.  Do we care about the tradeoffs?

Lastly, environmental context can encourage or discourage candor. 
Examples include the genders of the asker and respondent, any 
relationship they might have, or the presence of others.  Would you 
really provide candid answers about possible problems with your sex life 
when being asked with your partner present?  Amongst a group of 
co-workers?  Your parents?



The hums asked at the plenary were problematic along each of these lines.

The first question was theatre, essentially making the context 
political.  By way of example, note the difference between what was asked:

      The IETF is willing to respond to the pervasive surveillance attack?

which has loaded language with 'pervasive' and 'attack', versus a more 
neutral and purely technical question meant to cover the same basic concern:

     The IETF is willing to improve its specifications to be more 
resistant to surveillance?

But this isn't exactly a balanced question either.  By that, I mean that 
the answer really is already known.  A good question is one that has a 
chance of getting some support for each choice.  So perhaps a better 
example would be:

      The IETF is willing to require adding resistance to surveillance 
to all of its protocols?

The questions typically also did not offer "don't know" or "don't care" 
choices.  Some folk probably knew that they don't know enough yet, 
limiting their ability to support the kinds of questions being asked.

The IETF's doing anything privacy-related that is useful is going to 
require considering tradeoffs and some of those tradeoffs might reduce 
the utility of a service. So the actual choices that will be made might 
turn out to be quite different from what was implied by the dominant 
answers to the plenary questions.

And lastly, consider carefully the context of the room and ask whether 
everyone actually felt completely free to give a "no" hum to the initial 
questions.  I suggest that the emotions of the room created a strong 
bias against no's.   Maybe not for you.  Maybe not for me.  But probably 
for many of the folk sitting near you.

We now find ourselves with a set of hums that appears to establish a 
direction but which can't survive even basic analysis, as the later 
postings on the ietf mailing list demonstrate.



Here's what I suggest:  A single, simple, conceptual question that 
supplies all of the 'guidance' we can legitimately offer, at this stage:

      The IETF needs to press for careful attention to privacy
      concerns in its work, including protection against surveillance.

           [ ]  No
           [ ]  Yes
           [ ]  Don't Yet Know
           [ ]  Don't Care



d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net