Re: Hum theatre

[Tim Bray <[e-mail]>]

You’re entitled to your opinion, but I entirely disagree.  I thought each
of those made an important point and highlighted some areas where consensus
is broadly held.  I appreciated Russ’ composition of the issues and think
he deserves our thanks.

 -Tim


On Wed, Nov 6, 2013 at 6:23 PM, Dave Crocker <[e-mail]> wrote:

> Folks,
>
> An IETF hum is a method of surveying a group for its views. Unfortunately
> the hums that were taken at the end of this week's IAB plenary do not
> permit any meaningful interpretation.
>
>
> Here's why...
>
> Surveys are extremely sensitive to the phrasing of the questions, the
> phrasing and range of the response choices, the sequencing of the
> questions, and the context of the asking.  Get any of these wrong and you
> can get the wrong information, or even just the appearance of information
> -- that is, misunderstandings -- but nothing actually useful.
>
> A common response to such a concern is "well, at least we'll get some
> answers", but that's like saying "well, at least we'll get some noise."
>  The fact that the noise is misunderstood to be signal does not actually
> make it signal.
>
>
>
> The different phrasings of a question can produce very different
> understandings by responders.  The challenge is to formulate a question
> that is likely to be interpreted similarly amongst responders (and the
> person asking.)  It's also a challenge to ask a question that captures
> something that is actually meaningful (and was intended) rather than merely
> sounding good.
>
> The offered response choices can bias the responses.  A set of choices
> like (Good, Excellent) obviously leaves out (Bad, Don't Care, Don't Know.)
>  Or they can have bias in their phrasing by making some choices more or
> less appealing (Could be better, Excellent), rather than equivalent
> vocabulary in tone (Bad, Good).  So it's a challenge to make sure that
> choices cover the proper range and with equanimity to the alternative
> choices.
>
> A sequence of questions also needs to be carefully orchestrated.  For
> example today's questions took as a given that surveillance is an attack.
>  Due diligence might expect establishing that relationship explicitly.  And
> yes, it is possible that some IETF attendees do not see it as an attack.
>  Another example of sequencing is dealing with subtleties and complexities.
>  For example some anti-surveillance mechanisms are certain to defeat
> popular operational management mechanisms.  Do we care about the tradeoffs?
>
> Lastly, environmental context can encourage or discourage candor. Examples
> include the genders of the asker and respondent, any relationship they
> might have, or the presence of others.  Would you really provide candid
> answers about possible problems with your sex life when being asked with
> your partner present?  Amongst a group of co-workers?  Your parents?
>
>
>
> The hums asked at the plenary were problematic along each of these lines.
>
> The first question was theatre, essentially making the context political.
>  By way of example, note the difference between what was asked:
>
>      The IETF is willing to respond to the pervasive surveillance attack?
>
> which has loaded language with 'pervasive' and 'attack', versus a more
> neutral and purely technical question meant to cover the same basic concern:
>
>     The IETF is willing to improve its specifications to be more resistant
> to surveillance?
>
> But this isn't exactly a balanced question either.  By that, I mean that
> the answer really is already known.  A good question is one that has a
> chance of getting some support for each choice.  So perhaps a better
> example would be:
>
>      The IETF is willing to require adding resistance to surveillance to
> all of its protocols?
>
> The questions typically also did not offer "don't know" or "don't care"
> choices.  Some folk probably knew that they don't know enough yet, limiting
> their ability to support the kinds of questions being asked.
>
> The IETF's doing anything privacy-related that is useful is going to
> require considering tradeoffs and some of those tradeoffs might reduce the
> utility of a service. So the actual choices that will be made might turn
> out to be quite different from what was implied by the dominant answers to
> the plenary questions.
>
> And lastly, consider carefully the context of the room and ask whether
> everyone actually felt completely free to give a "no" hum to the initial
> questions.  I suggest that the emotions of the room created a strong bias
> against no's.   Maybe not for you.  Maybe not for me.  But probably for
> many of the folk sitting near you.
>
> We now find ourselves with a set of hums that appears to establish a
> direction but which can't survive even basic analysis, as the later
> postings on the ietf mailing list demonstrate.
>
>
>
> Here's what I suggest:  A single, simple, conceptual question that
> supplies all of the 'guidance' we can legitimately offer, at this stage:
>
>      The IETF needs to press for careful attention to privacy
>      concerns in its work, including protection against surveillance.
>
>           [ ]  No
>           [ ]  Yes
>           [ ]  Don't Yet Know
>           [ ]  Don't Care
>
>
>
> d/
>
> --
> Dave Crocker
> Brandenburg InternetWorking
> bbiw.net
>