Re: [rtcweb] Unacceptable - (Re: Straw Poll on Video Codec Alternatives)

[Dave Crocker <[e-mail]>]

Magnus,


On 12/10/2013 1:34 AM, Magnus Westerlund wrote:
> I agree that it is difficult to design questionnaires and will agree
> that this one may have some flaws. You argue for a balanced list of

Apparently, you misunderstood my note.  I wasn't expressing an opinion 
in the nature of a preference.  I was noting a methodological error.  In 
other words, I was citing something broken, not merely distasteful.


> choices where there is equally positive and negative options. The goal
> was that the Acceptable would be the in the middle option, between I

I would have thought that the goal would be meaningful results.

One does not usually have a structural choice as a "goal".

As I noted, 2/3 of the choices were positive.  That biases response 
patterns.  I went to the effort of providing citations specifically so 
that you would not think I was inventing the concern or expressing an 
"opinion".


> favor or I don't favor it. Thus I suggested Acceptable, a I can live
> with this but don't favor it or disfavor it. Using a word like neutral
> would also be strange in regards to the formulation of the question.

There are other issues in providing such a long and undifferentiated 
list of questions to folk, including respondent fatigue.  I was merely 
trying to raise the flag with one of the more obvious and 
well-understood flaws.


> Looking at your proposal I fail to understand the meaning of your options:
>> Given the intent of 'acceptable', what you need is something like:
>>
>>     Yes
>>     Prefer yes
>>     Don't care
>>     Prefer no
>>     No
>
> What does it mean to say "I Don't Care about the option: "There is no
> MTI video codec". Similar it appears difficult to interpret using
> "Prefer no".

Actually, it's semantics match your use of "Acceptable".  That is, the 
respondent doesn't actively want the choice, but can tolerate it.  The 
label I'm suggesting is a more accurate semantic since it makes the 
respondents starting point of a negative assessment explicit whereas 
yours does not.

The reason to have both prefer yes and prefer no is to balance "can live 
with its being chosen" with "can live with its not being chosen".  Hence 
the range of responses gives a more accurate sense of response strengths 
and potential points of resistance.


> In such a case a 1-5 (Least favored to Most favored) rating
> would be less problematic as it would allow people to think in their own
> terms.

Except that least/most semantics don't match the nature of the question. 
  The fact that both are 5-point scales misses the potential differences 
in respondent interpretation of the label semantics.  Such a difference 
affects respondent behavior quite a bit.


> Yes, we can spend a lot of emails on discussing what the questions
> should be. I don't think this would be productive unless there is a wide
> belief that the question asked is so flawed that we really are missing
> important information or skewing the results significantly.

If the text said 2+2=5 and only one person noted the error and no one 
responded, would that make it acceptable to ignore the problem?

Again, you seem to be confusing the difference between prefer and broken.


As basic as the methodology problem is to the survey instrument, it's 
actually a pretty minor concern, compared with the continuing failure to 
recruit wg participation in deciding how to proceed.  Imposing the 
survey before getting agreement from the wg to /do/ the survey and use 
the proffered questions is an example of what seems to be a pattern in 
the wg's management.

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net