Re: [rtcweb] Unacceptable - (Re: Straw Poll on Video Codec Alternatives)

[Magnus Westerlund <[e-mail]>]

On 2013-12-10 15:40, Dave Crocker wrote:
> Magnus,
> 
> 
> On 12/10/2013 1:34 AM, Magnus Westerlund wrote:
>> I agree that it is difficult to design questionnaires and will agree
>> that this one may have some flaws. You argue for a balanced list of
> 
> Apparently, you misunderstood my note.  I wasn't expressing an opinion
> in the nature of a preference.  I was noting a methodological error.  In
> other words, I was citing something broken, not merely distasteful.

I don't think I fully misunderstood it. Yes, I may not have understood
all aspects of what you wanted convey. However, my interpretation of
what you are saying is: This methodological error introduces a degree of
brokenness in your survey.

I personally have a view of how bad this brokenness is. I asked for
others to express their considerations on how bad they consider this error.

Because from my perspective it comes down to the issue. Is this so
broken that we must fix it, or can we live with it. Nothing ever is
perfect and works fully as intended.

> 
> 
>> favor or I don't favor it. Thus I suggested Acceptable, a I can live
>> with this but don't favor it or disfavor it. Using a word like neutral
>> would also be strange in regards to the formulation of the question.
> 
> There are other issues in providing such a long and undifferentiated
> list of questions to folk, including respondent fatigue.  I was merely
> trying to raise the flag with one of the more obvious and
> well-understood flaws.

Seriously, I don't think respondent fatigue is going to be a serious
issue for this. It is 16 alternatives with 2 questions per alternative.
There is no time pressure on doing and getting it done. In fact I
recommend that people take their time to work through the options and
consider their response well.

> 
> 
>> Looking at your proposal I fail to understand the meaning of your
>> options:
>>> Given the intent of 'acceptable', what you need is something like:
>>>
>>>     Yes
>>>     Prefer yes
>>>     Don't care
>>>     Prefer no
>>>     No
>>
>> What does it mean to say "I Don't Care about the option: "There is no
>> MTI video codec". Similar it appears difficult to interpret using
>> "Prefer no".
> 
> Actually, it's semantics match your use of "Acceptable".  That is, the
> respondent doesn't actively want the choice, but can tolerate it.  The
> label I'm suggesting is a more accurate semantic since it makes the
> respondents starting point of a negative assessment explicit whereas
> yours does not.

Okay, so "Tolerable" or some other English word that conveys a more
neutral position would address this issue?

> 
> The reason to have both prefer yes and prefer no is to balance "can live
> with its being chosen" with "can live with its not being chosen".  Hence
> the range of responses gives a more accurate sense of response strengths
> and potential points of resistance.

Sorry, but I don't get your labels. They doesn't make sense to me in
this context. But, I don't see it worth spending more time on the
particular labels to use if one would like to have five values in this
survey. Not, unless we really need to change it.

> 
> 
>> In such a case a 1-5 (Least favored to Most favored) rating
>> would be less problematic as it would allow people to think in their own
>> terms.
> 
> Except that least/most semantics don't match the nature of the question.
>  The fact that both are 5-point scales misses the potential differences
> in respondent interpretation of the label semantics.  Such a difference
> affects respondent behavior quite a bit.

Isn't this the core of the issue we are discussing. Either you are using
labels that will be interpreted and you get peoples interpretation of
the individual labels. Which introduce a certain potential bias due to
the labels. Or if you use a range and only anchor the edges of the
range, you get floating interpretations of what the mid values mean.

Different types of errors.

> 
> 
>> Yes, we can spend a lot of emails on discussing what the questions
>> should be. I don't think this would be productive unless there is a wide
>> belief that the question asked is so flawed that we really are missing
>> important information or skewing the results significantly.
> 
> If the text said 2+2=5 and only one person noted the error and no one
> responded, would that make it acceptable to ignore the problem?

No.

> 
> Again, you seem to be confusing the difference between prefer and broken.

No, but I am trying to arrive at a judgment if the brokenness is so
severe that you have to stop and call a tow truck, drive immediately to
the car shop, or if you can continue to drive for the next month and
have it addressed in your planned maintenance.

For the moment my personal interpretation is that this is something I
will do my best to remember the next time I am part of design a survey.

> 
> 
> As basic as the methodology problem is to the survey instrument, it's
> actually a pretty minor concern, compared with the continuing failure to
> recruit wg participation in deciding how to proceed.  Imposing the
> survey before getting agreement from the wg to /do/ the survey and use
> the proffered questions is an example of what seems to be a pattern in
> the wg's management.

I will not try to deny that we have made some errors and misjudgment.
Clear the WG hasn't caught some of them either, which was on WG review.
Looking in the rear view mirror it is easy to say how one should or
could have done it to make things better. We also have a role of running
and driving the process and do need to make decision at some points.

We chairs have discussed to put the straw poll and next steps out as a
proposal for the WG review. We felt that the information the straw poll
would give would be good independent for the whole WG. Thus we
implemented rather than have a week or two of discussion on it before
being able to implement it.

The high level process we intent to follow afterwards have been
provided. You are welcome to provide feedback on it.


Cheers

Magnus Westerlund

----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  [phone]
Färögatan 6                | Mobile [phone]
SE-164 80 Stockholm, Sweden| mailto: [e-mail]
----------------------------------------------------------------------