Re: [mpowr] Experiment design

[John C Klensin <[e-mail]>]

Harald,

--On Friday, 09 January, 2004 15:25 -0800 Harald Tveit 
Alvestrand <[e-mail]> wrote:

> When I think of running an experiment, I think of something
> like:
>
> - Define the ruleset we want to test
> - Declare that from March 1(?), in all WGs with the first
> letter from A to K, the new ruleset applies; in WGs from L to
> Z, the old rule of IESG decision applies.
> - At the end of 3 (or 6) months, the WG chairs get asked:
>   - Did you remove anyone from the list?
>   - Was there a time when you wanted to remove a person from
> the
>     list, but did not?
>   - Was there a time when you removed someone from the list,
> and
>     afterwards decided that you made the wrong decision?
>
> That should give us some real data - or some real data saying
> that this isn't a problem......

I'd add two things to your list, which I consider fairly 
important:

	(1) A request to the WG participants for input on
	whether they feel happier or unhappier with the changes
	in place, with "happier" specifically focused on "feel
	that the WG is more effective", "feel that it is easier
	to contribute", "feel that the S/N ratio in the WG's
	discussions is better", etc., and "unhappier" including
	the opposite of these.   We shouldn't just be asking the
	Chairs about impacts.
	
	(2) A mechanism for early discontinuance of the
	"experiment" if there are severe and obvious
	consequences.  While I personally think that is
	unlikely, if the process led to a lot of appeals with
	which the IESG agreed, it would be time to immediately
	stop, and either give it up or readjust the rules in
	some fashion.

Using the mailing list management situation as an example, I 
would consider "fewer disruptions than before and an improved 
S/N ratio" to make the procedural change a wild success.   But 
it would give zero data under your proposal and, indeed, under 
that proposal as written, might convince people that there was 
no problem, since no one was removed nor was there the strong 
temptation to remove anyone.  Similarly, I would consider 
"warned someone in private and they decided to behave, where 
previous warnings had been unheeded" to be a sign of huge 
success, and your list of questions is unlikely to capture that 
one either.

        john

<tirade>
p.s. To anyone inclined to raise the "it isn't really an 
experiment, since there is no design control group and no 
placebo" argument, please give it a rest.  I am _not_ singling 
anyone out here -- that comment is made by someone, usually 
someone different, every time someone uses the word "experiment" 
around the IETF.  Virtually any experiment (e.g., a change in 
the rules with a monitored and evaluated outcome) on a 
social/behavioral system requires working with the subjective 
evaluations of participants and/or observers as to what 
happened.  If you exclude anything that is not subject to a 
carefully managed cross-plot or split-plot design, or even 
everything on which no exogeneous variates can unexpectedly 
occur in the system, then there are no experiments except in 
agriculture, and fewer of those than is generally believed.  In 
more careful designs than I am suggesting above (because I don't 
think the design and implementation time is justified), there 
are sometimes-complex techniques to lay a basis for comparing 
pre- and post-test opinions but they still don't involve design 
control groups or placebo controls.  The latter, in their purest 
form, are applicable only if _all_ variation not measured by the 
experiment can be excluded and that is, in practice, nearly 
impossible (double-blind placebo-based tests are just intended 
to minimize the most obvious causes).

If you need some really good examples from the "experimental" 
literature, read some of the papers evaluating treatments the 
claim to reduce perceived pain levels.  Then, for a change in 
pace, read Wittgenstein's discussions on how someone accurately 
communicates levels of pain to someone else.

I try to discourage my statistician colleagues who are not 
trained in network design from doing their own networks, and 
believe that this community should return the favor.
</tirade>


_______________________________________________
mpowr mailing list
[e-mail]
https://www1.ietf.org/mailman/listinfo/mpowr