RE: IETF Website Redesign Effort

["Hallam-Baker, Phillip" <[e-mail]>]

Actually this represents currently accepted engineering practice in the
usability field.

According to Nielsen you can identify 80% of issues with small scale
studies of 5 people. The point here is not to create a statistically
representative sample, it is to identity the chief issues that may be of
concern.

You should always do a pre-survey before you do the main run or else you
will fail to ask the right questions.

Now I do have a big problem with the practice of publishing 'scientific
studies' on the basis of small sample sizes. You can get a paper
published in the security usability field on the basis of three samples
of 9 persons each. That is actually one of the larger studies. But that
is a different matter and in any case they have rather more
methodological issues to deal with than sample size.



> -----Original Message-----
> From: [e-mail] [mailto:[e-mail]] On 
> Behalf Of Eric Gray
> Sent: Thursday, May 08, 2008 8:09 AM
> To: Mark Nottingham
> Cc: [e-mail]
> Subject: RE: IETF Website Redesign Effort
> 
> Mark,
> 
> 	Unfortunately, the way I read this, it is effectively 
> not the latter.  The impression I am left with is that 10 
> volunteers are being asked for feedback on changes.  Assuming 
> this is an iterative process, their collective feedback could 
> have almost any arbitrary degree of impact on the final 
> result - depending on how well developed the current plan is 
> and how responsive the change makers expect to be to feedback.
> 
> 	Hence the value of Brian's observation that there 
> should be some effort to determine if the general outline of 
> planned changes is something that most people agree to.
> 
> 	Also, as has been politely hinted at already, the 
> sample selection method is likely to "poison" the sample.  
> Why the first 10 starting from a specific message at a 
> specific time convenient to only a narrow set of time zones?  
> Why only 10?  Why publish the names of the 
> victim^H^H^H^H^Holunteers?  None of these seem to be the sort 
> of factors one would like to see in getting a reasonable 
> cross section of the (probably) tens of thousands of people 
> who use the IETF website on a regular basis (where is that 
> visitor counter, or the web-site at the moment, anyway?).
> 
> --
> Eric Gray
> Principal Engineer
> Ericsson  
> 
> > -----Original Message-----
> > From: [e-mail] [mailto:[e-mail]] On 
> > Behalf Of Mark Nottingham
> > Sent: Wednesday, May 07, 2008 9:53 PM
> > To: Russ Housley
> > Cc: [e-mail]
> > Subject: Re: IETF Website Redesign Effort
> > 
> > The message on-announce was worded as if these people would be  
> > *performing* the redesign, but I read your message below as saying  
> > that they'll serve a a pool of testers.
> > 
> > I sincerely hope the truth is closer to the latter...
> > 
> > 
> > On 08/05/2008, at 6:11 AM, Russ Housley wrote:
> > 
> > > Stephane:
> > >
> > >>> We are inviting the first ten (10) interested members 
> of the IETF
> > >>> community who respond to this email to become a part of 
> > the website
> > >>> redesign team. If you are interested in assisting with 
> > this effort,
> > >>> please respond to this email as soon as possible.
> > >>
> > >> It is no longer "rough consensus and running code", it is 
> > now "quick
> > >> click and no SMTP latency".
> > >
> > > This approach was taken at my suggestion.
> > >
> > > These early reviewers are so that the Secretariat can get some
> > > feedback about the new design without have a large number 
> of people
> > > pile on.  Think of it as a product test group.  Once 
> these folks are
> > > comfortable that the new design is an improvement, there 
> will be an
> > > opportunity for broader comment.
> > >
> > > Russ
> > > IETF Chair
> > >
> > >
> > > _______________________________________________
> > > IETF mailing list
> > > [e-mail]
> > > https://www.ietf.org/mailman/listinfo/ietf
> > 
> > 
> > --
> > Mark Nottingham     http://www.mnot.net/
> > 
> > _______________________________________________
> > IETF mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/ietf
> > 
> _______________________________________________
> IETF mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/ietf
> 
_______________________________________________
IETF mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/ietf