Re: Survey design and interpretation (was: Re: Final reminder: Please fill out our survey on IETF authoring tools)

[[e-mail]]

John C Klensin <[e-mail]> wrote:

> I filled the survey out last night.

I filled it out the other day, although after completing it I wondered why
I bothered.

> Partially prompted by other
> recent events and discussions, I have several comments and
> thoughts that I think deserve community, as well as LLC,
> attention.  So, while I would normally send a note like this off
> list (as I have done in the past), I think this one needs to be
> public.  In no particular order:

> (1) The experience would presumably be different for those for
> whom the number of I-Ds or RFCs is small, but it took me close
> to 45 minutes to make even the crudest of estimates for the
> first page.

I gave up on figuring out accurate values a little sooner than you did.

> Just as some people (not a random sample of the
> community) find my long analytical notes discouraging and won't
> read to the end (or even start), surveys that require that sort
> of time investment will discourage some people (not a random
> sample of the community) from completing them and may even
> discourage them from opening your surveys in the future.  That
> means non-representative results and a survey that requires
> great care to interpret especially in a way that does not just
> reinforce prior biases or assumptions.

It is necessarily the case that anyone who has authored/coauthored a fair
number of drafts and RFCs has arrived at a process that works for them - and as
you note, this includes always having someone else do the editing. What is less
clear is that knowledge of the process someone like you or I use - which owes a
lot more to to historical factors than to any rational selection process
conducted ab inito - is of any use in deciding the direction to take on
future tool development.

Which is why if I was doing this I would be far more interested in the views of
new and even prospective contributers.

> (2) Again probably different for those with small numbers of
> I-Ds and, especially, RFCs but for those of us with large
> numbers of the latter, the survey is not, as claimed above,
> anonymous.  You don't need to collect IP addresses or data about
> computers - the number of people who are authors or co-authors
> of 51+ (if I remember correctly) RFCs is small enough that
> examination of answers to a few questions after that would make
> identifying us rather easy.

(It was at the point of trying to determine if I had crossed the 50 threshhold
that I decided to stop googling and start guessing.)

And perhaps more to the point, why ask the question at all? Does authoring more
stuff mean you're better judge of the tools to use? Or worse?

> (3) Our rules (or lack thereof) about RFC and I-D "authorship"
> sometimes don't predict well to actual construction of
> documents.  There are exceptions, but the typical number of
> people who actually interact with the tools and key or copy
> words into their machine-usable representations is often one
> independent of the number of listed authors.  When you include
> someone in a survey like this who has not actually interacted
> with a document or tools at a technical level, you get weird
> results if they complete the survey and an artificially low
> response rate if they do not.  At the other extreme, many of us
> have helped relative newcomers by providing outline I-Ds in
> xml2rfc form based on their notes but are not listed as authors
> or even contributors -- efforts that don't count for survey
> purposes (I would not be surprised to learn that similar
> outlines have been provided in MS Word).  And, fwiw, the larger
> amount of understanding required of a newcomer to flesh out an
> outline in xml2rfc v3 as compared to v2 is one objection to the
> new format that has not been discussed enough.

Having just embarked on a couple of new drafts, I've experienced this 
directly.

> I'm not going to go on further except to say that, for those of
> us who are trying to do technical work in the IETF, surveys like
> this (and at least some of the "Consultations" you have posted)
> take time away from that available to do that work.   That might
> be worthwhile if the surveys/ consultations are infrequent and
> designed well and carefully enough to inspire confidence in
> their interpretation and use.  But, even then, I suggest that at
> current levels they may be interfering with progress on
> standards-related activities, which is one thing the LLC, at
> least IMO, should absolutely not be doing.

+1

				Ned