Re: [Mtgvenue] New Draft: draft-elkins-mtgvenue-participation-metrics

["Fred Baker (fred)" <[e-mail]>]

Thanks for this. On a quick read, it gives a pretty reasonable summary of the questions.

One issue I have with metrics is figuring out how to consistently and accurately measure and report them. For example, your paper says that posting to a working group list is participation but posting to the IETF list is not, and that specifically the posts that are relevant to an Internet Draft are "participation", while others may not be. That sort of implies that we should analyze list discussion (probably after the fact, processing archives) looking for posts relevant to an internet draft, and count those.

Two issues:

(1) Some threads put the draft name into the subject line or mention it in the body, but I can't guarantee that all do; when a thread mutates, the subject line is usually changed, and the new one usually identifies the topic, not the draft name. So I'm not sure that I can identify the draft a post is about, in an automated way, and I'm not sure I want to impose the tooling burden to create a way.

(2) IESG Last Calls happen on the IETF List, in order to get opinions from people that have not necessarily been active in the working group. As a result, there is often discussion of Internet Drafts that happens on the IETF list.

So I don't know how to calculate the information that your drafts suggests be used as a metric.

Rather than add complexity or overhead, I would prefer a simpler metric, one that is a lot more readily calculated and reported on. If you post to a mailing list @ietf.org, it's IETF participation. The comment may or may not persuade others, may or may not be seminal, and may or may not be relevant to a specific draft. It may be as short as "+1", with most of the content implied being in someone else's post and expressing support. But the IETF is first and foremost a body in which ideas are discussed, and that post is part of the discussion.


I have a similar comment on drafts as participation. As you note, a draft is an extended expression of an idea. It might be a bad idea or a good idea, it might be mathematically impossible (we have had some of those) or it might reflect operational realities that non-operators tend to miss, it might be a number of other things. I have heard it said that a person that knows "how" will always have a job - working for someone that knows "why". If you look at the Internet Drafts historically, or the statistics of the RFC series, there are drafts and RFCs on both "how" and "why", and the latter are often the most seminal.

Yoav commented a few days ago that the drafts that should count as "participation" are those that "progress", which is to say "become Working Group drafts", or at least promote substantive discussion. I have some sympathy for the position, in the sense that it is something I think a lot about as a working group chair. However, in terms of participation, I would think that a person that posts a draft, however poorly or well received, has expended effort in an attempt to participate, and put an idea out for consideration. That, in my book, is participation.


I might ask how we would classify the discussion of draft-elkins-mtgvenue-participation-metrics. Imagine that you never actually post it - someone else posts a better one that you defer to, or you get disgusted with the resulting email conversation, or whatever other scenario might cause that. By the argument that mailing list discussion of posted drafts is "participation" and other comments are not, your email, this response, and the others in the thread would not be "participation". I'm pretty sure that's not what you intend or what I would want, but a strictly legal reading would say that. It's not uncommon for several individual submissions to bat variations on an idea around, and a subsequent working group draft combine them. I would think that each of those individual submissions was a contribution to discussion and the ultimate working group draft. So I would want to count them all as participation.

So on drafts, I would similarly go for "simple": if you post a draft, you have participated.

In general (we can discuss this f2f next week if you like) any draft that proposes metrics should describe how they might be measured, so that we have some idea of the mechanism and hoop-jumping involved.

My two yen.