Re: [Mtgvenue] New Draft: draft-elkins-mtgvenue-participation-metrics

[<[e-mail]>]

Fred,

>Thanks for this. On a quick read, it gives a pretty reasonable summary of the questions.


>One issue I have with metrics is figuring out how to consistently and accurately measure and report them. 

Yes.  This is a big issue.  I am trying to start with written material that can be perused via a tool.  For example, WG minutes, drafts and email list contributions.   I believe that once we agree on the principles (which list, how / if to weight items, then we (someone) can develop software that uses the written material as input.   I wanted to agree on principles first.   And, whether we should even do it!

>For example, your paper says that posting to a working group list is participation but posting to the IETF list is not, and that >specifically the posts that are relevant to an Internet Draft are "participation", while others may not be. That sort of implies >that we should analyze list discussion (probably after the fact, processing archives) looking for posts relevant to an internet >draft, and count those.
>Two issues:

>(1) Some threads put the draft name into the subject line or mention it in the body, but I can't guarantee that all do; >when a thread mutates, the subject line is usually changed, and the new one usually identifies the topic, not the draft >name. So I'm not sure that I can identify the draft a post is about, in an automated way, and I'm not sure I want to impose >the tooling burden to create a way.


Good point but I don't know that we need to be that specific.  Why do we need to have a metric per contributor (participant) per draft?  Why not just per WG email list per contributor?  Then, it doesn't matter what the subject line is.

>(2) IESG Last Calls happen on the IETF List, in order to get opinions from people that have not necessarily been active in >the working group. As a result, there is often discussion of Internet Drafts that happens on the IETF list.


True.  So, maybe contributions to the IETF list need to be counted somehow. 
>So I don't know how to calculate the information that your drafts suggests be used as a metric.

>Rather than add complexity or overhead, I would prefer a simpler metric, one that is a lot more readily calculated and >reported on. If you post to a mailing list @ietf.org, it's IETF participation. The comment may or may not persuade others, >may or may not be seminal, and may or may not be relevant to a specific draft. It may be as short as "+1", with most of the >content implied being in someone else's post and expressing support. But the IETF is first and foremost a body in which >ideas are discussed, and that post is part of the discussion.


Sure.

>I have a similar comment on drafts as participation. As you note, a draft is an extended expression of an idea. It might be >a bad idea or a good idea, it might be mathematically impossible (we have had some of those) or it might reflect >operational realities that non-operators tend to miss, it might be a number of other things. I have heard it said that a >person that knows "how" will always have a job - working for someone that knows "why". If you look at the Internet Drafts >historically, or the statistics of the RFC series, there are drafts and RFCs on both "how" and "why", and the latter are >often the most seminal.

>Yoav commented a few days ago that the drafts that should count as "participation" are those that "progress", which is to >say "become Working Group drafts", or at least promote substantive discussion. I have some sympathy for the position, in >the sense that it is something I think a lot about as a working group chair. However, in terms of participation, I would >think that a person that posts a draft, however poorly or well received, has expended effort in an attempt to participate, >and put an idea out for consideration. That, in my book, is participation.

>I might ask how we would classify the discussion of draft-elkins-mtgvenue-participation-metrics. Imagine that you never >actually post it - someone else posts a better one that you defer to, or you get disgusted with the resulting email >conversation, or whatever other scenario might cause that. By the argument that mailing list discussion of posted drafts is >"participation" and other comments are not, your email, this response, and the others in the thread would not be >"participation". I'm pretty sure that's not what you intend or what I would want, but a strictly legal reading would say >that. It's not uncommon for several individual submissions to bat variations on an idea around, and a subsequent working >group draft combine them. I would think that each of those individual submissions was a contribution to discussion and the >ultimate working group draft. So I would want to count them all as participation.

>So on drafts, I would similarly go for "simple": if you post a draft, you have participated.


I am good on simple but I am trying to do metrics that somehow reflect our intuitive understanding of the level of contribution.  It is of course true that it may be that a one line contribution in one RFC is equivalent in impact to 10 RFCs which see no implementation in reality.   Quality is hard to quantify.

What IS possible to quantify is how many people refer to your work.  I suppose also how many people list you in the acknowledgments.  As you well know, there is much interesting discussion that happens in private before a draft is posted.  Such people, if they don't become co-authors, are often acknowledged.   These IMO are very important contributions.

Again, I am trying to propose metrics that can be gathered if we see drafts, email lists, etc. as data sources which can be mined for information.

>In general (we can discuss this f2f next week if you like) any draft that proposes metrics should describe how they might be >measured, so that we have some idea of the mechanism and hoop-jumping involved.

Yes.  Let's discuss F2F.  I am nearly done with the presentation & will send off to WG chairs later today.

>My two yen.

Thank you for thinking about this problem.

Nalini