Measuring IETF and IESG trends (Was: Re: Appeal against IESG blocking DISCUSS on draft-klensin-rfc2821bis)

[Jari Arkko <[e-mail]>]

Bernard, Russ,

I changed the subject line, I think the thread has continued long enough 
:-)  Indeed, I collect a set of measurements. These are based on pulling 
information from the tracker and the documents. The reason for setting 
this up was to try to better understand what is happening in the IETF 
process at various levels: for each document, my own AD work, how the 
IESG works, where is the time spent in the IETF process as a whole.

   http://www.arkko.com/tools/admeasurements/ (start page and disclaimers)
   http://www.arkko.com/tools/admeasurements/stat/base.html (main IESG 
measurements)
   
http://www.arkko.com/tools/lifecycle/draft-ietf-mobike-protocol-timing.html 
(exists for each approved draft)

The data is current and is being updated every week or so. But before 
everyone looks at the graphs and jumps to conclusions, I wanted to 
insert a few disclaimers. First, this is based on tracker data and it 
does not go into that far into the past -- and some of the older data 
might even not be all that accurate. Second, I recently added a lot of 
measurements and I don't yet have 100% confidence that my tool gets 
everything correct. This is particularly true of the items looking at 
various aspects of Discusses. Third, some of the measurements look at 
tracker substates, which are not used by all ADs and no one uses them 
completely accurately. And last but not least, I don't want anyone to 
blindly look at the numbers without considering what's behind them. 
There are significant variations between documents. Tracker status and 
real world situation may be different. Areas differ. There is limited 
amount of information about documents prior to them becoming WG 
documents. And its not necessarily the goal of the IETF to produce a 
large number of documents as fast as it can; I have no way to measure 
quality in these numbers. These are so hard issues that its not even 
clear that the statistics have more value than acting as entertaining 
trivia about the process.

That being said, it is beneficial to understand what is happening and 
what changes are occurring in the process. Or understand where 
improvements might have a negligible vs. visible impact.

One of the things that the IESG has been concerned about recently is 
that the number of outstanding Discusses is growing. We talked about 
this in our May retreat and identified some actions to help reduce this 
problem. For instance, better tool support so that the ADs would better 
see the different things that are waiting for their action, getting the 
document shepherds better involved in the resolution process, informing 
authors how they should respond to Discusses, using RFC Editor notes to 
make small changes when the authors are slow in producing a new version, 
better checks of documents before they come to telechats (e.g., to 
ensure that formal syntax in a document is free of errors), etc. These 
would be in addition to the usual things we'd do, like debate whether 
the Discuss was within the Discuss criteria, whether the issue is real, 
try to ensure that the AD and the authors are being responsive over 
e-mail, etc.

Another interesting area to think about is the time that our processes 
take. For instance, documents that go through me take on the average 
five months from publication request to approval. But there is a lot of 
variation. This time includes everything from AD review, IETF Last Call 
to IESG review and waiting for document revisions, etc. One interesting 
piece of information is that documents that require no revision during 
this process are very rare, but they go through the system about five 
times as fast. If we look at the (unreliable) substates, they appear to 
indicate that the IESG processing time is divided roughly 1:2:1 for 
waiting on AD, authors, and mandatory process waits like last call periods.

I am working to extend the analysis a little bit further by including 
individual draft and WG document stages. You see some of the results of 
this in the third URL above, but I'd like to understand what fraction of 
the overall draft-smith-00 to RFC time is taken by the different stages 
for all IETF work, and how the stages have developed over time.

Comments and suggestions on what would be useful to measure are welcome.

Jari

_______________________________________________
IETF mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/ietf