[Iccrg] Procedure for obtaining ICCRG reviews

[[e-mail] (Mark Allman)]

> > clear to me (if I put my reviewers hat on for a moment) how to make  
> > that decision in a consistent way - perhaps the cautious thing is  
> > simply to default to 2a if an algorithm passes basic tests.   ID  
> > draft-ietf-tsvwg-cc-alt-04.txt deals with general principles but  
> > these still need to be turned into practice. Worth discussing on the  
> > iccrg surely ?
> 
> Hmm... I honestly think that these general principles are as good as
> it gets - I don't see how we could reasonably agree on more practical,
> detailed guidelines that talk about things like timeout thresholds
> etc, like you mention.
> 
> I think that it's general principles + common sense (which will be
> quite subjective, but this will be okay because we consider you an
> expert :) )

I agree with Michael here, except that I'd change his "common sense" to
"engineering judgment".  CC-ALT is a set of things that researchers
should think about, talk about and generally address when proposing new
CC schemes.  CC-ALT is also a set of things that IETF reviewers should
use to *guide* their thinking about some proposal (CC-ALT is not
exhaustive, however).  I doubt very much that we could come up with a
hard-and-fast checklist of things that would turn the process into
deciding "how safe?" into a mechanical process.  We have the high-level
guidance (IMO, but I co-wrote it, so ...).  As we consider proposals we
need to carefully consider how they *address* the topics in the
high-level guidance and whether we buy that a solid evaluation has been
performed or whether it is somehow lacking.  (And, of course, in the
latter case by describing what we think is lacking about the
evaluation.)

I don't know if that helps or not.

allman



-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 185 bytes
Desc: not available
Url : http://oakham.cs.ucl.ac.uk/pipermail/iccrg/attachments/20070806/363f755e/attachment.bin
>From [e-mail]  Tue Aug  7 00:49:42 2007
From: [e-mail] (Mark Allman)
Date: Tue Aug  7 02:55:13 2007
Subject: [Iccrg] Re: Procedure for obtaining ICCRG reviews
In-Reply-To: <[e-mail]>
Message-ID: <[e-mail]>


> Before I use it for the CTCP draft (which is the only one
> on our table so far), 

Note, as far as I can tell CTCP has not been brought to the IETF.
Perhaps I have that wrong, but it seems that someone should be
announcing such things to the TCPM list at least in parallel with some
ICCRG effort that is supposedly for the IETF's benefit.

> As you probably know, ICCRG has agreed to obtain reviews
> on experimental congestion control proposals before they
> are brought to the IETF (specifically the TCPM group). While
> the competence to actually decide about acceptance or not is
> with TCPM, it is expected that they will take our reviews
> into account. This process is outlined here:
> http://www.ietf.org/IESG/content/ions/drafts/ion-tsv-alt-cc.txt

As far as templates go I think "TCPM" should be an "XXX" because it is
possible for things to be tracked elsewhere within the IETF.  E.g., it
seems likely that TSVWG---instead of TCPM---would handle something like
XCP since it is not TCP.

> If you're interested in doing a review, please send a note
> to Wes and me. Reviews can be sent to the list, or directly
> to us for anonymization before reflecting them out to the
> list, if desired.

Since this is a community effort and not peer review, I'd like to see
your note encourage folks to make their reviews open and signed.  (I.e.,
to use the anonymization scheme as a 'last resort'.)

> * consider not only the draft alone, but also papers referenced
>   therein, where the authors should have carried out a performance 
>   evaluation of their mechanism, including studies which show the

Nit: I'd change "performance evaluation" to "evaluation". 

>   impact of the new mechanism on standard TCP. When looking at
>   such studies, this document is recommended to be used for guidance:
>   http://www.ietf.org/internet-drafts/draft-irtf-tmrg-metrics-09.txt
> 
> Reviews should include one of the following final recommendations:
> 
> * Experimental 1:
>   The proposed mechanism is judged to be safe to deploy for
>   best-effort traffic in the global Internet and further
>   investigated in that environment.
> 
> * Experimental 2 A:
>   While promising, the proposed mechanism is not deemed safe
>   enough for widespread deployment as best-effort traffic on
>   the Internet, but can be specified to facilitate investigations
>   in simulation, testbeds, or controlled environments.
> 
> * Experimental 2 B:
>   The IETF does not yet have sufficient understanding to decide
>   if the algorithm is or is not safe for deployment on the Internet.
> 
> * Informational:
>   The proposed mechanism does not meet the criteria for recommending
>   publication of the draft as an Experimental RFC. It should be
>   considered for publication as an Informational RFC for the benefit
>   of the IETF and IRTF communit, provided that it carries an
>   explicit warning against using the scheme in the global Internet.
>   
> * Not ready:
>   The draft is not ready for publication as an Experimental or
>   Informational RFC.
> 
> A review can be as short as a selection from the list of options,
> although preferably it would include at least some brief
> discussion or justification. No matter how long the feedback
> is, we expect reviewers to have thoroughly read all the
> necessary material.

First, I would note in here that if the TCPM group (or at least this
*participant*) gets a review that is simply a selection from the above
list then that review may not carry much weight.  The more careful,
complete and descriptive a review is the more credence it will be given,
I strongly suspect.  It might be good to note this in your summation
above.

Second, I am not sure if I like the list.  Every time I try to make a
'which option do you like best?' list people always start adding
options.  Maybe I am lousy with lists.  ;)  The CC-ALT document (in the
rfc-ed queue) that Sally and I wrote says that draft authors have to
make statements in their documents about the "safeness" of their scheme
for use in the global Internet.  The text from our document:

    Each alternate congestion control algorithm published is required to
    include a statement in the abstract indicating whether or not the
    proposal is considered safe for use on the Internet.  Each alternate
    congestion control algorithm published is also required to include a
    statement in the abstract describing environments where the protocol
    is not recommended for deployment.  There may be environments where
    the protocol is deemed *safe* for use, but still is not
    *recommended* for use because it does not perform well for the user.

So, perhaps instead of choosing which option the reviewers think is
appropriate from a canned list that might not be quite right for a
particular proposal, the question for reviewers should be whether or not
they buy the statement-on-safeness included in the draft itself.  Of
course, with the reviewers showing their work at how they arrived at
their decision.

allman



-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 185 bytes
Desc: not available
Url : http://oakham.cs.ucl.ac.uk/pipermail/iccrg/attachments/20070806/53f203b0/attachment.bin
>From [e-mail]  Tue Aug  7 07:01:46 2007
From: [e-mail] (Murari Sridharan)
Date: Tue Aug  7 07:04:30 2007
Subject: [Iccrg] Re: Procedure for obtaining ICCRG reviews
In-Reply-To: <[e-mail]>
References: <[e-mail]>
	<[e-mail]>
Message-ID: <[e-mail]>

Mark, we presented CTCP at the Chicago IETF and also sent the draft on the mailing lists.

Thanks
Murari

-----Original Message-----
From: [e-mail] [mailto:[e-mail]] On Behalf Of Mark Allman
Sent: Monday, August 06, 2007 4:50 PM
To: Michael Welzl
Cc: [e-mail]
Subject: [Iccrg] Re: Procedure for obtaining ICCRG reviews


> Before I use it for the CTCP draft (which is the only one on our table
> so far),

Note, as far as I can tell CTCP has not been brought to the IETF.
Perhaps I have that wrong, but it seems that someone should be announcing such things to the TCPM list at least in parallel with some ICCRG effort that is supposedly for the IETF's benefit.

> As you probably know, ICCRG has agreed to obtain reviews on
> experimental congestion control proposals before they are brought to
> the IETF (specifically the TCPM group). While the competence to
> actually decide about acceptance or not is with TCPM, it is expected
> that they will take our reviews into account. This process is outlined
> here:
> http://www.ietf.org/IESG/content/ions/drafts/ion-tsv-alt-cc.txt

As far as templates go I think "TCPM" should be an "XXX" because it is possible for things to be tracked elsewhere within the IETF.  E.g., it seems likely that TSVWG---instead of TCPM---would handle something like XCP since it is not TCP.

> If you're interested in doing a review, please send a note to Wes and
> me. Reviews can be sent to the list, or directly to us for
> anonymization before reflecting them out to the list, if desired.

Since this is a community effort and not peer review, I'd like to see your note encourage folks to make their reviews open and signed.  (I.e., to use the anonymization scheme as a 'last resort'.)

> * consider not only the draft alone, but also papers referenced
>   therein, where the authors should have carried out a performance
>   evaluation of their mechanism, including studies which show the

Nit: I'd change "performance evaluation" to "evaluation".

>   impact of the new mechanism on standard TCP. When looking at
>   such studies, this document is recommended to be used for guidance:
>   http://www.ietf.org/internet-drafts/draft-irtf-tmrg-metrics-09.txt
>
> Reviews should include one of the following final recommendations:
>
> * Experimental 1:
>   The proposed mechanism is judged to be safe to deploy for
>   best-effort traffic in the global Internet and further
>   investigated in that environment.
>
> * Experimental 2 A:
>   While promising, the proposed mechanism is not deemed safe
>   enough for widespread deployment as best-effort traffic on
>   the Internet, but can be specified to facilitate investigations
>   in simulation, testbeds, or controlled environments.
>
> * Experimental 2 B:
>   The IETF does not yet have sufficient understanding to decide
>   if the algorithm is or is not safe for deployment on the Internet.
>
> * Informational:
>   The proposed mechanism does not meet the criteria for recommending
>   publication of the draft as an Experimental RFC. It should be
>   considered for publication as an Informational RFC for the benefit
>   of the IETF and IRTF communit, provided that it carries an
>   explicit warning against using the scheme in the global Internet.
>
> * Not ready:
>   The draft is not ready for publication as an Experimental or
>   Informational RFC.
>
> A review can be as short as a selection from the list of options,
> although preferably it would include at least some brief discussion or
> justification. No matter how long the feedback is, we expect reviewers
> to have thoroughly read all the necessary material.

First, I would note in here that if the TCPM group (or at least this
*participant*) gets a review that is simply a selection from the above list then that review may not carry much weight.  The more careful, complete and descriptive a review is the more credence it will be given, I strongly suspect.  It might be good to note this in your summation above.

Second, I am not sure if I like the list.  Every time I try to make a 'which option do you like best?' list people always start adding options.  Maybe I am lousy with lists.  ;)  The CC-ALT document (in the rfc-ed queue) that Sally and I wrote says that draft authors have to make statements in their documents about the "safeness" of their scheme for use in the global Internet.  The text from our document:

    Each alternate congestion control algorithm published is required to
    include a statement in the abstract indicating whether or not the
    proposal is considered safe for use on the Internet.  Each alternate
    congestion control algorithm published is also required to include a
    statement in the abstract describing environments where the protocol
    is not recommended for deployment.  There may be environments where
    the protocol is deemed *safe* for use, but still is not
    *recommended* for use because it does not perform well for the user.

So, perhaps instead of choosing which option the reviewers think is appropriate from a canned list that might not be quite right for a particular proposal, the question for reviewers should be whether or not they buy the statement-on-safeness included in the draft itself.  Of course, with the reviewers showing their work at how they arrived at their decision.

allman