Re: [Tsvwg] Re: Last Call: draft-ietf-tsvwg-cc-alt (Specifying New Congestion Control Algorithms) to BCP

[Pekka Savola <[e-mail]>]

Hi Mark,

Thanks for the prompt response, and sorry for a delay in responding.

On Wed, 23 May 2007, Mark Allman wrote:
>> 1) the document appears to be slightly inconsistent wrt. what it's
>> actually specifying, or there are nuances that may not be sufficiently
>> well articulated.  The introduction speaks of "..evaluating suggested
>> CC algorithms that _significantly differ_ from the general CC
>> principles outlined in [RFC2914]" (emphasis mine).  The abstract does
>> not mention 'significantly differ', and there is Rule (0) which seems
>> to imply that this BCP could be applied both to the mechanisms that
>> significantly differ from the RFC2914 guidelines and other congestion
>> control mechanisms that still honor the RFC 2914 guidelines.  Which is
>> it?  Does it have implications on the different 'tracks'?
>
> I think you are reading a little much into this.  I have just tweaked
> the language in the document to include 'significantly different' in the
> abstract and have changed (0) to this:
>
>    (0) Differences with Congestion Control Principles [RFC2914]
>
>        Proposed congestion control mechanisms that do should include a
>        clear explanation of the deviations from [RFC2914].
>
> Is that more clear?

Yes (though '..that do should..' seemed weird on the first read).

>> Section 2 also says "Each alternate CC algorithm published is
>> required to include a statement in the abstract indicating whether
>> or not the proposal is considered safe for use on the Internet".
>> Which documents this applies to is not clear enough: all IETF
>> documents (through WG or through an AD)? IAB documents?  IRTF
>> documents?  Individual RFC-editor submissions? It is not clear to me
>> whether this document would have the authority (without explicit
>> discussion within the RFC-editor publications constituencies) to
>> impose further requirements on non-IETF document streams especially
>> if it doesn't include 'Updates: 3932' in its header.  I suspect the
>> document only means to affect the IETF RFC publication stream but is
>> not clear enough about it.
>
> The document is intended to apply to IETF-produced documents.  I added
> the following to the 'Status' section.  Does this help?
>
>    Note: we are not changing RFC publication process for non-IETF
>    produced documents (e.g., those from the IRTF or independent
>    RFC-Editor submissions).  However, we would hope the guidelines in
>    this document inform the IESG as they consider whether to add a note
>    to such documents.

Yes.

>> Section 4 only talks of 'minimum requirements for a document to be
>> approved as Experimental with approval for widespread deployment in
>> the global Internet.' and further down "..approval for widespread
>> deployment".  What about the rest?  Also, is omitting "in the global
>> Internet" intentional? The flow of text doesn't go well as it is if
>> that's the case, and if it's intentional, I'd rather use two
>> entirely different wordings to make 'encouraged for widespread
>> deployment' and 'encouraged for widespread deployment in the global
>> Internet' more clearly distinct from each other. (Also, it isn't
>> clear if the text is out of sync regarding guideline (2) compared to
>> what it says in section 3 and what it says here.)
>
> I agree the text is a little weird.  I beat it a little.  Does this:
>
>    The minimum requirements for approval for widespread deployment in
>    the global Internet include guidelines (1) on assessing the impact
>    on standard congestion control, (3) on investigation of the proposed
>    mechanism in a range of environments, guideline (4) on protection
>    against congestion collapse and guideline (8), discussing whether
>    the mechanism allows for incremental deployment.
>
>    For other guidelines, i.e., (2), (5), (6), and (7), evidence
>    that the proposed mechanism has significantly more problems
>    than those of TCP should be a cause for concern in approval for
>    widespread deployment in the global Internet.
>
> work better?

(Maybe you meant to add 'following' before 'guidelines' in the 2nd 
line?)

If 'following' or some such is added, the first paragraph is clearer 
on what the authors and reviewers should do: they should read through 
the listed guidelines and do what's required there.  It might not be 
optimally clear whether the author _must_ do as 'shoulds' say in those 
guidelines or are those just suggestions.  I'd guess you mean either 
"must do" or "must do, or justify why you haven't" but not "may do".

What isn't clear enough is what the authors and reviewers should do 
for the case of 2nd paragraph's guidelines.  Can the authors skip them 
completely, and the burden of proof that the proposal causes 
significantly more problems on the reviewer?

Or do you mean that the proposers should do everything guidelines (2), 
(5), (6) and (7) say, but shortcomings in the results of these 
guidelines (e.g., proposal being only slightly more troublesome than 
TCP) should not block the approval for widespread deployment in the 
global Internet.  Also the same point as above wrt 'shoulds'.

I suspect you mean the latter, but this should be clearer.

>> 2) the document requires that 'serious scientific study .. needs to have
>> been done'.  Yet there is no metrics an outsider could use to evaluate
>> whether this test is met or not.  It may be that TCP congestion
>> control community has de-facto oral tradition what needs to be
>> evaluated before an algorithm can be looked at seriously, but based on
>> this proposed BCP, such metrics are not clear enough.
>>
>> Unless we can define clearer requirements and metrics for evaluation,
>> perhaps the IETF should not attempt to make such an evaluation but
>> leave it to the scientific community.  I'm not sure how the IETF can
>> liaise with the scientific community on that, e.g., whether it's
>> possible to get a consensus statement from IRSG or an IRTF RG or
>> something that could meet this requirement (if we don't want specify
>> these criteria within the IETF).
>
> The IETF and the IRTF have worked out a procedure for working on these
> sorts of things (and Lars has documented it in an ION).

(Note: this is still a draft version if you're referring to
http://www.ietf.org/IESG/content/ions/drafts/ion-tsv-alt-cc.txt)

This model raises one fundamental question issue of the scope of this 
document.

Who should be evaluating section 3 and 4 of this document?  Is this 
solely meant for ICCRG, the IETF or both?  If both, would both parties 
do everything described in those documents?

It would be clearer if the reviewer was ICCRG, and the IETF would not 
attempt to perform the same review, and the IETF wouldn't be allowed 
to second-guess the proposal, e.g., that it's research has not been 
done well enough if it was already positively evaluated by ICCRG.

> At the end of the day, IMO, the IETF needs to be able to make decisions
> about new CC schemes.  The document we wrote lays out a set of areas in
> which authors of such proposals need to provide information to the IETF
> community such that the IETF community can make sound engineering
> judgments about the proposals.  For a document that intends to live for
> a long time it is pretty hard to go beyond "sound scientific evaluation"
> and a big-picture list of *areas* where we know we want information.

Maybe including a reference to this (even changing) list of evaluation 
criteria would help to mitigate my concern of enabling a 'rock fetch'. 
That way the authors and reviewers can look at it in advance to 
strengthen their case and afterwards also make an easier evaluation if 
the review was fair and according to the documented criteria.

>> Also, checklist (2) has "A minimum goal for experimental mechanisms
>> proposed for widespread deployment in the Internet should
>> be that they do not perform significantly worse than TCP
>> in these environments."
>>
>> However, 'these environments' refers to a non-exhaustive list of
>> scenarios.  This checklist does not provide adequate information to
>> evaluate whether sufficient testing in the non-exhaustively defined
>> "difficult environments" has taken place or not.  I do not believe we
>> can require assessments in various environments unless it's better
>> specified what which environmets that various refers to.
>
> You are right that we may want to better enumerate things.  But, we
> cannot necessarily illuminate all such environments that one might
> encounter.  I hacked out the following text (to be added, not replacing
> anything that is there now).
>
>        While it is impossible to enumerate all possible "difficult
>        environments", we note that the IETF has previously grappled
>        with paths with long delays [RFC2488], high delay bandwidth
>        products [RFC3649], high packet corruption rates [RFC3155],
>        packet reordering [RFC4653] and significantly slow links
>        [RFC3150].  Aspects of alternate congestion control that impact
>        networks with these characteristics should be detailed.
>
> Is that better?

Yes, thanks.

What I want to avoid that when someone comes up with a proposal, folks 
that are for some reason opposed to it will try to seek specific (but 
globally not necessarily compelling) scenarios where the proposal 
doesn't behave very well, and argue that that's a major shortcoming. 
I think above should help on making the process more deterministic 
though with some experience it could maybe go even further.

>> 3) The first evaluation criteria includes ".. should be evaluated when
>> competing with standard IETF congestion control."
>>
>> What is that standard IETF congestion control referred to?  RFC 793
>> plus RFC 1122?  These are the only two IETF _Standard_ specifications
>> I can think of. Or does this include proposed standards as well?  What
>> constitutes "IETF congestion control" or "standard congestion control"
>> (in another place) that a CC algorithm should be evaluated against?
>>
>> Please do not cite the TCP roadmap RFC on this.  It's Informational
>> and not an IETF consensus statement, and as such has no authority to
>> define what constitutes "standard congestion control".
>
> Added refs:
>
>        Proposed congestion control mechanisms should be evaluated when
>        competing with standard IETF congestion control
>        [RFC2581,RFC2960,RFC4340].

OK.

>> 4) The first evaluation criteria also includes "We also note that this
>> guideline does not constrain the fairness offered for non-best-effort
>> traffic."
>
> I don't understand your point here.  All we are saying is that if---by
> some means---we know that some flow is not best effort then it can be
> subjected to some other criteria then it need not be constrained by the
> guideline.

What I try to say is that I can't figure out how operationally and 
practically this could be achieved.  Do you have examples in mind how 
how this could apply in some specific scenario?

If not -- and it isn't a practical concern right now -- maybe we 
should not overengineer the BCP to address best-effort vs 
non-best-effort at all.

If yes, there are some approaches to this, I'd be interested in 
hearing how they'd work, but more even more than that, I'd require the 
proposers of such a mechanism to provide an evaluation how reliable 
that knowledge of best vs non-best-effort is.

What I'm afraid of is that the specification would say that it's 
applied for traffic with certain DSCP values but typically that alone 
is not IMHO a very reliable indicator of BE vs non-BE because DSCPs 
can be rewritten and used for other purposes than those envisaged by 
the proposer of the mechanism.

>
>> 5) Normative references is empty, yet this is a BCP document which, among
>> other things, referred to "standard congestion control" without
>> providing a reference (as raised above).  Please move some
>> informational references over here and/or add applicable references.
>
> I have now put the references to RFCs 2581, 2914, 2960 and 4340 under
> normative references.
>
> (And, splitting the references was the dumbest thing we ever did---it
> causes no end to the haggling.)

(Well.. Some will argue that the current IESG's policy requires 
such a split.  Some have interpreted that in certain classes of 
Informational documents, a split may not be necessary, but I 
believe Standards Track and BCPs do need it.)

>> 2.  Status
>>
>> ==> this title seems too short, and a somewhat longer and a more
>> descriptive one would be useful.  Actually the section seems to be a
>> mix and match of different topics and a better organization might help
>> the document considerably.  E.g., if there was a numbered list or
>> subsection of different "publication paths" and descriptions of each
>> the scope of the document and the intent of this section would likely
>> be much clearer.
>
> I changed the title to "Document Status", but I am not inclined to
> re-arrange it further because lots of people have read this now and
> nobody has complained.
>
> The major caveat here is that *I* hacked out the changes described
> above.  Sally may want to wordsmith or just plain disagree.

OK.  I'd have liked to change it more but I realize it's a bit late in 
the process and the other clarifications made above should help 
already.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings