Re: [aqm] WGLC on draft-ietf-aqm-eval-guidelines

[Wesley Eddy <[e-mail]>]

On 8/18/2015 6:03 PM, Roland Bless wrote:
> Hi,
> 
> Am 10.08.2015 um 15:43 schrieb Wesley Eddy:
>> As chairs, Richard and I would like to start a 2-week working
>> group last call on the AQM characterization guidelines:
>>
>> https://datatracker.ietf.org/doc/draft-ietf-aqm-eval-guidelines/
>>
>> Please make a review of this, and send comments to the list or
>> chairs.  Any comments that you might have will be useful to us,
>> even if it's just to say that you've read it and have no other
>> comments.
> 
> "Unfortunately", we (Polina and I) did a thorough review,
> which is attached. TL;DR: from our point-of-view
> the I-D needs a major revision.
> 


Many thanks for the detailed review.

I think a majority of the comments could be addressed in an update, if
the authors agree.

There were only a couple of the "major issues" that I thought I should
comment on as a co-chair of the WG:


> 3) the overall number of tests and parameter combinations is really
>   high

Are there particular permutations (or classes of permutations) that you
can suggest to remove?  There's a balancing act between including enough
to satisfy people that want to find edge cases and thoroughly
characterize an algorithm, and the desire for a more easily tractable
suite of tests.


> 4) from the discussed end-to-end metrics only latency/goodput metrics
>   are used in the scenarios and for some of the scenarios these metrics
>   are not suitable to show the desired behavior

It would be easier for the editors to improve this if you could suggest
specific metrics to add to specific scenarios, I think.


> 5) some sections in this document (e.g., 7.3, 10, 13) specify requirements
>   for an AQM standard(/draft) and not requirements for a performance
>   evaluation, so these sections should be moved to
[draft-ietf-aqm-recommendation]

That one is now an RFC (7567), so hopefully they're already reflected
if they were critical requirements.

In any case, I agree with you that requirements themselves should not
be conveyed in this document, but rather it should be just aimed at
characterizing algorithm behavior with regard to the requirements
(for ones that are applicable to verification by testing).

-- 
Wes Eddy
MTI Systems