Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines-11: (with DISCUSS and COMMENT)

Here's one area which could use more detail:

   ...The Flow Completion Time (FCT) is
   related to the flow size (Fs) and the goodput for the flow (G) as
   follows:

   FCT [s] = Fs [Byte] / ( G [Bit/s] / 8 [Bit/Byte] )

What protocol layers are included and excluded from Fs?

Also, G needs to be measured at the same layer, and the 
definition in RFC 2647 is a bit vague about layers, too.
It would be good to clarify which bytes to count here.

Al

> -----Original Message-----
> From: Mirja Kuehlewind (IETF) [mailto:ietf@kuehlewind.net]
> Sent: Wednesday, June 08, 2016 5:40 AM
> To: MORTON, ALFRED C (AL)
> Cc: Benoit Claise; wes@mti-systems.com; aqm-chairs@ietf.org;
> aqm@ietf.org; draft-ietf-aqm-eval-guidelines@ietf.org; The IESG
> Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-
> guidelines-11: (with DISCUSS and COMMENT)
> 
> Hi Al,
> 
> what kind of detail are you looking for? Because I thought with the
> given equation this one was pretty clear.
> 
> Do you have a reference to the benchmarking work?
> 
> Mirja
> 
> 
> > Am 08.06.2016 um 11:18 schrieb MORTON, ALFRED C (AL)
> <acmorton@att.com>:
> >
> > Hi Mirja,
> >
> > That sounds fairly reasonable to me.
> > Would it be possible to ask the authors provide a bit more
> > detail on Flow Completion Time?
> >
> >>>> Flow Completion Time is close to a definition for a new metric,
> >>>> and could benefit from more attention, perhaps a few more details.
> >>>> RFC6390 will provide some areas for improvement.
> >
> > I imagine that related benchmarking efforts may wish to measure this
> > metric, and there would be independent implementations based on
> > the description provided here.
> >
> > regards from Geneve'
> > Al
> >
> >> -----Original Message-----
> >> From: Mirja Kuehlewind (IETF) [mailto:ietf@kuehlewind.net]
> >> Sent: Wednesday, June 08, 2016 4:46 AM
> >> To: MORTON, ALFRED C (AL); Benoit Claise
> >> Cc: wes@mti-systems.com; aqm@ietf.org; The IESG; draft-ietf-aqm-eval-
> >> guidelines@ietf.org; aqm-chairs@ietf.org
> >> Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-
> >> guidelines-11: (with DISCUSS and COMMENT)
> >>
> >> Hi Benoit,
> >>
> >> I finally had another look at the document as well as at RFC6390. I
> >> guess the metrics in question are Flow completion time (sec 2.1.),
> Flow
> >> start up time (2.2.) and Packet loss synchronization (2.4.). And you
> are
> >> right that these metric could be see as Application-Specific
> Performance
> >> Metric as defined in RFC6390. However, I agree with Al that given the
> >> scope of this document is providing
> >> "a generic list of scenarios against which an
> >>   AQM proposal should be evaluated, considering both potential
> >>   performance gain and safety of deployment.“,
> >> I don’t think these metric need to be defined and registered this
> way.
> >>
> >> I guess as soon as we have a registry, maybe there is someone
> interest
> >> in IPPM to catch up these metrics again and provide a RFC6390
> definition
> >> but I would rather not like this document doing it.
> >>
> >> Is that acceptable for you?
> >>
> >> We could add one more sentence in the abstract to make the scope on
> lab
> >> testing during development and before deployment clear. Would that
> help?
> >>
> >> Mirja
> >>
> >>
> >>
> >>> Am 25.05.2016 um 14:22 schrieb Mirja Kuehlewind (IETF)
> >> <ietf@kuehlewind.net>:
> >>>
> >>> Hi Al, Benoit, hi all,
> >>>
> >>> thanks for the feedback. Sorry for me delaying this maybe a little
> but
> >> I need t have another look at the document which will be next week at
> >> this point. In general I agree that this does not need to only rely
> on
> >> registered metrics because is mostly for lab tests; further this
> might
> >> probably not the right doc to register new metrics. However, I would
> >> still like to have another look at the doc and see if we can improve
> >> anything or figure out if any of the ’new’/non-registed metrics
> >> should/could be taken up by e.g. ippm.
> >>>
> >>> Mirja
> >>>
> >>>
> >>>
> >>>> Am 20.05.2016 um 14:53 schrieb MORTON, ALFRED C (AL)
> >> <acmorton@att.com>:
> >>>>
> >>>> All,
> >>>> a few replies in-line below,
> >>>> Al
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Benoit Claise [mailto:bclaise@cisco.com]
> >>>>> Sent: Thursday, May 19, 2016 5:38 AM
> >>>>> To: The IESG
> >>>>> Cc: draft-ietf-aqm-eval-guidelines@ietf.org; wes@mti-systems.com;
> >> aqm-
> >>>>> chairs@ietf.org; wes@mti-systems.com; aqm@ietf.org; linda Dunbar;
> >>>>> MORTON, ALFRED C (AL)
> >>>>> Subject: Benoit Claise's Discuss on draft-ietf-aqm-eval-
> guidelines-
> >> 11:
> >>>>> (with DISCUSS and COMMENT)
> >>>>>
> >>>>> Benoit Claise has entered the following ballot position for
> >>>>> draft-ietf-aqm-eval-guidelines-11: Discuss
> >>>>>
> >>>> ...
> >>>>> ------------------------------------------------------------------
> --
> >> --
> >>>>> DISCUSS:
> >>>>> ------------------------------------------------------------------
> --
> >> --
> >>>>>
> >>>>> Has a RFC6390 performance directorate review done for the 2.X
> >> metrics?
> >>>>> It
> >>>>> should.
> >>>> [ACM]
> >>>> I reviewed this draft about 18 months ago.
> >>>> Mostly, it points to existing RFCs for fundamental metrics,
> >>>> and discusses others.  I read this:
> >>>> ...This document provides characterization guidelines that
> >>>> can be used to assess the deployability of an AQM, whether it is
> >>>> candidate for standardization at IETF or not.
> >>>> as restricted to lab testing.
> >>>>
> >>>>> See http://www.ietf.org/iesg/directorate/performance-metrics.html
> >>>>> I guess that the metrics will be recorded in the future (See
> >>>>> draft-ietf-ippm-metric-registry-06
> >>>>> ), right?
> >>>> [ACM]
> >>>> That's up to the authors, they might simply point to
> >>>> metrics in the registry contributed by others
> >>>> (when following these guidelines at a future time).
> >>>>
> >>>>> For example, Flow Completion Time and Packet Loss Synchronization
> >> are
> >>>>> new, I believe.
> >>>> [ACM]
> >>>> Flow Completion Time is close to a definition for a new metric,
> >>>> and could benefit from more attention, perhaps a few more details.
> >>>> RFC6390 will provide some areas for improvement.
> >>>>
> >>>> Packet loss sync full methodology is described in [JAY006],
> >>>> according to the text.
> >>>>
> >>>>> And some other metrics are already documented in RFC6390 compliant
> >>>>> documents. Pointers should be provided.
> >>>> [ACM]
> >>>> Most others are discussion sections and provide references.
> >>>>
> >>>>> See
> >>>>> https://tools.ietf.org/html/draft-ietf-xrblock-independent-burst-
> >> gap-
> >>>>> discard-01#appendix-A
> >>>>> for an example
> >>>>>
> >>>>>
> >>>>> ------------------------------------------------------------------
> --
> >> --
> >>>>> COMMENT:
> >>>>> ------------------------------------------------------------------
> --
> >> --
> >>>>>
> >>>>> - Random Early Detection (RED), BLUE, and Proportional Integral
> >>>>> controller (PI)
> >>>>> Would you have references?
> >>>>>
> >>>>> - BDP is mentioned a few times. Please expand.
> >>>>>
> >>>>> - Glossary section = terminology section, right? If we want to be
> >>>>> consistent across documents
> >>>>>
> >>>>> - section 12.2. Why not a MUST below?
> >>>>> In order to understand an AQM's deployment considerations and
> >>>>> performance under a specific environment, AQM proposals SHOULD
> >>>>> describe the parameters that control the macroscopic AQM behavior,
> >>>>> and identify any parameters that require tuning to operational
> >>>>> conditions.
> >>>>>
> >>>>
> >>>
> >>> _______________________________________________
> >>> aqm mailing list
> >>> aqm@ietf.org
> >>> https://www.ietf.org/mailman/listinfo/aqm
> >