Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines-11: (with DISCUSS and COMMENT)

[Benoit Claise <[e-mail]>]

Hi Mirja,
> Hi Benoit,
>
> I would not expect to see a bis doc here. As I said I would rather 
> expect a new, separate document to specify and register the metrics 
> (and providing more details). Given the very different scope and use 
> case, I would not even see it as a problem if those two documents 
> don't align fully.
>
> I can go back to the authors and ask if they would be interested in a 
> directorate review. But I guess that would also mean that they first 
> have to change the text in section 2.X to the format/structure 
> guidelines in RFC6390 (sec 5.4.2 only in this case, I'd say), correct?
Not necessarily.
The only question is that important in my mind is:
Al, assuming that someone would like to register this metric in a 
registry (RFC6390), are they any grey areas in the performance metric 
definitions in the draft?
 From what I understand, a point such this one (from Al) is:

    Because we are using Goodput, G, I take as given that there
    must be a protocol with retransmission capability.
    Otherwise, further simplification is possible (with dummy traffic).

    But yes, Fs and G need to be reported on payload
    at the same layer, so the protocol layer chosen is
    an input parameter for this metric. 


>
> I still have to say that looking at RFC6390 again I find it not fully 
> applicable for this case; e.g. Measurement Timing is not a great fit 
> when you talk about a one time measurement (for one test run) of the 
> flow completion time. I'd fear that using the template provided in 
> RFC6390 would make it rather more confusing than clear.
Fine with that.

Regards, Benoit
>
> Mirja
>
> On 10.06.2016 09:36, Benoit Claise wrote:
>> Thanks Al,
>> This is exactly the type of feedback requested from the Performance 
>> Metric
>> Directorate.
>>
>> Mirja,
>> I buy into your argument:
>>
>>     I guess as soon as we have a registry, maybe there is someone 
>> interest in IPPM to catch up these metrics again and provide a 
>> RFC6390 definition but I would rather not like this document doing it.
>>
>> As such, we probably don't need the RFC6390 template in this 
>> document. Fair
>> enough.
>> However, what would be a pity is that if/when someone would like to 
>> register
>> this metric, we end with questions such Al's one below, because the 
>> metric
>> definitions in the published RFC are not precise. That could result 
>> in a RFCbis.
>>
>> Therefore, going through the Performance Metric Directorate feedback 
>> now is
>> important IMO.
>>
>> Regards, Benoit
>>> Because we are using Goodput, G, I take as given that there
>>> must be a protocol with retransmission capability.
>>> Otherwise, further simplification is possible (with dummy traffic).
>>>
>>> But yes, Fs and G need to be reported on payload
>>> at the same layer, so the protocol layer chosen is
>>> an input parameter for this metric.
>>>
>>> Al
>>>
>>>> -----Original Message-----
>>>> From: Mirja Kuehlewind (IETF) [mailto:[e-mail]]
>>>> Sent: Wednesday, June 08, 2016 7:21 AM
>>>> To: MORTON, ALFRED C (AL)
>>>> Cc:[e-mail];[e-mail]; The IESG; draft-ietf-aqm-
>>>> [e-mail]; Benoit Claise;[e-mail]
>>>> Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-
>>>> guidelines-11: (with DISCUSS and COMMENT)
>>>>
>>>> Actually, it really doesn't matter that much in this case, I’d say. As
>>>> we are talking about a lab environment, you might use dummy traffic 
>>>> that
>>>> has some headers or not, that you might take into account of not, 
>>>> mostly
>>>> depending on which information can be more easily accessed. What is
>>>> important is that you do the same thing for all schemes that you
>>>> compare.
>>>>
>>>> I guess one could add a note that there are different ways to measure
>>>> this and that it is important to measure G at the same layer. Does 
>>>> that
>>>> make sense?
>>>>
>>>> Mirja
>>>>
>>>>
>>>>> Am 08.06.2016 um 13:03 schrieb MORTON, ALFRED C (AL)
>>>> <[e-mail]>:
>>>>> Here's one area which could use more detail:
>>>>>
>>>>>    ...The Flow Completion Time (FCT) is
>>>>>    related to the flow size (Fs) and the goodput for the flow (G) as
>>>>>    follows:
>>>>>
>>>>>    FCT [s] = Fs [Byte] / ( G [Bit/s] / 8 [Bit/Byte] )
>>>>>
>>>>> What protocol layers are included and excluded from Fs?
>>>>>
>>>>> Also, G needs to be measured at the same layer, and the
>>>>> definition in RFC 2647 is a bit vague about layers, too.
>>>>> It would be good to clarify which bytes to count here.
>>>>>
>>>>> Al
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Mirja Kuehlewind (IETF) [mailto:[e-mail]]
>>>>>> Sent: Wednesday, June 08, 2016 5:40 AM
>>>>>> To: MORTON, ALFRED C (AL)
>>>>>> Cc: Benoit Claise;[e-mail];[e-mail];
>>>>>> [e-mail];[e-mail]; The IESG
>>>>>> Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-
>>>>>> guidelines-11: (with DISCUSS and COMMENT)
>>>>>>
>>>>>> Hi Al,
>>>>>>
>>>>>> what kind of detail are you looking for? Because I thought with the
>>>>>> given equation this one was pretty clear.
>>>>>>
>>>>>> Do you have a reference to the benchmarking work?
>>>>>>
>>>>>> Mirja
>>>>>>
>>>>>>
>>>>>>> Am 08.06.2016 um 11:18 schrieb MORTON, ALFRED C (AL)
>>>>>> <[e-mail]>:
>>>>>>> Hi Mirja,
>>>>>>>
>>>>>>> That sounds fairly reasonable to me.
>>>>>>> Would it be possible to ask the authors provide a bit more
>>>>>>> detail on Flow Completion Time?
>>>>>>>
>>>>>>>>>> Flow Completion Time is close to a definition for a new metric,
>>>>>>>>>> and could benefit from more attention, perhaps a few more
>>>> details.
>>>>>>>>>> RFC6390 will provide some areas for improvement.
>>>>>>> I imagine that related benchmarking efforts may wish to measure 
>>>>>>> this
>>>>>>> metric, and there would be independent implementations based on
>>>>>>> the description provided here.
>>>>>>>
>>>>>>> regards from Geneve'
>>>>>>> Al
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Mirja Kuehlewind (IETF) [mailto:[e-mail]]
>>>>>>>> Sent: Wednesday, June 08, 2016 4:46 AM
>>>>>>>> To: MORTON, ALFRED C (AL); Benoit Claise
>>>>>>>> Cc:[e-mail];[e-mail]; The IESG; draft-ietf-aqm-
>>>> eval-
>>>>>>>> [e-mail];[e-mail]
>>>>>>>> Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-
>>>>>>>> guidelines-11: (with DISCUSS and COMMENT)
>>>>>>>>
>>>>>>>> Hi Benoit,
>>>>>>>>
>>>>>>>> I finally had another look at the document as well as at 
>>>>>>>> RFC6390. I
>>>>>>>> guess the metrics in question are Flow completion time (sec 2.1.),
>>>>>> Flow
>>>>>>>> start up time (2.2.) and Packet loss synchronization (2.4.). And
>>>> you
>>>>>> are
>>>>>>>> right that these metric could be see as Application-Specific
>>>>>> Performance
>>>>>>>> Metric as defined in RFC6390. However, I agree with Al that given
>>>> the
>>>>>>>> scope of this document is providing
>>>>>>>> "a generic list of scenarios against which an
>>>>>>>>   AQM proposal should be evaluated, considering both potential
>>>>>>>>   performance gain and safety of deployment.“,
>>>>>>>> I don’t think these metric need to be defined and registered this
>>>>>> way.
>>>>>>>> I guess as soon as we have a registry, maybe there is someone
>>>>>> interest
>>>>>>>> in IPPM to catch up these metrics again and provide a RFC6390
>>>>>> definition
>>>>>>>> but I would rather not like this document doing it.
>>>>>>>>
>>>>>>>> Is that acceptable for you?
>>>>>>>>
>>>>>>>> We could add one more sentence in the abstract to make the 
>>>>>>>> scope on
>>>>>> lab
>>>>>>>> testing during development and before deployment clear. Would that
>>>>>> help?
>>>>>>>> Mirja
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Am 25.05.2016 um 14:22 schrieb Mirja Kuehlewind (IETF)
>>>>>>>> <[e-mail]>:
>>>>>>>>> Hi Al, Benoit, hi all,
>>>>>>>>>
>>>>>>>>> thanks for the feedback. Sorry for me delaying this maybe a 
>>>>>>>>> little
>>>>>> but
>>>>>>>> I need t have another look at the document which will be next week
>>>> at
>>>>>>>> this point. In general I agree that this does not need to only 
>>>>>>>> rely
>>>>>> on
>>>>>>>> registered metrics because is mostly for lab tests; further this
>>>>>> might
>>>>>>>> probably not the right doc to register new metrics. However, I
>>>> would
>>>>>>>> still like to have another look at the doc and see if we can
>>>> improve
>>>>>>>> anything or figure out if any of the ’new’/non-registed metrics
>>>>>>>> should/could be taken up by e.g. ippm.
>>>>>>>>> Mirja
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Am 20.05.2016 um 14:53 schrieb MORTON, ALFRED C (AL)
>>>>>>>> <[e-mail]>:
>>>>>>>>>> All,
>>>>>>>>>> a few replies in-line below,
>>>>>>>>>> Al
>>>>>>>>>>
>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>> From: Benoit Claise [mailto:[e-mail]]
>>>>>>>>>>> Sent: Thursday, May 19, 2016 5:38 AM
>>>>>>>>>>> To: The IESG
>>>>>>>>>>> Cc:[e-mail]; wes@mti-
>>>> systems.com;
>>>>>>>> aqm-
>>>>>>>>>>> [e-mail];[e-mail];[e-mail]; linda
>>>> Dunbar;
>>>>>>>>>>> MORTON, ALFRED C (AL)
>>>>>>>>>>> Subject: Benoit Claise's Discuss on draft-ietf-aqm-eval-
>>>>>> guidelines-
>>>>>>>> 11:
>>>>>>>>>>> (with DISCUSS and COMMENT)
>>>>>>>>>>>
>>>>>>>>>>> Benoit Claise has entered the following ballot position for
>>>>>>>>>>> draft-ietf-aqm-eval-guidelines-11: Discuss
>>>>>>>>>>>
>>>>>>>>>> ...
>>>>>>>>>>> ---------------------------------------------------------------- 
>>>>>>>>>>>
>>>> -- 
>>>>>> -- 
>>>>>>>> -- 
>>>>>>>>>>> DISCUSS:
>>>>>>>>>>> ---------------------------------------------------------------- 
>>>>>>>>>>>
>>>> -- 
>>>>>> -- 
>>>>>>>> -- 
>>>>>>>>>>> Has a RFC6390 performance directorate review done for the 2.X
>>>>>>>> metrics?
>>>>>>>>>>> It
>>>>>>>>>>> should.
>>>>>>>>>> [ACM]
>>>>>>>>>> I reviewed this draft about 18 months ago.
>>>>>>>>>> Mostly, it points to existing RFCs for fundamental metrics,
>>>>>>>>>> and discusses others.  I read this:
>>>>>>>>>> ...This document provides characterization guidelines that
>>>>>>>>>> can be used to assess the deployability of an AQM, whether it is
>>>>>>>>>> candidate for standardization at IETF or not.
>>>>>>>>>> as restricted to lab testing.
>>>>>>>>>>
>>>>>>>>>>> Seehttp://www.ietf.org/iesg/directorate/performance-
>>>> metrics.html
>>>>>>>>>>> I guess that the metrics will be recorded in the future (See
>>>>>>>>>>> draft-ietf-ippm-metric-registry-06
>>>>>>>>>>> ), right?
>>>>>>>>>> [ACM]
>>>>>>>>>> That's up to the authors, they might simply point to
>>>>>>>>>> metrics in the registry contributed by others
>>>>>>>>>> (when following these guidelines at a future time).
>>>>>>>>>>
>>>>>>>>>>> For example, Flow Completion Time and Packet Loss
>>>> Synchronization
>>>>>>>> are
>>>>>>>>>>> new, I believe.
>>>>>>>>>> [ACM]
>>>>>>>>>> Flow Completion Time is close to a definition for a new metric,
>>>>>>>>>> and could benefit from more attention, perhaps a few more
>>>> details.
>>>>>>>>>> RFC6390 will provide some areas for improvement.
>>>>>>>>>>
>>>>>>>>>> Packet loss sync full methodology is described in [JAY006],
>>>>>>>>>> according to the text.
>>>>>>>>>>
>>>>>>>>>>> And some other metrics are already documented in RFC6390
>>>> compliant
>>>>>>>>>>> documents. Pointers should be provided.
>>>>>>>>>> [ACM]
>>>>>>>>>> Most others are discussion sections and provide references.
>>>>>>>>>>
>>>>>>>>>>> See
>>>>>>>>>>> https://tools.ietf.org/html/draft-ietf-xrblock-independent-
>>>> burst-
>>>>>>>> gap-
>>>>>>>>>>> discard-01#appendix-A
>>>>>>>>>>> for an example
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------- 
>>>>>>>>>>>
>>>> -- 
>>>>>> -- 
>>>>>>>> -- 
>>>>>>>>>>> COMMENT:
>>>>>>>>>>> ---------------------------------------------------------------- 
>>>>>>>>>>>
>>>> -- 
>>>>>> -- 
>>>>>>>> -- 
>>>>>>>>>>> - Random Early Detection (RED), BLUE, and Proportional Integral
>>>>>>>>>>> controller (PI)
>>>>>>>>>>> Would you have references?
>>>>>>>>>>>
>>>>>>>>>>> - BDP is mentioned a few times. Please expand.
>>>>>>>>>>>
>>>>>>>>>>> - Glossary section = terminology section, right? If we want to
>>>> be
>>>>>>>>>>> consistent across documents
>>>>>>>>>>>
>>>>>>>>>>> - section 12.2. Why not a MUST below?
>>>>>>>>>>> In order to understand an AQM's deployment considerations and
>>>>>>>>>>> performance under a specific environment, AQM proposals SHOULD
>>>>>>>>>>> describe the parameters that control the macroscopic AQM
>>>> behavior,
>>>>>>>>>>> and identify any parameters that require tuning to operational
>>>>>>>>>>> conditions.
>>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> aqm mailing list
>>>>>>>>> [e-mail]
>>>>>>>>> https://www.ietf.org/mailman/listinfo/aqm
>>
> .
>