Re: [CCAMP] [ippm] [Fwd: IPPM expert review request]

["Weiqiang Sun" <[e-mail]>]

Hi Reinhard,

Thanks for responding. Please see inline.

>
> Thanks for sending this part. See below for my responses.
>
> [RS7]
> Is there any help or suggestion on how to actually compute a usable
> Lambda_m?
> [sunwq]
> No. Depending on the performance of the implementation, the lambda_m can 
> be
> very different, even under the same topology. A proper value of Lambda_m 
> can
> only be tuned in the testing.
>
> [RS7a]
> I am a bit at loss here.
>
> It was my understanding that you use a metric to measure certain network
> quantities, e.g. throughput, set up delays, etc.
> In a second step you'd use these measured metrics to determine whether the
> systems under test meet your previously expected or defined quality
> parameters. If not, you'd make changes to the components or network 
> design,
> measure again, and compare the previous set of data to the one after the
> change and see whether you have improved and are moving into the right
> direction.
>
> You seem to be coming from the other end, i.e. tweak the input parameters 
> to
> suit your system under test.
[sunwq]
OK, I get your point here.
What you said is true. I had thought by saying ``a usable lambda_m", you are 
saying that a lambda_m that is *best achievable* by the DUT/SUT.
If you are expecting certain performance from the DUT/SUT, then very likely 
you will have in mind an "expected" lambda_m, won't you?
The methodology here does not suggest how to select lambda_m. This in fact 
can provide flexibility. Either the test will use an expected lambda_m, or 
tune the lambda_m that reflects the best system performance, depending on 
the purpose of the test. Does this make sense?

>
> [RS8]
> How to compare metrics that have been generated by two different Poisson
> processes, i.e. e.g. two different arrival rates?
> [sunwq]
> No way. Do we need to do that anyway?
>
> [RS8a]
> Pls also see my comments above.
> If you cannot compare two sets of measurements how will you then arrive at
> an objective study of the network components under test?
[sunwq]
Maybe I misunderstood you again here. The Poisson arrival rate is an 
approximation of real traffic load. Of course it is worthwhile to find out 
how DUT/SUT performs when traffic load changes. But what else do you expect 
by comparing two data sets from different traffic load? Did I miss something 
here?

>
>
> [RS9]
> These sample tuples are dependent on the arrival rate Lambda_m. How to
> deduct this input parameter from a resulting sample?
> For instance, if two result sample sets are given, how can we be sure that
> they can be compared?
> [sunwq]
> My understanding of the sample is that it is a sequence of singleton 
> values
> obtained under a set of parameters. The parameters, such as Lambda_m, are
> not visible in the sample. However, when reporting the sample, the
> parameters must also be reported.
>
> [RS9a]
> OK. Noted and understood.
[sunwq]
Great, thanks.

>
>
> [RS10]
> A superposition of two different Poisson processes is being used here.
> How do you map the resulting RESV messages to the correct process, i.e. 
> how
> to differentiate e.g. a late RESV message resulting from third PATH 
> message
> in fourth invocation from an early RESV message solicited by first PATH
> message in fourth invocation?
> [sunwq]
> RESV messages can be differentiated by either MSG ID or non-overlapping 
> LSP
> ID, or combined. The RSVP-TE protocol does that automatically. So I 
> believe
> this is an RSVP-TE implementation issue, rather than a testing methodology
> issue.
>
> [RS10a]
> OK. Noted and understood.
[sunwq]
Great, thanks.

>
> [RS11]
> ????
> [sunwq]
> Well, the full text is like: "The percentile of Metric is defined as: 
> given
> a metric and a percent X between 0% and 100%, the Xth percentile of all 
> the
> dT values in the sample." Would it help a bit if sentence is replaced by 
> the
>
> full text above?
> Please be noted that this text is inherited from the IPPM docs (See RFC
> 2681, page 16, 4.1).
>
> [RS11a]
> The way it was written in the doc it was just a fragment of a sentence.
> Unfortunately RFC 2681, page 16, 4.1. also does not give a proper
> definition, as it too is just a fragmented sentence.
> RFC 2330, p.26, 11.3 gives a better explanation, as:
> "We will use the term 'percentile' to refer to the smallest value of x for
> which the empirical distribution function is greater or equal a given
> percentage"
[sunwq]
Well, to tell the truth I was troubled by this too. Later I realized that 
one reason that might lead to such a definition is that the authors of 2681 
had assumed that ``percentile" is a ``well known" term, and does not 
inheriting its definition from RFC 2330. And quite surprisingly wikipedia 
says "there is no standard definition of percentile." But wikipedia also 
says ``however all definitions yield similar results when the number of 
observations is large."

I will replace the original example with the following to avoid such a 
dilemma (of conflicting calculations in two RFCs).
============original text===============
   Example: suppose we take a sample and the results are: Stream1 = <
   <T1, 100 msec>, <T2, 110 msec>, <T3, undefined>, <T4, 90 msec>, <T5,
   500 msec> >

   Then the 50th percentile would be 110 msec, since 90 msec and 100
   msec are smaller, and 110 and 500 msec are larger (undefined values
   are not counted in).
============end of original text========

============new text===============
   Example: suppose we take a sample and the results are: Stream1 = <
   <T1, 100 msec>, <T2, 110 msec>, <T3, undefined>, <T4, 90 msec>, <T5,
   500 msec>, <T6, 350msec> >

   Then the 60th percentile would be 110 msec, since 90 msec and 100
   msec are smaller, and 110, 350 and 500 msec are larger (undefined values
   are not counted in).

   A more detailed definition of percentile may be found in RFC 2330.
============end of new text========


> [RS12]
> As stipulated in RFC2330 the EDF is defined as a function F(x) which for 
> any
>
> x gives the fractional proportion of the total measurements that were <= 
> x.
> (see RFC2330 11.3 p26)
> Hence F(90)=0.25, F(100)=0.5,F(110)=0.75, F(500)=1 and therefore the 50th
> percentile is defined as 100 as opposed to 110.
> [sunwq]
> Yes you are right with regard to RFC 2330. But we have used exactly the 
> same
>
> example as in RFC 2681 (page 16, section 4.1). The good thing is that the
> difference of the two calculations is very small when the number of
> measurement is large. Change needed?
>
> [RS12a]
> As outlined above RFC 2330 as opposed to RFC2681 gives a proper definition
> of percentile. Therefore we should stick to RFC2330 and change 
> accordingly.
>
> It should be noted though, that also RFC2330 defines percentile 
> differently
> than it is done in the statistical literature, but we can live with this, 
> as
> it makes the definition easier to apply to measurement samples.
Good point, thanks for explaining. I hope the change above is satisfactory 
for you.


Thanks and best regards,
Weiqiang

>
> [------------------End of new comments---------------------]
>
>
> And I realized that the points you have raised covered the main part of 
> the
> text. We would expect your further comments regarding our responses.
> Otherwise we will assume that you are happy with it and will proceed to
> change the text based on the revision you sent in your previous email :)
>
> Thank you again for your time in reviewing the draft so carefully.
>
> Best regards,
> Weiqiang
>
> --
> Weiqiang Sun
> Shanghai Jiao Tong University
> http://front.sjtu.edu.cn/~sunwq/
>
> --------------------------------------------------
> From: "Rschrage" <[e-mail]>
> Sent: Wednesday, August 19, 2009 9:57 PM
> To: "'Weiqiang Sun'" <[e-mail]>; "'Lou Berger'" <[e-mail]>;
> "'Henk Uijterwaal'" <[e-mail]>
> Cc: <[e-mail]>; <[e-mail]>; "'IETF IPPM WG'"
> <[e-mail]>
> Subject: Re: [CCAMP] [ippm] [Fwd: IPPM expert review request]
>
>> Hi all,
>>
>> pls find attached a further updated edition of previous revised doc
>> draft-ietf-ccamp-lsp-dppm-06.
>>
>> I believe this is as far as we can go for the moment before a mutual
>> satisfactory understanding has been reached and a new draft has been
>> issued
>> by the authors.
>>
>> Again, pls feel free to comment.
>>
>> Many thanks.
>> Brgds.
>> Reinhard Schrage
>> Tel: [phone]
>> Mobile: [phone]
>> [e-mail]
>>
>> -----Original Message-----
>> From: [e-mail] [mailto:[e-mail]] On Behalf Of
>> Weiqiang Sun
>> Sent: Mittwoch, 19. August 2009 09:28
>> To: Rschrage; 'Lou Berger'; 'Henk Uijterwaal'
>> Cc: [e-mail]; [e-mail]; 'BRUNGARD, DEBORAH A,
>> ATTLABS'; 'IETF IPPM WG'
>> Subject: Re: [ippm] [Fwd: IPPM expert review request]
>>
>> More responses below:
>>
>> RS4.
>> Should be more specific as to why this metric is no simple function of
>> single uni-directional LSP Setup Delay metric, i.e. why can it not be
>> deduced from single LSP Setup Delay?
>> [sunwq]
>> This point is raised regarding the sentence "The time needed to setup a
>> large number of LSPs during a short time period can not be deduced by
>> single
>>
>> LSP setup delay."
>> One reason is that when a large number of LSPs in being setup during a
>> short
>>
>> period of time, the control plane may be crowded or even overwhelmed by
>> the
>> large number of signaling and routing messages. This may significantly
>> slow
>> down the processing of a particular signaling message, which will results
>> in
>>
>> longer LSP provisioning delays.
>> Are you suggesting that we add similar explanations to the document? We'd
>> love to do that if you feel it is necessary.
>>
>> RS5.
>> The current definition only addresses multiple uni-directional LSP setups
>> between two nodes ID0 and ID1. What about the the case when ID0 is to
>> setup
>> multiple uni-directional LSP setups to different egress nodes ID1, ID2,
>> ID3,.,IDN?
>> [sunwq]
>> Yes, it is a useful case. Likewise you would expect a case to establish a
>> varied number of (instead of one) LSPs to a set of destinations. Our
>> suggestion is that this case is approached by integrating the defined
>> methodologies of single/multiple LSPs provisioning delay, in the form of 
>> a
>> particular testing case, rather than a formal testing methodology. This 
>> is
>> in fact a trade-off between complexity and accuracy.
>>
>> Best regards,
>> Weiqiang
>>
>> --
>> Weiqiang Sun
>> Shanghai Jiao Tong University
>> http://front.sjtu.edu.cn/~sunwq/
>>
>> --------------------------------------------------
>> From: "Weiqiang Sun" <[e-mail]>
>> Sent: Wednesday, August 19, 2009 2:48 PM
>> To: "Rschrage" <[e-mail]>; "'Lou Berger'"
>> <[e-mail]>; "'Henk Uijterwaal'" <[e-mail]>
>> Cc: <[e-mail]>; "'BRUNGARD, DEBORAH A, ATTLABS'"
>> <[e-mail]>; "'IETF IPPM WG'" <[e-mail]>; <[e-mail]>
>> Subject: Re: [ippm] [Fwd: IPPM expert review request]
>>
>>> Hi Reinhard,
>>>
>>> Many thanks for doing the careful review. We appreciate your many 
>>> wording
>>> suggestions. We will go through these one by one and adopt those we feel
>>> appropriate. In this email I will not list all the comments since these
>>> will not lead to major technical changes.
>>>
>>> Below I try to respond to the points you raised in the revision.
>>>
>>> RS1.
>>> This would imply that the metric may produce a range of several values 
>>> at
>>> one time with the minimum value to be taken from that range.
>>> I believe what the authors are trying to say is, that the LSP Setup 
>>> Delay
>>> Metric provides a lower bound on all possible LSP Delay Metrics of
>>> actually instantiated LSPs, or in other words: an actual LSP Delay 
>>> Metric
>>> cannot get smaller than an LSP Setup Delay Metric.
>>> [sunwq]
>>> This point is raised regarding the motivation of the sigleton definition
>>> of Single Uni-directional LSP Setup Delay (ie. 4.1, para 2). We are
>>> trying
>>
>>> to say that the minimum value of this metric reflects the (likely) 
>>> single
>>> lsp setup delay when the control plane is lightly loaded. The formal
>>> definition of the minimum value is in "14.1 The minimum of metric"
>>> In fact we have inherited this from the IPPM documents (see RFC 2681,
>>> page
>>
>>> 2 section 1.1 bullet 3).
>>>
>>> RS2.
>>> How? Should there be a methodology be defined for this?
>>> [sunwq]
>>> This point is raised regarding the sentense in our methodologies
>>> sections - "Make sure that the network has enough resource to set up the
>>> requested LSP."
>>> We have assumed that test personnel should have adequate expertise in
>>> allocating enough resources for the testing. The allocation of resources
>>> for the testing purpose can be very test specific and we believe it is
>>> quite outside the scope of this document.
>>>
>>> RS3.
>>> Too vague? As both timestamps are to be taken on the same ingress node,
>>> they should be taken at the same application/network level.
>>> [sunwq]
>>> This point is raised regarding the sentense "If the corresponding RESV
>>> message arrives within a reasonable period of time, take the timestamp
>>> (T2) as soon as possible upon receipt of the message."
>>> Yes, ideally the timestamp should be taken immediately after the
>>> reception
>>
>>> of the RESV message. The wording "as soon as possible" is used to allow
>>> for some flexibility when RESV message needs to be propagated to certain
>>> modules where timestamp-taking is most appropriate.
>>> And again, this text is inherited from the IPPM documents (see RFC 2681
>>> page 7, bullet 3).
>>>
>>> Thanks again and looking forward to your further comments and revisions.
>>>
>>> Weiqiang
>>>
>>> --
>>> Weiqiang Sun
>>> Shanghai Jiao Tong University
>>> http://front.sjtu.edu.cn/~sunwq/
>>>
>>> --------------------------------------------------
>>> From: "Rschrage" <[e-mail]>
>>> Sent: Tuesday, August 18, 2009 8:23 PM
>>> To: "'Lou Berger'" <[e-mail]>; "'Henk Uijterwaal'"
>>> <[e-mail]>
>>> Cc: <[e-mail]>; "'BRUNGARD, DEBORAH A, ATTLABS'"
>>> <[e-mail]>; <[e-mail]>; "'IETF IPPM WG'" <[e-mail]>;
>>> <[e-mail]>
>>> Subject: RE: [ippm] [Fwd: IPPM expert review request]
>>>
>>>> Hello,
>>>>
>>>> sorry for late response-
>>>>
>>>> Nevertheless here is a first edition of my revision of doc
>>>> draft-ietf-ccamp-lsp-dppm-06.
>>>>
>>>> The doc has been saved in Word 2003 format to easily track changes.
>>>> I am using Kaspersky Anti Virus 2010 software so I trust there should 
>>>> be
>>>> no
>>>> unpleasant surprises when downloading the attached word doc.
>>>>
>>>>
>>>> I thought it would be advisable to first address the suggested comments
>>>> and
>>>> questions upto and including the uni-directional LSP setup delay metric
>>>> and
>>>> after that to continue with the bi-directional and sample definitions 
>>>> as
>>>> covered in the doc.
>>>>
>>>> Please let me have your thoughts on this.
>>>>
>>>> Many thanks.
>>>> Reinhard Schrage
>>>> Tel: [phone]
>>>> Mobile: [phone]
>>>> [e-mail]
>>>>
>>>> -----Original Message-----
>>>> From: [e-mail] [mailto:[e-mail]] On Behalf Of
>>>> Lou
>>>> Berger
>>>> Sent: Dienstag, 28. Juli 2009 14:59
>>>> To: Henk Uijterwaal
>>>> Cc: [e-mail]; BRUNGARD, DEBORAH A, ATTLABS;
>>>> [e-mail]; IETF IPPM WG
>>>> Subject: Re: [ippm] [Fwd: IPPM expert review request]
>>>>
>>>> Hank/Reinhard,
>>>>
>>>> Thank you very much for undertaking this review.  Please cc
>>>> [e-mail] on any comments you may have.
>>>>
>>>> Lou
>>>>
>>>> On 7/28/2009 6:47 AM, Henk Uijterwaal wrote:
>>>>> Lou, others,
>>>>>
>>>>>> We received the request for a review of a document currently under
>>>>>> discussion in the CCAMP WG, please see below.  Is there anybody who
>>>>>> has time to do this review in the near future?
>>>>>
>>>>> Reinhard Schrage ([e-mail]) has voluntered to do
>>>>> this.
>>>>>
>>>>> Reinhard: please post anything you find to both the list and the
>>>>> authors.
>>>>> And thank you for doing this.
>>>>>
>>>>> Henk
>>>>>
>>>>>>
>>>>>> Matt & Henk
>>>>>>
>>>>>> -------- Original Message --------
>>>>>> Subject: IPPM expert review request
>>>>>> Date: Fri, 24 Jul 2009 15:57:41 -0400
>>>>>> From: Lou Berger <[e-mail]>
>>>>>> To: [e-mail]
>>>>>> CC: Brungard, Deborah A, ALABS <[e-mail]>, [e-mail],
>>>>>> zhangguoying <[e-mail]>
>>>>>>
>>>>>> Hi,
>>>>>>     We, the CCAMP WG chairs, would like to request that the IPPM WG
>>>>>> review
>>>>>> a draft that is progressing through the CCAMP WG.  This work applies
>>>>>> IPPM approaches to GMPLS.  The document we'd like reviewed is
>>>>>> available at:
>>>>>>
>>>>>> http://tools.ietf.org/html/draft-ietf-ccamp-lsp-dppm-06
>>>>>>
>>>>>> Is this acceptable?  Can you undertake this review?  Alternatively, 
>>>>>> we
>>>>>> can just last call the document in your WG (it has already passed
>>>>>> CCAMP
>>>>>> WG LC).
>>>>>>
>>>>>> Thank you,
>>>>>> Lou (and Deborah)
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> ippm mailing list
>>>> [e-mail]
>>>> https://www.ietf.org/mailman/listinfo/ippm
>>>>
>> _______________________________________________
>> ippm mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/ippm
>>
>
>
>
>> _______________________________________________
>> CCAMP mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/ccamp
>>
>
>