Re: [rmcat] RMCAT Testing Topology and Initial Testing

["Michael Ramalho (mramalho)" <[e-mail]>]

Mirja,

Thanks for your email.

Comments in-line below (with "MAR:").

Regards,

Michael Ramalho

-----Original Message-----
From: Mirja Kuehlewind [mailto:[e-mail]] 
Sent: Tuesday, May 07, 2013 11:24 AM
To: [e-mail]
Cc: Michael Ramalho (mramalho)
Subject: Re: [rmcat] RMCAT Testing Topology and Initial Testing

Hi Michael,

thanks you for writting this up. What is your plan with this document? Are you planning to merge this document with draft-singh-rmcat-cc-eval-02 or is this the starting point for an own draft?

MAR: The RMCAT design team (as discussed in Orlando meeting) agreed in Orlando to document a network topology model and a testing methodology by which we would compare RMCAT candidates - with the belief that some of the testing criteria would be helpful in defining RMCAT properties. For example, the eventual RMCAT self-fairness definition (to be specified in Randell's requirements document - now a RMCAT WG draft) might well fall out of the experience gathered out of the testing methodology.

MAR: I have not discussed with the design team whether the document I have authored on behalf of the design team should be merged with Varun's (/Joerg's) draft-singh-rmcat-cc-eval-0X draft (not yet adopted by RMCAT). I am open to whatever makes sense. Until we get a little more convergence on the testing methodology, I'd rather not spend too much effort on my ASCII art skills ;-). I do plan to convert it to an easily group-editable form (Google docs or easier-to-deal-with wiki than the RMCAT wiki).

Two comment on the document as an individual contributor:

1) Regarding the passing criteria we should be more careful. I believe you had already a certain solution in mind while writting the document. But in fact the solution does not have to be a pure delay based approach. Probability it actually will not be only delay based because we have to compete with standard TCP. 

MAR: There is no explicit assumption on "a certain solution" during the writing of the document OTHER THAN the working group assumption that when competing with other RMCAT flows that the RMCAT solution would primarily adapt based on delay whenever possible. Of course, if there is a short (in time) queue on the bottleneck link and some subset of the flows progressing through it have sufficiently long RTT - one cannot guarantee timely adaption based solely on delay variation alone. That is why the first test scenario is proposed to have a large queue (in time) in the bottleneck - to test the bounds attainable for a delay-based adaption given some reasonable topology bounds.

MAR: It is well understood that the eventual RMCAT candidate protocol operation MUST also have a loss-based component - as we cannot guarantee that a RMCAT flow would not mix with other loss-only-based flow adaptation transports. These comments were voiced at the RMCAT meeting in Orlando. The fusion of the information available - available delay-variation history and recent loss characteristics - is potentially the most interesting part of the fundamental adaptation algorithm/design.

More specifically regarding the criteria 2 (no packet loss): I don't think this is the right criteria as we only what to have a small (average) queue. 
If there e.g. is a little spike in the queue (maybe in the start-up phase of a competing flow) that might be toleratable. While the packet loss itself (if this is only a few packets) should not be a problem for the application.

MAR: Who is the "we" in "we only what to have a small (average) queue"?

MAR: I agree that the eventual RMCAT protocol should admit a small queue size for RMCAT flow aggregates whenever possible. Thus the eventual protocol should never use all of the (proposed for testing) 500 ms queue for the first testing scenario documented. This is mentioned in the document. However at our last design team meeting it was agreed that at least three queue depths be used in the testing: 1) a nominal network case (of which 500 ms queue depth was agreed, as it should never overflow), 2) a bufferbloated network (over a second of queue depth) and 3) a very low maximum queuing (representative of AQMs such as CODEL or PIE set to ~ 70 ms). I think queue depth scenario 3 above addresses your concern of short-term bursts potentially causing loss on bottlenecks with short queues.

Also regarding the capacity sharing: I don't see fair or equal sharing as a criteria; not even a sharing which is close to equal (as defined in criteria 1). The thing that is important is that each flow gets at least some part of the capacity or maybe there is even something like a minimum rate each flow needs to achieve (as long as N * min_rate < link capacity). I guess it is probably the easiest solution to come up with an algorithm with achieves something like more or less equal sharing but it a too strong requirement (in case someone has a more fancy idea).

MAR: Thank you for your excellent comment. Please join us on the design teams to add your voice (they are bi-weekly, the next one is scheduled for next Monday - details already sent to the list by Mo Zanaty).

To sum up: when formulating evaluation criteria, we should not be more restrictive that the requirements in draft-jesup-rmcat-reqs.

MAR: Agreed - but only when the requirements draft is in WGLC or an RFC ;-). In other words, I believe that the design team is using the requirements draft as a framework until work progresses further.

MAR:  I think it is fair to say that we do not know what are the bounds of fairness possible at this stage. And, for some/most of the RMCAT deployments, we will not have any form of admission control; and thus a given RMCAT endpoint cannot know if sum{min_rate_i} < link capacity ... because a given endpoint cannot know how many other flows i exist (answer: N-1)  or the link capacity of the bottleneck (answer: C) or even the min_rate_i of the other i applications running over RMCAT protocol!.

MAR: Although TCP does a reasonable job at equal sharing considering that individual flows can have vastly different RTTs (with the proviso that long RTTs limit the maximum rate of the long RTT session) - it only does so only in the aggregate average and only over relatively large averaging intervals. That is, an individual TCP session can get "unlucky" (statistically) and stall on any particularly small time frame (that is a primary reason why ABR schemes open more than one TCP). My intuition at this stage is to open up the definition of fairness somewhat (my guess was to within a factor of three) - but that we would prefer the variance of the rate (over any reasonably small averaging interval) less than TCP (for those cases where we are successful in adapting via delay variance - I've got no clue what to require in other traffic mixes).

MAR: Please join us on the design team meetings for your valued input.

2) This (simple) scenario is a good starting point but it only focuses on capacity sharing and small delays. There are many algorithm out there already which can fullfill these requirement (e.g. TCP Vegas). The main challenge we have to worry about is that we also need to compete with TCP. This is not handled at all in your document so far.

MAR: Yours is the first comment on the list to infer a window-based congestion control (TCP Vegas) solution in the group, I also expected TFWC and variants thereof also to be more mentioned. TCP Vegas, by its compatibility to existing on-the-wire TCP constraint, doesn't meet some of our other design goals (e.g., lower rate variability than TCP and less probability of stalling). Some of these design goals have not been incorporated in Randall's draft yet.

MAR: As to the challenge to compete with TCP, I agree. As per the text in the introduction, we plan to "layer in" TCP, LEDBAT and other transports in time as other sources we intend to compete against. As of now, the document only specifies the first of many scenarios. As you and I both know, RMCAT will not attain its desire for low delay when competing with TCP in bottlenecks with large (in time) queues.

MAR: Please join us on the design team - it takes less time than replying to emails ;-) .

Mirja

MAR: I will be generating an updated version of the document prior to the next design team on Monday.

On Friday 05 April 2013 18:54:19 Michael Ramalho (mramalho) wrote:
> RMCAT Design Team,
>
> At the RMCAT Design Team meeting on 28 March, I volunteered to 
> document an initial network topology model and an initial testing 
> methodology model for our RMCAT work. The goal is to have a topology 
> and an associated testing methodology so that we  could determine if 
> the RMCAT requirements are being met.
>
> Attached is my first draft for such a proposal. It is a work in progress.
>
> Comments are welcome, but please be sure to raise a technology-based 
> objection and associated rationale with any item within it you find 
> objectionable.
>
> We have our work cut out for us, as designing an acceptable control 
> system (i.e. rate adaptation algorithm) to meet all of our 
> requirements will be tough. As can be seen, my initial testing 
> proposal only addresses the simplest part of our work: RMCAT self-friendliness.
>
> I would like to poll this audience, when should the next RMCAT design 
> meeting be held?
>
> I would be willing to review various aspects of my proposal at the 
> next meeting.
>
> Comments? Fire away!
>
> Best Regards,
>
> Michael Ramalho



--
-------------------------------------------------------------------
Dipl.-Ing. Mirja Kühlewind
Institute of Communication Networks and Computer Engineering (IKR) University of Stuttgart, Germany Pfaffenwaldring 47, D-70569 Stuttgart

tel: [phone]
email: [e-mail]
web: www.ikr.uni-stuttgart.de
-------------------------------------------------------------------