Re: [rmcat] AD review of draft-ietf-rmcat-eval-test

[Zaheduzzaman Sarker <[e-mail]>]

Hi Mirja,

OK, I will give it a go ASAP. See inline below.

BR
Zahed

On 2019-01-28, 18:12, "Mirja Kühlewind" <[e-mail]> wrote:

    Hi Zahed,
    
    I just started the IETF last call as none of these comments where blocking. If you quickly want to push a new version with those changes that are easy to resolve, just do that, otherwise these things can also wait until after the end of IETF last call.
    
    See two comments below.
    
    Mirja
    
    
    > Am 28.01.2019 um 17:59 schrieb Zaheduzzaman Sarker <[e-mail]>:
    > 
    > Hi Mirja,
    > 
    > Thanks for your review, and sorry for not an early response. I think all of the comments you have can be resolved in the next version. Please see inline below.
    > 
    > BR
    > 
    > Zahed
    > 
    > On 2019-01-24, 21:12, "Mirja Kühlewind" <[e-mail]> wrote:
    > 
    >    Hi all,
    > 
    >    thanks for the well-written doc and shepherd write-up. I think the doc is ready for last call but I have a few minor comments/questions and I will wait for your potential reply till Monday before I start IETF last call.
    > 
    >    Some more editorial comments first:
    > 
    >    1) Maybe add a informational references to PIE and FQ-Codel (section 3)…?
    > 
    > Ok will do that.
    > 
    >    2) I’m not sure about the use of normative language in this doc. As this doc doesn’t specify anything that is important for interoperability or such, I guess you could also just not use any normative language at all. However, the way it is used currently seems a bit random and in some cases doesn’t really seem to be necessary while other cases are missed out. I don’t think that’s a bit issue but wanted to discuss this with the authors (and shepherd) quickly. Maybe the authors can at least have another look at the doc and double-check all used normative language and see if the usage makes really sense in each case (or alternatively just remove the normative language entirely).
    > 
    > Agreed, that the use of normative language need to be consistent. Personally, I don’t think we need the normative language, it was in fact added after we got some comments (now I don’t remember when, some authors may have more information). If is OK with others we can just remove the normative language.
    > 
    >    3) In section 4.3: "We recommend to use Foreman video sequence.“
    >    Maybe add the respective reference here again.
    > 
    > ok
    > 
    >    4) section 5.5:
    >    " Expected behavior: It is expected that the competing flows will
    >       converge to bit rates to accommodate all the flows with minimum
    >       possible latency and loss.  Specifically, the test introduces five
    >       media flows at the same time instance.“
    >    Not sure I fully understand what this text is supposed to tell me. Especially the second sentence… I guess the flows are not expected to share the capacity equally but every flow is supposed to get some share…?
    > 
    > Hmm, this is odd (specially the last sentence). I think the last sentence is supposed to support the uniqueness of the test compared to 5.2. I will propose text in the next version.
    > 
    > Regarding fairness, the test is about the fair share of the link, hence they should share the available bandwidth fairly irrespective of the RTT.
    > 
    > 
    >    5) Figure 6 is a bit misaligned in the middle. Btw. given the figures are all too long (see nits: "There are 9 instances of too long lines in the document, the longest one being 4 characters in excess of 72.“ https://www.ietf.org/tools/idnits?url=https://www.ietf.org/archive/id/draft-ietf-rmcat-eval-test-08.txt), can you maybe adapt the figures accordingly with the next revision..?
    > 
    > Yes. 
    > 
    >    6) s/one (1) and long-lived TCP/one (1) long-lived TCP flow/
    > 
    >    7) Section 6.1:
    >    "The candidate algorithm can use a coupled congestion control
    >       mechanism…“
    >    I recommend to add an informational reference to the coupling RFC.
    > 
    >    8) And regarding references in general I’m not sure if RFC6679 must be a normative references as this is an optional test, however, draft-ietf-rmcat-cc-requirements is pointed to quite often, describing some details and might need to be normative.
    > 
    > We add this as normative as RFC6679 is required for the test itself. I don’t have strong opinion about it. 
    > Again I don’t have any strong opinion about moving req- draft to normative ref either. If there is no other comments we can move it.
    > 
    >    And finally three more technical questions:
    > 
    >    1) Section 3 has a reference to draft-ietf-rmcat-video-traffic-model which also defines synthetic traffic generation. Then right after this reference there is the Media content value specified. However, that value only probably makes sense for trace-driven approaches while a synthetic model would need also in input parameters as specified in Figure 2 in draft-ietf-rmcat-video-traffic-model…? In general the the parameters in section 4.3 seems to rather (only) have the trace-driven model in mind.
    > 
    > The intention here is to define the targeted video scenario to be use for the test run. The movement present in the video sequence impacts how the video is encoded ( t.ex. fremesize ), hence it is important to know about the nature of the video sequence used. So, no the description is not really there only for trace-based model. If you look in to the description of the Media content, then is say - "The media content should represent a typical video conversational scenario with head and shoulder movement". Now for as trace-file based synthetic model one can use any video sequence that satisfies that or for statistical synthetic model, one need to configure the model to produce similar behaviour. 
    > 
    > I think we need to update the description of Media Content to be clearer. 
    > 
    >    2) in Section 4.1 you have:
    >    " A.  End-to-end delay for the congestion controlled media flow(s).“
    >    Wouldn’t you need to further specify this e.g. say _average: delay, as you also do for the queuing delay right afterwards?
    >    Or otherwise as the text says that these metric must be logged "at a fine enough time granularity“ (whatever that means), does this sentence for the queuing delay actually make sense:
    >    "+  average over the length of the session.“?
    >    Btw. I guess it would be also useful to discuss a bit further what a fine enough time granularity for logging is, e.g. just say that often information are logged per packet or e.g. once per ms depending on the implementation.
    > 
    > Not sure, what exactly you are suggesting here. If you are saying, we need to write the different representation of end to end delay metric in details, we can perhaps do that. Looking at the different presentation of results people have been presented end to end delay in terms of OWD, frame delay, IP packet delay etc at the media flow level. we can list them all. But Just wondering now if we should really restrict how people should present their results. So, far I don’t think we had problem in interpreting results.
    
    No, I wanted to say that you probably have multiple samples in your logging interval and you need to say if you want to log only the average of these samples, or also the min/max, or (what I guess would make more sense), log all samples (on the granularity that you can depending on where you take the samples), but then it doesn’t make to much sense anymore to talk about an logging interval at all.
    
    The second part of the comment was that it would probably be good to say more about the granularity of the logging interval/give a recommendation, e.g every 100ms or per-packet/per frame, just to say something.

OK understood. Right now, I am thinking it is best to describe (at a higher level) what metric we would like to see when people present their results and leave the logging intensity open to the proponents. Does this make sense? If yes, then I will update the text accordingly. 
    
    > 
    >    3) Also for the converge time in section 4.1, is that just covering the converge time at the beginning of the test or also recovering after a bandwidth increase/cross traffic decrease. E.g. in the test with short TCP cross flows, you could measure convergence every time whenever a flow disappears. I think it would be good to clarify that. However, effectively converge time is also not something you actually log but derive later on from a series of logged rate values… I have the feeling more explanation is needed here.
    > 
    > Do you have any text suggestion here?
    
    Depending what your intention is: „Convergence time is only log at the start of a connection“ or „Converge time is logged every time the network or traffic conditions changes in a test until the congestion control behavior has stabilized again.“
    
    And maybe also: „It may not be always possible to log conference time during a test, however, it can usually be derived in retrospect from logged throughput samples over time.“

The usual case would be  to log bitrate and then look into the convergence on a time series plot. At least that is how it is done for our tests. The current text does not tell how to do it rather what it is.  We can add that, one need to show each convergence period (i.e. after each time flows  fluctuate from steady state) present during the period test. Will that work?

BR
Zahed