Content-Type: text/plain;
	charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\))
From: =?utf-8?Q?Mirja_K=C3=BChlewind?= <mirja.kuehlewind@tik.ee.ethz.ch>
In-Reply-To: <E2212833-702B-4771-97B4-7682666D527B@ericsson.com>
Date: Mon, 28 Jan 2019 18:12:01 +0100
Cc: "draft-ietf-rmcat-eval-test.all@ietf.org"
 <draft-ietf-rmcat-eval-test.all@ietf.org>, 
 "rmcat@ietf.org" <rmcat@ietf.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <1D385CCF-7035-487F-A60B-E92444A27575@tik.ee.ethz.ch>
References: <D0F202A9-670A-483B-8AE3-F0091A4C30BE@tik.ee.ethz.ch>
 <E2212833-702B-4771-97B4-7682666D527B@ericsson.com>
To: Zaheduzzaman Sarker <zaheduzzaman.sarker@ericsson.com>
Archived-At: <https://mailarchive.ietf.org/arch/msg/rmcat/h_jUEWcakW9HiRnHIQZ-YiCjGS4>
Subject: Re: [rmcat] AD review of draft-ietf-rmcat-eval-test
Precedence: list

Hi Zahed,

I just started the IETF last call as none of these comments where =
blocking. If you quickly want to push a new version with those changes =
that are easy to resolve, just do that, otherwise these things can also =
wait until after the end of IETF last call.

See two comments below.

Mirja


> Am 28.01.2019 um 17:59 schrieb Zaheduzzaman Sarker =
<zaheduzzaman.sarker@ericsson.com>:
>=20
> Hi Mirja,
>=20
> Thanks for your review, and sorry for not an early response. I think =
all of the comments you have can be resolved in the next version. Please =
see inline below.
>=20
> BR
>=20
> Zahed
>=20
> =EF=BB=BFOn 2019-01-24, 21:12, "Mirja K=C3=BChlewind" =
<mirja.kuehlewind@tik.ee.ethz.ch> wrote:
>=20
>    Hi all,
>=20
>    thanks for the well-written doc and shepherd write-up. I think the =
doc is ready for last call but I have a few minor comments/questions and =
I will wait for your potential reply till Monday before I start IETF =
last call.
>=20
>    Some more editorial comments first:
>=20
>    1) Maybe add a informational references to PIE and FQ-Codel =
(section 3)=E2=80=A6?
>=20
> Ok will do that.
>=20
>    2) I=E2=80=99m not sure about the use of normative language in this =
doc. As this doc doesn=E2=80=99t specify anything that is important for =
interoperability or such, I guess you could also just not use any =
normative language at all. However, the way it is used currently seems a =
bit random and in some cases doesn=E2=80=99t really seem to be necessary =
while other cases are missed out. I don=E2=80=99t think that=E2=80=99s a =
bit issue but wanted to discuss this with the authors (and shepherd) =
quickly. Maybe the authors can at least have another look at the doc and =
double-check all used normative language and see if the usage makes =
really sense in each case (or alternatively just remove the normative =
language entirely).
>=20
> Agreed, that the use of normative language need to be consistent. =
Personally, I don=E2=80=99t think we need the normative language, it was =
in fact added after we got some comments (now I don=E2=80=99t remember =
when, some authors may have more information). If is OK with others we =
can just remove the normative language.
>=20
>    3) In section 4.3: "We recommend to use Foreman video sequence.=E2=80=
=9C
>    Maybe add the respective reference here again.
>=20
> ok
>=20
>    4) section 5.5:
>    " Expected behavior: It is expected that the competing flows will
>       converge to bit rates to accommodate all the flows with minimum
>       possible latency and loss.  Specifically, the test introduces =
five
>       media flows at the same time instance.=E2=80=9C
>    Not sure I fully understand what this text is supposed to tell me. =
Especially the second sentence=E2=80=A6 I guess the flows are not =
expected to share the capacity equally but every flow is supposed to get =
some share=E2=80=A6?
>=20
> Hmm, this is odd (specially the last sentence). I think the last =
sentence is supposed to support the uniqueness of the test compared to =
5.2. I will propose text in the next version.
>=20
> Regarding fairness, the test is about the fair share of the link, =
hence they should share the available bandwidth fairly irrespective of =
the RTT.
>=20
>=20
>    5) Figure 6 is a bit misaligned in the middle. Btw. given the =
figures are all too long (see nits: "There are 9 instances of too long =
lines in the document, the longest one being 4 characters in excess of =
72.=E2=80=9C =
https://www.ietf.org/tools/idnits?url=3Dhttps://www.ietf.org/archive/id/dr=
aft-ietf-rmcat-eval-test-08.txt), can you maybe adapt the figures =
accordingly with the next revision..?
>=20
> Yes.=20
>=20
>    6) s/one (1) and long-lived TCP/one (1) long-lived TCP flow/
>=20
>    7) Section 6.1:
>    "The candidate algorithm can use a coupled congestion control
>       mechanism=E2=80=A6=E2=80=9C
>    I recommend to add an informational reference to the coupling RFC.
>=20
>    8) And regarding references in general I=E2=80=99m not sure if =
RFC6679 must be a normative references as this is an optional test, =
however, draft-ietf-rmcat-cc-requirements is pointed to quite often, =
describing some details and might need to be normative.
>=20
> We add this as normative as RFC6679 is required for the test itself. I =
don=E2=80=99t have strong opinion about it.=20
> Again I don=E2=80=99t have any strong opinion about moving req- draft =
to normative ref either. If there is no other comments we can move it.
>=20
>    And finally three more technical questions:
>=20
>    1) Section 3 has a reference to =
draft-ietf-rmcat-video-traffic-model which also defines synthetic =
traffic generation. Then right after this reference there is the Media =
content value specified. However, that value only probably makes sense =
for trace-driven approaches while a synthetic model would need also in =
input parameters as specified in Figure 2 in =
draft-ietf-rmcat-video-traffic-model=E2=80=A6? In general the the =
parameters in section 4.3 seems to rather (only) have the trace-driven =
model in mind.
>=20
> The intention here is to define the targeted video scenario to be use =
for the test run. The movement present in the video sequence impacts how =
the video is encoded ( t.ex. fremesize ), hence it is important to know =
about the nature of the video sequence used. So, no the description is =
not really there only for trace-based model. If you look in to the =
description of the Media content, then is say - "The media content =
should represent a typical video conversational scenario with head and =
shoulder movement". Now for as trace-file based synthetic model one can =
use any video sequence that satisfies that or for statistical synthetic =
model, one need to configure the model to produce similar behaviour.=20
>=20
> I think we need to update the description of Media Content to be =
clearer.=20
>=20
>    2) in Section 4.1 you have:
>    " A.  End-to-end delay for the congestion controlled media =
flow(s).=E2=80=9C
>    Wouldn=E2=80=99t you need to further specify this e.g. say =
_average: delay, as you also do for the queuing delay right afterwards?
>    Or otherwise as the text says that these metric must be logged "at =
a fine enough time granularity=E2=80=9C (whatever that means), does this =
sentence for the queuing delay actually make sense:
>    "+  average over the length of the session.=E2=80=9C?
>    Btw. I guess it would be also useful to discuss a bit further what =
a fine enough time granularity for logging is, e.g. just say that often =
information are logged per packet or e.g. once per ms depending on the =
implementation.
>=20
> Not sure, what exactly you are suggesting here. If you are saying, we =
need to write the different representation of end to end delay metric in =
details, we can perhaps do that. Looking at the different presentation =
of results people have been presented end to end delay in terms of OWD, =
frame delay, IP packet delay etc at the media flow level. we can list =
them all. But Just wondering now if we should really restrict how people =
should present their results. So, far I don=E2=80=99t think we had =
problem in interpreting results.

No, I wanted to say that you probably have multiple samples in your =
logging interval and you need to say if you want to log only the average =
of these samples, or also the min/max, or (what I guess would make more =
sense), log all samples (on the granularity that you can depending on =
where you take the samples), but then it doesn=E2=80=99t make to much =
sense anymore to talk about an logging interval at all.

The second part of the comment was that it would probably be good to say =
more about the granularity of the logging interval/give a =
recommendation, e.g every 100ms or per-packet/per frame, just to say =
something.

>=20
>    3) Also for the converge time in section 4.1, is that just covering =
the converge time at the beginning of the test or also recovering after =
a bandwidth increase/cross traffic decrease. E.g. in the test with short =
TCP cross flows, you could measure convergence every time whenever a =
flow disappears. I think it would be good to clarify that. However, =
effectively converge time is also not something you actually log but =
derive later on from a series of logged rate values=E2=80=A6 I have the =
feeling more explanation is needed here.
>=20
> Do you have any text suggestion here?

Depending what your intention is: =E2=80=9EConvergence time is only log =
at the start of a connection=E2=80=9C or =E2=80=9EConverge time is =
logged every time the network or traffic conditions changes in a test =
until the congestion control behavior has stabilized again.=E2=80=9C

And maybe also: =E2=80=9EIt may not be always possible to log conference =
time during a test, however, it can usually be derived in retrospect =
from logged throughput samples over time.=E2=80=9C


>=20
>    Let me know what you think!
>=20
>    Thanks!
>    Mirja
>=20
>=20
>=20
>=20
>=20