Re: [rmcat] AD review of draft-ietf-rmcat-eval-test

Zaheduzzaman Sarker <zaheduzzaman.sarker@ericsson.com> Mon, 28 January 2019 17:00 UTC

Return-Path: <zaheduzzaman.sarker@ericsson.com>
X-Original-To: rmcat@ietfa.amsl.com
Delivered-To: rmcat@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 22AE7131067 for <rmcat@ietfa.amsl.com>; Mon, 28 Jan 2019 09:00:03 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -8.853
X-Spam-Level:
X-Spam-Status: No, score=-8.853 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-4.553, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=ericsson.com header.b=Dm+wv3Un; dkim=pass (1024-bit key) header.d=ericsson.com header.b=aBJgPY+m
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7fYq33uvDz6o for <rmcat@ietfa.amsl.com>; Mon, 28 Jan 2019 09:00:00 -0800 (PST)
Received: from sessmg23.ericsson.net (sessmg23.ericsson.net [193.180.251.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A7470131062 for <rmcat@ietf.org>; Mon, 28 Jan 2019 08:59:59 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; d=ericsson.com; s=mailgw201801; c=relaxed/relaxed; q=dns/txt; i=@ericsson.com; t=1548694797; x=1551286797; h=From:Sender:Reply-To:Subject:Date:Message-ID:To:CC:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=xwQBLrq5pEaxCCnwGVyKSnfkmWazGdeV/4qczc/szOM=; b=Dm+wv3Unc2hj1DAojiKb7iL6lDZ0IKc55j5E1d4nVAl5do3bSgNQVTUUeCR2D5qY S8VFwVtVef/v3Ykm0KNMYdtDgadisYXkuVtJ9fEP1e6EvnDjMNnpImwjjWQHIRdX j53FiJ8TCcuhi6V03uVsQ+tEdvb73ZTVP9fMuUrLM4c=;
X-AuditID: c1b4fb2d-db5ff7000000062f-18-5c4f350dbaf6
Received: from ESESSMB502.ericsson.se (Unknown_Domain [153.88.183.120]) by sessmg23.ericsson.net (Symantec Mail Security) with SMTP id 98.DF.01583.D053F4C5; Mon, 28 Jan 2019 17:59:57 +0100 (CET)
Received: from ESESSMR502.ericsson.se (153.88.183.110) by ESESSMB502.ericsson.se (153.88.183.120) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1466.3; Mon, 28 Jan 2019 17:59:57 +0100
Received: from ESESBMB503.ericsson.se (153.88.183.170) by ESESSMR502.ericsson.se (153.88.183.110) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1466.3; Mon, 28 Jan 2019 17:59:57 +0100
Received: from EUR02-AM5-obe.outbound.protection.outlook.com (153.88.183.157) by ESESBMB503.ericsson.se (153.88.183.170) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1466.3 via Frontend Transport; Mon, 28 Jan 2019 17:59:56 +0100
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xwQBLrq5pEaxCCnwGVyKSnfkmWazGdeV/4qczc/szOM=; b=aBJgPY+mt52QDy6uV+7nEAdY/RjQPdVxhZ7YUWfiG2lpL+0uUCL2FWX8Lj53kJxL/PbFhPhqTLyXs867kL/MJQgTaEoBo80Ij1knBftfMvxUnRzgkX9Pc7sngt0YzXnrpFT81XoB1s0WXR0mHNM+XnGLVWfaETLp67iX7JxQkG4=
Received: from HE1PR0701MB3020.eurprd07.prod.outlook.com (10.168.93.142) by HE1PR0701MB3033.eurprd07.prod.outlook.com (10.168.93.143) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1580.11; Mon, 28 Jan 2019 16:59:56 +0000
Received: from HE1PR0701MB3020.eurprd07.prod.outlook.com ([fe80::f958:b09:cfd7:5878]) by HE1PR0701MB3020.eurprd07.prod.outlook.com ([fe80::f958:b09:cfd7:5878%6]) with mapi id 15.20.1558.021; Mon, 28 Jan 2019 16:59:56 +0000
From: Zaheduzzaman Sarker <zaheduzzaman.sarker@ericsson.com>
To: Mirja Kühlewind <mirja.kuehlewind@tik.ee.ethz.ch>, "draft-ietf-rmcat-eval-test.all@ietf.org" <draft-ietf-rmcat-eval-test.all@ietf.org>
CC: "rmcat@ietf.org" <rmcat@ietf.org>
Thread-Topic: AD review of draft-ietf-rmcat-eval-test
Thread-Index: AQHUtCEvQvpZHZAhW0OHZlHFEUkaWKXE/kqA
Date: Mon, 28 Jan 2019 16:59:55 +0000
Message-ID: <E2212833-702B-4771-97B4-7682666D527B@ericsson.com>
References: <D0F202A9-670A-483B-8AE3-F0091A4C30BE@tik.ee.ethz.ch>
In-Reply-To: <D0F202A9-670A-483B-8AE3-F0091A4C30BE@tik.ee.ethz.ch>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-MacOutlook/10.15.0.190115
authentication-results: spf=none (sender IP is ) smtp.mailfrom=zaheduzzaman.sarker@ericsson.com;
x-originating-ip: [85.238.211.27]
x-ms-publictraffictype: Email
x-microsoft-exchange-diagnostics: 1; HE1PR0701MB3033; 6:gT8WIgpr/HrG3sGXEYBrGgIC8AB3kIiTnC2sxKFAURMu0k57W74t5MhHA/8Eb5j5OXzVtyKnB05+MiIpIxdGezii1NCZx+s1WEWRHqVDHhOupEQRzQsTOFIV040pjLMs3bPXRmAPyeTTdyrtwBgyxg2F2grkghdbISSPyliCIpK75N69Ky2Er+3wtEFDkP9XNuRdOz3TDQUBl135ZvdSt4dCrl6fhHpW/YCeYdpeS18mlDKgh3GZm6okGbQqBKYOsHVm9T8+ATcJsPUU0vTb9aqYLOVEsT2e128cy70PEENaktjaUJ8fKBb+PuW5JQOakwB9LXAEPIDyCszicOhlfQMbU38+14wdJf2PybbY+PiNAGvKbaGTRcVqVTeaBDyGMaZrp0GHhwYb2vbRdLi6gAPcY7q9KJbLh9xXMbdpom3VXQPD8xlLze3y+LF9GiR+UPCjseTTD2m0Df5IYyt/0g==; 5:XdBtGxm4rlIly/pey36RMe3P3YQRyAlSAjhoXY4hmv7S8mVL4RXx2W9AG1G9xgGh8Tg2x6RimEdzS2cDnqDNseuRn2eTOikNK1SkL5p1a4mR0BoEWhL/vzcH3E4rUv7IXduY4fdEDmaL2n5g/ey09UjG1BNeGdcivKpXQD6qKQ4qHFSgAsZ99ygQZlPZMsMI9V8/RpCLzd/dn6WsmpwJyw==; 7:9YeBmhXpFdukhrWif+RQQv8snVGoVsAmVP94BkLWgEEbkSNmuyyQfQbzV0F982HJuLNKvIqUpe26qbOR6/Tn50LogPjJaGNiQKbZa+NVbi3hElVJOk698mlAdbFQ1q0qsHb0wP+u5DClgfpXMNcHZw==
x-ms-exchange-antispam-srfa-diagnostics: SOS;
x-ms-office365-filtering-correlation-id: bfa610f7-6374-4947-cfd3-08d6854207a7
x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600110)(711020)(4605077)(2017052603328)(7153060)(7193020); SRVR:HE1PR0701MB3033;
x-ms-traffictypediagnostic: HE1PR0701MB3033:
x-microsoft-antispam-prvs: <HE1PR0701MB3033C8BF38C3F1F26023DCC19F960@HE1PR0701MB3033.eurprd07.prod.outlook.com>
x-forefront-prvs: 0931CB1479
x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(346002)(396003)(39860400002)(136003)(376002)(366004)(189003)(199004)(51914003)(53754006)(6486002)(6436002)(66066001)(486006)(83716004)(14454004)(6506007)(186003)(102836004)(76176011)(26005)(97736004)(229853002)(82746002)(446003)(11346002)(66574012)(71200400001)(71190400001)(476003)(2616005)(44832011)(86362001)(3846002)(105586002)(6116002)(6306002)(7736002)(36756003)(110136005)(8936002)(81166006)(4326008)(478600001)(966005)(33656002)(6512007)(305945005)(316002)(58126008)(81156014)(53936002)(14444005)(256004)(2906002)(99286004)(25786009)(2501003)(6246003)(106356001)(68736007)(8676002); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR0701MB3033; H:HE1PR0701MB3020.eurprd07.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1;
received-spf: None (protection.outlook.com: ericsson.com does not designate permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam-message-info: 5475JCPWo2QIP6LrVLKygUtidYAJDu+XhGi/WKp/bfqGveZq0T6RzLJcpU/KK8eIOQmIsx/K87iIKXQ/mJSKyrLaD/t9LlksclC/IwAwaz5HFViY3V+0AtOr0s3j9OrLrgLUdottuf3iHvGElFRnruE20rr8xzWXmFmk31kVdv+gKD0rzU8aX7pbzGITY71G+UzeYKEUi8KaehNXq2cbKv0JkTYE3mbHq0DAYIll8LJUHEwgXhr6h7tSpiQKQClK/KnrGdZBn37yuspxkJvb48mSfoAMHKYsWibiQGIG4oW+ROsk55duYhdsT2R56W3UkPAH7zWZT0xZxsS3qdmZcZ/pzpM6MPTWrRn/u5zX/20Up1url1UQfyd0j3S4d0qgPZ2MO2YBjTPjIdgq6Mkt9gdOmNn3few9ldmXJKtKE9g=
Content-Type: text/plain; charset="utf-8"
Content-ID: <7F128E79507D004DB8792F3EBCA0419F@eurprd07.prod.outlook.com>
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: bfa610f7-6374-4947-cfd3-08d6854207a7
X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Jan 2019 16:59:56.0356 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 92e84ceb-fbfd-47ab-be52-080c6b87953f
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0701MB3033
X-OriginatorOrg: ericsson.com
X-Brightmail-Tracker: H4sIAAAAAAAAA02Sa0hTYRjHec9lHqUDr8vLk7dwkNBl3gU/WGb4QcxLQYnIoIYe1NJNzzGZ 9UUrM3XFJm7m1BQcGGlYoigmZMsMDZEUvA0kcWQWyvqgppm27azo2+///p/ry8OQUiMdwBSq yjhepSySSbyo5uxBjZyNy1RE9v4+Ef+03UTHv+xupOK7F+2S82SK2bxLpIzbtySXiByvhDyu qLCc4yPOXfcq6Hn9mSwxp2nWBzfISvQstQ55MoBjwaLdoZwsxWMILBOaOuTl4G0EVu205J/4 0PUIicJMwHTdD9opKKwjYXF3iRadJgIG9qyUKGwIvhn1DodhJDgBVjsznO8+uBPBpm3G1ZHE J0C3ppM4+ahjkndt87STfXAc6D/tUyJHw05/i4eTKUf87hedqyaLE2F39oY4eBKMNhhJJ3vi C2B6MuJKRdgPdiZ7CLGVPyzZ2glxaQzmkWlSZF9YXz1wtfXFEVC985MWc3PBsFUtEWNCoatq zJ0bDDPt9a6vAHzfA+bm9tyF5GA3GNycDvcste4gK4LhqUq3cQY2G+5SIqvB2j5F6FCU6b8B TY7dSHwSeocjxOcUME7qPUQOhcb6FRez2Bsmmm1UB6KfI1+BE4Ti/OiYcI4vzBUEtSpcxZX1 Ice1vO3/JR9C3d+TLAgzSHaEjTqdqZDSynKhotiCgCFlPqxhPkMhZfOUFbc5Xn2Nv1XECRYU yFAyf3Zf6q2Q4nxlGXeT40o4/q9LMJ4BjhuK/BpiXkmypg7lrAenb5FnX13Miwlb5Q4Jvrd0 ubjqqs10XDf1Jnbp8qR50K7PaolIPHg8u1arTS7Z9ruzMJqKmjODYrpeVLXVzIZkvU8O3M+v ObQ+HFjQBn2cLV1oEDbk2ft9Tcut40MbqsIroUn1bR3+WrXsQSt5LCxNw8oooUAZdYrkBeUf OYTucSkDAAA=
Archived-At: <https://mailarchive.ietf.org/arch/msg/rmcat/pH2ZbemAhvK1OJiG4mLFYLV0li4>
Subject: Re: [rmcat] AD review of draft-ietf-rmcat-eval-test
X-BeenThere: rmcat@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "RTP Media Congestion Avoidance Techniques \(RMCAT\) Working Group discussion list." <rmcat.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rmcat>, <mailto:rmcat-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/rmcat/>
List-Post: <mailto:rmcat@ietf.org>
List-Help: <mailto:rmcat-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rmcat>, <mailto:rmcat-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 28 Jan 2019 17:00:03 -0000

Hi Mirja,

Thanks for your review, and sorry for not an early response. I think all of the comments you have can be resolved in the next version. Please see inline below.

BR

Zahed

On 2019-01-24, 21:12, "Mirja Kühlewind" <mirja.kuehlewind@tik.ee.ethz.ch> wrote:

    Hi all,
    
    thanks for the well-written doc and shepherd write-up. I think the doc is ready for last call but I have a few minor comments/questions and I will wait for your potential reply till Monday before I start IETF last call.
    
    Some more editorial comments first:
    
    1) Maybe add a informational references to PIE and FQ-Codel (section 3)…?

Ok will do that.
    
    2) I’m not sure about the use of normative language in this doc. As this doc doesn’t specify anything that is important for interoperability or such, I guess you could also just not use any normative language at all. However, the way it is used currently seems a bit random and in some cases doesn’t really seem to be necessary while other cases are missed out. I don’t think that’s a bit issue but wanted to discuss this with the authors (and shepherd) quickly. Maybe the authors can at least have another look at the doc and double-check all used normative language and see if the usage makes really sense in each case (or alternatively just remove the normative language entirely).

Agreed, that the use of normative language need to be consistent. Personally, I don’t think we need the normative language, it was in fact added after we got some comments (now I don’t remember when, some authors may have more information). If is OK with others we can just remove the normative language.
    
    3) In section 4.3: "We recommend to use Foreman video sequence.“
    Maybe add the respective reference here again.

ok
    
    4) section 5.5:
    " Expected behavior: It is expected that the competing flows will
       converge to bit rates to accommodate all the flows with minimum
       possible latency and loss.  Specifically, the test introduces five
       media flows at the same time instance.“
    Not sure I fully understand what this text is supposed to tell me. Especially the second sentence… I guess the flows are not expected to share the capacity equally but every flow is supposed to get some share…?

Hmm, this is odd (specially the last sentence). I think the last sentence is supposed to support the uniqueness of the test compared to 5.2. I will propose text in the next version.

Regarding fairness, the test is about the fair share of the link, hence they should share the available bandwidth fairly irrespective of the RTT.

    
    5) Figure 6 is a bit misaligned in the middle. Btw. given the figures are all too long (see nits: "There are 9 instances of too long lines in the document, the longest one being 4 characters in excess of 72.“ https://www.ietf.org/tools/idnits?url=https://www.ietf.org/archive/id/draft-ietf-rmcat-eval-test-08.txt), can you maybe adapt the figures accordingly with the next revision..?

Yes. 
    
    6) s/one (1) and long-lived TCP/one (1) long-lived TCP flow/
    
    7) Section 6.1:
    "The candidate algorithm can use a coupled congestion control
       mechanism…“
    I recommend to add an informational reference to the coupling RFC.
    
    8) And regarding references in general I’m not sure if RFC6679 must be a normative references as this is an optional test, however, draft-ietf-rmcat-cc-requirements is pointed to quite often, describing some details and might need to be normative.

We add this as normative as RFC6679 is required for the test itself. I don’t have strong opinion about it. 
Again I don’t have any strong opinion about moving req- draft to normative ref either. If there is no other comments we can move it.
    
    And finally three more technical questions:
    
    1) Section 3 has a reference to draft-ietf-rmcat-video-traffic-model which also defines synthetic traffic generation. Then right after this reference there is the Media content value specified. However, that value only probably makes sense for trace-driven approaches while a synthetic model would need also in input parameters as specified in Figure 2 in draft-ietf-rmcat-video-traffic-model…? In general the the parameters in section 4.3 seems to rather (only) have the trace-driven model in mind.

The intention here is to define the targeted video scenario to be use for the test run. The movement present in the video sequence impacts how the video is encoded ( t.ex. fremesize ), hence it is important to know about the nature of the video sequence used. So, no the description is not really there only for trace-based model. If you look in to the description of the Media content, then is say - "The media content should represent a typical video conversational scenario with head and shoulder movement". Now for as trace-file based synthetic model one can use any video sequence that satisfies that or for statistical synthetic model, one need to configure the model to produce similar behaviour. 

I think we need to update the description of Media Content to be clearer. 
    
    2) in Section 4.1 you have:
    " A.  End-to-end delay for the congestion controlled media flow(s).“
    Wouldn’t you need to further specify this e.g. say _average: delay, as you also do for the queuing delay right afterwards?
    Or otherwise as the text says that these metric must be logged "at a fine enough time granularity“ (whatever that means), does this sentence for the queuing delay actually make sense:
    "+  average over the length of the session.“?
    Btw. I guess it would be also useful to discuss a bit further what a fine enough time granularity for logging is, e.g. just say that often information are logged per packet or e.g. once per ms depending on the implementation.

Not sure, what exactly you are suggesting here. If you are saying, we need to write the different representation of end to end delay metric in details, we can perhaps do that. Looking at the different presentation of results people have been presented end to end delay in terms of OWD, frame delay, IP packet delay etc at the media flow level. we can list them all. But Just wondering now if we should really restrict how people should present their results. So, far I don’t think we had problem in interpreting results.
    
    3) Also for the converge time in section 4.1, is that just covering the converge time at the beginning of the test or also recovering after a bandwidth increase/cross traffic decrease. E.g. in the test with short TCP cross flows, you could measure convergence every time whenever a flow disappears. I think it would be good to clarify that. However, effectively converge time is also not something you actually log but derive later on from a series of logged rate values… I have the feeling more explanation is needed here.

Do you have any text suggestion here?
    
    Let me know what you think!
    
    Thanks!
    Mirja