Re: [rmcat] Roman Danyliw's Discuss on draft-ietf-rmcat-eval-criteria-12: (with DISCUSS and COMMENT)

Roman Danyliw <rdd@cert.org> Wed, 18 March 2020 23:18 UTC

Return-Path: <rdd@cert.org>
X-Original-To: rmcat@ietfa.amsl.com
Delivered-To: rmcat@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 587973A1E14; Wed, 18 Mar 2020 16:18:56 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.099
X-Spam-Level:
X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=cert.org
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PVov0bhnnLbp; Wed, 18 Mar 2020 16:18:53 -0700 (PDT)
Received: from veto.sei.cmu.edu (veto.sei.cmu.edu [147.72.252.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 90F743A09EE; Wed, 18 Mar 2020 16:18:53 -0700 (PDT)
Received: from delp.sei.cmu.edu (delp.sei.cmu.edu [10.64.21.31]) by veto.sei.cmu.edu (8.14.7/8.14.7) with ESMTP id 02INIhpF046791; Wed, 18 Mar 2020 19:18:43 -0400
DKIM-Filter: OpenDKIM Filter v2.11.0 veto.sei.cmu.edu 02INIhpF046791
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cert.org; s=yc2bmwvrj62m; t=1584573523; bh=RbQM/ctQxRBfpBwdSaGs+oMDbXh+Sq78qburUqV9UGs=; h=From:To:CC:Subject:Date:References:In-Reply-To:From; b=NdIZ04dmIkkZrSLCd8k9XtcKKdHCJgjgu78Sl208amG4qneUKN0p2bMTb3nrAyA3U jQu8yVzb4OcAER3G4JjzcvfV7cwyyQmJRKYx4ABGpj0+Dm3qIfSCKgxX2qAw1qR0kp xQceISJGx+eCxFoYsICNt8UOsxZqC37DueakC/HI=
Received: from CASCADE.ad.sei.cmu.edu (cascade.ad.sei.cmu.edu [10.64.28.248]) by delp.sei.cmu.edu (8.14.7/8.14.7) with ESMTP id 02INIeoM005257; Wed, 18 Mar 2020 19:18:41 -0400
Received: from MARCHAND.ad.sei.cmu.edu ([10.64.28.251]) by CASCADE.ad.sei.cmu.edu ([10.64.28.248]) with mapi id 14.03.0487.000; Wed, 18 Mar 2020 19:18:40 -0400
From: Roman Danyliw <rdd@cert.org>
To: Joerg Ott <ott@in.tum.de>, The IESG <iesg@ietf.org>
CC: "rmcat-chairs@ietf.org" <rmcat-chairs@ietf.org>, "varun.singh@iki.fi" <varun.singh@iki.fi>, "draft-ietf-rmcat-eval-criteria@ietf.org" <draft-ietf-rmcat-eval-criteria@ietf.org>, Martin Stiemerling <mls.ietf@gmail.com>, "rmcat@ietf.org" <rmcat@ietf.org>, Colin Perkins <csp@csperkins.org>
Thread-Topic: Roman Danyliw's Discuss on draft-ietf-rmcat-eval-criteria-12: (with DISCUSS and COMMENT)
Thread-Index: AQHV8Ww/wwzONoRIjkWQ0TdfC+oiIahA+bgAgA4aGNA=
Date: Wed, 18 Mar 2020 23:18:39 +0000
Message-ID: <359EC4B99E040048A7131E0F4E113AFC0216FA0696@marchand>
References: <158324752207.7683.15663429456876412351@ietfa.amsl.com> <05afa3e4-786b-f8b0-670b-510cfd961482@in.tum.de>
In-Reply-To: <05afa3e4-786b-f8b0-670b-510cfd961482@in.tum.de>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [10.64.22.6]
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
Archived-At: <https://mailarchive.ietf.org/arch/msg/rmcat/-4nQd3zCroflklB971bXhR9vuMU>
Subject: Re: [rmcat] Roman Danyliw's Discuss on draft-ietf-rmcat-eval-criteria-12: (with DISCUSS and COMMENT)
X-BeenThere: rmcat@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "RTP Media Congestion Avoidance Techniques \(RMCAT\) Working Group discussion list." <rmcat.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rmcat>, <mailto:rmcat-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/rmcat/>
List-Post: <mailto:rmcat@ietf.org>
List-Help: <mailto:rmcat-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rmcat>, <mailto:rmcat-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 18 Mar 2020 23:18:57 -0000

Hi Joerg!

Thanks for the update.  -13 addresses my DISCUSS points.  One non-blocking comment on a COMMENT inline ...

Thanks,
Roman

-----Original Message-----
From: iesg <iesg-bounces@ietf.org> On Behalf Of Joerg Ott
Sent: Monday, March 9, 2020 3:56 PM
To: Roman Danyliw <rdd@cert.org>; The IESG <iesg@ietf.org>
Cc: rmcat-chairs@ietf.org; varun.singh@iki.fi; draft-ietf-rmcat-eval-criteria@ietf.org; Martin Stiemerling <mls.ietf@gmail.com>; rmcat@ietf.org; Colin Perkins <csp@csperkins.org>
Subject: Re: Roman Danyliw's Discuss on draft-ietf-rmcat-eval-criteria-12: (with DISCUSS and COMMENT)

Dear Roman,

thanks much for your comments.  I just posted an updated draft (-13) to address those.  See below.

> ----------------------------------------------------------------------
> DISCUSS:
> ----------------------------------------------------------------------
> 
> Section 5.2.  Per “Sample video test sequences are available at: 
> [xiph-seq] and [HEVC-seq].  The following two video streams are the 
> recommended minimum for
> testing: Foreman and FourPeople.”, these test sequences seems underspecified.
> 
> ** Is the “recommended” here intended to be normative?  There is no 
> RFC2119 boiler plate in this document to guide the parsing of the text.

Recommended is recommended as per English language and it not used as a verb but as an adjective, indicating a suggestion.

> ** From the text, there wasn’t much precision in where to find these 
> recommended videos (Foreman and FourPeople).  At the url pointed to by 
> [HEVC-seq], I found the filenames “FourPeople_1280x720_60.yuv” and 
> “foreman15_4000.yuv”, is that them?

This is fixed in the rev now, also the link that disappeared is removed.

> ** Is it expected for
> http://www.netlab.tkk.fi/~varun/test_sequences/foreman15_4000.yuv to 
> be 0 bytes?  I tried on 03/03/2020 at ~0950 EST

The broken link is gone now.  Both videos can be found in a single place now.  It is clearer specified where to find them.

> ** Give that that one of the recommended urls doesn’t work even before 
> this draft is published, I have great reservation with keeping a 
> normative “recommended” to such external repositories.  However, 
> providing pointers to repositories of “sample video test sequences” makes sense to me and is helpful.

Should be addressed by now.

> ----------------------------------------------------------------------
> COMMENT:
> ----------------------------------------------------------------------
> 
> ** Section 3.1.  The purpose of this log isn’t clear.  What is the 
> relationship of it to the metrics described in the previous section?  
> Where does it fit into the measurement workflow?  Is this constructed 
> on a per packet capture file basis?

Added a clarifying sentence:

These data should suffice to compute the media-encoding independent metrics described above.  Use of a common log will allow simplified post-processing and analysis across different implementations.

> ** References:
> -- Section 3.  Consider adding a citation for tcpdump and wireshark

done

> -- Section 4.4.  Consider adding a citation for the “Bilbert-Elliot” 
> model

We got this comment also from Mirja, to which I replied:

I could include a reference here for Gilbert-Elliot such as:
http://www.ohohlfeld.com/paper/hasslinger_hohlfeld-mmb_2008.pdf

[Roman]  Since it would be informative, I don't think it would hurt to add this citation.

However, this not mandatory text and the second part of the sentence is even less specific and does (deliberately) not give a specific example.  We don't want to create any prejudice here beyond random losses.  The Internet changes, after all.

In order to avoid this being a showstopper here (we don't have enough hour tonight for another rev round, I am adding this in now.  Please provide feedback if this is really needed.  I could still remove it in two weeks when I-D submission reopens.

> ** Editorial:
> -- Section 3.  Recommend explicitly spelling out PCAP as packet capture.

This is spelled out in brackets; since this also refers to a commonly used file format, I would keep it as is.

> -- Section 4.5. s/is is/is/

Fixed.

Thanks much,
Jörg