[video-codec] Roman Danyliw's Discuss on draft-ietf-netvc-testing-08: (with DISCUSS and COMMENT)

[Roman Danyliw via Datatracker <[e-mail]>]

Roman Danyliw has entered the following ballot position for
draft-ietf-netvc-testing-08: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)


Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.


The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-netvc-testing/



----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

(1) There appear to be deep and implicit dependencies in the document to the
references [DAALA-GIT] and [TESTSEQUENCES].  I applaud the intent to provide
tangible advice on testing and evaluation to the community with them.  I have a
few questions around their use.

(1.a) Why aren’t [DAALA-GIT] and [TESTSEQUENCES] normative references as they
are needed to fully understand the testing approach and provide the test data?

(1.b) What should readers of the RFC do should these external references no
longer be available?  How is the change control of these references handled?

(1.c) In the case of [DAALA-GIT] which version of the code in the repo should
be used?  Formally, what version of C is in that repo?

(1.d) Per the observation that there are implicit assumptions made by the
document about familiarity with [DAALA-GIT] and [TESTSEQUENCES], here are a few
places where additional clarity is required:

-- Section 4.3, Per “For individual feature changes in libaom or libvpx , the
overlap BD-Rate method with quantizers 20, 32, 43, and 55 must be used”, what
are libaom and libvpx and what is their role?

-- Section 5.3.  Multiple subsection in 5.3.* list what look like settings for
tools (e.g., “av1: -codec=av1 -ivf -frame-parallel=0 …”). What exactly are
those?  How to read them/use them?

(2) The full details of some of the testing regimes need to be more fully
specified (or cited as normative): -- Section 3.1. The variable MAX is not
explained in either equation.

-- Section 3.1.  This section doesn’t explain or provide a reference to
calculate PSNR.  I’m not sure how to calculate or implement it.

-- Section 4.2.  Reference needed for Bjontegaard rate difference to explain
its computation

-- The references [SSIM], [MSSIM], [CIEDE2000] and [VMAF] are needed to fully
explain a given testing metric so they need to be normative

(3) An IANA Considerations section isn’t present in the document.

(4) A Security Considerations sections isn’t present in the document.


----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

A few comments:

(5) Consider qualifying the title to more accurately capture the substance of
this draft “Video Codec Testing and Quality Measurement {using the Daala Tool
Suite or Xiph Tools and Data}.”

(6) Section 3.1 and 3.3.  Cite a reference for the source code files names in
question – dump_psnr.c and dump_pnsrhvs.c (which are somewhere in the
[DAALA-GIT] repo?)

(7) Editorial Nits
-- Section 2.1.  Expand PMF (Probability Mass Function) on first use.

-- Section 2.1. Explain floor.

-- Section 2.2.  Typo.  s/vidoes/videos/

-- Section 2.2. Typo. s/rewatched/re-watched/

-- Section 2.3.  Typo.  s/comparisions/comparisons/

-- Section 3.1.  Expand PSNR (Peak signal to noise ratio) on first use.

-- Section 3.1.  Typo.  s/drived/derived/