Re: [video-codec] Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT)

[Roman Danyliw <[e-mail]>]

Hi!

> -----Original Message-----
> From: iesg [mailto:[e-mail]] On Behalf Of Filippov Alexey
> Sent: Thursday, June 13, 2019 1:15 PM
> To: Roman Danyliw <[e-mail]>; Adam Roach <[e-mail]>;
> The IESG <[e-mail]>
> Cc: [e-mail]; [e-mail]; draft-ietf-netvc-
> [e-mail]; Mo Zanaty <[e-mail]>
> Subject: RE: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09:
> (with DISCUSS and COMMENT)
> 
> Dear Roman,
> 
> Thank you a lot for your comments and raising the important questions.
> 
> > How should a set of QPs be specified?
> As you probably know, a set of available QP values can be codec specific.
> Usually, a QP set is selected to cover medium bit-rate range that is
> considered to be the most complex for compressing. We proposed to set up
> the assessment process not only for this range but also for low and high QP
> ranges to cover a wider range of applications. According to my
> understanding, concrete QP values should be specified for candidate codecs
> in draft-ietf-netvc-testing

See below.  Can you help with a specific pointer into draft-ietf-netvc-testing which lists candidate codecs+QPs.

> 
> > How should the quality values be calculated?
> It is explained in Section 4.1, namely: “To assess the quality of output
> (decoded) sequences, two indexes, PSNR [3] and MS-SSIM [3,11] are
> separately computed. In the case of the YCbCr color format, PSNR should be
> calculated for each color plane whereas MS-SSIM is calculated for luma
> channel only. In the case of the RGB color format, both metrics are
> computed for R, G and B channels. Thus, for each sequence, 30 RD-points for
> PSNR (i.e. three RD-curves, one for each channel) and 10 RD-points for MS-
> SSIM (i.e. one RD-curve, for luma channel only) should be calculated in the
> case of YCbCr. If content is encoded as RGB, 60 RD-points (30 for PSNR and
> 30 for MS-SSIM) should be calculated, i.e. three RD-curves (one for each
> channel) are computed for PSNR as well as three RD-curves (one for each
> channel) for MS-SSIM.” In references [3, 11], these 2 quality assessment
> metrics and the ways of how to calculate them are described in detail.

Got it.  Thanks.

> >-- What does the text “(in a separate document on Internet video codec
> testing)” mean?
> > Per “A list of video sequences that should be used for testing as well as the
> 10 QP values for the reference codec are defined in a separate document ",
> what document is that?  Is it draft-ietf-netvc-testing?
> According to my understanding, it is draft-ietf-netvc-testing.

Can you please help me with the specific section references in draft-ietf-netvc-testing as I'm not seeing it -- where is the explicit reference codec being named and its associate 10 QP values?  For what it's worth I see Section 4.1 of this draft saying that:

   As
   the reference for evaluation, state-of-the-art video codecs such as
   HEVC/H.265 [4,5] or VP9  must be used. The reference source code of
   the HEVC/H.265 codec can be found at [6]. The HEVC/H.265 codec must
   be configured according to [13] and Table 9.

I'm looking for something that either unambiguously points to the reference codec+QPs; or clearer language that says that all of this is out of scope in this and/or the -testing draft, but is clear on what input into the processes of this draft is required.

> >What does “codec implementation (for both an encoder and a decoder)
> should cover the worst case of computational complexity, memory
> bandwidth, and physical memory size” mean?
> I’d like to thank Adam for his comment. I also think that his formulation
> ("...should take into consideration the worst-case...") is clearer.
> A more detailed explanation on this Section can be found in my response to
> the secdir reviewer: https://mailarchive.ietf.org/arch/msg/secdir/QYx-
> hl07aec5XhgqkldqgWFG3Is

I'd like to offer a further refinement of what Adam proposed:

Original:
However, it is worth noting that a codec implementation (for both an encoder and a decoder) should cover the worst case of computational complexity, memory bandwidth, and physical memory size (e.g., for  decoded pictures used as references).

Adam:
However, it is worth noting that a codec implementation (for both an encoder and a decoder) should take into consideration the worst-case computational complexity, memory bandwidth, and physical memory size (e.g., for decoded pictures used as references).

Roman+Adam:
However, it is worth noting that a codec implementation (for both an encoder and a decoder) should take into consideration the worst-case computational complexity, memory bandwidth, and physical memory size needed to processes the input (e.g., the decoded pictures used as references).

> >Please add additional language that codec should be written in a defensive
> style as they will be processing untrusted input.
> Good point. Thank you. I’ll add it.

Regards,
Roman

> 
> --
> Best regards,
> Alexey Filippov
> 
> -----Original Message-----
> From: Roman Danyliw via Datatracker [mailto:[e-mail]]
> Sent: Wednesday, June 12, 2019 4:25 AM
> To: The IESG <[e-mail]>
> Cc: [e-mail]; Mo Zanaty
> <[e-mail]>; [e-mail]; [e-mail]; video-
> [e-mail]
> Subject: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: (with
> DISCUSS and COMMENT)
> 
> Roman Danyliw has entered the following ballot position for
> draft-ietf-netvc-requirements-09: Discuss
> 
> When responding, please keep the subject line intact and reply to all email
> addresses included in the To and CC lines. (Feel free to cut this introductory
> paragraph, however.)
> 
> 
> Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
> for more information about IESG DISCUSS and COMMENT positions.
> 
> 
> The document, along with other ballot positions, can be found here:
> https://datatracker.ietf.org/doc/draft-ietf-netvc-requirements/
> 
> 
> 
> ----------------------------------------------------------------------
> DISCUSS:
> ----------------------------------------------------------------------
> 
> (1) I worry that the level of detail in the in the Compression Performance
> Evaluation (Section 4.1) is insufficient for implementation.  Specifically:
> 
> (a) Per “Initially, for the codec selected as a reference one (e.g., HEVC or
> VP9), a set of 10 QP quantization parameter) values should be specified (in a
> separate document on Internet video codec testing) and corresponding
> quality values should be calculated.”
> 
> -- How should a set of QPs be specified?
> 
> --How should the quality values be calculated?
> 
> -- What does the text “(in a separate document on Internet video codec
> testing)” mean?
> 
> (b) Per “A list of video sequences that should be used for testing as well as
> the 10 QP values for the reference codec are defined in a separate document
> ", what document is that?  Is it draft-ietf-netvc-testing?
> 
> (2) Per the Security Considerations Section (Section 5)
> 
> -- What does “codec implementation (for both an encoder and a decoder)
> should cover the worst case of computational complexity, memory
> bandwidth, and physical memory size” mean?
> 
> -- Please add additional language that codec should be written in a
> defensive style as they will be processing untrusted input.
> 
> 
> ----------------------------------------------------------------------
> COMMENT:
> ----------------------------------------------------------------------
> 
> (1) There is something odd about the document formatting – the title and
> the first author last name in the footer is wrapped in “< … >”
> 
> (2) It would be helpful to forward reference that acronyms are explained in
> Appendix A.
> 
> (3) This draft uses the words should and must to prescribe action.  Why
> wasn’t
> RFC2119 cited to explain these words?
> 
> (4) Section 2.0.  A reference to explain “YCbCr 4:2:0” would be helpful
> 
> (5) Section 2.1. Per “high encoder complexity” and “decoding complexity”, I
> initiate read that as a qualitative measure.  However, the text says “up to
> 10x and more” so that implies some quantitative measure.  What is that?
> 
> (6) Section 2.1.  Expand QP values on first use
> 
> (7) Section 2.x.  The language around content doesn’t appear to be consist.
> For example:
> 
> -- Section 2.1, Internet Video Streaming says “movies, TV-series and shows,
> and animation.”
> 
> -- Section 2.2, IPTV says “television content”
> 
> -- Section 2.5, Screen casting says “business presentations …, animation
> (cartoons), gaming content, data visualization, …, virtual desktop
> infrastructure (VDI), screen/desktop sharing and collaboration, supervisory
> control and data acquisition (SCADA) display, automotive/navigation
> display, cloud gaming, factory automation display, wireless display, display
> wall, digital operating room (DiOR), etc.
> 
> What the difference between Section 2.1’s animation and Section 2.5’s
> cartoons?
> 
> What’s the different between Section 2.1’s “movies, TV series …” and Section
> 2.2’s “television content”?
> 
> (8) Section 2.5.  The sentence “Currently, …” is very challenging to parse as it
> includes inline “i.e.,” and “etc”.
> 
> (9) Section 2.5.  Per “powerpoint, word documents”, these are specific
> Microsoft products.  I recommend using more generic names.
> 
> (10) Section 4.  I found it confusing that an evaluation methodology was in a
> requirements document.  I would have expected it in the draft-ietf-netvc-
> testing
> 
> (11) Section 4.1.  VP9 needs a reference.
> 
> (12) Section 6.  I don’t think this entire section is necessary.
> 
> (13) Editorial Nits:
> -- Section 3.  Style nit.  s/chapter/section/
> 
> -- Section 4.1. Typo.  s/computged/computed
>