Re: [rtcweb] Finishing up the Video Codec document

[Stephan Wenger <[e-mail]>]

Hi,

I meant to copy the mailing list on two emails regarding my (and
Bernard’s) action items.  I mis-spelled rtcweb.  So here the content
integrated into a single email.  (Bernard and Adam: there’s nothing new
herein.)

Hi,
So here my proposal, with respect to H.264 baseline:

MUST:
Filler payload.  You see this occasionally from mostly early generation)
encoder chips.  It¹s basically a way to waste bits so to fulfill buffer
constraints.  Implementation complexity is zero: throw away bits.

Full frame freeze: used in video switching MCUs, to ensure a stable
decoded displayed picture while switching channels in probably a
media-unaware way.  Commonly used in H.32x legacy environments and also in
some SIP MCUs.


SHOULD:
vacat

MAY:
User data T.35 registered, and user data unregistered
Especially the latter you see occasionally for proprietary extensions.
It¹s a good idea to advise implementers that a bitstream may contain that
kind of stuff, even if they don¹t act on it.

Display orientation SEI, as it is referenced in section 4.




With respect to freeze request, below a slightly edited version of an
email that I sent to Adam when he was preparing the pre IETF-91 draft.

Stephan

Freeze-request and freeze-release is something used in the video conf
industry for ages.
Consider a four-way conference with endpoints A, B,  C, and D, and a
video-switching MCU.  A, B, C are all sending pictures to the MCU.  For
receiving endpoint A, the MCU selects either bitstream sent by B or by
endpoint C for forwarding to A.  The selection can be based on voice
activation, conf control, manual request, whatever.  When switching from
B¹s video to C¹s video, the following happens:

1. In-band signaling ³freeze picture request² inserted by the MCU into the
bitstream forwarded to A.  This freezes the display (though not
necessarily the decoding) of the bitstream.
2. MCU sends full intra request to endpoint A, using a signaling protocol.
Meanwhile, it continues to forward all the bitstreams as usual.  Service
to endpoints B, C, and D continue uninterrupted.
3. In some systems supporting dirty refresh points, the MCU starts
forwarding bitstream C to A, resulting in garbage (until the dirty refresh
has cleaned up the picture.  The garbage is luckily not displayed, thanks
to the freeze request.  In other systems, the MCU waits for an intra
picture from A.
4. As soon as the MCU knows that the picture buffer in receiving endpoint
A has been cleaned up (though intra, or dirty refresh), it
sends‹-typically via a signaling protocol like H.245--a freeze-release.
At that point, endpoint A starts displaying again.

The freeze request is in the H.xxx series standards an SEI message.  In
H.261, it is a bit in the the picture header.  The reason is that the
freeze request has to be dealt with synchronously with the bitstream (on a
per picture granularity) so to avoid displaying garbage.  The
freeze-release, OTOH, can be in the signaling, because it does not matter
overly much when the picture is released--faster is better from an user
experience viewpoint, but the freeze only affects the one decoding
endpoint that has decided it wants to see something else, so let this one
suffer the consequences (resulting from delay in the signaling).

This is why pretty much all profiling specs in the video conferencing
industry require freeze request to be supported in the video bitstream,
and it is also why it is available in ITU and MPEG specs since
standardized video compression was developed.

One could argue that pure video switching MCUs are a thing of the past,
and I would mostly concur.  However, in the context of large video
conferences (dozens or hundreds of participants), and also on systems with
limited screen real-estate, video switching continues to be required.
With respect to state-of-the-art technologies, the advent of selective
forwarding units and layered coding also requires freeze requests (in the
enhancement layer) for similar reasons as described above.


>
>
>On 11/25/14, 15:33, "Adam Roach" <[e-mail]> wrote:
>
>>I've updated the draft-ietf-rtcweb-video based on the minutes from
>>Hawaii. Now that we have a clear path to success, I'd like to get the
>>document finalized and published as quickly as possible. This means I
>>need your feedback on the remaining open issues (1, 4, and 5, below). If
>>you are CCed on this mail, it's because you took an action item in
>>Hawaii, and I'm waiting to hear back from you.
>>
>>New version: http://datatracker.ietf.org/doc/draft-ietf-rtcweb-video/
>>Diff: https://tools.ietf.org/rfcdiff?url2=draft-ietf-rtcweb-video-03.txt
>>____
>>
>>Action item for Harald Alvestrand from the minutes:
>>> Mo Zanaty: An IANA registered header with reference to 3GPP document.
>>> Codec specific mechanism way of doing this.
>>> Harald: Will send some suggestions on doing this.
>>
>>Action item for Bernard Aboba and Stephan Wenger: collect list of H.264
>>SEI messages we want to call out as MUST and/or SHOULD.
>>____
>>
>>Open Issue 1: If you don't want sRGB to be the default color space, say
>>something now.
>>
>>Open Issue 2 (closed): Took "screen source video metadata" issue out of
>>the document, as this is an RTP issue, not a codec issue.
>>
>>Open Issue 3 (closed): New text in "stream orientation" section calling
>>out various MUST/SHOULD/MAY levels, based on what I took away from the
>>mic in Hawaii. If you care about this, double-check that these are what
>>you expect them to be.
>>
>>Open Issue 4: In Toronto, it was suggested that we don't need to specify
>>support for "bilinear" and "none" filter styles, since they're already
>>required. I'm happy to remove the statement if someone can point me to
>>where this is otherwise mandated. If no one comes forward with a
>>citation, I'm going to close this open issue as harmless.
>>
>>Open Issue 5: Waiting on feedback from Bernard and Stefan, as detailed
>>above.
>>
>>Open Issue 6 (closed): Please see the text in section 5 of the new
>>document, which tracks the text from the Novel Proposal email pretty
>>closely.
>>
>>/a
>