Re: [AVTCORE] [payload] Clarifying the H.265 RTP payload format specification's text on when to set the RTP "M" bit

["Wang, Ye-Kui" <[e-mail]>]

Regardless of the following, I still think that the timestamp approach should be the way to go. A good system should have a clear mapping of each video data unit (NAL unit in the context of H.265 and H.264) to an output timestamp, be it real-time encoding or pre-encoded content. For real-time encoding, the encoder knows what data units are generated for each input raw picture and thus it clearly knows the timestamp for each data unit. A pre-encoded content should have been stored in some kind of video storage format that has timestamp associated with each data unit, e.g. in a file container according to ISO base media file format, MPEG-2 systems, or whatever other file storage format. Trying to generate timestamps from a raw coded video bitstream without timing information for each coded picture based on a guess of frame rate is really just a hack to me, as frame rate can be adaptive, particularly at the beginning of a session in low-delay conversational applications or at locations of scene change or of need of full intra refresh or random accessibility, wherein your rate control algorithm can easily require skipping the encoding of some frames.

BR, YK

From: [e-mail] [mailto:[e-mail]] On Behalf Of Wang, Ye-Kui
Sent: Saturday, August 10, 2013 6:55 AM
To: Ross Finlayson; [e-mail]; [e-mail]
Subject: Re: [AVTCORE] [payload] Clarifying the H.265 RTP payload format specification's text on when to set the RTP "M" bit

Hi Ross,

You just need one more little step to get everything right here, instead of thinking of compromising the implementation :)

That is, read subclause 7.4.2.4.4 of the HEVC spec. The most relevant paragraphs are copied below for convenience (the first paragraph itself contains the answer to your question, but the second paragraph may also provide helpful information):

The first of any of the following NAL units after the last VCL NAL unit of a coded picture specifies the start of a new access unit:
-     access unit delimiter NAL unit (when present),
-     VPS NAL unit (when present),
-     SPS NAL unit (when present),
-     PPS NAL unit (when present),
-     Prefix SEI NAL unit (when present),
-     NAL units with nal_unit_type in the range of RSV_NVCL41..RSV_NVCL44 (when present),
-     NAL units with nal_unit_type in the range of UNSPEC48..UNSPEC55 (when present),
-     first VCL NAL unit of a coded picture (always present).

The order of the coded pictures and non-VCL NAL units within an access unit shall obey the following constraints:
-     When an access unit delimiter NAL unit is present, it shall be the first NAL unit. There shall be at most one access unit delimiter NAL unit in any access unit.
-     When any prefix SEI NAL units are present, they shall not follow the last VCL NAL unit of the access unit.
-     NAL units having nal_unit_type equal to FD_NUT or SUFFIX_SEI_NUT, or in the range of RSV_NVCL45..RSV_NVCL47 or UNSPEC56..UNSPEC63 shall not precede the first VCL NAL unit of the coded picture.
-     When an end of sequence NAL unit is present, it shall be the last NAL unit in the access unit other than an end of bitstream NAL unit (when present).
-     When an end of bitstream NAL unit is present, it shall be the last NAL unit in the access unit.

NOTE - VPS NAL units, SPS NAL units, PPS NAL units, prefix SEI NAL units, or NAL units with nal_unit_type in the range of RSV_NVCL41..RSV_NVCL44 or UNSPEC48..UNSPEC55, may be present in an access unit, but cannot follow the last VCL NAL unit of the coded picture within the access unit, as this condition would specify the start of a new access unit.

BR, YK

From: [e-mail]<mailto:[e-mail]> [mailto:[e-mail]] On Behalf Of Ross Finlayson
Sent: Friday, August 09, 2013 9:50 PM
To: [e-mail]<mailto:[e-mail]>; [e-mail]<mailto:[e-mail]>
Subject: Re: [AVTCORE] [payload] Clarifying the H.265 RTP payload format specification's text on when to set the RTP "M" bit

If you are able to set the RTP timestamp reliably, then you must already have some sort of frame boundary identification.

Yes, we have a reliable way of determining when *VCL* NAL units cross a frame boundary: Look at the first slice header bit.  The one remaining problem (that I identified at the end of my last message) is: How to classify any *non-VCL* NAL units that appear between one VCL NAL unit and another VCL unit that has the first slice header bit.  Which of these belong to the last frame, and which belong to the next frame.

But perhaps this doesn't really matter?  Perhaps it's OK (from the point of view of receivers that check and try to optimize for the 'M' bit) if we always set the 'M' bit for the last VCL NAL unit of a frame, even if some subsequent non-VCL NAL units (appearing before the first VCL NAL unit of the next frame) also belong logically to the current frame??


But you should realize that, in doing this, you have defeated the marker bit optimization. You have just shifted the latency from the receiver to the sender

No, not necessarily.  If the encoded video comes from a file (i.e., pre-recorded), then the latency involved in requiring the sender to examine the 'next' frame will usually be negligible.  But in any case, the situation we're talking about here (senders working from a sequence of pre-encoded NAL units) is one that usually occurs only in non-interactive environments, for which latency isn't crucial.


So there is not much benefit in making the marker bit reliable in this case, and there is even the risk that it will add unnecessary latency.

That's certainly another possibility, I suppose: Recommend that if a sending implementation cannot determine whether or not a NAL unit ends an 'access unit' (frame) without having to scan subsequent NAL units, then it should not set the 'M' bit at all.  I don't like this, though (especially because it adds receiver latency without necessarily reducing sender latency (note above)).  But it's a possibility to consider...
Ross Finlayson
Live Networks, Inc.
http://www.live555.com/