Re: [AVTCORE] I-D Action: draft-ietf-avtext-framemarking-08.txt

["Mo Zanaty (mzanaty)" <[e-mail]>]

Dale,

Thank you very much for the detailed review and comments.
Version -10 has updates for all your feedback below.

https://tools.ietf.org/html/draft-ietf-avtext-framemarking-10


See Mo: inline below for details of the update.
And please confirm the updates in -10 adequately resolve all your issues.
Finally, sincere apologies for not getting this update out sooner.

Thanks,
Mo

On 4/3/19, 7:32 AM, "Dale R. Worley" <[e-mail]> wrote:

>I'm no expert on this field, but as the frame marking extension is
>intended to be used broadly over many different video encodings, I think
>it can be usefully critiqued relative to its ambition to be a
>*generalized* frame marking mechanism.  In particular, a number of its
>features seem to reference ideas which generally apply to multiple
>encodings.  But this leaves a great deal of room for lack of alignment
>as to the exact semantics of the features, which could easily lead to a
>lot of subtle interoperation problems.  So I am here pushing for a
>clearer definition of what is and is not meant by the features.
>
>There is some oddity in how the sections are structured.  The short form
>is defined in 3.1, and the long form is defined in 3.2.  The three
>mapping for specific codecs are listed in 3.2.1.1, 3.2.1.2, and 3.2.1.3,
>and 3.2.1.4.  It would be better to group the two definitional sections
>together and group the four example sections together.

Mo: This structure was intentional to show that Layer ID mappings only
apply to Scalable Streams.

Section 3.1 defines the Short Extension for Non-Scalable Streams, where no
Layer IDs apply.

Section 3.2 defines the Long Extension for Scalable Streams, which
contains Layer IDs.

Section 3.2.1 defines the Layer ID Mappings for Scalable Streams, with
subsections for specific codecs. Note these subsections are not merely
examples; they are normative / "definitional".

I added the following text to section 3.2.1 to help clarify this:

"This section maps the specific Layer ID information contained in specific
scalable codecs to the generic LID and TID fields. Note that non-scalable
streams have no Layer ID information and thus no mappings."

Does this grouping make sense now, given the clarifying text, or do you
still think it is odd?

>
>Also, 3.2.1.3 (H264 (AVC) LID Mapping) and 3.2.1.4 (VP8 LID Mapping)
>don't specify how the S, E, I, D, and B bits are determined from the
>codec's output packets.

Mo: This is specified in section 3.2.
>
>Regarding the multiple (four, actually) formats of the extension, it
>helps specifying them if they can all be mapped into the same semantic
>data structure.  For example,
>
>    TID is the temporal layer index.  It is implicitly 0 if the short
>    format is used.
>
>    LID is the (spacial) layer index.  It is implicitly 0 if the short
>    format is used or the L=1 form of the long format is used.
>
>    TL0PICIDX:  When TID is 0:  If present, it is a cyclic counter
>    labeling the frames.  If not present, the frames have no such labels.
>    When TID is not 0, it indicates that this frame in this layer
>    depends on the frame with this label in the layer with TID 0.

Mo: Correct. I added this wording to help clarify things in section 3.2.
>o  TID: Temporal ID (3 bits) - The base temporal layer starts with 0,
>      and increases with 1 for each higher temporal layer/sub-layer.  If
>      no scalability is used, this MUST be 0.  It is implicitly 0 in the
>      short extension format.
>   o  LID: Layer ID (8 bits) - Identifies the spatial and quality layer
>      encoded, starting with 0 and increasing with higher fidelity.  If
>      no scalability is used, this MUST be 0 or omitted to reduce
>      length.  When omitted, TL0PICIDX MUST also be omitted.  It is
>      implicitly 0 in the short extension format or when omitted in the
>      long extension format.
>o  TL0PICIDX: Temporal Layer 0 Picture Index (8 bits) - When TID is 0
>      and LID is 0, this is a cyclic counter labeling base layer frames.
>      When TID is not 0 or LID is not 0, this indicates a dependency on
>      the given index, such that this frame in this layer depends on the
>      frame with this label in the layer with TID 0 and LID 0.  If no
>      scalability is used, or the cyclic counter is unknown, this MUST
>      be omitted to reduce length.  Note that 0 is a valid index value
>      for TL0PICIDX.
>
>
>
>Notice that a missing TL0PICIDX has different semantics than a missing
>TID or LID.
>
>Given the similarity of "temporal layer index" and "layer ID", it seems
>like you want a more distinctive phrase for the latter.  Could it be
>changed to "spatial layer" or "resolution layer"?

Mo: Earlier codecs called this a "dependency layer", which could be a
spatial or quality layer. Later codecs called this simply a layer! (And
demoted temporal layers to "sublayers".) We should probably stay aligned
with the later codecs.
>
>There seems to be no way to signal whether the short form is used
>vs. the L=0 version of the long form (if B=0 and TID=0) -- The ID value
>for both is signaled in SDP by
>
>      a=extmap:3 urn:ietf:params:rtp-hdrext:framemarking

Mo: Correct.
>
>This doesn't cause a problem, as the semantics of the two alternatives
>are the same, but it prevents the 4 reserved bits in the short form from
>being defined in the future for any purpose other than B and LID. --
>Alternatively, is the short form simply what the extension is reduced to
>when using non-scalable streams, as those must necessarily have B=0 and
>LID=0?  (Also see my query below regarding the "default" value of B.)

Mo: Correct, except I think you meant TID=0 (not LID=0) here. I have
clarified that the 4 reserved bits are fixed to 0 and not for future
use/extension.

>It seems that the intention is that the video stream can be divided into
>substreams of RTP packets, called "layers", each of which is identified
>by a particular TID and LID, that is, TID/LID defines a
>*two-dimensional* hierarchy.  "They convey a layer hierarchy with [the
>layer with] TID=0 and LID=0 identifying the base layer."
>
>My guess is that the special case for interpreting the TL0PICIDX value
>is actually when both TID and LID = 0, that is, the base layer, not just
>TID = 0 as stated in the text.  (If I'm wrong, the structure here is

Mo: Correct. I fixed the text to explicitly state this (TID=LID=0).

>more complicated than I'm describing, with the TL0PICIDX labels of an
>upper layer referring to the label with the *same* LID but TID = 0.)
>
>The idea seems to be that one can "efficiently" discard layers from the
>RTP stream, as long as: if one keeps a layer with a particular TID and
>LID, one keeps all layers with lesser or equal TID and LID.  I can't
>quite see how best to define "efficiently" here, but it seems to be the
>central reason for labeling the layers -- that a receiver can
>successfully decode all of the data in all of the layers that remain
>present.

Mo: Correct.
>
>Things are more interesting in regard to what packets can be discarded
>from a layer "efficiently".  The S, E, I, D, and B bits seem to be
>intended to guide a device that needs to discard packets.  The use of D
>bits is specified:
>
>   When an RTP switch needs to discard a received video frame due to
>   congestion control considerations, it is RECOMMENDED that it
>   preferably drop frames marked with the D (Discardable) bit set [...]
>
>And I suspect that it is implied that if packets are dropped from one
>frame, further packets from the same frame are preferred to be dropped.
>The S and E bits are intended to help with this process.
>
>But dropping whole frames to some degree conflicts with the fact that
>small losses from video layers can often be recovered from, either due
>to redundancy in the layer, or by loss-reconstruction strategies in the
>receiver.  However, if one drops a *lot* of packets from one frame, one
>might as well discard the remainder of them.

Mo: Correct. We decided against being normative or even prescriptive about
any aspect of this as it is highly implementation dependent and subjective.
>
>The I bit suggests that there are provisions for dependency between the
>frames in a single layer, and dependency between frames is not the same
>for all frames.  It appears that if one frame of a layer is dropped, the
>following frames are preferred to be dropped until a frame with I = 1 is
>seen.

Mo: Again, that is highly subjective and implementation dependent, so we
don't attempt guidance or requirements here.

>I am less clear on what the B bit means -- presumably all layers with
>lower TID and LID than the layer containing the frame in question are
>retained, B doesn't seem to carry useful information.

Mo: B is used to break intra-layer dependencies in upper layers that have
loss, and fallback to the base layer which is often protected better
(FEC/RTX) and thus more reliable. Again, this is more dark art that we
can't prescribe or even describe accurately for most implementations (as
they vary widely).
>
>And there seems to be a problem with "defaulting" the value of B to 0
>when there is no scalability.
>
>As stated:
>
>   o  B: Base Layer Sync (1 bit) - MUST be 1 if the sender knows this
>      frame only depends on the base temporal layer; otherwise MUST be 0.
>
>This can be stated equivalently:
>
>      MUST be 1 if the sender knows this frame does not depend on any
>      frames that do not have TID=0.
>
>Now if the frame itself has TID=0, then it cannot (by the ordering of
>the layers) depend on any frame that does not also have TID=0.  The
>consequence is that the "natural" value of B in TID=0 layers is 1.  And
>when there is no scalability, the only layer has TID=0.

Mo: I fixed this unfortunate circumstance by forcing B=0 when TID=0. B is
useless when TID=0, so we can force it to 0 or 1 arbitrarily. The
"natural" value for something to ignore is 0, so I just enforced this in
the following text.
o B: Base Layer Sync (1 bit) - When TID is not 0, this MUST be 1 if
the sender knows this frame only depends on the base temporal
layer; otherwise MUST be 0. When TID is 0 or if no scalability is
used, this MUST be 0.


>I think what is going on is that there's an implicit structure of
>dependencies between the frames of a layer (a frame depends on earlier
>frames), and between the frames of different layers (a frame can depend
>on frames with lower TID/LID and no later in time), and the various bits
>are used to signal the *lack* of certain possible dependencies, but how
>the bits do this needs to be clarified.  (The meaning of TL0PICIDX
>particularly needs to be specified.)  But the implicit dependency
>structure isn't spelled out.  That makes things harder in two ways:  (1)
>it is not clear what sorts of dependencies future codecs are *not*
>allowed tointroduce, and (2) it is difficult to state exactly what
>dependencies are *removed* by particular signaling.

Mo: It is impossible to capture and signal all the dependencies in a
modern video stream efficiently. Far too much context required, and far
too much variation allowed within video specs, although most
implementations settle on simpler dependency structures. We decided to
limit the scope in 3.4.2 to "nested hierarchical temporal layering
structures" since that is the most widely deployed, and more complex
structures would require more complex and inefficient signaling to
describe all dependencies. This was a WG decision that was revisited and
upheld several times.
>
>3.2.1.  Layer ID Mappings for Scalable Streams
>
>All of the descriptions for specific codecs contain "ID=2", whereas the
>generic descriptions of the extension formats show "ID=?".  The latter
>is correct, since the ID value is negotiated for every RTP stream.

Mo: Good catch. It doesn't even match the example extmap:3. I replaced 2
with ?.

>3.4.  Usage Considerations
>
>The switching of video streams is recommended to be done this way:
>
>   When an RTP switch wants to forward a new video stream to a receiver,
>   it is RECOMMENDED to select the new video stream from the first
>   switching point with the I (Independent) bit set in all spatial
>   layers and forward the same.  An RTP switch can request a media
>   source to generate a switching point by sending Full Intra Request
>   (RTCP FIR) as defined in [RFC5104], for example.
>
>This is difficult to implement in general, as it requires the switch to
>keep track of all the layer IDs that have been seen, then look ahead in
>the stream to see if, over a narrow range of time, all of the layers
>that have been seen have packets with I set.  If the fundamental purpose
>of I is to signal the best points to switch streams, it would be better
>to define its semantics to be that.  E.g., "If a switch intends to start
>forwarding a video stream, and within that stream, transmitting all
>frames with TID and LID less than or equal to certain values, it should
>start forwarding the stream beginning with a packet within that layer
>that has I set."  That is, I signals that at this point, the coming
>frames of this layer and all layers with lesser TID/LID can be decoded
>without dependency on any previous frames.

Mo: Media switches already do this, but by deep inspection of the payload
rather than simple inspection of a header extension. The I bit does not
signal anything about lower layers, only this layer. That is what media
switches want and expect.
>
>3.4.2.  Scalability Structures
>
>It would be more effective to state that that for "complex or irregular
>scalability structures", subdivision by TID and LID is not effective and
>so such structures should mark all packets with TID=0 and LID=0.  The
>current text suggests that the switch is required to know whether such a
>structure is in use, and if so, ignore the TID and LID fields, which
>suggests that the sender can put various values in those fields.  This
>would lead to requiring the switch to know what encoding is in use, and
>avoiding that is the point of this document.

Mo: Complex structures also use TIDs and LIDs but not necessarily in a
clean nested hierarchy. The most complex structures are total anarchy
(dynamic, unpredictable) but could still use TIDs and LIDs (within the
codec payloads, but not in this header extension). We don't see any good
way to support such complexity, so the extension is not useful for such
streams. This decision must happen at the source, not any later node like
the switch. Only the stream source truly knows the full structure of its
streams, so it decides whether or not to use this extension. Later nodes
like the switch can't reliably detect/infer this properly.
>
>Dale