Re: [AVTCORE] I-D Action: draft-ietf-avtext-framemarking-08.txt

[[e-mail] (Dale R. Worley)]

In attempt to clarify a particular point:

The whole point of draft-ietf-avtext-framemarking is to provide
information to packet-handling devices on how to manipulate streams of
RTP packets containing encoded video, even when the device cannot
understand the payload of the RTP packets, either because they are
encrypted or they are in a video format that the device does not
understand.

Three typical operations are:
1) Routers dropping packets due to congestion, trying to determine the
least "costly" packets to drop.
2) Routers trying to "shape" the bandwidth demand of a video stream by
removing one or more highest-resolution layers from the video encoding.
3) RTP switches wanting to splice from one video stream to another,
looking for an "efficient" place to switch to the new stream.

In order for draft-ietf-avtext-framemarking to work well, the
significance of the extension data must be well-defined, so devices know
what the extension data tells about the RTP packets.  Inevitably, this
means that the extension data is interpreted within a model about the
packets and how they are related.

The fundamental structure seems to be the "frame-in-layer", the set of
packets that have the same SSRC, RTP timestamp, LID, and TID values.  A
frame-in-layer is assumed to encode a particular image at a particular
temporal and spatial resolution.

The remaining extension data seems to always encode dependencies between
frames-in-layer, that is, if the receiver is to successfully decode one
particular frame-in-layer, it needs all or most of the packets in
certain other frames-in-layer.

E.g., the D bit says that this frame-in-layer is not depended upon by
any other frame-in-layer.  The implication is that dropping these
packets is "low cost" compared to dropping packets within a
frame-in-layere that is depended on by another frame-in-layer.

There is an implication that if one frame-in-layer has the same SSRC,
same timestamp, no higher LID, and no higher TID than another
frame-in-layer, then the latter frame-in-layer depends on the former
frame-in-layer.

Within this framework, the algorithm for applying the extension to any
particular video encoding attempts to capture the actual dependency
structure of the video packets within the model that the extension data
can express.  There can be two sorts of mismatch:  "false positive",
where the extensions express a dependency not present in the video
encoding, and "false negative", where the extension do not express a
dependency which is present in the video encoding.  The extreme case is
"complex, irregular scalability structures that do not conform to
common, fixed patterns of inter-layer dependencies and referencing
structures."  In that case, using TID and LID is likely to not be
beneficial, and the extension data will tend to express a lot of "false
positive" dependencies.

What I'm pushing for is that all of this machinery be clearly stated,
especially exactly what dependencies are signaled by the extension data.
If those are left to the common intuitive understanding, we're likely to
have a lot of edge cases implemented differently by different devices,
leading to poor user experience (although probably not outright
non-interoperability).

Dale