Re: [MMUSIC] Remarks from today

in-line

On Fri, Apr 22, 2011 at 10:55 PM, Bert Greevenbosch <
Bert.Greevenbosch@huawei.com> wrote:

> Hi Ted, Stephen, all,
>
> I have thought a bit more about the frame rate issue, especially why it is
> originally defined as a maximum frame rate rather than an exact frame rate.
>
> The main problem with the negotiation of the frame rate is, that it is
> likely that the server can only provide a limited number of fixed frame
> rates. It could have the coded video ready for these frame rates, but it
> would be hard to provide any other frame rate. With the negotiation of a max
> frame rate, the source can choose a lower frame rate, as it already has the
> video stream with that frame rate available.
>
I suspect you are thinking mostly about video streaming of stored media.

There are video sources where you get substantially lower bit rates if you
use a dynamic frame rate when encoding - particularly PC screens which often
are relatively unchanging.  Digital signage might have similar
characteristics.

In video conferencing (or other real-time encoding applications) it is
common for encoders to drop frames during heavy motion or scene changes in
order to maintain a bit rate constraint (trading off image sharpness for
motion handling).

If the video encoding is layered (for instance H.264 Annex G), it can be
routed through a MANE.  In this case, the frame rate (and the spatial
resolution) are dynamically adjusted by the MANE in response to network
conditions, with different receivers getting different resolutions and frame
rates.

>
> So a fixed frame rate negotiation would only work if there is only a
> limited number of common shutter glass frequencies and associated video
> frame rates. In that case the server can have video representations with
> matching frame rates available.
>
> I think a common shutter glass frequency is 120 Hz (60 Hz/view). Common
> frame rates are 24 fps, 30 fps, 50 fps, 60 fps. If this is the case in
> general, negotiation with an exact frame rate can work. Notice that the 24
> fps and 50 fps are not divisors of 60 Hz, but by displaying some of the
> frames one time more than the other frames, it is still relatively easy to
> convert the frame rates to shutter glass frequencies.
>

@Stephen: do you want to keep the meaning of the maximum frame rate because
> you want to be able to choose a lower frame rate, for example because the
> video is to be multiplexed with other video in the same RTP stream?
>

Part of my thinking is simply that it is a bad design practice to overload a
parameter by giving it different meanings (even if the differences appear
slight).

Design principles aside, I believe that real-time applications (particularly
video conferencing) will need variable frame rate for 3D (as they already do
for 2D), for the reasons I pointed out above.  As you point out, there is
one easy (though sub-optimal) way to upsample / downsample frame rate to
match the needed refresh rate - simple duplicating / dropping of frames.
Since frames can be lost in transmission (and because sender and receiver
clocks are not precisely matched), receivers will need to employ such
techniques anyway.

And shuttered glasses are only one way of rendering 3D;  it is clear that
shuttering, polarization and lenticular techniques will all co-exist. For
many receivers, shutter rate doesn't apply at all.

So I am still thinking that negotiation of shuttering rate (or fixed frame
rate) is not required for 3D interoperability.  An encoder is already free
to send fixed frame rate video if it wishes; I think that is enough.

Best Regards,
Stephen

>
> @Ted: given the common frame rates in practise, do you think they still
> need to be negotiated or will the shutter glasses be able to accept all of
> these? Then we could also add some informative text about this in the draft.
>
> Best regards,
> Bert
>
>
>
> From: Stephen Botzko [mailto:stephen.botzko@gmail.com]
> Sent: 22 April 2011 01:30
> To: Ted Hardie
> Cc: Bert Greevenbosch; lennox@cs.columbia.edu; mmusic@ietf.org
> Subject: Re: [MMUSIC] Remarks from today
>
> in-line
> Stephen Botzko
> On Thu, Apr 21, 2011 at 12:35 PM, Ted Hardie <ted.ietf@gmail.com> wrote:
> On Mon, Apr 18, 2011 at 7:52 PM, Bert Greevenbosch <
> Bert.Greevenbosch@huawei.com> wrote:
> Hi Ted,
>
> If I understand you well, you would like to be able to negotiate the video
> frequency, such that it can easily be converted to the shutter glass
> frequency.
>
> So for example, if the shutter glass frequency is 120Hz (60Hz/view), then
> reasonable video frequencies would be 60Hz, 40Hz, 30Hz, i.e. divisors of
> 60Hz (or in the Frame Sequential case, 120Hz). In this case it is easy to
> upsample the signal by repeating each frame sequence {Lk Rk} an integer
> number of times.
>
> I have a preference to use the "framerate" attribute for this, as adding
> another rather similar parameter to the "3dFormat" attribute seems somehow
> double. However, I also see your point in that the "framerate" attribute
> only indicates a maximum frequency, not a guaranteed frequency. We could
> solve this in the draft by mandating that in case of 3D streaming, the
> frequency the video MUST equal the value of the "framerate" attribute, in
> case the "framerate" attribute is included.
>
> Well, I agree that this solves the problem.  I'm a little leary of having
> the same parameter have two related but distinct meanings, depending on the
> context.  But if no one else objects to this, I'm certainly willing to go
> along.  I think you are correct that the context is easy enough to
> determine.
>
> regards,
>
> So this means that this video frequency negotiation can't be done for 3D
> video conferencing?  Providing application-specific definitions for a
> parameter seems like a problem to me, as many stacks / systems support
> multiple applications, and good definitions of the application might be hard
> to craft.
>
> Are we sure we really need this negotiation to achieve interoperability?
> I'm not convinced myself.  If we do, I think a new parameter is a better
> approach.
>
>
> Ted Hardie
>
>
>
> What do you think?
>
> Best regards,
> Bert
>
> From: Ted Hardie [mailto:ted.ietf@gmail.com]
> Sent: 16 April 2011 08:36
>
> To: Bert Greevenbosch
> Cc: lennox@cs.columbia.edu; mmusic@ietf.org
> Subject: Re: [MMUSIC] Remarks from today
>
> Hi,
>
> Some further comments in-line.
> On Thu, Apr 14, 2011 at 11:21 PM, Bert Greevenbosch <
> Bert.Greevenbosch@huawei.com> wrote:
> Hi Ted,
>
> My suspicion is that there are probably practical limits to some system's
> ability to either upsample or interpolate.  Would it make sense to add
> something to the 3dFormat line to indicate what the resulting per-view rate
> will be?  So that
>
> a=3dFormat:FP SbS
>
> had an additional parameter:
>
> a=3dFormat:FP SbS 30Hz
> We could also consider using the SDP attribute "framerate" (RFC 4566). This
> attribute would give the frame rate of the 2D channel stream, but we can
> write in the draft that in the frame sequential case the frame rate per view
> is half of it.
>
> The relevant text from RFC 4566 is:
> a=framerate:<frame rate>
>
>
>
>
>
>
>
>          This gives the maximum video frame rate in frames/sec.  It is
>
>
>
>
>
>
>
>          intended as a recommendation for the encoding of video data.
>
>
>
>          Decimal representations of fractional values using the notation
>
>
>
>          "<integer>.<fraction>" are allowed.  It is a media-level
>
>
>
>
>
>
>
>          attribute, defined only for video media, and it is not
>
>
>
>          dependent on charset.
> While I think it could be re-used here, it seems to me to make more sense
> to use a separate indicator.  This is partly because it is more for the
> decoder/upsampler/interpolation check (Can I upsample this to something
> reasonable?  If not, don't select this choice).  And it's partly because I
> worry a bit about the interaction with existing code.
>
>
>  Or do you want to indicate the target shutter glass frequency?
>
>
> No, I think you're right that it's more likely that things will get
> adjusted to a standard shutter speed by upsampling than they will change the
> shutter speed.
>
> regards,
>
> Ted Hardie
>
> Best regards,
> Bert
>
>
> From: Ted Hardie [mailto:ted.ietf@gmail.com]
> Sent: 15 April 2011 02:49
>
> To: Bert Greevenbosch
> Cc: lennox@cs.columbia.edu; mmusic@ietf.org
> Subject: Re: [MMUSIC] Remarks from today
>
> On Wed, Apr 13, 2011 at 10:44 PM, Bert Greevenbosch <
> Bert.Greevenbosch@huawei.com> wrote:
> I agree, the 30Hz (per view) case would give a flickering effect. Since
> normal TVs have a frequency of 60Hz or more, for a similar quality with
> shutter glasses the frame rate should be 120Hz (=60 Hz/view) or more.
>
> But the receiver can also upsample the received signal to a higher display
> rate. For example, if it receives the following frame sequence at 30Hz/view:
>
>       L1,R1,L2,R2,L3,R3,...,
>
> then it could display at 60Hz/view:
>
>       L1,R1,L1,R1,L2,R2,L2,R2,L3,R3,L3,R3,....
>
> The receiver can also use some interpolation technique to generate the
> frames between Lk and L(k+1).
>
> I think that in praxis the receiving device will process the received
> stream to make it fit the display system. This allows different
> types/standards of shutter glasses to be used, independent on the format in
> which the 3D stream was received. It also allows other display types such as
> polarisation or autostereoscopic displays.
>
>
> My suspicion is that there are probably practical limits to some system's
> ability to either upsample or interpolate.  Would it make sense to add
> something to the 3dFormat line to indicate what the resulting per-view rate
> will be?  So that
>
> a=3dFormat:FP SbS
>
> had an additional parameter:
>
> a=3dFormat:FP SbS 30Hz
> >I know only know one manufacturer's system at all well, and my
> understanding is that the timing signals which synchronize them are really
> matching the frame start times to the shuttering, but don't adjust the frame
> rates.  Are there glasses which can modify the rate substantially as well?
>
> Can you please explain a bit more about the system you are referring too,
> especially about not modifying the frame rate? Do you mean the frame rate
> between the TV screen and the signalling device (that transmits the signal
> to the shutter glasses) or the frame rate of the video stream as it is
> received before it is decoded (by the codec)?
>
>
> I probably should have said "shuttering rate", sorry.  I mean that it
> matches the shutters to the alternating left and right view, but that the
> shuttering rate is consistent (resulting in 60Hz per view, in this case).
>
> regards,
>
> Ted Hardie
>
>
>
> _______________________________________________
> mmusic mailing list
> mmusic@ietf.org
> https://www.ietf.org/mailman/listinfo/mmusic
>
>