Re: [Hls-interest] Image-based subtitles and trickplay tracks

["Law, Will" <[e-mail]>]

Some additional data for this thread from a recent study just conducted for Apple device playback against Akamai CDN by the Client Optimization Team last month. You can see that for this ATV playback session, over the time period monitored, there were 22K requests in total, of which 98% were small range requests for thumbnails/trickplay.

Every request has a fulfilment cost at some point. Tiled images would serve to lower the request rate against the edge, in this case, by an order of magnitude.   From a CDN perspective, we would lend our support towards the development of an optional tiled-image-based thumbnail solution for HLS.

Cheers
Will


[cid:image001.png@01D6336D.43ED3800]

From: "Weil, Nicolas" <[e-mail]>
Date: Tuesday, May 26, 2020 at 12:22 PM
To: "May, Bill" <[e-mail]>, Roger Pantos <rpantos=[e-mail]>
Cc: "[e-mail]" <[e-mail]>
Subject: Re: [Hls-interest] Image-based subtitles and trickplay tracks

Comments inline.

From: May, Bill <[e-mail]>
Sent: Tuesday, May 26, 2020 9:31 AM
To: Roger Pantos <rpantos=[e-mail]>
Cc: Weil, Nicolas <[e-mail]>; [e-mail]
Subject: RE: [Hls-interest] Image-based subtitles and trickplay tracks


On May 22, 2020, at 11:17 AM, Roger Pantos <rpantos=[e-mail]<mailto:rpantos=[e-mail]>> wrote:

Hello Nicholas. Thanks for bringing these up. I have some questions:

On May 20, 2020, at 3:12 PM, Weil, Nicolas <[e-mail]<mailto:[e-mail]>> wrote:

Hello,

We are often seeing two image-related topics causing interoperability problems as they are not currently covered by the HLS spec. Normalizing the implementations around an official specification for these two points would be great:

Image-based subtitles tracks
For workflow reasons and charset reasons, some content owners don't include text-based subtitles in the live channels sources that they provide to distributors, but rather image-based subtitles (like DVB-Sub). While it's possible to transform these subtitles as IMSC1 Image Profile as per DASH-IF IOP section 6.4.4, there is no equivalent IMSC1 Image Profile support in the HLS RFC, which means that companies will continue to rely on proprietary forks of the HLS RFC to support these use cases. Even if it wasn't supported by Apple players, it would be tremendously helpful for interoperability in the rest of the HLS ecosystem.

I'd like to understand how widely validated the Image Profile of IMSC1 has been. Can anyone volunteer some examples where it’s been commercially deployed successfully? (Specifically IMSC1, vs. some other fork of TTML.)
[NW] IMSC1 Image profile is now supported by ATSC3, DASH-IF IOP (with support in dash.js) and IMF (with support in IMFTool<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_IMFTool_IMFTool&d=DwMGaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=KkevKJerDHRF9WRs8nW8Ew&m=dBy7sHrrIgjIMBRCpRh9IfMnwq8RKZHcgJLKmlBotj0&s=fBrrNLMD_eSxO9MYInkAMOhyaX6Zicdvk3HTD6p5mho&e=> which development has been sponsored by Netflix and other studios initially).

Another question: there are requirements in the US at least that require the ability to change font sizes, colors, etc.   And, TBH, these are changes that help people world-wide.

How would you meet those requirements with bit mapped subtitles?  Wouldn’t it be better to work to eliminate bitmapped subtitles completely?
[NW] I believe these font size/color change requirements can be satisfied with IMSC1 Text Profile which is supported in rfc8216bis since 2017.
As much as I’d like to get rid of bitmap subtitles, sometimes the content owners cannot provide anything else than bitmaps in the source feed. And it’s very challenging to apply a reliable OCR pass on it, for all target languages (Latin/Cyrillic/Asian/… charsets). IMSC1 Image Profile has got a decent industry support, and the Text Profile is already supported in HLS, so I would expect it to be a natural extension for HLS to support also the Image Profile.

Image-based trickplay tracks
For player resources optimization reasons, the use of a video track as a trickplay artefact is not always possible, and a lot of player providers recommend the use of image thumbnails tracks instead of special low framerate video tracks. DASHIF IOP section 6.2.6 covers this use case but there is equivalent support in the HLS RFC. There is the Image Media playlists HLS extension proposal from Roku/Disney/WarnerMedia here https://github.com/image-media-playlist/spec<https://urldefense.proofpoint.com/v2/url?u=https-3A__nam12.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fgithub.com-252Fimage-2Dmedia-2Dplaylist-252Fspec-26data-3D02-257C01-257Cbill.may-2540disneystreaming.com-257C49edde2508884eb51cca08d7fe7c807e-257C65f03ca86d0a493e9e4ac85ac9526a03-257C1-257C0-257C637257683002059478-26sdata-3D9lV-252BiNvxiK5feFCV1hYgo5I1lW0TEpMlWl-252BpwKwVKZU-253D-26reserved-3D0&d=DwMGaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=KkevKJerDHRF9WRs8nW8Ew&m=dBy7sHrrIgjIMBRCpRh9IfMnwq8RKZHcgJLKmlBotj0&s=yKAoFpJAUPAHylJJXOXaXVSeWHIwBnO5F9MOWl5tgWc&e=> but its relevance/adoption is currently limited by the fact that it's not part of the RFC. Same logic here: even if not supported by Apple players which don't need it as they can leverage I-frame tracks, it would be super useful for the rest of the HLS ecosystem to get this officially part of the RFC.

I'd like to better understand what’s driving this. Is the limitation essentially one of not being able to support an AVC decoder for i-frame display?

If that’s the case then it seems that putting JPEG images into fMP4 containers and using EXT-X-I-FRAME-STREAM-INF would be a smaller extension to HLS, both in terms of departure from the existing approach and less new spec to invent.

One of the things I don’t love about the image-media-playlist spec is that it doesn’t follow the regular HLS timing model, where the media presentation time is defined in the media data itself. Instead it relies on precise synchronization of the EXTINF values, which seems like a recipe for long-term accumulation of floating point error, as well as difficult to achieve with multiple geographically-dispersed packagers for live.


The limitation is exactly that.  A second decoder (AVC or HEVC) is not available on many devices.  This also makes mid-fragment switching difficult as well and makes switching between codecs impossible as well.

The image-media-playlist spec does rely somewhat on floating point; no more so that a seek to date or seek to time does in a regular HLS playlist, however.  I’m not sure that anyone is asking for precise millisecond switching from these images to regular AV.

I see 2 solutions to this problem: give a PTS/timescale in the HLS playlist (something like we did for transport stream to webVTT timing, but in the playlist), or wrap the jpeg in some sort of wrapper with timing (fmp4?).  It would be good, if that is the route, to have guidance from Apple on what specification to use.

The thing about JPEGs is that they are easy; almost any software decodes them; wrapping them in FMP4 doesn’t make it easier or better.

[NW] Thumbnails in the DASH-IF IOP are simple jpeg images and it makes it easy to produce and to manipulate on the service side (like aggregating several live thumbnails into tiles of thumbnails when a program is transitioned from Live to VOD). Using the same simple image container would allow direct interoperability with DASH, without requiring an additional CMAF+DASH specification cycle. As regards HLS, I was hoping that the use of EXT-PROGRAM-DATE-TIME would become mandatory, as per the preliminary LL-HLS specification. That would give use the millisecond-accurate time reference that we need to avoid drifts if we keep images in a simple image container.