Re: [Hls-interest] Image-based subtitles and trickplay tracks

> On May 26, 2020, at 9:31 AM, May, Bill <Bill.May=40disneystreaming.com@dmarc.ietf.org> wrote:
> 
> 
> 
>> On May 22, 2020, at 11:17 AM, Roger Pantos <rpantos=40apple.com@dmarc.ietf.org <mailto:rpantos=40apple.com@dmarc.ietf.org>> wrote:
>> 
>> RPANTOS=40APPLE.COM@DMARC.IETF.ORG <mailto:RPANTOS=40APPLE.COM@DMARC.IETF.ORG> appears similar to someone who previously sent you email, but may not be that person. Learn why this could be a risk <http://aka.ms/LearnAboutSenderIdentification>	Feedback <http://aka.ms/SafetyTipsFeedback>
>> Hello Nicholas. Thanks for bringing these up. I have some questions:
>> 
>>> On May 20, 2020, at 3:12 PM, Weil, Nicolas <nicoweil@elemental.com <mailto:nicoweil@elemental.com>> wrote:
>>> 
>>> Hello,
>>>  
>>> We are often seeing two image-related topics causing interoperability problems as they are not currently covered by the HLS spec. Normalizing the implementations around an official specification for these two points would be great:
>>>  
>>> Image-based subtitles tracks
>>> For workflow reasons and charset reasons, some content owners don't include text-based subtitles in the live channels sources that they provide to distributors, but rather image-based subtitles (like DVB-Sub). While it's possible to transform these subtitles as IMSC1 Image Profile as per DASH-IF IOP section 6.4.4, there is no equivalent IMSC1 Image Profile support in the HLS RFC, which means that companies will continue to rely on proprietary forks of the HLS RFC to support these use cases. Even if it wasn't supported by Apple players, it would be tremendously helpful for interoperability in the rest of the HLS ecosystem.
>> 
>> I'd like to understand how widely validated the Image Profile of IMSC1 has been. Can anyone volunteer some examples where it’s been commercially deployed successfully? (Specifically IMSC1, vs. some other fork of TTML.)
> 
> Another question: there are requirements in the US at least that require the ability to change font sizes, colors, etc.   And, TBH, these are changes that help people world-wide.
> 
> How would you meet those requirements with bit mapped subtitles?  Wouldn’t it be better to work to eliminate bitmapped subtitles completely?
> 
>> 
>>>  
>>> Image-based trickplay tracks
>>> For player resources optimization reasons, the use of a video track as a trickplay artefact is not always possible, and a lot of player providers recommend the use of image thumbnails tracks instead of special low framerate video tracks. DASHIF IOP section 6.2.6 covers this use case but there is equivalent support in the HLS RFC. There is the Image Media playlists HLS extension proposal from Roku/Disney/WarnerMedia here https://github.com/image-media-playlist/spec <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fimage-media-playlist%2Fspec&data=02%7C01%7Cbill.may%40disneystreaming.com%7C49edde2508884eb51cca08d7fe7c807e%7C65f03ca86d0a493e9e4ac85ac9526a03%7C1%7C0%7C637257683002059478&sdata=9lV%2BiNvxiK5feFCV1hYgo5I1lW0TEpMlWl%2BpwKwVKZU%3D&reserved=0> but its relevance/adoption is currently limited by the fact that it's not part of the RFC. Same logic here: even if not supported by Apple players which don't need it as they can leverage I-frame tracks, it would be super useful for the rest of the HLS ecosystem to get this officially part of the RFC.
>> 
>> I'd like to better understand what’s driving this. Is the limitation essentially one of not being able to support an AVC decoder for i-frame display? 
>> 
>> If that’s the case then it seems that putting JPEG images into fMP4 containers and using EXT-X-I-FRAME-STREAM-INF would be a smaller extension to HLS, both in terms of departure from the existing approach and less new spec to invent.
>> 
>> One of the things I don’t love about the image-media-playlist spec is that it doesn’t follow the regular HLS timing model, where the media presentation time is defined in the media data itself. Instead it relies on precise synchronization of the EXTINF values, which seems like a recipe for long-term accumulation of floating point error, as well as difficult to achieve with multiple geographically-dispersed packagers for live.
>> 
> 
> The limitation is exactly that.  A second decoder (AVC or HEVC) is not available on many devices.  This also makes mid-fragment switching difficult as well and makes switching between codecs impossible as well.
> 
> The image-media-playlist spec does rely somewhat on floating point; no more so that a seek to date or seek to time does in a regular HLS playlist, however.  I’m not sure that anyone is asking for precise millisecond switching from these images to regular AV.

The difference is that with seek to date or seek to time you only need to get close enough to download the media segment. After that the precise timestamp is in the media, which will correct any accumulated  floating-point error.

I don’t agree that “close is good enough” for all HLS clients - I know of many apps where a disagreement between the i-frame in the scrubber and what is full-screen would be a must-fix bug.

> 
> I see 2 solutions to this problem: give a PTS/timescale in the HLS playlist (something like we did for transport stream to webVTT timing, but in the playlist), or wrap the jpeg in some sort of wrapper with timing (fmp4?).  It would be good, if that is the route, to have guidance from Apple on what specification to use.

I see the second approach as requiring (considerably) less new specification. It would basically be a SHOULD that clients accept jpeg-in-fMP4 for i-frame-only playlists. 

ISOBMFF is a little light on jpeg-in-mp4 specification — essentially because it’s just MPEG-4 with a particular codec — but there’s certainly Annex H of ISO/IEC 23008-12.

> 
> The thing about JPEGs is that they are easy; almost any software decodes them; wrapping them in FMP4 doesn’t make it easier or better.

It’s about equally easy to wrap JPEG in fmp4, particularly for anything that is already producing fMP4. ffmpeg, for instance, has supported it for ages.

What fmp4 brings to the party is the well-trodden MPEG-4 / CMAF timing model, which means you don’t need to invent something new (and possibly broken in ways that are, at least initially, difficult to appreciate).

Roger.

> -- 
> Hls-interest mailing list
> Hls-interest@ietf.org
> https://www.ietf.org/mailman/listinfo/hls-interest