Re: [Hls-interest] Image-based subtitles and trickplay tracks

[Nigel Megitt <[e-mail]>]

A somewhat off-topic point, but something I'd like to pick up on:

On 26/05/2020, 13:22, "Weil, Nicolas" <[e-mail]> wrote:

>     Another question: there are requirements in the US at least that require the ability to change font sizes, colors, etc.   And, TBH, these are changes that help people world-wide.
    
That isn't the way it looks from this side of the Atlantic! Subtitles globally have a lot of localised cultural idioms, and allowing unconstrained modification of settings like this, or making it seem okay for the client device or player code to decide can cause well thought-through authorial choices to be discarded in ways that break the experience for viewers.

For example in the UK colours are used to indicate changes of speaker, sometimes multiple times per line, and in France I understand they are used to indicate different sources or types of sound. 

In examining the claim about the benefit of those options, I've so far been unable to find good published evidence demonstrating the benefit of all of those customisations, but I did commission research for the BBC that indicates some user preferences around customising text size.

The point is, please don't design technical solutions on the _assumption_ that those US requirements are needed or wanted globally. Of course I'm not saying there's anything wrong with designing technical solutions to _accommodate_ such requirements.

Returning to the main thread, the BBC does broadcast bitmap subtitles according to the DVB specifications mentioned elsewhere, and that is considered a reasonable accessibility solution for televisions, for audience members who cannot hear the sound. I realise this does not answer Roger's question about use of image based subtitles based on IMSC Image Profile specifically.

One important factor in favour of bitmap subtitles is that the client side work needed to modify the image to include the subtitles is minimised, which can help with synchronisation requirements. For example there is no question about installing fonts, using processor cycles to layout and rasterise text etc. For lower-end devices, this can be a helpful part of the solution.

Nigel



On 26/05/2020, 13:22, ""Weil, Nicolas" <[e-mail] <mailto:[e-mail]=" <[e-mail] <mailto:[e-mail]=> wrote:

     
    
    Comments inline. 
    
     
    
    From: May, Bill <[e-mail] <mailto:Bill.May@disneystream=
    ing.com> > 
    Sent: Tuesday, May 26, 2020 9:31 AM
    To: Roger Pantos <rpantos=[e-mail] <mailto:rpantos=3D40=
    [e-mail]> >
    Cc: Weil, Nicolas <[e-mail] <mailto:[e-mail]> >=
    ; [e-mail] <mailto:[e-mail]> 
    Subject: RE: [Hls-interest] Image-based subtitles and trickplay tracks
    
     
    
     
    
    On May 22, 2020, at 11:17 AM, Roger Pantos <rpantos=[e-mail]=
    f.org <mailto:rpantos=[e-mail]> > wrote:
    
     
    
    Hello Nicholas. Thanks for bringing these up. I have some questions:
    
     
    
    On May 20, 2020, at 3:12 PM, Weil, Nicolas <[e-mail] <mailto:=
    [e-mail]> > wrote:
    
     
    
    Hello,
    
     
    
    We are often seeing two image-related topics causing interoperability probl=
    ems as they are not currently covered by the HLS spec.. Normalizing the imp=
    lementations around an official specification for these two points would be=
     great:
    
     
    
    Image-based subtitles tracks
    For workflow reasons and charset reasons, some content owners don't include=
     text-based subtitles in the live channels sources that they provide to dis=
    tributors, but rather image-based subtitles (like DVB-Sub). While it's poss=
    ible to transform these subtitles as IMSC1 Image Profile as per DASH-IF IOP=
     section 6.4.4, there is no equivalent IMSC1 Image Profile support in the H=
    LS RFC, which means that companies will continue to rely on proprietary for=
    ks of the HLS RFC to support these use cases. Even if it wasn't supported b=
    y Apple players, it would be tremendously helpful for interoperability in t=
    he rest of the HLS ecosystem.
    
     
    
    I'd like to understand how widely validated the Image Profile of IMSC1 has =
    been. Can anyone volunteer some examples where it=E2=80=99s been commercial=
    ly deployed successfully? (Specifically IMSC1, vs. some other fork of TTML.=
    )
    
    [NW] IMSC1 Image profile is now supported by ATSC3, DASH-IF IOP (with suppo=
    rt in dash.js) and IMF (with support in  <https://urldefense.proofpoint.com=
    /v2/url?u=3Dhttps-3A__github.com_IMFTool_IMFTool&d=3DDwMGaQ&c=3D96ZbZZcaMF4=
    w0F4jpN6LZg&r=3DKkevKJerDHRF9WRs8nW8Ew&m=3DdBy7sHrrIgjIMBRCpRh9IfMnwq8RKZHc=
    gJLKmlBotj0&s=3DfBrrNLMD_eSxO9MYInkAMOhyaX6Zicdvk3HTD6p5mho&e=3D> IMFTool w=
    hich development has been sponsored by Netflix and other studios initially)=
    =2E
    
     
    
    Another question: there are requirements in the US at least that require th=
    e ability to change font sizes, colors, etc.   And, TBH, these are changes =
    that help people world-wide.
    
     
    
    How would you meet those requirements with bit mapped subtitles?  Wouldn=E2=
    =80=99t it be better to work to eliminate bitmapped subtitles completely?
    
    [NW] I believe these font size/color change requirements can be satisfied w=
    ith IMSC1 Text Profile which is supported in rfc8216bis since 2017. 
    As much as I=E2=80=99d like to get rid of bitmap subtitles, sometimes the c=
    ontent owners cannot provide anything else than bitmaps in the source feed.=
     And it=E2=80=99s very challenging to apply a reliable OCR pass on it, for =
    all target languages (Latin/Cyrillic/Asian/=E2=80=A6 charsets). IMSC1 Image=
     Profile has got a decent industry support, and the Text Profile is already=
     supported in HLS, so I would expect it to be a natural extension for HLS t=
    o support also the Image Profile.
    
     
    
    Image-based trickplay tracks
    For player resources optimization reasons, the use of a video track as a tr=
    ickplay artefact is not always possible, and a lot of player providers reco=
    mmend the use of image thumbnails tracks instead of special low framerate v=
    ideo tracks. DASHIF IOP section 6.2.6 covers this use case but there is equ=
    ivalent support in the HLS RFC. There is the Image Media playlists HLS exte=
    nsion proposal from Roku/Disney/WarnerMedia here https://github.com/image-m=
    edia-playlist/spec but its relevance/adoption is currently limited by the f=
    act that it's not part of the RFC. Same logic here: even if not supported b=
    y Apple players which don't need it as they can leverage I-frame tracks, it=
     would be super useful for the rest of the HLS ecosystem to get this offici=
    ally part of the RFC.
    
     
    
    I'd like to better understand what=E2=80=99s driving this. Is the limitatio=
    n essentially one of not being able to support an AVC decoder for i-frame d=
    isplay? 
    
     
    
    If that=E2=80=99s the case then it seems that putting JPEG images into fMP4=
     containers and using EXT-X-I-FRAME-STREAM-INF would be a smaller extension=
     to HLS, both in terms of departure from the existing approach and less new=
     spec to invent.
    
     
    
    One of the things I don=E2=80=99t love about the image-media-playlist spec =
    is that it doesn=E2=80=99t follow the regular HLS timing model, where the m=
    edia presentation time is defined in the media data itself. Instead it reli=
    es on precise synchronization of the EXTINF values, which seems like a reci=
    pe for long-term accumulation of floating point error, as well as difficult=
     to achieve with multiple geographically-dispersed packagers for live.
    
     
    
     
    
    The limitation is exactly that.  A second decoder (AVC or HEVC) is not avai=
    lable on many devices.  This also makes mid-fragment switching difficult as=
     well and makes switching between codecs impossible as well.
    
     
    
    The image-media-playlist spec does rely somewhat on floating point; no more=
     so that a seek to date or seek to time does in a regular HLS playlist, how=
    ever.  I=E2=80=99m not sure that anyone is asking for precise millisecond s=
    witching from these images to regular AV.
    
     
    
    I see 2 solutions to this problem: give a PTS/timescale in the HLS playlist=
     (something like we did for transport stream to webVTT timing, but in the p=
    laylist), or wrap the jpeg in some sort of wrapper with timing (fmp4?).  It=
     would be good, if that is the route, to have guidance from Apple on what s=
    pecification to use.
    
     
    
    The thing about JPEGs is that they are easy; almost any software decodes th=
    em; wrapping them in FMP4 doesn=E2=80=99t make it easier or better.
    
     
    
    [NW] Thumbnails in the DASH-IF IOP are simple jpeg images and it makes it e=
    asy to produce and to manipulate on the service side (like aggregating seve=
    ral live thumbnails into tiles of thumbnails when a program is transitioned=
     from Live to VOD). Using the same simple image container would allow direc=
    t interoperability with DASH, without requiring an additional CMAF+DASH spe=
    cification cycle. As regards HLS, I was hoping that the use of EXT-PROGRAM-=
    DATE-TIME would become mandatory, as per the preliminary LL-HLS specificati=
    on. That would give use the millisecond-accurate time reference that we nee=
    d to avoid drifts if we keep images in a simple image container.