Re: [Hls-interest] Image-based subtitles and trickplay tracks

[Alex Giladi <[e-mail]>]

Hi Nigel,
This is very interesting!
How much bandwidth do the DVB subtitles take per language?
Best,
Alex.

On Fri, May 29, 2020 at 10:09 AM Nigel Megitt <[e-mail]>
wrote:

> A somewhat off-topic point, but something I'd like to pick up on:
>
> On 26/05/2020, 13:22, "Weil, Nicolas" <[e-mail]> wrote:
>
> >     Another question: there are requirements in the US at least that
> require the ability to change font sizes, colors, etc.   And, TBH, these
> are changes that help people world-wide.
>
> That isn't the way it looks from this side of the Atlantic! Subtitles
> globally have a lot of localised cultural idioms, and allowing
> unconstrained modification of settings like this, or making it seem okay
> for the client device or player code to decide can cause well
> thought-through authorial choices to be discarded in ways that break the
> experience for viewers.
>
> For example in the UK colours are used to indicate changes of speaker,
> sometimes multiple times per line, and in France I understand they are used
> to indicate different sources or types of sound.
>
> In examining the claim about the benefit of those options, I've so far
> been unable to find good published evidence demonstrating the benefit of
> all of those customisations, but I did commission research for the BBC that
> indicates some user preferences around customising text size.
>
> The point is, please don't design technical solutions on the _assumption_
> that those US requirements are needed or wanted globally. Of course I'm not
> saying there's anything wrong with designing technical solutions to
> _accommodate_ such requirements.
>
> Returning to the main thread, the BBC does broadcast bitmap subtitles
> according to the DVB specifications mentioned elsewhere, and that is
> considered a reasonable accessibility solution for televisions, for
> audience members who cannot hear the sound. I realise this does not answer
> Roger's question about use of image based subtitles based on IMSC Image
> Profile specifically.
>
> One important factor in favour of bitmap subtitles is that the client side
> work needed to modify the image to include the subtitles is minimised,
> which can help with synchronisation requirements. For example there is no
> question about installing fonts, using processor cycles to layout and
> rasterise text etc. For lower-end devices, this can be a helpful part of
> the solution.
>
> Nigel
>
>
>
> On 26/05/2020, 13:22, ""Weil, Nicolas" <[e-mail] <mailto:
> [e-mail]=" <[e-mail] <mailto:
> [e-mail]=> wrote:
>
>
>
>     Comments inline.
>
>
>
>     From: May, Bill <[e-mail] <mailto:
> Bill.May@disneystream=
>     ing.com> >
>     Sent: Tuesday, May 26, 2020 9:31 AM
>     To: Roger Pantos <rpantos=[e-mail] <mailto:rpantos
> =3D40=
>     [e-mail]> >
>     Cc: Weil, Nicolas <[e-mail] <mailto:
> [e-mail]> >=
>     ; [e-mail] <mailto:[e-mail]>
>     Subject: RE: [Hls-interest] Image-based subtitles and trickplay tracks
>
>
>
>
>
>     On May 22, 2020, at 11:17 AM, Roger Pantos
> <rpantos=[e-mail]=
>     f.org <mailto:rpantos=[e-mail]> > wrote:
>
>
>
>     Hello Nicholas. Thanks for bringing these up. I have some questions:
>
>
>
>     On May 20, 2020, at 3:12 PM, Weil, Nicolas <[e-mail]
> <mailto:=
>     [e-mail]> > wrote:
>
>
>
>     Hello,
>
>
>
>     We are often seeing two image-related topics causing interoperability
> probl=
>     ems as they are not currently covered by the HLS spec.. Normalizing
> the imp=
>     lementations around an official specification for these two points
> would be=
>      great:
>
>
>
>     Image-based subtitles tracks
>     For workflow reasons and charset reasons, some content owners don't
> include=
>      text-based subtitles in the live channels sources that they provide
> to dis=
>     tributors, but rather image-based subtitles (like DVB-Sub). While it's
> poss=
>     ible to transform these subtitles as IMSC1 Image Profile as per
> DASH-IF IOP=
>      section 6.4.4, there is no equivalent IMSC1 Image Profile support in
> the H=
>     LS RFC, which means that companies will continue to rely on
> proprietary for=
>     ks of the HLS RFC to support these use cases. Even if it wasn't
> supported b=
>     y Apple players, it would be tremendously helpful for interoperability
> in t=
>     he rest of the HLS ecosystem.
>
>
>
>     I'd like to understand how widely validated the Image Profile of IMSC1
> has =
>     been. Can anyone volunteer some examples where it=E2=80=99s been
> commercial=
>     ly deployed successfully? (Specifically IMSC1, vs. some other fork of
> TTML.=
>     )
>
>     [NW] IMSC1 Image profile is now supported by ATSC3, DASH-IF IOP (with
> suppo=
>     rt in dash.js) and IMF (with support in  <
> https://urldefense.proofpoint.com=
>
> /v2/url?u=3Dhttps-3A__github.com_IMFTool_IMFTool&d=3DDwMGaQ&c=3D96ZbZZcaMF4=
>
> w0F4jpN6LZg&r=3DKkevKJerDHRF9WRs8nW8Ew&m=3DdBy7sHrrIgjIMBRCpRh9IfMnwq8RKZHc=
>     gJLKmlBotj0&s=3DfBrrNLMD_eSxO9MYInkAMOhyaX6Zicdvk3HTD6p5mho&e=3D>
> IMFTool w=
>     hich development has been sponsored by Netflix and other studios
> initially)=
>     =2E
>
>
>
>     Another question: there are requirements in the US at least that
> require th=
>     e ability to change font sizes, colors, etc.   And, TBH, these are
> changes =
>     that help people world-wide.
>
>
>
>     How would you meet those requirements with bit mapped subtitles?
> Wouldn=E2=
>     =80=99t it be better to work to eliminate bitmapped subtitles
> completely?
>
>     [NW] I believe these font size/color change requirements can be
> satisfied w=
>     ith IMSC1 Text Profile which is supported in rfc8216bis since 2017.
>     As much as I=E2=80=99d like to get rid of bitmap subtitles, sometimes
> the c=
>     ontent owners cannot provide anything else than bitmaps in the source
> feed.=
>      And it=E2=80=99s very challenging to apply a reliable OCR pass on it,
> for =
>     all target languages (Latin/Cyrillic/Asian/=E2=80=A6 charsets). IMSC1
> Image=
>      Profile has got a decent industry support, and the Text Profile is
> already=
>      supported in HLS, so I would expect it to be a natural extension for
> HLS t=
>     o support also the Image Profile.
>
>
>
>     Image-based trickplay tracks
>     For player resources optimization reasons, the use of a video track as
> a tr=
>     ickplay artefact is not always possible, and a lot of player providers
> reco=
>     mmend the use of image thumbnails tracks instead of special low
> framerate v=
>     ideo tracks. DASHIF IOP section 6.2.6 covers this use case but there
> is equ=
>     ivalent support in the HLS RFC. There is the Image Media playlists HLS
> exte=
>     nsion proposal from Roku/Disney/WarnerMedia here
> https://github.com/image-m=
>     edia-playlist/spec but its relevance/adoption is currently limited by
> the f=
>     act that it's not part of the RFC. Same logic here: even if not
> supported b=
>     y Apple players which don't need it as they can leverage I-frame
> tracks, it=
>      would be super useful for the rest of the HLS ecosystem to get this
> offici=
>     ally part of the RFC.
>
>
>
>     I'd like to better understand what=E2=80=99s driving this. Is the
> limitatio=
>     n essentially one of not being able to support an AVC decoder for
> i-frame d=
>     isplay?
>
>
>
>     If that=E2=80=99s the case then it seems that putting JPEG images into
> fMP4=
>      containers and using EXT-X-I-FRAME-STREAM-INF would be a smaller
> extension=
>      to HLS, both in terms of departure from the existing approach and
> less new=
>      spec to invent.
>
>
>
>     One of the things I don=E2=80=99t love about the image-media-playlist
> spec =
>     is that it doesn=E2=80=99t follow the regular HLS timing model, where
> the m=
>     edia presentation time is defined in the media data itself. Instead it
> reli=
>     es on precise synchronization of the EXTINF values, which seems like a
> reci=
>     pe for long-term accumulation of floating point error, as well as
> difficult=
>      to achieve with multiple geographically-dispersed packagers for live.
>
>
>
>
>
>     The limitation is exactly that.  A second decoder (AVC or HEVC) is not
> avai=
>     lable on many devices.  This also makes mid-fragment switching
> difficult as=
>      well and makes switching between codecs impossible as well.
>
>
>
>     The image-media-playlist spec does rely somewhat on floating point; no
> more=
>      so that a seek to date or seek to time does in a regular HLS
> playlist, how=
>     ever.  I=E2=80=99m not sure that anyone is asking for precise
> millisecond s=
>     witching from these images to regular AV.
>
>
>
>     I see 2 solutions to this problem: give a PTS/timescale in the HLS
> playlist=
>      (something like we did for transport stream to webVTT timing, but in
> the p=
>     laylist), or wrap the jpeg in some sort of wrapper with timing
> (fmp4?).  It=
>      would be good, if that is the route, to have guidance from Apple on
> what s=
>     pecification to use.
>
>
>
>     The thing about JPEGs is that they are easy; almost any software
> decodes th=
>     em; wrapping them in FMP4 doesn=E2=80=99t make it easier or better.
>
>
>
>     [NW] Thumbnails in the DASH-IF IOP are simple jpeg images and it makes
> it e=
>     asy to produce and to manipulate on the service side (like aggregating
> seve=
>     ral live thumbnails into tiles of thumbnails when a program is
> transitioned=
>      from Live to VOD). Using the same simple image container would allow
> direc=
>     t interoperability with DASH, without requiring an additional
> CMAF+DASH spe=
>     cification cycle. As regards HLS, I was hoping that the use of
> EXT-PROGRAM-=
>     DATE-TIME would become mandatory, as per the preliminary LL-HLS
> specificati=
>     on. That would give use the millisecond-accurate time reference that
> we nee=
>     d to avoid drifts if we keep images in a simple image container.
>
>
>
>
> --
> Hls-interest mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/hls-interest
>