Re: [Cellar] Matroska support for closed captions?

Hendrik Leppkes <h.leppkes@gmail.com> Mon, 27 April 2020 19:15 UTC

Return-Path: <h.leppkes@gmail.com>
X-Original-To: cellar@ietfa.amsl.com
Delivered-To: cellar@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 917663A1AD0 for <cellar@ietfa.amsl.com>; Mon, 27 Apr 2020 12:15:42 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.098
X-Spam-Level:
X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id En1Fs91uiq-K for <cellar@ietfa.amsl.com>; Mon, 27 Apr 2020 12:15:40 -0700 (PDT)
Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id D3A033A1ACD for <cellar@ietf.org>; Mon, 27 Apr 2020 12:15:39 -0700 (PDT)
Received: by mail-wr1-x42a.google.com with SMTP id i10so21869936wrv.10 for <cellar@ietf.org>; Mon, 27 Apr 2020 12:15:39 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :content-transfer-encoding; bh=X3ZgfKERyiSorGosz3KGpiP6/0qbCTnPT1sq4V4LNU8=; b=hbz76NjqFlY5P+d0IAWvYgDcpJXeIXZXb3ytumXZR41hT1FvcCFEKoxdoje8C6/0FD JUByqEdgXV/c7dMqDTc4jcRaPU5BxFdC2jxNtzK5niWOpJ+IoFu5qd1JYT05QZ1PSU3h QLsvyosJeIeJxyr42CNsHeUnVKhnMz6yEqDLoUYfrNIrVgNOr91Lzqu5gI0SkA7VD0OD iyI/yXbJAZ5J2ZHwncr0ZMzYKyiJKUTJLTOZm97NbOX1l1BRRYbUbTyld6I+d/KnTho1 V8EyXArtG02xsOv7G2u9G9mtZJkLPX1LbjNOpjSMoZHgunfnzrKsxCw0kLkoehDOkK83 KkGQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:content-transfer-encoding; bh=X3ZgfKERyiSorGosz3KGpiP6/0qbCTnPT1sq4V4LNU8=; b=cn2B7XG+Lu8wyyIxIUZQir2XDgO1vdUJkUeCBVw82EuaICEs+kNLXl89xX/kEDMbWw mVc0XJ9K1VyCQ6EPyC9VkjbQ76EFkOPzyFb0PE2Mkb2L89hvUU7IGBfZPtNCL4Qtbtnb 0LD1Q8UL70Y8yM3bYK/fGUlmlb6AY9Bhj3Hmf0np8WkvQ5zmLfEr6gOKrYdFiiT5+v7m hJWsXWp1QoB9Lj3FCU0/SndE8nlRFq1d232vH4c5yyLqt6tsVed4bx8rOpWi4t4YeRCf u8L3SsqlivBP9C0/dSoMIIIwvIxkCw4hto0p7AeKoBbBBUBc/CqIlAreMPiyQOQObblx CKTQ==
X-Gm-Message-State: AGi0PuaXlyMj6oVIjLMikBqiCUxGz58dg10xoHcTI8F4gMs74b9Sy80Z xMJLgyZixRWya0nuj+7IQvI2n9XQCsM0Rslpn4whgsh0NYo=
X-Google-Smtp-Source: APiQypJHwqBNU1zw0AVsptoe5UQe0aCPgyZHjMt2kxrkefkpuYJiiOFfZ+tDnpVkDRD43wuxMcsBH6uTq+6hBHay3QQ=
X-Received: by 2002:a5d:45cf:: with SMTP id b15mr28210660wrs.78.1588014937657; Mon, 27 Apr 2020 12:15:37 -0700 (PDT)
MIME-Version: 1.0
References: <CY4PR04MB072989A9F039E5DFCBBF1E40C9D20@CY4PR04MB0729.namprd04.prod.outlook.com> <514D57FC-BD77-45E7-846A-046363BED953@dericed.com> <6e93c666-4414-abab-0f2e-fbe6beaec904@mediaarea.net> <CAGoCfixnxp6zxSRyze3DVT=hTGK81TDUgZjWMagZAkHUcn0kEw@mail.gmail.com>
In-Reply-To: <CAGoCfixnxp6zxSRyze3DVT=hTGK81TDUgZjWMagZAkHUcn0kEw@mail.gmail.com>
From: Hendrik Leppkes <h.leppkes@gmail.com>
Date: Mon, 27 Apr 2020 21:15:27 +0200
Message-ID: <CA+anqdyMd1bKhnJBYLxf1N3hssB2Mh1f0V+TJc+Z7nFvAWZWKA@mail.gmail.com>
To: CELLAR list <cellar@ietf.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: <https://mailarchive.ietf.org/arch/msg/cellar/9AL80A8wN2U39vyLs74aPvlHf8o>
Subject: Re: [Cellar] Matroska support for closed captions?
X-BeenThere: cellar@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Codec Encoding for LossLess Archiving and Realtime transmission <cellar.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/cellar>, <mailto:cellar-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/cellar/>
List-Post: <mailto:cellar@ietf.org>
List-Help: <mailto:cellar-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/cellar>, <mailto:cellar-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 27 Apr 2020 19:15:43 -0000

On Mon, Apr 27, 2020 at 6:54 PM Devin Heitmueller
<dheitmueller@kernellabs.com> wrote:
>
> On Mon, Apr 27, 2020 at 12:09 PM Jerome Martinez <jerome@mediaarea.net> wrote:
> >
> > On 27/04/2020 17:46, Dave Rice wrote:
> > >
> > > For EIA-608 and EIA-708 data I suppose, either a new type of Codec Mapping could be written (and then these would be a new type of subtitle track) or they could be support as side-data with a BlockAdditionalMapping as described at https://www.ietf.org/id/draft-ietf-cellar-codec-04.html#name-block-additional-mapping. I’m unsure which technique the community would recommend but curious for some discussion on it.
> >
> >
> > As MXF have JPEG-2000, I guess that captions are in a standalone
> > ancillary data (SMPTE ST 291) track.
> > As a first shot, I would not be in favor of BlockAdditionalMapping
> > because the track does not depend on data in another bistream.
> >
> > then there would (at least) 2 choices:
> > - demux of the track from MXF and mux of this track as is; advantage is
> > that we lose no other data inside the ancillary data, disadvantage is
> > that lot of different content (several formats of captions, bar data,
> > VBI, time codes, Acquisition Metadata...) are muxed in the "raw" stream,
> > more difficult for a demuxer/player.
> > - demux of the captions from Ancillary data from MXF; advantage is that
> > we have 1 track per content stream, disadvantage is that we may lose
> > some other "opaque" data from the ancillary data.
> >
> > That said, we could propose both.
> >
> > Jérôme
>
> For what it's worth, we have the same general problem within ffmpeg.
> File formats like MXF and MP4 treat captions and timecodes as a
> separate track, while other formats like TS expect it to be tied to
> the individual video frames as side data.  There's no easy answer, and
> in particular it's a huge pain when you need to convert from one to
> the other (e.g. extracting captions from the SEI in an H.264 TS where
> it gets treated as video frame side data, and creating an MP4 where
> captions need their own subtitle track).
>

Well, its more like TS does not handle closed captions at all, which
is why they get thrown in with the video. Its not a TS concept, but a
video one. If we have the option to carry them seperately, then I
believe we should. It makes it so much easier for everything involved
- and there is precedent for that in eg. MP4 already as well.
You can always go the "TS way" with Matroska as well if you must,
since its entirely container-agnostic, but if container support is
being build, I see no reason to handle them differently then "proper"
subtitles.

- Hendrik