Re: [Cellar] Matroska support for closed captions?

Devin Heitmueller <dheitmueller@kernellabs.com> Mon, 27 April 2020 16:54 UTC

Return-Path: <dheitmueller@kernellabs.com>
X-Original-To: cellar@ietfa.amsl.com
Delivered-To: cellar@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 65B303A0F87 for <cellar@ietfa.amsl.com>; Mon, 27 Apr 2020 09:54:00 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.899
X-Spam-Level:
X-Spam-Status: No, score=-1.899 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=kernellabs-com.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hWjkCZ6At1YO for <cellar@ietfa.amsl.com>; Mon, 27 Apr 2020 09:53:58 -0700 (PDT)
Received: from mail-io1-xd2b.google.com (mail-io1-xd2b.google.com [IPv6:2607:f8b0:4864:20::d2b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id AA9C23A0F70 for <cellar@ietf.org>; Mon, 27 Apr 2020 09:53:58 -0700 (PDT)
Received: by mail-io1-xd2b.google.com with SMTP id y26so9722085ioj.2 for <cellar@ietf.org>; Mon, 27 Apr 2020 09:53:58 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernellabs-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=MiMlDzStkYfnGUh7kcyAVwsn9apR6hhaQOiDvP4WbP8=; b=bwSexdHbbv0KszsgcrXBqAIWXGwXrTLsXapH8G4iNW7URnusSWUf7AJrL2xmuVli9B 6VFXF9IDm3rvrzWDX7QRfHoxACZUDCfglbDpC3KB6czD8kWekr894+B9xRO1GTCuBZf1 5t42iSA7aIgc829GfEn4hTot2PRMDqXRAU4uxkhx1VJSTJvfvwT52O1YX9yLRJQWGMne aXdtFT2KI9EhIhQVsowkfTe/FBYneY8TcQfl9eDZgEkmtLaS7+MS6I5bkT1mWYT7g/k1 rsxEQKovO+XboKznbIAqkC22Kl6Iai1ahJn+V5wmqk2l0tbuaQmmT+RDybAFOtteJp2H SCYw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=MiMlDzStkYfnGUh7kcyAVwsn9apR6hhaQOiDvP4WbP8=; b=hkss+FTxjVSsgf7GSIfuN4Zwx/10wPW+GE7/goIStxu7ocKILG86qblSEFpKu9cF70 stU7ACWapxkhHKcGxJ8QaWcu1rgARxQTdOJiploSYg8k0ti8LMa2qrEYFP2kI7iPb+MV lEJLbs85lJ8flMupTqkZQDYC7WKkODKjU8ilmAzYvvQn/0iJ9cJaV6ZOGhu52YXwFVrd OqME9rBoEsRoPGjoPhCLv69DIy0wBanatu/ws1aVwH0QtGnu4Jd8jV05Q6lf5HI/S+ut Cu8LWBFSpzkXQf6d8mBUQkGKDWkafNhTgHHHxVSHIm6bk9HDQEyfrq2UyI87dXeATVxn ZMUg==
X-Gm-Message-State: AGi0PubOaX1OwBSJgFZ8C3vBbhxD53CKUcD2UN1D3RHj8fM7gGPDTiWO u1GtiwW7VciZAAyExkZaOfT98LzydhsLxL+u24XKw9x5
X-Google-Smtp-Source: APiQypLUSNQtUbIzWqHRwIawkf9HUZqdvCee3OimL/zQEaJ/DNKGViMVaTAVgJkvFQ6qWSX4WPUtBHmy5V9xcYsCZw8=
X-Received: by 2002:a5e:9b13:: with SMTP id j19mr21508846iok.86.1588006437537; Mon, 27 Apr 2020 09:53:57 -0700 (PDT)
MIME-Version: 1.0
References: <CY4PR04MB072989A9F039E5DFCBBF1E40C9D20@CY4PR04MB0729.namprd04.prod.outlook.com> <514D57FC-BD77-45E7-846A-046363BED953@dericed.com> <6e93c666-4414-abab-0f2e-fbe6beaec904@mediaarea.net>
In-Reply-To: <6e93c666-4414-abab-0f2e-fbe6beaec904@mediaarea.net>
From: Devin Heitmueller <dheitmueller@kernellabs.com>
Date: Mon, 27 Apr 2020 12:53:46 -0400
Message-ID: <CAGoCfixnxp6zxSRyze3DVT=hTGK81TDUgZjWMagZAkHUcn0kEw@mail.gmail.com>
To: Jerome Martinez <jerome@mediaarea.net>
Cc: cellar@ietf.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: <https://mailarchive.ietf.org/arch/msg/cellar/432mX7N9YrQyh8EtU2D8yXUgHVo>
Subject: Re: [Cellar] Matroska support for closed captions?
X-BeenThere: cellar@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Codec Encoding for LossLess Archiving and Realtime transmission <cellar.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/cellar>, <mailto:cellar-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/cellar/>
List-Post: <mailto:cellar@ietf.org>
List-Help: <mailto:cellar-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/cellar>, <mailto:cellar-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 27 Apr 2020 16:54:01 -0000

On Mon, Apr 27, 2020 at 12:09 PM Jerome Martinez <jerome@mediaarea.net> wrote:
>
> On 27/04/2020 17:46, Dave Rice wrote:
> >
> > For EIA-608 and EIA-708 data I suppose, either a new type of Codec Mapping could be written (and then these would be a new type of subtitle track) or they could be support as side-data with a BlockAdditionalMapping as described at https://www.ietf.org/id/draft-ietf-cellar-codec-04.html#name-block-additional-mapping. I’m unsure which technique the community would recommend but curious for some discussion on it.
>
>
> As MXF have JPEG-2000, I guess that captions are in a standalone
> ancillary data (SMPTE ST 291) track.
> As a first shot, I would not be in favor of BlockAdditionalMapping
> because the track does not depend on data in another bistream.
>
> then there would (at least) 2 choices:
> - demux of the track from MXF and mux of this track as is; advantage is
> that we lose no other data inside the ancillary data, disadvantage is
> that lot of different content (several formats of captions, bar data,
> VBI, time codes, Acquisition Metadata...) are muxed in the "raw" stream,
> more difficult for a demuxer/player.
> - demux of the captions from Ancillary data from MXF; advantage is that
> we have 1 track per content stream, disadvantage is that we may lose
> some other "opaque" data from the ancillary data.
>
> That said, we could propose both.
>
> Jérôme

For what it's worth, we have the same general problem within ffmpeg.
File formats like MXF and MP4 treat captions and timecodes as a
separate track, while other formats like TS expect it to be tied to
the individual video frames as side data.  There's no easy answer, and
in particular it's a huge pain when you need to convert from one to
the other (e.g. extracting captions from the SEI in an H.264 TS where
it gets treated as video frame side data, and creating an MP4 where
captions need their own subtitle track).

Devin

-- 
Devin J. Heitmueller - Kernel Labs
http://www.kernellabs.com