qlog: new streaming serialization format

Robin MARX <robin.marx@uhasselt.be> Wed, 20 October 2021 14:19 UTC

Return-Path: <robin.marx@uhasselt.be>
X-Original-To: quic@ietfa.amsl.com
Delivered-To: quic@ietfa.amsl.com
Received: from localhost (localhost []) by ietfa.amsl.com (Postfix) with ESMTP id EFC843A0970 for <quic@ietfa.amsl.com>; Wed, 20 Oct 2021 07:19:30 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.099
X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=uhasselt.be
Received: from mail.ietf.org ([]) by localhost (ietfa.amsl.com []) (amavisd-new, port 10024) with ESMTP id TlUKYP9LjMOB for <quic@ietfa.amsl.com>; Wed, 20 Oct 2021 07:19:25 -0700 (PDT)
Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 48F8C3A0898 for <quic@ietf.org>; Wed, 20 Oct 2021 07:19:24 -0700 (PDT)
Received: by mail-wm1-x334.google.com with SMTP id 77-20020a1c0450000000b0030da2439b21so2203473wme.0 for <quic@ietf.org>; Wed, 20 Oct 2021 07:19:24 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uhasselt.be; s=google; h=mime-version:from:date:message-id:subject:to; bh=NtNjqCGMwOScsllQDg8/hWz/sQK7lsEqPXvElFt8ORM=; b=AsyyBtda0ho9wV9QPuw1mzHpDOiKD0GjQni177NA4IRmvW1IRoMVaMQGI2rGMh8s/7 WkfvOn6gPydcnVhn/ADLnZ52HT0xgCFOSWpZlKFxlixBzBBLVmqYGavXttjUfKtuosjg rRfse9ZCQuMA9EjDuPkmcKuVMLzQggCvdSL3E=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=NtNjqCGMwOScsllQDg8/hWz/sQK7lsEqPXvElFt8ORM=; b=NeeWQkcF6N7xW4FE2zklYIsvlSDHJ+7zhK9Eq8Cb40pbDrEOcvSNxq6Vrd4tr/IUkv czOLfwcWLT/93RhXv9Oj8hFOTrrkY3mBdFp82qQjvvY6lAYKIEgbrmruVvEtNxOeQTlS OE0LjCstTmZI0yteEtALlErnvO9tDFIebFpqgxRkD32zIRty2LiqiK/RdVAFjSR4tLIj n/4NTX4BYSovimz3BDWVSY1KhZWQ0ufcN4iBLpnBrDLc4+t3PDJ5uk5iMprCQeZggfdr 5+stmnvOUMrvYpsEcbA54BF7BnA4WQtbVXLzO76XntSP0VwFXvoTByMP5+c5yKSvo+pW 9Hhw==
X-Gm-Message-State: AOAM5320YtC/et0e4uW6HWOnzDxalR3P8VJZgwZFoz0uCSBjSKX8+Vn3 kVDT+vLX1j5o9bP8N7aBLxreTUaBQihuM1hWWzGAK5rqe/8=
X-Google-Smtp-Source: ABdhPJzK/1yuXlK9HZIflBw4IlAeMZLSbgfGktTlW1DHvXH5HD+6U84tzMR94pix5esMvP5j8C/Mix6iizV8FcMLezM=
X-Received: by 2002:a05:6000:1563:: with SMTP id 3mr66954wrz.20.1634739560622; Wed, 20 Oct 2021 07:19:20 -0700 (PDT)
MIME-Version: 1.0
From: Robin MARX <robin.marx@uhasselt.be>
Date: Wed, 20 Oct 2021 16:19:16 +0200
Message-ID: <CAC7UV9ZqyDKAVctj6Q=NfAG=-sXvBJ+2fwScrWd8qhkUkKjPLw@mail.gmail.com>
Subject: qlog: new streaming serialization format
To: IETF QUIC WG <quic@ietf.org>
Content-Type: multipart/alternative; boundary="000000000000cfc14205cec97606"
Archived-At: <https://mailarchive.ietf.org/arch/msg/quic/ikWj1UHov7Lbp1kO6TjlyChWIWo>
X-BeenThere: quic@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <quic.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/quic>, <mailto:quic-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/quic/>
List-Post: <mailto:quic@ietf.org>
List-Help: <mailto:quic-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/quic>, <mailto:quic-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Oct 2021 14:19:31 -0000

Hello everyone,

As discussed during IETF 111, we are looking to update qlog to move to a
new streaming serialization format.

The current version uses the "Newline Delimited JSON" (NDJSON) format for
event streaming, which, while it works well, is not an official standard
(in the IETF or elsewhere).

As such, it was proposed we move to the "JSON text-sequences" format
defined in RFC 7464 [1]. json-seq separates each record (qlog event in our
case) with the Record Separator character (\x1E) and a \n at the end. As
such, it should be easy to update an existing NDJSON implementation (as it
just requires outputting one additional character per event).

We have prototyped and tested this approach in qvis and various other tools
like jq and the early outcomes are looking good [2].

As such, we will be preparing a PR and associated draft-01 for IETF 112
that introduces this change.

In the meantime, it would be great if people interested in event streaming
or with existing NDJSON qlog stacks could look at the proposal/early
results [2] and provide feedback.

[1]: https://datatracker.ietf.org/doc/html/rfc7464
[2]: https://github.com/quicwg/qlog/issues/172

With best regards,
the qlog editors

P.S. note that we also still plan to keep the normal JSON non-streaming
format intact, leading to two "main" formats defined in the spec: 1) normal
JSON and 2) JSON text sequences.


dr. Robin Marx
Postdoc researcher - Web protocols
Expertise centre for Digital Media

*Cellphone *+32(0)497 72 86 94

Universiteit Hasselt - Campus Diepenbeek
Agoralaan Gebouw D - B-3590 Diepenbeek
Kantoor EDM-2.05