Re: [Qlog] qlog in pcap-ng format

Robin MARX <> Mon, 10 May 2021 08:35 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id BB3EF3A0FE2 for <>; Mon, 10 May 2021 01:35:40 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.097
X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (1024-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id PNY5SAxBfNtf for <>; Mon, 10 May 2021 01:35:36 -0700 (PDT)
Received: from ( [IPv6:2a00:1450:4864:20::436]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id B94353A0FE5 for <>; Mon, 10 May 2021 01:35:35 -0700 (PDT)
Received: by with SMTP id a4so15669966wrr.2 for <>; Mon, 10 May 2021 01:35:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=liEMTGBRQIfuznf17+tk4lmFMbnSSKGHob1cNqKsRo4=; b=vI25X42lazAtnYpGZXqamKCX6DNQnIxe6rPMliuAToxdBykS6/+/UEIybVazgvkbBg iGkjeyh8Gka3sl329Y++jpDFohv9na8rMmaCm+iYMqZMnh3mb0fnY9mG/yUop2U9UniR qV6sXNI9CT5Nf7aNohIuja2ddvtAXSyg1UU/c=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=liEMTGBRQIfuznf17+tk4lmFMbnSSKGHob1cNqKsRo4=; b=j+ISoN+4gTfrRSHIXbC1qByKvB1ri3f/5XVwD1FUOGQbJfLfVDebAFz35fqyOU3FY/ yuOcQEl5547YPUb2ARj883Fs3BXxUTslF/jnXLoL5zeyhsI0B9TIKbRYGkmhMz5I1CPY kASUeTygVDJUwpuMCzU/R+KXDHu5qJKACUCPNVvt4PnQ3ZGw+nwvjldVLpj02Kw1CnS3 lyP9cFqnCONZmerJV7cSTr0yhMeygbD5/24fgJI9XTXS9+EQ4b5uRebuTeD1fhYU5F4B 4e8eVwmuxyX0KSL/pyafR6EI+QjJxLAvixWQKDITWdElln/xsUMkK5LIoV15rRQu8wzj +dfg==
X-Gm-Message-State: AOAM532F0kNaN+fXqXK49VzeIJHJM+1m6BHr8FSBPj0w4sE2G2+XR4g7 oDlxSDm/uUiA07hQS6IkeujS4jfWJOmUtpnq+Zg6Cw==
X-Google-Smtp-Source: ABdhPJzuLv45GCgt2qbftgWR+p+vTy5ITMrWCuVKbGi20Y241UtiIEbxk3eeMWx2m0Em5+1yB5qUpnEs5q8x15//jcM=
X-Received: by 2002:adf:ffcc:: with SMTP id x12mr29565871wrs.162.1620635732612; Mon, 10 May 2021 01:35:32 -0700 (PDT)
MIME-Version: 1.0
References: <32242.1620502978@localhost>
In-Reply-To: <32242.1620502978@localhost>
From: Robin MARX <>
Date: Mon, 10 May 2021 10:35:16 +0200
Message-ID: <>
Subject: Re: [Qlog] qlog in pcap-ng format
To: Michael Richardson <>
Content-Type: multipart/alternative; boundary="000000000000272c6605c1f5a90b"
Archived-At: <>
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Mon, 10 May 2021 08:35:51 -0000

Hello Michael,

Thank you for your comments.

The qlog drafts are set to be adopted by the QUIC wg soon and this is
indeed still an open point of discussion.
This is tracked at
with already some thoughts on the matter mentioned there.

I agree there is significant overlap between PCAP and qlog conceptually
(though I did not know pcap was actually considered for adoption at the
However, from my understanding, the PCAP format is strongly oriented around
logging (full) raw packet contents with little support for omission of
fields / anonymization besides rough, high-level truncation (-s, snaplen?)
As such, while it would be possible to support
it would be much more difficult to support *decrypted*-quic-with-extra-comments
taking into account privacy/file size/ease of capture/...
(I'm not even sure pcap allows the logging of decrypted protocol data
as-is? And even if it does, is it still compatible with e.g., wireshark or
other pcap tools?)

Additionally, again from my understanding, pcap is a custom binary format
that's not all that easy to process/use.
IIUC, even tools like pyshark utilize tshark's XML output instead of
parsing the PCAP itself  (which has been a source of much frustration on my
This seems to go somewhat against one of the main goals of qlog, which is
to make the creation of tools and the analysis of data easier (which is one
of the main reasons to choose JSON in the first place).

That being said, I'm far from an expert on PCAP/PCAP-NG and could be wrong
on all these counts. Please feel free to educate me.
However, if I'm not, I feel it's still best to continue qlog as a separate,
complementary approach to pcaps.

Finally, I've added the QUIC wg in CC, as that's where most of the
work/discussion will likely be done in the future.

With best regards,

On Sat, 8 May 2021 at 21:43, Michael Richardson <>

> I saw part of this presentation a few weeks ago someplace.
> I just watched the SAAG recording.
> 1) I think that JSON is too verbose for this stuff, but I think that CBOR
>    (described by CDDL) is probably exactly right for what you are doing.
> 2) PCAP-NG, which wireshark reads/writes, and libpcap only reads,
>    accomodates a multitude of different streams and formats, not just
>    ethernet dumps.
>    (We have USB captures, nflog captures, and stuff all sorts of
>    instruments).
> PCAP and PCAP-NG are candidates for adoption by the OPSAWG, but if
> QUIC would like to adopt them, that's okay with me.
> I think that there is a significant value here, particular when there may
> be
> a need to capture both TCP, UDP and internal logic at the same time.
> Imagine something doesn't work, because something is eating UDP packets
> with
> a particular checksum (or ethernet CRC).  It's happened more than once.
> Having the internal state that says, "I retransmitted foo", and then the
> capture from end A, that shows the encrypted QUIC packet going out, and
> then the capture from end B, which says, "I didn't get foo, I asked for it
> again"
> would be very useful to see.  That requires merging streams from both ends,
> and interleaving actual captured traffic at the same time.
> (And still doesn't require access to cleartext to debug)
> --
> Michael Richardson <>   . o O ( IPv6 IøT consulting )
>            Sandelman Software Works Inc, Ottawa and Worldwide
> --
> Qlog mailing list


dr. Robin Marx
Postdoc researcher - Web protocols
Expertise centre for Digital Media

*Cellphone *+32(0)497 72 86 94
Universiteit Hasselt - Campus Diepenbeek
Agoralaan Gebouw D - B-3590 Diepenbeek
Kantoor EDM-2.05