Broader discussion - limit dictionary encoding to one compression algorithm?
Patrick Meenan <patmeenan@gmail.com> Tue, 21 May 2024 15:02 UTC
Return-Path: <ietf-http-wg-request+bounce-httpbisa-archive-bis2juki=ietf.org@listhub.w3.org>
X-Original-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Delivered-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 58791C14F6A7 for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Tue, 21 May 2024 08:02:47 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.85
X-Spam-Level:
X-Spam-Status: No, score=-2.85 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249, HTML_MESSAGE=0.001, MAILING_LIST_MULTI=-1, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=w3.org header.b="NBaJX1zk"; dkim=pass (2048-bit key) header.d=w3.org header.b="dpbGpWQ3"; dkim=pass (2048-bit key) header.d=gmail.com header.b="Eq2mmEo/"
Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mcDFprQWzSD5 for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Tue, 21 May 2024 08:02:46 -0700 (PDT)
Received: from mab.w3.org (mab.w3.org [IPv6:2600:1f18:7d7a:2700:d091:4b25:8566:8113]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 6DF33C14F61A for <httpbisa-archive-bis2Juki@ietf.org>; Tue, 21 May 2024 08:02:41 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=w3.org; s=s1; h=Subject:Content-Type:To:Message-ID:Date:From:MIME-Version:Cc:Reply-To :In-Reply-To:References; bh=oXrndNzEjnhOTXcERgZEed8RgBsiZtXGOX/5Poh4TsM=; b=N BaJX1zkzaWqMwST+DiOIFl/XrURRcBPKfFl5OxdTTEOBEM+K7MYn1lw+icJINz4+30H+6ZA1x/vai w4nkSkbQkCZl9LvD62LOt+3rU0e+uSTLzmtUQaiVOIo8HlZ1B27T0qsYlFBCR4AiOycRlPlcBGM4y gNLt7cRq/A6JywrjwcIZKyvX5mKFSe201o0m24E4H3/Xfn6UAvrHb67uYOm6j62VoYK4aH0lbp2+C R5SaTLG7+VTklx/lRE6efAQvJl7ySgO2EFYmWm6Zp5rjqyEQCLZ+m5++eLoyvWGTy5bqpg8ruHXVE 6wnXCE0UMbvY7qxcjHu38Kt++r/LwpTLg==;
Received: from lists by mab.w3.org with local (Exim 4.96) (envelope-from <ietf-http-wg-request@listhub.w3.org>) id 1s9Qzi-00Ey6f-1Q for ietf-http-wg-dist@listhub.w3.org; Tue, 21 May 2024 15:01:38 +0000
Resent-Date: Tue, 21 May 2024 15:01:38 +0000
Resent-Message-Id: <E1s9Qzi-00Ey6f-1Q@mab.w3.org>
Received: from ip-10-0-0-224.ec2.internal ([10.0.0.224] helo=puck.w3.org) by mab.w3.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from <patmeenan@gmail.com>) id 1s9Qzf-00Ey5k-15 for ietf-http-wg@listhub.w3.internal; Tue, 21 May 2024 15:01:35 +0000
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=w3.org; s=s1; h=Content-Type:To:Subject:Message-ID:Date:From:MIME-Version:Cc:Reply-To :In-Reply-To:References; bh=oXrndNzEjnhOTXcERgZEed8RgBsiZtXGOX/5Poh4TsM=; t=1716303695; x=1717167695; b=dpbGpWQ3Dped4e4mtusMTGnyXHVE2HlDGFLmJiPiyqol5Bv BMEvxCjHtmjEXFAGlEMAzd5LOGWWjE/gOscmwJDQcn1w6+56UMfmv6tTd1JD6DNFd87+U6l3o4MAs hFdegfLtrp/n10BHP3+KycM5pchZvHnxD86uMjF/zW8nqP9vY2G7y+w1qiQkwFfRs+YUZFqJQZICV MnsBviYnxVsBGYz8+DJ4C7POlAVXWkwSj78FGMFm2ZvUe7SVAcRAouWf/hVW3XNgj3lc1Fnzw5V7B KH0AkMUcNW6LMmhEXtjlAYx4lN2d5MUNX24lqk67LxCEDzOT78DYhs7AbX90HoFA==;
Received-SPF: pass (puck.w3.org: domain of gmail.com designates 2a00:1450:4864:20::52b as permitted sender) client-ip=2a00:1450:4864:20::52b; envelope-from=patmeenan@gmail.com; helo=mail-ed1-x52b.google.com;
Received: from mail-ed1-x52b.google.com ([2a00:1450:4864:20::52b]) by puck.w3.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.96) (envelope-from <patmeenan@gmail.com>) id 1s9Qze-00H4NO-20 for ietf-http-wg@w3.org; Tue, 21 May 2024 15:01:35 +0000
Received: by mail-ed1-x52b.google.com with SMTP id 4fb4d7f45d1cf-56e6affdd21so3126830a12.3 for <ietf-http-wg@w3.org>; Tue, 21 May 2024 08:01:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716303690; x=1716908490; darn=w3.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=oXrndNzEjnhOTXcERgZEed8RgBsiZtXGOX/5Poh4TsM=; b=Eq2mmEo/hZKGDMyL5Ks9BqLEHm+9F6YfllO/VJEsDDcpopwhw+5koe6X2WOMuVZHXV VoM5ATLGWCFsmYkuteWUms2Ids048wP5NctyeKf4+OOkIOzGUUwtSqT9W72pSfSHkCyO rBoaFWrXgbWZdaAcwNpjg/u2zS6Y02ZHuL05E8yeCOO27n7B51Q8Okws56WbRAMGjpZM YNfnXIsCFg7XY663I9/7Zwm/uAF52195vR1POOnnh4L23BWDOkebTDGlAMI4qxAIrYTB sLEXUIXM+J/w1lQ8gsKrrnwEMxJ8eyHG1VTf6LuWZU64M+kkb9Ud7eCCoTLkRAfJ5Pcf nU4g==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716303690; x=1716908490; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=oXrndNzEjnhOTXcERgZEed8RgBsiZtXGOX/5Poh4TsM=; b=pYzXgWRL9gIsGcEeLFA7xUF02a+tfm3Mt1OFacw+YXqPqrJfkkxWdcq3TLthbGZbi5 l4/V+Dm4jYMcNoWiABer2NOgEycGKx+CyfnFNJ6ShdbeEyXn0W++JLG7bpocdLWhUWsX C9kEC8cRMY7LUU/WrVVqROHlkivdMpTJwB/hNflK4IaU6FKqvZuzgGjH0Eu+LQp4VmgI gwtMD9aAfTn3A8Jv9akJOiwJP6PMcLUgAUW+xZAE/NSSfiBD+WJ+fdeL4j9jwdegs7Eo NTICemnk80DLriQmf9jSHRILg0Me+5WqTc0fsjduTjBg4zEkB/uJwWdWNgvZVzsU3XOY 163w==
X-Gm-Message-State: AOJu0Yw2zzQmOZZjwRknBOj+be5vORl0N0AOOgtxw+7Rz1KwakeAXEbP hFgxKGqXDvVwDve2o7fJlE3l15Rr+p8+kRKaHxBzU+Lp40Znx7o/4v9OyL9+CO1ssYBb0aU1xES p9Ihf2FXhu3R538Kjd8ZUhH0HQiPvft9Z
X-Google-Smtp-Source: AGHT+IEAkZb57vgwkbyGdAP4XLStgyKBslU2jg0tHhgN4y7+DGBI5c6Rt7qyfR+IX4bb9M/RbdYNfvnWlMEWJ5lfJfY=
X-Received: by 2002:a50:ab12:0:b0:572:3f41:25aa with SMTP id 4fb4d7f45d1cf-5734d5c0eb9mr29284524a12.11.1716303689690; Tue, 21 May 2024 08:01:29 -0700 (PDT)
MIME-Version: 1.0
From: Patrick Meenan <patmeenan@gmail.com>
Date: Tue, 21 May 2024 11:01:17 -0400
Message-ID: <CAJV+MGzjUnZZ=XFn5veOvuhVWyZNP2b9U0fxpS3UmrDC_bc_wQ@mail.gmail.com>
To: HTTP Working Group <ietf-http-wg@w3.org>
Content-Type: multipart/alternative; boundary="000000000000c0357b0618f817f1"
X-W3C-Hub-DKIM-Status: validation passed: (address=patmeenan@gmail.com domain=gmail.com), signature is good
X-W3C-Hub-Spam-Status: No, score=-5.1
X-W3C-Hub-Spam-Report: BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, DMARC_PASS=-0.001, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, W3C_AA=-1, W3C_DB=-1, W3C_WL=-1
X-W3C-Scan-Sig: puck.w3.org 1s9Qze-00H4NO-20 83d8397dc7033fca86ca2b57c52afb8f
X-Original-To: ietf-http-wg@w3.org
Subject: Broader discussion - limit dictionary encoding to one compression algorithm?
Archived-At: <https://www.w3.org/mid/CAJV+MGzjUnZZ=XFn5veOvuhVWyZNP2b9U0fxpS3UmrDC_bc_wQ@mail.gmail.com>
Resent-From: ietf-http-wg@w3.org
X-Mailing-List: <ietf-http-wg@w3.org> archive/latest/51950
X-Loop: ietf-http-wg@w3.org
Resent-Sender: ietf-http-wg-request@w3.org
Precedence: list
List-Id: <ietf-http-wg.w3.org>
List-Help: <https://www.w3.org/email/>
List-Post: <mailto:ietf-http-wg@w3.org>
List-Unsubscribe: <mailto:ietf-http-wg-request@w3.org?subject=unsubscribe>
The last open issue on the work for the compression dictionary transport draft is around the compression algorithms (content-encodings) that should be defined in the draft. Specifically, if we should spec both Brotli and Zstandard or pick one. It would be helpful to get the wider working group's opinion (and maybe some prior experience where this may have come up with br/zstd and deflate/gzip). I have a reasonably faithful summary of the two options below but if you'd like to read the full discussion, the issue is here: https://github.com/httpwg/http-extensions/issues/2756 ** The case for a single content-encoding: A single option will result in broader interoperability. Multiple choices may lead to fragmentation where clients may be forced to support both Zstandard and Brotli if there is a wide mix of servers only implementing one or the other. On the other hand, if different clients only implement one (and there is no intersection of support), then servers/sites will need to implement both to get broad benefit. If Brotli and Zstandard have similar capabilities then converging to a single encoding would be better for scaling adoption and interop. ** The case for both Brotli and Zstandard: Zstandard and Brotli are both already dictionary-aware outside of a specified content-encoding. We already have content-encoding support for both of them so why should the dictionary variants be given special treatment? Specifying the format for content-encoding the dictionary variants of both allows for interop with either format, reducing the need for a one-off private implementation if someone wants to use the format that wasn't picked. There are some tangible differences that may lead someone to choose one over the other. Some of them are implementation differences in the current libraries that may be able to improve upon and some are fundamental to the format: For the delta-encoding case, the "dictionary" is the previous version of the resource so any limits around the dictionaries end up being limits for the size of resources that are being delta-compressed. - Brotli is limited to 50MB dictionaries, Zstandard can go up to 128MB. - Brotli uses 16MB of ram for the window while compressing/decompressing independent of the dictionary size, Zstandard requires a window (RAM) as large as the resource being compressed (for the delta case). - Brotli at max compression is ~10-20% smaller than Zstandard at max compression with dictionary (current implementations). - Zstandard benefits from dictionary use across all compression levels, Brotli only benefits from dictionaries at level 5 and above (current implementations). As things stand right now, if you have resources > 50MB and < 128MB you can't use brotli to delta-encode them (even in the web case we have already seen this with some large WASM apps). If you have static resources < 50MB and can do the compression at build time you would benefit from an additional 10-20% savings by using brotli (current cli anyway). If you are compressing dynamic responses and need to limit CPU, you may benefit from using Zstandard at low compression levels (the amount of brotli level-1 that is on the web may indicate this is a common constraint). If you have existing infrastructure plumbed (security approved, etc) to support one or the other, your preference might be to use the dictionary version of the same algorithm rather than pull in a new library. Thanks, -Pat
- Broader discussion - limit dictionary encoding to… Patrick Meenan
- Re: Broader discussion - limit dictionary encodin… Poul-Henning Kamp
- Re: Broader discussion - limit dictionary encodin… Patrick Meenan
- Re: Broader discussion - limit dictionary encodin… Glenn Strauss
- Re: Broader discussion - limit dictionary encodin… Roy T. Fielding
- Re: Broader discussion - limit dictionary encodin… Jyrki Alakuijala
- Re: Broader discussion - limit dictionary encodin… Patrick Meenan
- Re: Broader discussion - limit dictionary encodin… Jyrki Alakuijala