IETF Logo Mail Archive

Re: [Sidrops] Publication Point -> RP synchronization in bandwidth constrained environments (note for RRDP v2)

Ties de Kock <tdekock@ripe.net> Mon, 12 June 2023 18:47 UTC

Return-Path: <tdekock@ripe.net>
X-Original-To: sidrops@ietfa.amsl.com
Delivered-To: sidrops@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 83909C14CF05; Mon, 12 Jun 2023 11:47:23 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.095
X-Spam-Level:
X-Spam-Status: No, score=-2.095 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=ripe.net
Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CIRZ1U7KVKdN; Mon, 12 Jun 2023 11:47:19 -0700 (PDT)
Received: from mail-mx-1.ripe.net (mail-mx-1.ripe.net [IPv6:2001:67c:2e8:11::c100:1311]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 66C7EC14CF18; Mon, 12 Jun 2023 11:47:19 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ripe.net; s=s1-ripe-net; h=To:Message-Id:Cc:Date:From:Subject:Mime-Version:Content-Type ; bh=hC0mLYHYVz28Qgf2NZ1E/NIPs2vP9oZFjSjplqaafzc=; b=Ov20LtnLIAmu7a8+eGe+MSmY s5Ozova7NK/2wKJpd0hdmXAvUKar26YLTP8KCGcNc8ehSVFrLr7n4ELAU0Sn7Q9CKEAMFGLdY7L87 2FGOqrKf7ZjU0Tp0KiJjOoc3FpkSb9cU7e8xSev6+KFAVDZYIGOOvdXxxfOc4IbbpFjXe/Z4js4QU jWNxusuNpOwqj1nWScAssgEBwE/OV7fftwza9Gl3XoReOypqZ+pxKEJjDly4IiDShcN6zWSGCJPtv NyeGgSZulUOJuVpi4KmPh1mfcS6r/+Sb5gcvK6xI2GdNgnOXa75fitbs/QMFU9iiDE/3IfKnS5GJ+ E5BvDi340g==;
Received: from bufobufo.ripe.net ([2001:67c:2e8:23::c100:170d]:55544) by mail-mx-1.ripe.net with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from <tdekock@ripe.net>) id 1q8mZR-005vDH-3C; Mon, 12 Jun 2023 18:47:18 +0000
Received: from sslvpn.ipv6.ripe.net ([2001:67c:2e8:9::c100:14e6] helo=smtpclient.apple) by bufobufo.ripe.net with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from <tdekock@ripe.net>) id 1q8mZR-0000RO-1f; Mon, 12 Jun 2023 18:47:17 +0000
Content-Type: text/plain; charset="utf-8"
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.600.7\))
From: Ties de Kock <tdekock@ripe.net>
In-Reply-To: <ZIdndx6tgayGx17d@feather.sobornost.net>
Date: Mon, 12 Jun 2023 11:47:04 -0700
Cc: Mikhail Puzanov <mpuzanov@ripe.net>, sidrops@ietf.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <32C15EB4-CD79-4312-921C-90A3F6D27C50@ripe.net>
References: <ZHYYt77xdtrkNV1a@snel> <955CFF67-8D19-4B38-8585-3754F3119EDF@ripe.net> <ZIdndx6tgayGx17d@feather.sobornost.net>
To: Job Snijders <job=40fastly.com@dmarc.ietf.org>
X-Mailer: Apple Mail (2.3731.600.7)
X-RIPE-Signature: 059faafd1cc22ebb05e1592c815fe1e13dd55877d3624f6f6d889ae38eac63b0
Archived-At: <https://mailarchive.ietf.org/arch/msg/sidrops/zby0IyetZPyeL6606hUVN1dViKg>
Subject: Re: [Sidrops] Publication Point -> RP synchronization in bandwidth constrained environments (note for RRDP v2)
X-BeenThere: sidrops@ietf.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: A list for the SIDR Operations WG <sidrops.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/sidrops>, <mailto:sidrops-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/sidrops/>
List-Post: <mailto:sidrops@ietf.org>
List-Help: <mailto:sidrops-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/sidrops>, <mailto:sidrops-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 12 Jun 2023 18:47:23 -0000

Hi Job,

> On 12 Jun 2023, at 11:44, Job Snijders <job=40fastly.com@dmarc.ietf.org> wrote:
> 
> Dear all,
> 
> On Thu, Jun 08, 2023 at 01:45:46PM +0200, Mikhail Puzanov wrote:
>> I think compression is probably the quickest way of mitigating the
>> size problem:
> 
> Yes. Support for gzip and deflate HTTP Content-Encoding has now been
> added in OpenBSD rpki-client and will be available in a next release.
> 
>> Even the good ol’ gzip can shrink RIPE NCC’s snapshot more than
>> twofold.  Also compression should take care of the repetitive parts,
>> i.e.  XML tags, newlines, etc.
> 
> And as a nice bonus: compression (unsurprisingly) reduces the time spend
> waiting for the network.
> 
>> Extra 2 cents as an RP implementer: rpki-prover fetches every
>> repository in a separate OS process with constrained heap, constrained
>> download size and a timeout, so it pretty much ignores all
>> CVE-2021-43174-related considerations and compression was never
>> disabled in it. If the fetching process crashes or runs out of some
>> resource the impact is limited to the specific repository. So there is
>> a way to have compression and tolerate crashes.
> 
> In the rpki-client implementation a similar approach is used:
> establishing RRDP TLS connections, and HTTP decompression happens in a
> dedicated subprocess [1] which runs in a restricted-service operating
> mode using pledge() [2].
> 
> The fetches are timeboxed and size constrained (both in original and
> inflated form). Inflation happens in a streaming fashion, so even if
> HTTP Chunked transfer encoding is used, the size is checked on the fly
> and to impose the constraint. Special attention needs to be taken to
> stop the remote side from continuing to send data after Z_STREAM_END.


That is a really nice update! And a very quick turnaround. The next rpki-client
release sounds like a significant one :)

If people are interested I can give an update on the impact of compression that
we see in practice as a repository.

Kind regards,
Ties

v2.23.0 | Report a Bug | By Email