Re: [Sidrops] trying to limit RP processing variability

Stephen Kent <> Thu, 09 April 2020 19:59 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 6CE6B3A0D39 for <>; Thu, 9 Apr 2020 12:59:30 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.101
X-Spam-Status: No, score=-2.101 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id OpJJx_cCQUpx for <>; Thu, 9 Apr 2020 12:59:29 -0700 (PDT)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 2CA443A0D37 for <>; Thu, 9 Apr 2020 12:59:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=a2048; t=1586462368; bh=ZcoQhYql2wrqT7LPXuxGio2YzPrMF2geEdLjzdL4IPw=; h=Subject:To:References:From:Date:In-Reply-To:From:Subject; b=imrel3mSnWSWhqr09xL5R26DjDZCG4hCDqPfqvqjRvFy3uhevLMM8uNVFZZrFs24lAETFJJaA5+YIrU4HcxLIMoXKKLrMguaoDHk4oyz8GSGUXFQE+BKyomrTGfPpGpa+YXO7K1JkN1+krAC1H3oo0adyIHakY95CAwltuX472BvUTE+KT1ITdNCsnhsaxSOnmrZKf9pzWBn9I9DQwiKjMLgG1MJ2uD5hOhMjrVqFMe0ismT3ZlRpjfrZ/mLJUDrRQSECd62o1QEutY1BphBFdEGrnfukWd6EiyocYsyuhJathLVkWDICiM2adOBFEZ5zXKicrp7jrM5rqI/I99whA==
X-YMail-OSG: 75gq400VM1nTaDyi.UsSVTXcqRh9GKyLbXLTsUlN02SHaUnKOJovvkd809qsUjS iLL0ivBLIH687Ho1cuihPf8RRP8yjMKM4DS2DmOxH5GCbrNdG2uNs4cXCxnUk.MFyh50WlcWaQ42 4cyox.R6qeoXV_6ISIHMSURcJwQC0JhsBaHsUU0PojKxIXdNhaStn5qqkoCqorWdRZai9PUatbdL RT2iU6UaiwWftA1.7BFFVjQh59E6.2ag1JZ0SD8yhkpjN23NGHflQlQCfOFtajM2.Pn4q4OZ2.Vx Xz_S3Hspq.5K4OJPe8ZbZbjWMK6RYStwzXm58NzDyBbWwnqYXcOQzjUKRwl_4TMjtijXAY2_zH54 NVglA5zk7IjQiAioO9yL5mN1Z05_y3pbjKQlS_BpBuaTM8o_2.yCyY6hniluHbSr5Fx_n8GotiVN 8RCdprYVEkqqtDHjtttmvsHKpIyMVuVrzzVv3iRSK9dwf38xg86IGVYhmVH4.qDRTIX4b2LTSQpO mSCeKoqJiupoKGwPxU68Tz10jwMAZLTsVKSmv8tWSUrN4fqYM8Iw0JSw..IXOO_HBgaC_N4MyTMi 9jxmS52RvQBxy2tBWHOaz0J1mGKjvtFhSgTbhO1aujDFjVfhavsPfQG2qK.AMSzXsplTmyuv28Bk QQFIZVeQ78kFfz1kZxolC3Z6CmzGUBeSPm3nHmyhT_9mx3tjQ1Vg.2H_eosSo8NCFLl.Pq4ihwKy fGRywb9NIVeEpfOZ60SSW41rsAUb5v2GDX6ziCbXSKeliZj_4dHGs2SKhSG2aKQ9AescXndXCjJz U4jEQdzL.uEHEpJ.6u01a83MpwzT5LqFQYGDVrD4PXgixltfN.GGJ3FO2hVADAxqcq2pfxbXu4SO 7QE8CQ8EA_hcLnXrN_NETB07cdkzAsQTVlCBzaG0rP.F.iOWRdQHyx7yF6ZJSWQBsBMwUZ7Fo32y XQVax1Asekdntgu8f2YhtLTJ_58sltpnsoN7giVsoNpM4pMCVNF6gBT9CYqZfPWaUR0TqWsloR.u bvnYtHuJhych0aPoWrXT.IfyAhYniq8JYfb8xaQfrdgEvs4s8aBo79LGjn0nXw8h2ISxFnz0.e6H hKnzJq_lqB9ulX4Zh57RjdCDoVYCsg91qrbxY7ZptpVGo_6Xmv2i9jGXynLmcCDFVPU7A5B8cWjL j0RHDT0KKnxv7_cyRWhxlPlKNPaWuOsl0Ioko7h3Pl.D1x76iJ4Wr.e820U4v1TGejFLoeK.cteO AyZFEylS6PhnNZL9LkHqqXKLGL0kU1F2M82_ZIsd8wdNhWIXxg.JKE93ugb5JaJISOgu68y5K74O icvcPDOPCFCWXbKC8mJjrMdMWuYVCPF0F2xkUe.znlG6FsQdm5YjwY0RT9JktzWtqGmHh6XesVyG uL4BXySK39Ed3xncolOZuEjFq7qmqR7SFfw--
Received: from by with HTTP; Thu, 9 Apr 2020 19:59:28 +0000
Received: by (Oath Hermes SMTP Server) with ESMTPA ID 16db1356950edc71143b80d88bc03477; Thu, 09 Apr 2020 19:59:26 +0000 (UTC)
To: Robert Kisteleki <>, "" <>
References: <> <> <>
From: Stephen Kent <>
Message-ID: <>
Date: Thu, 09 Apr 2020 15:59:26 -0400
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Thunderbird/68.7.0
MIME-Version: 1.0
In-Reply-To: <>
Content-Type: text/plain; charset="utf-8"; format="flowed"
Content-Transfer-Encoding: 8bit
Content-Language: en-US
X-Mailer: WebService/1.1.15620 hermes Apache-HttpAsyncClient/4.1.4 (Java/11.0.6)
Archived-At: <>
Subject: Re: [Sidrops] trying to limit RP processing variability
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: A list for the SIDR Operations WG <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Thu, 09 Apr 2020 19:59:31 -0000

> Hi,
>> Missing: an object named in a Manifest, but not available for download
>> from a PP, is termed *missing*. An RP has no obvious way to acquire
>> missing objects, but operators SHOULD be warned about which objects are
>> missing.
> IMO an "RP has no obvious way to acquire missing objects" is not
> entirely true.
> If, at the previous run, the RP fetched the relevant (now missing)
> object, then I see no reason to not use it again. Think of the previous
> run as an object a cache if you will: if you're looking for an object
> mentioned in the manifest, and you have it already (hash / name / etc.
> matches) then you can reuse it.
I probably should have said that an RP views an object as "missing" if 
the the object is not present in the RP's cache and cannot be retrieved 
from the relevant PP. Because RPs retrieve objects only if the objects 
have changed (newly added or updated), an RP would not be aware that an 
object is not present at a PP if the object is present in the RP's 
cache. I recall a famous quote about whether an object that has not been 
updated makes any noise when it is deleted from a PP, or something like 
that :-)
> Of course it can be useful to check if it still exists in the PP, but it
> seems to me the only benefit is to detect that it is missing from there
> and perhaps warn the PP operator. Otherwise the RP has a hard time
> arguing "no idea what this is since it's not there!".
I think I agree about the utility of detecting an object that has gone 
missing from a PP, when the RP has the object locally cached. But, in my 
experience, RP software was not designed to detect this case.
> For bonus points: an implementation that fetched a PP's manifest,
> detected that it's exactly the same as before, and therefore reused the
> validation outcome from the previous run would not even have to fetch
> *any* other objects from the same PP.
If the manifest was not changed, at all, then it would not be fetched, 
> The downside is that this process
> will not be able to point out the omission. The upside is saves a lot of
> resources (bandwidth, CPU and all). A further upside is that as a
> side-effect this protects against a malicious attacker (selectively)
> hiding objects.
If the only things that changed in the manifest were the updates and 
manifest number, then there would be no need to retrieve any additional 
objects, and with rsync I don't believe there would be any other file 
retrievals. I can't be sure, but I think thge BBN RPSTIR software 
behaved that way.  Do we get bonus points?
> Where this comes back to the current discussion is: would this behaviour
> be mandated, recommended, or considered a big no-no?

Which behavior? I would say recommended.