Re: [Doh] [DNSOP] Do53 vs DoT vs DoH Page Load Performance Study at ANRW

Rob Sayre <> Fri, 19 July 2019 05:56 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id EFBFA120157; Thu, 18 Jul 2019 22:56:34 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.998
X-Spam-Status: No, score=-1.998 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id NG2tw7_0IKga; Thu, 18 Jul 2019 22:56:33 -0700 (PDT)
Received: from ( [IPv6:2607:f8b0:4864:20::d41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id D98EA120025; Thu, 18 Jul 2019 22:56:32 -0700 (PDT)
Received: by with SMTP id h6so56004149iom.7; Thu, 18 Jul 2019 22:56:32 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=U5NEklj6YBO0mXKdughThqpRKwbiePRInGqQ2xZeAZI=; b=W7rt73uSqywcC0abL+rogDxF8XHvzsqc3tsPgb9g85q5+yrVG7M7b5SHOMwQZN3J0O 9JEAnSUIv97fEL/R5iYZ4kPve6oU1FjjSdl4I0r03u4gGIyaDC6OnPFAgjHEgi8klMqN +xpE9+J2Ct9P2i4bk0CyCGMnQDAFL5hmC53I74aYtE7cY9IbV/nWIUIgPkrjdEeGtYxG 2S63VM+RpagzeszCmmkgN3eZI5kuAVppYEHalyJqvBiFr4+LYqChgKHQV6zIyf3CSmqq Lyk/vEejBbhcTbjd4MslIzshpfHiRx1rWMAyQkjyWbplvSoJdHLlesx1WPHMZsJQmbl2 na3g==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=U5NEklj6YBO0mXKdughThqpRKwbiePRInGqQ2xZeAZI=; b=g5FpOq7T52kUHLOJlhwgLgqAGoHO2BMA0ASVIiursSqU4SSSWJh2bihM5g/TrsMcty fLY/qrNaVtyFqfPqBE+GQ5nFqGuVgGoKJbhJ+NJZo6MCSyEKd5C0mDYjnvEKYvuRZ8bf mz8laM5DfuxESt91/FP9gVk3o8aGLC2wzXFiYXgyVec1h4v+MiBqjnTLXjyN8PwHtCrg 12UT0wJSBk3G8KvtHDCWtiddqIuEudNgGHzrLu8khHhDzADCBes6OUUaNkyTjR4BrGFu 3h6t2YRBZmkPU3IauxP0VPkFiFfwTPwYpy3GbWameMxq/C0Orv2JBigX7SFNq53Ml1lr SrFg==
X-Gm-Message-State: APjAAAWygPrj73lafHCMiDn4bXNeIawIrEDXqIf/LQ8vi/GtP78e1UBP ex6aareKTiAe6Vc6D0Mlr0VPaNG1CZns+WmcNa4=
X-Google-Smtp-Source: APXvYqzRtCN3Xq2JMrbI2vHeYBYTYUBr5YQxKT6rm28koAdBPj9bGkxlxbDAbKBhjS64UySpzvypXuogWvXj6mkQY/s=
X-Received: by 2002:a5e:d618:: with SMTP id w24mr12275671iom.73.1563515791141; Thu, 18 Jul 2019 22:56:31 -0700 (PDT)
MIME-Version: 1.0
References: <> <> <>
In-Reply-To: <>
From: Rob Sayre <>
Date: Thu, 18 Jul 2019 22:56:19 -0700
Message-ID: <>
To: Kevin Borgolte <>
Cc:, DoH WG <>, dnsop WG <>,,,,,
Content-Type: multipart/alternative; boundary="00000000000054ef23058e026351"
Archived-At: <>
Subject: Re: [Doh] [DNSOP] Do53 vs DoT vs DoH Page Load Performance Study at ANRW
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DNS Over HTTPS <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Fri, 19 Jul 2019 05:56:35 -0000

On Thu, Jul 18, 2019 at 10:42 PM Kevin Borgolte <> wrote:

> The list of websites is attached. It is extracted from the top 1,000 and
> 99,000 to 100,000 of a Tranco list.

Thanks for attaching the list. Having seen a fair a number of these, I
think it looks reasonable.

But, I think you should add the list and the reason for the range choice to
the paper. For example, I can't tell what range you actually used from your
description (although that might just be due to a hurried reply).

Another issue is that, while your paper might accurately capture the
network conditions on your local network, it's probably doesn't capture
network variation as well as a large scale test along the lines of what
Mozilla did. For example, if the university used a single router brand,
this could skew the test. As one data point, I've never seen the various
network-throttling apps match a real-user-metrics test very well, although
they do catch really problematic situations.

This test is a welcome contribution, but given the data in the paper, it
would be difficult to reproduce.