Re: [Doh] [DNSOP] Do53 vs DoT vs DoH Page Load Performance Study at ANRW

Rob Sayre <sayrer@gmail.com> Fri, 19 July 2019 05:56 UTC

Return-Path: <sayrer@gmail.com>
X-Original-To: doh@ietfa.amsl.com
Delivered-To: doh@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id EFBFA120157; Thu, 18 Jul 2019 22:56:34 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.998
X-Spam-Level:
X-Spam-Status: No, score=-1.998 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NG2tw7_0IKga; Thu, 18 Jul 2019 22:56:33 -0700 (PDT)
Received: from mail-io1-xd41.google.com (mail-io1-xd41.google.com [IPv6:2607:f8b0:4864:20::d41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id D98EA120025; Thu, 18 Jul 2019 22:56:32 -0700 (PDT)
Received: by mail-io1-xd41.google.com with SMTP id h6so56004149iom.7; Thu, 18 Jul 2019 22:56:32 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=U5NEklj6YBO0mXKdughThqpRKwbiePRInGqQ2xZeAZI=; b=W7rt73uSqywcC0abL+rogDxF8XHvzsqc3tsPgb9g85q5+yrVG7M7b5SHOMwQZN3J0O 9JEAnSUIv97fEL/R5iYZ4kPve6oU1FjjSdl4I0r03u4gGIyaDC6OnPFAgjHEgi8klMqN +xpE9+J2Ct9P2i4bk0CyCGMnQDAFL5hmC53I74aYtE7cY9IbV/nWIUIgPkrjdEeGtYxG 2S63VM+RpagzeszCmmkgN3eZI5kuAVppYEHalyJqvBiFr4+LYqChgKHQV6zIyf3CSmqq Lyk/vEejBbhcTbjd4MslIzshpfHiRx1rWMAyQkjyWbplvSoJdHLlesx1WPHMZsJQmbl2 na3g==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=U5NEklj6YBO0mXKdughThqpRKwbiePRInGqQ2xZeAZI=; b=g5FpOq7T52kUHLOJlhwgLgqAGoHO2BMA0ASVIiursSqU4SSSWJh2bihM5g/TrsMcty fLY/qrNaVtyFqfPqBE+GQ5nFqGuVgGoKJbhJ+NJZo6MCSyEKd5C0mDYjnvEKYvuRZ8bf mz8laM5DfuxESt91/FP9gVk3o8aGLC2wzXFiYXgyVec1h4v+MiBqjnTLXjyN8PwHtCrg 12UT0wJSBk3G8KvtHDCWtiddqIuEudNgGHzrLu8khHhDzADCBes6OUUaNkyTjR4BrGFu 3h6t2YRBZmkPU3IauxP0VPkFiFfwTPwYpy3GbWameMxq/C0Orv2JBigX7SFNq53Ml1lr SrFg==
X-Gm-Message-State: APjAAAWygPrj73lafHCMiDn4bXNeIawIrEDXqIf/LQ8vi/GtP78e1UBP ex6aareKTiAe6Vc6D0Mlr0VPaNG1CZns+WmcNa4=
X-Google-Smtp-Source: APXvYqzRtCN3Xq2JMrbI2vHeYBYTYUBr5YQxKT6rm28koAdBPj9bGkxlxbDAbKBhjS64UySpzvypXuogWvXj6mkQY/s=
X-Received: by 2002:a5e:d618:: with SMTP id w24mr12275671iom.73.1563515791141; Thu, 18 Jul 2019 22:56:31 -0700 (PDT)
MIME-Version: 1.0
References: <402781F4-33D8-4FD4-8087-FDCEFFF2D549@iseclab.org> <CAChr6SwBKOymQjKsN+GEnygn5ogJb6WUMd=jxRrV2eQWwdiBcg@mail.gmail.com> <267CBE74-9DC5-40D8-A61F-7C566644A1CB@iseclab.org>
In-Reply-To: <267CBE74-9DC5-40D8-A61F-7C566644A1CB@iseclab.org>
From: Rob Sayre <sayrer@gmail.com>
Date: Thu, 18 Jul 2019 22:56:19 -0700
Message-ID: <CAChr6SzzkvVkHvfHbO=1tBYqaaMVsdp43+it256f4WQmbCbQFw@mail.gmail.com>
To: Kevin Borgolte <kevin@iseclab.org>
Cc: add@ietf.org, DoH WG <doh@ietf.org>, dnsop WG <dnsop@ietf.org>, jordanah@princeton.edu, pschmitt@cs.princeton.edu, ahounsel@cs.princeton.edu, feamster@uchicago.edu, dns-privacy@ietf.org
Content-Type: multipart/alternative; boundary="00000000000054ef23058e026351"
Archived-At: <https://mailarchive.ietf.org/arch/msg/doh/2uMp0PtwRZPbPFOg_I29HjFni24>
Subject: Re: [Doh] [DNSOP] Do53 vs DoT vs DoH Page Load Performance Study at ANRW
X-BeenThere: doh@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DNS Over HTTPS <doh.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/doh>, <mailto:doh-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/doh/>
List-Post: <mailto:doh@ietf.org>
List-Help: <mailto:doh-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/doh>, <mailto:doh-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 19 Jul 2019 05:56:35 -0000

On Thu, Jul 18, 2019 at 10:42 PM Kevin Borgolte <kevin@iseclab.org> wrote:

> The list of websites is attached. It is extracted from the top 1,000 and
> 99,000 to 100,000 of a Tranco list.
>

Thanks for attaching the list. Having seen a fair a number of these, I
think it looks reasonable.

But, I think you should add the list and the reason for the range choice to
the paper. For example, I can't tell what range you actually used from your
description (although that might just be due to a hurried reply).

Another issue is that, while your paper might accurately capture the
network conditions on your local network, it's probably doesn't capture
network variation as well as a large scale test along the lines of what
Mozilla did. For example, if the university used a single router brand,
this could skew the test. As one data point, I've never seen the various
network-throttling apps match a real-user-metrics test very well, although
they do catch really problematic situations.

This test is a welcome contribution, but given the data in the paper, it
would be difficult to reproduce.

thanks,
Rob