[rrg] procedural aggregation

William Herrin <bill@herrin.us> Wed, 05 March 2014 00:12 UTC

Return-Path: <wherrin@gmail.com>
X-Original-To: rrg@ietfa.amsl.com
Delivered-To: rrg@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com []) by ietfa.amsl.com (Postfix) with ESMTP id D426C1A0063 for <rrg@ietfa.amsl.com>; Tue, 4 Mar 2014 16:12:06 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 1.423
X-Spam-Level: *
X-Spam-Status: No, score=1.423 tagged_above=-999 required=5 tests=[BAYES_50=0.8, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FM_FORGED_GMAIL=0.622, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_PASS=-0.001] autolearn=no
Received: from mail.ietf.org ([]) by localhost (ietfa.amsl.com []) (amavisd-new, port 10024) with ESMTP id NRtUOFE9Lxfx for <rrg@ietfa.amsl.com>; Tue, 4 Mar 2014 16:12:03 -0800 (PST)
Received: from mail-ve0-x22a.google.com (mail-ve0-x22a.google.com [IPv6:2607:f8b0:400c:c01::22a]) by ietfa.amsl.com (Postfix) with ESMTP id 587681A0062 for <rrg@irtf.org>; Tue, 4 Mar 2014 16:12:03 -0800 (PST)
Received: by mail-ve0-f170.google.com with SMTP id pa12so288145veb.15 for <rrg@irtf.org>; Tue, 04 Mar 2014 16:11:59 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:from:date:message-id:subject:to:content-type; bh=uyRER1aO4lCkkboM6722GsB1XWd/iRsuahaYkLdpA1w=; b=Y+TyezQ+2zy7vKJ5M68NLdlARtY1635y6kDpf+wXOE1Da0AAfNEu1dyRIp+4uNi6d1 OWiff698fSbYqXGWcJ/Rc9zxeyfDZeVgUYAWPT8kn7tk2uiD828kxIMwf8R6/Ekz9WDC 8zvs/tO+rYc4wibYqIpwqScI7eK/sc8R9qeohGBDhdke//vLrjneQTrt8fBone/PTvG3 i99Knm30hlGzG0J+5c4DB76pTw5Ke6jcxZaEjOZUotxaJFPnQ7NkIvsQsbfku7thqqdw W46gmPsDVECHNXU/dQD8MiIDJDOyRMr90D3Mjb0yVREPTtNk98rhc+Ntmeo/3QrPxDZT hhig==
X-Received: by with SMTP id w20mr2016730vco.18.1393978319685; Tue, 04 Mar 2014 16:11:59 -0800 (PST)
MIME-Version: 1.0
Sender: wherrin@gmail.com
Received: by with HTTP; Tue, 4 Mar 2014 16:11:39 -0800 (PST)
From: William Herrin <bill@herrin.us>
Date: Tue, 4 Mar 2014 19:11:39 -0500
X-Google-Sender-Auth: D5AMrkh_74tKTufueik9W1DMVTI
Message-ID: <CAP-guGXyxehmCiskATSLOouE0Cx1i9KFroK60r=xWe-Lu7peSA@mail.gmail.com>
To: RRG <rrg@irtf.org>
Content-Type: multipart/alternative; boundary=047d7b3435c091cfc404f3d0ded1
Archived-At: http://mailarchive.ietf.org/arch/msg/rrg/2HTgMWYeNn6lW5T2Y71EIwp2rfY
Subject: [rrg] procedural aggregation
X-BeenThere: rrg@irtf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: IRTF Routing Research Group <rrg.irtf.org>
List-Unsubscribe: <http://www.irtf.org/mailman/options/rrg>, <mailto:rrg-request@irtf.org?subject=unsubscribe>
List-Archive: <http://www.irtf.org/mail-archive/web/rrg/>
List-Post: <mailto:rrg@irtf.org>
List-Help: <mailto:rrg-request@irtf.org?subject=help>
List-Subscribe: <http://www.irtf.org/mailman/listinfo/rrg>, <mailto:rrg-request@irtf.org?subject=subscribe>
X-List-Received-Date: Wed, 05 Mar 2014 00:12:07 -0000

Hi Folks,

Given recent comments, I thought it'd be an opportune time to introduce a
concept I've been fiddling with for a couple years which I tentatively call
"procedural aggregation."

I have discovered no new math which would overcome the technical challenges
to aggregation identified here in the past. I come at the problem from a
different angle instead.

My basic premise is that with current techniques (not necessarily the exact
protocols) there is no practical upper limit to the number of routes which
can be introduced to the Internet provided that each recipient of a route
advertisement is sufficiently financially compensated. He must be paid
enough for carrying each route to allow him to purchase and operate
hardware that carries all of the routes for which he's being paid. With
that in place, total routes will self-limit based on what technology is
capable of at each price point.

>From there I develop administrative procedures for determining who
announces what aggregates and how to pay who for carriage of the
more-specifics in order to yield a technically useful network configuration.

I'll post a couple examples in the next few days. Before that, I want to
take the remainder of this message to review how the Internet currently
routes packets. I'll largely skip the technical part. You're all experts or
you wouldn't be here. Instead I will clarify the financial process.
Somebody pays each and every organization on the Internet to accept your
packets and forward them to the next hop. Every single Autonomous System.
Every single packet. I'll explain who and I'll explain how that money flows
with the technical parts of packet routing.

Understanding the money flow will be important when I ask you to consider
procedural aggregation. Exterior gateway protocols need not produce *all*
technically possible networks. They need only (must only) produce those for
which the participants are properly compensated.

__ Transit and Peering __

>From a financial point of view there are two types of Internet connections
between two organizations: transit and peering.

transit - Internet provider A provides access to "the entire Internet" to
organization B. B may be an end user or another service provider.

hosting - Internet provider A connects servers under the control of
organization B to "the entire Internet." Hosting is transit plus floorspace.

peering - Internet provider A provides Internet provider Z with access to
only its downstream transit customers and vice versa, not to the entire

settlement free peering - peering for which no no money changes hands.

For the purposes of this discussion:

Transit and hosting are the same thing. In each case, someone pays someone
else to connect them to "the Internet." I will only use the term transit.

Usually B pays A for Internet access. Sometimes it's indirect: buy coffee
get "free" Internet. Sometimes A's compensation is non-monetary, e.g. good
will from a charitable donation. Other times A is paid by a third party to
service B. For the purposes of the discussion, transit service is always
paid for: B always pays A for access to the Internet.

For simplicity's sake I will treat all peering as settlement free peering,
even though many service providers opt to behave as robber barons.

__ Routes flow with the money __

We have two types of connections between Autonomous Systems: transit and

Let's say Endpoint 1 (E1) and endpoint 2 (E2) are both connected to the
Internet. They pay their ISPs who pay ISPs who peer with each other. What
do those connections look like? Typically:

E1 -> B -> A - Z <- X <- E2

* E1 is a transit customer of ISP B. He pays ISP B to connect him to the
entire Internet. So, B is paid to send packets from and receive packets for

* B is a small ISP. He collects the payments from all of his customers like
E1, keeps some of it and spends some of it on transit service from a larger
ISP A to connect all of his customers to the Internet. A is expected to
send packets from and receive packets for all of B's customers, including

* E2's relationship with X is the same as E1's relationship with B. X's
relationship with Z is the same as B's relationship with A. The former buys
transit service from the latter.

* A buys transit service from several other large ISPs (not shown) but he
also has a peering relationship with Z. E2 has paid Z (indirectly via X) to
connect him to the Internet, so Z advertises to A that he's willing to
accept packets addressed to E2. That's what X is paying Z for.

So, a packet travels from endpoint 1 (E1) to endpoint 2 (E2):

E1 -> B -> A -> Z -> X -> E2

Follow the money:

* E1 sends a packet destined for E2 over to B because it's E1's packet and
he wants it sent. B accepts it because E1 paid B to accept it.

* B sends it to A because E1 paid B to send that packet. A accepts it from
B because B paid A to accept it.

* A sends it to Z because B paid A to send that packet. Z accepts it from A
because X paid Z to _receive_ packets for E2.

See the change there at the peering link? Now the money is flowing from the
other endpoint.

* Z sends the packet to X because X paid Z to receive packets for E2. X
accepts it because E2 paid X to accept it.

*  X sends the packet to E2 because E2 paid X to receive packets for E2. E2
accepts the packet because he wants to receive packets addressed to him.

And that's how everybody gets paid for E1 to send packets to E2. Comparable
payment relationships exist for every combination of two endpoints which
attempt to communicate with each other on the Internet.

__ Key Takeaways __

Sometimes E2 is a transit customer of A or of B instead of being on the
other side of a peering session. In that case A or B got paid twice and
banked a little extra money.
As often as not there's a peering session somewhere in the path. Either
way, some key takeaways are:

1. Everybody involved was directly or indirectly paid by E1, E2 or both to
forward that packet to the next hop.

2. Some or all of them were paid by only E1 or E2 but not both.

3. A packet may cross many paid transit links but no packet will cross two
free peering links.

Now you know who pays for the Internet. And how.

Finally, I offer you this estimate of the worldwide systemic cost of
carrying a BGP route. The estimate is out of date but the methodology is
sound enough to use for discussion purposes:

Before I dive in to how this structure might be beneficially changed with
procedural aggregation, are there any questions about any of this? Anything
I can clarify about how the money moves?

Bill Herrin

William D. Herrin ................ herrin@dirtside.com  bill@herrin.us
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004