Re: [rrg] Arguments in favour of Core-Edge Elimination vs. Separation?

[Robin Whittle <[e-mail]>]

Short version:    Continuing discussion of the adoption problems
                  inherent in CEE and whether CEE is needed if a
                  CES architecture can "do everything".  Also a
                  perceived problem with (ITR?) cache unreliability,
                  which I think may be based on a misunderstanding.


Hi patte,

Thanks for continuing the discussion:

>> From a routing scaling point of view (ignoring my concerns about
>> burdening all hosts with extra responsibilities) I think the trouble
>> with CEE approaches is that they only provide scaling benefits when
>> an adopting network can either:
>>
>>  1 - Give up its current PI space and work from the PA space from
>>      its one, two or more ISPs.
>>
>> or
>>
>>  2 - Migrate from reliance on its current (non-portable, no
>>      multi-homing) PI space to the new system of identity (AKA edge)
>>      addresses in a way which provides portability, multihoming and
>>      TE.

In point 2 I should have written PA space.  There are two classes of
existing end-user network which might want portability, multihoming
and TE:

  1 - Those who have it already with PI space.

  2 - Those who don't have it since they have PA space.

So my original 1 and 2 are not a choice an end-user network makes -
it is two types of transition for two classes of existing end-user
network.


> I prefer approach 2, think this is the best way to go since that what
> has been done in the cell-phone network - the subscriber owns the
> number and can choose which ever service provider he/she prefers. So
> we should keep this model in the Internet as well.

That is my point 1 - the end-user network already has "their own" PI
space.  But in CEE, the new kind of scalable space is not IP
addresses of the original kind.  So they can't keep their space and
make it portable, multihomable.  All CEE schemes involve two separate
namespaces - one for Locator (physical) and one for Identifier (logical).

As far as I know, for any CEE (I think including your hIPv4 proposal,
which I am yet to read in detail) no network at present has any space
of the new Identifier type which a CEE will provide.  So neither of
my two points involve the end-user network retaining their existing
address space as their new, portable, multihomable, Identifier space.

In point 1, they give up their PI space and adopt multiple prefixes
of PA space, for the "Locator (physical)" addresses for their hosts.
 They gain a patch of the new Identifier range of addresses, which is
in a totally different namespace from IP addresses - it might not
even be a binary number.  In point 2, they may keep their current PA
address space, and get another set of PA addresses from a second ISP
if they start multihoming.  They migrate their hosts and applications
to a new set of "addresses" of the Identifier kind, just as for point 1.


>>> Totally new host protocols might not be needed, you could get around
>>> most of the challenges by adding extensions to existing protocols.
>>
>> OK - I agree, but even altering applications is very difficult.
>> Also, I think it would be a nightmare to have an application with two
>> different ways of communicating with other hosts - the conventional
>> approach and some CEE approach.
> 
> You might be able to design the CEE architecture in such a way that
> most applications do not see the difference, if the approach is enough
> backwards compatible. Of course you get into trouble with applications
> that uses IP-addresses in their payload - but these applications are
> already having troubles with NAT traversal

I hope to review the CEE proposals which the RRG is considering, plus
HIP, in various respects - including what host changes would be required.

>> For a host to use the new CEE system, as far as I know, its
>> applications all need to be rewritten to use it as well, and the
>> stack and its API needs to be changed as well.
> 
> Nope, not all CEE approaches.

Which one can run with unmodified API and applications?  If they do
this, then the apps are using things which are the same number of
bits as IP addresses to refer to other hosts.  Yet the whole idea of
CEE is to have the apps using a new kind of address - an Identifier.
 If so, then the Identifiers are of the same numeric type and size as
the IP addresses which are used for "Locators" - however the two are
in different namespaces.  I can't think of such a system, but I need
to read about them in more detail.

>> There may be CEE architectures which work entirely with existing API
>> and applications, but I can't think of one now that would work.
> 
> Well, there is one that might work around the problem

The trouble is, I don't think there are any such CEE architectures.


>> So its not just a network deciding to adopt a new system, upgrading
>> their operating systems etc.  It also requires that all their
>> applications be rewritten.   I can't imagine how this would ever
>> happen - especially since I can see how mobility, scalable
>> portability, multihoming etc. can be done with a CES system and no
>> host changes at all.
> 
> Agree, if every application needs to be rewritten we are in serious trouble

Exactly.

Yet the purpose of a CEE is to have all applications using purely
Identifiers, and being generally unaware of how the stack decides
which physical address(es) - Locator(s) - to use when sending packets
to the identified host the application is communicating with.


>> I think the CEE proposals differ enormously from CES proposals in
>> their ability to meet the constraints which arise from the need for
>> voluntary adoption.  The only attempt I know of to state these
>> constraints is my page:
>>
>>  http://www.firstpr.com.au/ip/ivip/RRG-2009/constraints/
>>
>> This is my text, refined after some RRG discussions.  I don't know of
>> anyone who thinks these constraints are invalid, though 8, 9 and 10
>> are sliding scale constraints and the rest are absolute, so there is
>> debate about how much 8, 9 and 10 matter on a case-by-case basis.
>>
>> A CES architecture which does not significantly affect performance,
>> security etc. could satisfy all 10 constraints.  I think Ivip would
>> satisfy them all.  I think LISP-ALT or any system which relies on a
>> global query server system for mapping should be able to satisfy all
>> but point 9 (performance) - due to the "initial packet delay"
>> problem.  With ALT, this is actually a problem of "initial packets
>> get dropped until the one is sent after the ITR gets the mapping".
>>
>> No CEE architecture can meet any of the constraints 1 to 5.
> 
> Had a look on those and some comments

OK - thanks.


> 1. Think all proposals suffer from this, why upgrade in the first
> place - what do the end user gain? 

If they gain portability, multihoming and TE when they either
couldn't get it before, or couldn't afford it, then this is a good
motivating factor - however, to be "compelling" these benefits have
to work for packets coming from all hosts, not just those in networks
which have already adopted the CEE or CES architecture.


> It is really hard to find disruptive elements in the proposals

All the proposals involve effort.  CES involves no host changes and
minimal end-user network changes.  CEE involves upgrading at least
the stacks and (I think) probably the apps in all hosts - which is so
disruptive as to be generally prohibitive.

> 2. Yes, no problem

??

  2 - Therefore, it must provide multihoming, TE and
      portability for the adopting networks in a
      manner which fully supports communications with
      all hosts - including all hosts in non-upgraded
      networks.

A CES architecture such as LISP with its PTRs or Ivip with its DITRs
can do this: provide portability, multihoming and inbound TE for
packets sent by *all* hosts, not just those in networks which have
adopted LISP-mapped EID address space or Ivip-mapped SPI space.

No CEE can do it - since they only provide the benefits for packets
sent by hosts in networks which have already adopted the CEE
architecture.  This will initially be close to 0% and even if 90% or
99% of other networks adopted it, there would still be 10% or 1% of
hosts for which communications are not supported with portability,
multihoming and TE.

I think its fair to say that most networks would only regard the
benefits as being worthwhile once they applied to all communications,
or some very high percentage of them, like 99.99%.


> 3. Agree, has been taken into account and obeyed

Maybe I need to make this more explicit.  Its not just that a host in
a network which has adopted the CES or CEE architecture might be able
to do old-style communications with non-upgraded hosts - it must be
able to support portability, multihoming and TE with them too.  CES
architectures can do this.  CEE architectures can't.


> 4. Almost, I do have issues with some applications when it goes
> outside an ALOC realm

Googling "ALOC" leads me to your hIPv4 summary:

  http://www.ietf.org/mail-archive/web/rrg/current/msg05529.html

which I intend to write a critique of, once I have written some others.


> 5. Disagree here, both approaches should be adopted because there is
> the cache issue that is scaring the crap out of me.  More about the
> cache later on.

I guess by "both approaches" you mean a network-based CES and a
host-based CEE.

> Also other benefits than routing scalability in
> Internet can be achieved if you take this core-edge split to the
> hosts' stack. See what Microsoft Research have done with their VL2
> work
> http://research.microsoft.com/pubs/80693/vl2-sigcomm09-final.pdf
> I'm not working for MS but here is an example what can be done if you
> don't exclude a host stack upgrade

I don't clearly understand this.

All sorts of things are possible with host-upgrades.  Maybe not all
those things are a good idea to enforce on all Internet hosts,
especially those on slow wireless links.

The difficulty with host upgrades is if the system requires them to
be made to all hosts in the world in order to provide the benefits.
This is the case with all CEE architectures.


>>> Think we should remove the
>>> technical nerd hat for a while and put on the shoes of residential
>>> user and the trousers of a CIO in an enterprise, ask ourselves - do
>>> our proposals:
>>>
>>> - introduce any new services?
>>>
>>> - lower the cost of Internet equipment?
>>>
>>> - make Internet connectivity more or less complicated?
>>>
>>> - can you use any idealistic arguments, e.g. green values, to promote
>>>   a migration?
>>>
>>> - introduce some new e.g. API etc that could provide a path to create
>>>   new services on?
>>>
>>> These end user do not understand nor cares about routing scalability
>>> or that the depletion of IPv4 is soon occurring, simply because they
>>> have a very well working Internet at the moment - they also have a
>>> valid IP address space and the new address space do not provide any
>>> new services that can't be found in the current Internet.
>>
>>
>> I agree.  I think that in order to solve the routing scalability
>> problem, we need to construct a perfectly beneficial Trojan Horse.
>> We can generally assume they won't do anything to their networks
>> purely for the purpose of helping with scalable routing.  So we need
>> to give them something provides concrete and immediate benefits, and
>> which will also achieve our scalable routing goals if adopted
>> sufficiently widely.
> 
> Yes, and I believe that the  Trojan Horse is called MPTCP :-)
> My guess is that MPTCP offers something interesting for the end-users
> and we do have an opportunity to smuggle our Achilles in the hosts
> when MPTCP is getting deployed

The MPTCP horse wouldn't get through the castle gates.  It doesn't do
anything useful and its name is hard to pronounce.

It doesn't support existing applications.

Maybe if some people write applications which use it, then that's
great because for those applications, maybe the hosts can run OK on
scalable PA address space and multihome with PA space from two or
more ISPs.  But that is not enough to give real multihoming,
portability etc.  The only way to do it is in a way which supports
existing applications and existing host stacks.


>> The requirements for getting people to adopt the Horse depend on the
>> nature of the architecture.
>>
>> For CES, we only need it to be adopted by the subset of end-user
>> networks we are trying to serve - those which want or need
>> portability, multihoming and TE.  This is as long as the scheme is
>> self-contained with its own PTRs (LISP) or DITRs (Ivip).  In
>> practice, it would also be best if networks which weren't using EID
>> addresses (LISP) or SPI addresses (Ivip) also installed ITRs - since
>> that would make the whole thing work better than relying on PTRs and
>> DITRs.
>>
>> Still, there is no need, with a CES, to do any of these things:
>>
>>  1 - Change host stacks or apps in any hosts whatsoever.
>>
>>  2 - Change routers or hosts in any networks not adopting the new
>>      kind of space.
>>
>> in order to:
>>
>>  1 - Give full portability, multihoming etc. benefits to the
>>      adopting networks.
>>
>>  2 - Achieve routing scalability goals in direct proportion to
>>      the number of end-user networks which adopt the system.
>>
>> With a CEE, it is totally different.
>>
>> As far as I know, no network which adopts it can get portability,
>> multihoming etc. benefits for all its packets until all other hosts
>> in the world adopt the system too.
>
> Nope, not true. By using MPTCP you can migrate current multi-homing
> towards multi-pathing. No tweaking with BGP load-balancing anymore,
> let the transport protocol take care of the load-balancing and
> multi-homing. And this can be done in a smooth way, also it should
> lower the costs of networking devices.

Yes, but it only works with applications which have been written (or
rewritten) to use MPTCP - and some applications can't use MPTCP.

So there's nothing you can do with MPTCP to help with scalable
routing in a gutsy way, which will really provide multihoming,
portability etc. for existing applications.  To do that you need to
have a simple change, which supports all existing applications with
these benefits.

If this was a clean-slate design, then by all means, MPTCP would be a
part of the system - but we are working with a billion or more
end-users who each use dozens of applications.


> It is true that you would need a lot of host stacks to be upgraded
> before the benefits can be harvested, thus CEE needs a CES solution to
> speed up the adaption. 

But if the CES can provide all the portability, multihoming and TE
benefits, then why do we need CEE?

A CES architecture such as Ivip can optionally involve host changes
when it is the cheapest way of providing the ITR function - to build
it into the sending host.  But this is not a requirement - nor does
it resemble a CEE scheme.


> Then you argue, as you do, that it is
> sufficient with a CES solution only. But here I see a paradox and I
> might be wrong, so please correct me if I miss something.
> 
> The more popular a CES solution becomes the more stress it will create
> on the ETR (that becomes ITR for returning traffic). 

The device which performs the ETR functions is not necessarily being
an ITR for outgoing packets.  ITRs in Ivip could be in various
places, including in sending hosts.

In Ivip, the ETR normally only de-tunnels the traffic packet - and
then knows how to forward it to the destination network.  The ETR
sometimes engages in communication with the ITR for the purposes of
PMTUD management - but this is just for encapsulation tunneling.  In
the long term, or perhaps from the start, Ivip is intended to use
Modified Header Forwarding for tunneling - which involves no PMTUD
problems.  Then the ETR function is very simple - just revert the
header to normal and forward the packet to the destination network.


> If the CES
> solution becomes popular and gets widely deployed, e.g. if broadband
> routers will include an ITR stack it will increase cache sizes on the
> ETRs in front of popular sites. 

It would be possible to put an ITR function in DSL etc. routers.
This would be cheaper than building ITRs into the ISP's network.

I don't clearly understand what you wrote about cache sizes. ETRs
don't have caches - either in LISP or Ivip.  ITRs cache mapping.  In
Ivip, the mapping is for a micronet of SPI space (an arbitrary number
of IPv4 addresses or IPv6 /64s) to a single ETR address.  LISP maps
each EID prefix to one or more ETR addresses, with priorities to help
with multihoming and with weightings to implement load sharing
(inbound TE for the destination network).


> Today broadband address space is quite
> well aggregated in the DFZ but if you put an ITR on every broadband
> router the currently aggregated PA-address space will get
> de-aggregated on the ETRs in front of popular sites. 

I don't understand this at all.

If you have a million DSL customers like today, each DSL router has
an IPv4 address - a single IPv4 address of PA space from the ISP's
large (short) prefixes which it scalably advertised in the DFZ.

The ISP could decide to put an ITR in each DSL router (by specifying
the use of certain DSL modems etc.), rather than spend money on
adding ITR functionality to existing internal routers or by
implementing ITRs on servers in the internal routing system.  There's
no problem with this, at least for Ivip.

With Ivip, the ITRs request mapping directly or indirectly from full
database QSDs in the ISP - and so get the mapping reliably and within
a few tens of milliseconds.   Each such ITR function needs to cache
the ETR addresses of all the destination hosts which are on SPI space
 which any of the home/office computers are sending packets to.
That's fine - there may be 10 PCs on the LAN, and over any time
period such as 10 minutes, they may be sending packets to 1000 or
more hosts, most of which (in a nearly ubiquitous adoption scenario)
are on SPI addresses.  If each such host is in a different micronet
(some destination hosts would probably be on the same micronet), then
the ITR is caching mapping for 1000 micronets or so.  That's no
problem for the CPUs which are in broadband routers.  If it was
10,000 it might be trickier - and 100,000 would probably be excessive.

But the ITR function can always drop things from its cache and
request the mapping again if it needs it.

What you wrote:

  "the currently aggregated PA-address space will get
   de-aggregated on the ETRs in front of popular sites."

doesn't mean anything to me.  Can you give a fuller description of
the problem you foresee?


> Of course the
> core network will have a smaller FIB but how large cached FIB is
> needed on the popular ETRs? One million, two millions, will it scale?

I don't understand this.

> What about the housekeeping of the caches? What are the costs of these
> large cache ETRs - it is not trivial to do housekeeping with two
> millions cache entries, or is it?

ETRs don't cache anything.

When you are discussing a broadband router, I assume you mean a
customer like today, on a single IPv4 address via DSL, fibre or HFC
cable.  The broadband router does NAT to support potentially many PCs
on the local LAN.

This broadband router is not on SPI space.  It is not multihomed.  It
uses ordinary PA space like it does today, which is perfectly
scalable.  The fact that it has an ITR function built into it - which
is a choice, not a requirement in Ivip - just means the ITR function
needs to cache ETR addresses according to whatever is needed due to
the packets sent by the various PCs in the LAN.

This broadband router is not operating as an ETR.

If you wanted to multihome a network via DSL and HFC cable, then in
principle you could make a special "broadband router" to do it.

Then, the box would get IP address XX from ISP-A (DSL) and IP address
YY from ISP-B (HFC).  I assume for this example that these are fixed
IP addresses.  Then the box will perform ITR functions as previously
noted.  It would also perform the functions of two ETRs - one on XX
and one on YY.  An external company hired by the end-user network
would be given the credentials to control the mapping for this
network's one or more micronets - and then would control the mapping
to implement multihoming service restoration, and if both links were
working, inbound TE as noted below.

Now the box can support its PCs on the LAN with SPI addresses, which
are portable, multihomable etc.  With Ivip, it can do inbound TE by
splitting up the SPI addresses to which the incoming packets are
addressed into two or more different micronets and then mapping one
micronet to the XX ETR and the other to the YY.  Ivip can't
load-share traffic addressed to a single micronet.  (LISP can load
share traffic addressed to a single EID prefix, including potentially
a single IPv4 address.)  So there would need to be at least 2 SPI
addresses and the incoming packets would need to be addressed some to
one and the rest to the other.

There's no such thing as "cache" in ETRs.  The ITR function is just
the same as before without multihoming and ETR functions.  The box
would also be making outgoing TE decisions about its outgoing
packets, since it has two uplinks.

With encapsulation tunneling, the ITR function would have two
outgoing addresses - XX and YY.  It could decide which link to send
out encapsulated packets on as it chooses.  It doesn't matter to the
ETR at the destination network which ITR the packet comes through.
The host at the destination network doesn't know or care either.

Perhaps your understanding of what CES involves is different from mine.



> Will the content providers adapt to this technology - if the content
> sites becomes unreliable due to cache housekeeping will they adapt or
> do they rather stay with the current solution - so they don't risk
> their business model?

I don't understand the problem you are referring to.


> If the cache solution is not 100% reliable and proven the content
> providers will most likely not accept this technology and then the CES
> approaches will not be adapted.

Maybe you are asking what happens if ITR caches are unreliable.
There's no reason why they should be unreliable.  Do you have any
specific problems in mind, for Ivip or for LISP?


> Can you get a better story if you combine a CES with CEE from day one,
> explaining the benefits of it, also remove the address space
> constraints, so that the content providers can have simpler&cheaper
> Internet connectivity and they can reach more customers in the future?
> 
> Only more questions....and I might misunderstand something...

Maybe so - it is difficult discussing this stuff by email.
Face-to-face and pen and paper would be a lot easier!

If you describe the problem you foresee I will respond ASAP.

  - Robin