Re: [rrg] IRON (RANGER): description and draft critique

[Robin Whittle <[e-mail]>]

Short version:    I will leave this message for now as the record of
                  my understanding of IRON-RANGER, though I still
                  don't understand how IRON routers such as D and E
                  securely, continually, register the EUN (End User
                  Network) prefixes they handle with potentially
                  multiple VP routers, which could be anywhere.

                  I now understand the SEAL redirect message closely
                  resembles the Map Reply a LISP ITR would get from
                  the ALT network.

                  I think IRON can usefully be compared to LISP-ALT
                  with a completely flattened ALT network.  This is
                  a rough, but instructive, comparison.

                  I will write a V2 critique soon.


Hi Fred,

Thanks for your comments:

>> However, in the IRON ID, Fred my have implied a very much lower
>> number of VPs, because he mentions (page 5) IPv6 /8 prefixes.  Even
>> if the whole of IPv6's address space was used for global unicast
>> address space, this would imply no more than 256 VPs.  I would have
>> thought that 10,000 to 100,000 or 200,000 VPs would be a better way
>> of spreading the load over multiple VP IRON routers.
> 
> I don't want to stipulate and exact maximum prefix length,
> but O(10K) - O(100K) VPs in the IRON RIB sounds reasonable.

OK.


>> IRON routers
>> ------------
>>
>> IRON routers are not necessarily DFZ routers.  However, they are
>> probably located topologically close to DFZ routers, near the borders
>> of ISP networks and of other large networks such as PI-using
>> corporations and universities etc. who may have their own DFZ
>> routers, or who connect to the DFZ via one or more ISPs.
>>
>> In principle it would be possible to implement an IRON router in a
>> DFZ router, but the intention of IRON is that the IRON routers are
>> not DFZ routers.
> 
> Not necessarily; IRON routers can also be DFZ routers but
> then they would participate in two separate BGP instances;
> one for core RLOC prefixes and the other for edge EID
> prefixes (this latter being the IRON BGP control plane).
> 
> The purpose in mentioning that IRON routers need not also
> be DFZ routers was simply to show an incremental deployment
> strategy.

OK.



>> Fred describes IRON routers discovering nearby routers via PRLs
>> (Possible Router Lists) which are part of RANGER, or via some
>> DNS-based methods.  I am interested in understanding IRON with as
>> little as possible of RANGER, since I find RANGER vary open-ended,
>> complex and hard to understand.  To me, it would be acceptable if
>> each IRON router was manually configured with the IP addresses of a
>> handful of "nearby" IRON routers.
> 
> Manual config is always possible. Autoconfiguration can
> also be used when it is available, but static manual
> config should be the base case.

OK.


>> When the packet arrives at the Seattle router, it is decapsulated and
>> emerges from the VET interface, to be handled by the FIB.
>>
>> It is possible that a packet sent to the Seattle router is addressed
>> to a host in an EUN directly connected to that Seattle router.  In
>> this case, as usually, this is not true.
> 
> Not necessarily. There may be significant portions
> of the Seattle router's VP that are "at home" and
> really use the Seattle router as their point of
> attachment to the IRON. For those, the packet is
> simply delivered and no redirect is sent.

OK.


>> The Seattle router's FIB has a more-specific prefix which matches
>> this destination address - the prefix 43.0.56.76 /30 which has a best
>> path to IRON router D in the South Island - the one which has the DSL
>> link to the tour company.
>>
>> The Seattle router now tunnels the packet to the router D.  This is
>> on page 6:
>>
>> Translating the sentence:
>>
>>    'C' then forwards the packet to an IRON router 'D' which
>>    connects the RANGER network where 'E' currently resides.
>>
>> to represent the current example:
>>
>>    The Seattle router 'B' then forwards the packet to an IRON router
>>    'D' which connects the ISP-4 network where the tour company
>>    currently prefers its packets to be delivered.
>>
>> However, "forward" in this sentence is not, as far as I know,
>> ordinary forwarding in the DFZ.  The previous reference to "forward"
>> was "forwards the packet via VET automatic tunneling" - so I think
>> the second usage also implies VET automatic tunneling:
>>
>>    IRON router 'B' then consults its FIB and discovers a VP that
>>    covers the 'E' prefix, then forwards the packet via VET automatic
>>    tunneling to an IRON router 'C' that owns the VP.
>>
>> translated:
>>
>>    Auckland ISP IRON router 'A' then consults its FIB and discovers a
>>    VP 43.0.0.0 /16 that covers the destination address 43.0.56.78,
>>    then forwards the packet via VET automatic tunneling to an IRON
>>    router 'B' in Seattle that owns the VP.
> 
> Right.

OK.


>> So I think the Seattle router A uses VET tunneling to "forward" the
>> packet to the IRON router D in the South Island - which will deliver
>> it to the tour company's DSL service.
>>
>> The most obvious problem with this is that the packet had to traverse
>> the Pacific Ocean and the Equator back and forth to get from the
>> North to the South Island.
>>
>> This is where the RANGER "route optimization" comes into play.
>>
>> But how does the Seattle router B get the packet to router D in the
>> ISP-4 of the South Island?
>>
>> I thought that B would use VET tunneling to D.
>>
>> However, what Fred told me about router discovery made me think that
>> perhaps the tour company router, via D, does some kind of "bubble
>> blowing" process by which D winds up with an FIB entry for the
>> 43.0.56.76 /30 prefix, with a best path which leads to intermediate
>> routers including D.
>>
>> I don't know how this would work for IPv6, much less IPv4 - or how it
>> would scale considering there will be millions of EUN "edge"
>> prefixes, like the one used by the tour company in the South Island.
> 
> It would work very similar to Teredo [RFC4380].

This still doesn't give me any clear idea of how the D and E routers
securely, continually register the prefix with the one or more VP
routers, and how they convey their priorities and load sharing TE
information.



>> Route optimization
>> ------------------
>>
>> The B router in Seattle will send back a SEAL message, via a SEAL
>> tunnel from B to A, to the A router in the North Island.  This tells
>> the A router that for any packets addressed to the 43.0.56.76 /30
>> prefix, it should on longer forward them on the path to the B router
>> in Seattle, but should forward them directly to the IRON router D in
>> the South Island.
>>
>> This is, in effect, a route redirect message.  It would also come
>> with a caching time.
> 
> Right.

OK.


>> This results in the installation of a "more-specific" prefix in the
>> FIB of the A router in the North Island.  This has precedence over
>> the 43.0.0.0 /16 or 43.0.0.0 /14 prefix which all IRON routers have.
>>
>> As best I understand Fred's plans, the A router will have a locally
>> configured STALETIME, such as 120 seconds.  I understand that if no
>> traffic packets use this new "more-specific" FIB entry within any 120
>> second period, then it will be deleted.
>>
>> I understand that the A router also caches a SEAL_ID with this - the
>> SEAL_ID which came with the redirect message, which itself was copied
>> from the initial traffic packet which A sent to B.  So this SEAL_ID,
>> which A generated, enabled A to authenticate the SEAL redirect message.
>>
>> I think it could also be used to authenticate a second redirect
>> message from B, but as far as I know, B would not send such a
>> message, at least in respect of the initial traffic packet.
>>
>> Now, as long as traffic packets keep arriving for this prefix less
>> than 120 seconds after each other, and as long as the redirect's
>> cache time has not expired - and as long as nothing else happens -
>> the A router in the North Island will tunnel packets to the D router
>> in the South Island, and all will be well.
>>
>> If the D router becomes unreachable, or if it cannot reach the router
>> in the tour company (say the prodigious rainfall and stiff winds
>> bring down another tree and pull down a fibre cable line which the
>> DSL service depends upon), then the A router will delete this
>> more-specific entry and its cached SEAL_ID.  This would only occur if
>> the D router sent a destination unreachable message to the A router,
>> or if the D router was somehow unreachable - but that would require
>> some other router to send a destination unreachable, I think, since I
>> understood that all IRON routers are presumed to be reachable via the
>> VET interface.
> 
> Neighbor Unreachability Detection is used to detect
> unreachable RLOCs so that other RLOCs can be tried.

I suggest that in future versions of the IRON ID, that you provide an
example such as this, explaining exactly how everything happens.

I only have a vague idea what "Neighbor Unreachability Detection"
means.  I understand it is specified in:

  http://tools.ietf.org/html/rfc4861#section-7

so I don't understand how this applies to IPv4, between IRON routers
in separate locations, or between the D and E IRON routers and the
router in the office of the tour company.

But see below - it seems you may not be using this


>> The next time a packet arrives at the A router, with a destination
>> address matching the 43.0.56.76 /30 prefix, the A router will once
>> again tunnel the packet to the B router in Seattle and the process
>> will begin again.
>>
>> However, by now - by some means I don't fully understand - the B
>> router in Seattle knows that the packet should be tunneled to the E
>> router in the South Island, which uses a 3G link or whatever to the
>> tour company's network.  So that is where the packet is sent by B,
>> and the A router gets a redirect to the E router, rather than the D
>> router.
> 
> Here is a piece that you may be missing. The initial
> redirect from B would name *both* D and E as RLOCs
> of IRON routers that know how to reach the Fox Glacier
> network. So, if D goes down A will try E before it
> gives up on the route altogether.
> 
> So, B needs to discover *all* of the other IRON routers
> that service the Fox Glacier network - not just one.

OK - I assume a future revision of SEAL will include messages for
redirection.  This resembles a map reply in LISP, since it contains
multiple RLOC addresses with some kind of priority arrangement to
choose one if both are reachable.

So I guess the FIB of the A router now needs to store, for the
43.0.56.76 /30 prefix, quite a lot more information and to make
decisions about which address to tunnel to.  Part of this is
determining reachability to the D and E routers.

I suggest you explain, in a future IRON ID, exactly how this would
work, and what the scalability concerns might be with hundreds or
thousands of ITEs determining reachability to the one ETE.  Maybe
there are few such scaling problems - with the only form or
reachability testing being the ITE getting an ICMP message about the
destination not being reachable.   But is this going to be good enough?

I guess you can have the ITE request (with a flag in the SEAL header
of every encapsulated traffic packet) the initial packet to the D or
E router be acknowledged, and then repeat this every 30 seconds or
so.  Then the ITE needs more state, and needs decision-making
algorithms for how often to request this and how many failures will
result in sending packets to the next most preferred IRON router - E
in this case.

This extra complexity in turn limits how many ETEs an ITE with given
resources can be tunneling to at once, and to a certain extent how
many alternative ETE addresses it can store for all its currently
active EUN prefixes.  So, depending on the diversity of the traffic
destinations of packets emerging from a given network which
constitutes the "catchment area" of a given ITE, there may need to be
more such ITEs each with a smaller given "catchment area".

Fortunately P2P file sharing is declining, but each end-user desktop
host could easily be sending packets to dozens or perhaps hundreds of
destinations over a five minute period.  End-user networks which are
hosting farms will be sending packets to hundreds of thousands or
millions of separate destinations, so there would be real volume and
FIB entry scaling problems unless there were more IRON routers, each
handling packets from a smaller number of servers.

With Ivip, the best solution for a hosting company would probably be
to put an ITR function in each sending host.  This would cost nothing
and would work nicely - with only the mapping query and response
packets going back and forth between each ITR function and the query
servers in that network, or in the network of an ISP it is using.

In theory you could implement an IRON router in each sending host of
the hosting farm, also for no financial cost.  However, this would
require more careful management than the Ivip approach, I think, and
it would involve that host in a continual level of BGP messages.  It
would also expand the number of IRON routers.


>> Somehow:
>>
>>   1 - The VP router (B in Seattle) already knew about both D and E as
>>       being IRON routers which could accept packets addressed to the
>>       43.0.56.76 /30.
>>
>>   2 - The VP router initially knew that both D and E were reachable,
>>       and that they could reach the tour company's router.
>>
>>   3 - The VP router knew that the D router was preferred over the E
>>       router.  (I don't know if this is possible via Router
>>       Advertisements.)
>>
>>   4 - After the outage, the VP router was told that D could not be
>>       used any more, so it altered the path in its FIB for the more
>>       specific route 43.0.56.76 /30 to point the E router instead.
> 
> It doesn't need to be told that D can't be used anymore.
> The periodic RAs coming from D would cease, and the cache
> time for the RLOC for this FIB entry would expire. But,
> the FIB entry would still be retained because E is still
> reachable.

OK.  So D and E both use this RA (Router Advertisement) method to
tell the VP router (B in Seattle) - or potentially multiple VP
routers all around the world - that they are alive, that they can
forward packets to 43.0.56.76 /30 and that each has a certain
priority, which will be different, preferring the D router.

I don't yet understand how this RA process would work, either for
IPv6 or IPv4.


>> Let's say the outage happened a minute after the first packet, and
>> by some means the VP router in Seattle found out about it 10 seconds
>> later.  Could the VP router send a second redirect to the A router?
>> I guess it could, but as far as I know, this is not part of IRON.
> 
> No second redirects. Once the first redirect is sent
> out, the IRON router that receives the redirect is
> left to its own devices.

OK. If the redidirect is lost, then there will be a second tunneled
packet from A to B, and B will send another redirect.


>> The caching time in the redirect is to avoid the A router from
>> sending packets for too long according to the redirect, when it
>> should periodically forget the redirect and let the next packet(s) go
>> to the VP router in Seattle, and await any redirect which results.
>>
>> The STALETIME value is to reduce unwanted clutter in the A router's
>> FIB in the absence of them actually being used.
> 
> And also to more quickly discover any changes that may
> have occurred at B without having to first try to reach
> Fox Glacier directly using stale information.

OK.


>> Multiple VP routers
>> -------------------
>>
>> I understand there can be multiple routers such as the one in Seattle
>> which advertise the 43.0.0.0 /16 Virtual Prefix in the IRON BGP
>> control plane.
>>
>> This would have three advantages at least:
>>
>>   1 - The load for this prefix would be spread over more than one
>>       VP router.
>>
>>   2 - There would be natural failure recovery - if the Seattle
>>       router was down, whatever IRON routers had a path to it for
>>       this prefix would adapt by choosing a path to another VP
>>       router advertising the same prefix.
>>
>>   3 - Generally, subject to conditions discussed below, the A router
>>       would find the closest of multiple VP routers - so reducing
>>       total path lengths and delays for the first packet or packets.
>>
>>       There could be a flurry of packets sent from A to B before
>>       B's redirect gets to A - especially if one or more of the
>>       the redirect packets are lost.  So the B router in Seattle
>>       would need to get all those packets to the correct D or E
>>       router.
> 
> Right, but that's how redirects work on any link.
> There could be multiple initial packets that travel
> over the dogleg route before the direct route gets
> installed.

OK.


>> However, now the D and E routers need to communicate their
>> "ownership" of 43.0.56.76 /30 to multiple VP routers all over the
>> world.  Likewise their lack of ability to handle packets for this
>> prefix if there is an outage.
> 
> There would never be a need to explicitly communicate
> an outage after the initial reachability information
> was propagated. IRON routers that receive the redirects
> will be able to learn of outages via Neighbor
> Unreachability Detection and/or ICMP Destination
> Unreachables from other IRON routers.

OK.


>> These VP routers could be anywhere in the world.
>>
>> So how does the proposed "blowing bubbles" method (I think based on
>> IPv6 or RANGER Router Advertisements) scale properly?  Does it happen
>> over the IRON BGP control plane only - or is it somehow a process
>> which happens outside this?  The EUN router in the tour company
>> office is not part of this control plane.
>>
>> I understand that this process is a continual one - the D and E
>> routers need to keep doing it, based on some caching time in the VP
>> routers, I guess.
> 
> Yes.

OK.


>> There are going to be millions of these EUN prefixes, and for each
>> one, if it is multihomed to two ISPs, there are going to be two IRON
>> routers "blowing bubbles" in a manner which will continually reach
>> one or more IRON VP routers anywhere in the world.
> 
> Yes.

OK.  But I don't understand how this works, or how it would scale to
millions of routers such as D and E being able to do RA or whatever
to multiple VP routers which could be anywhere in the world.


>> Draft Critique
>> --------------

>> IRON-RANGER (hereafter "IRON") uses principles from RANGER, VET and
>> SEAL to construct a Core-Edge Separation scalable routing solution.
>> Separate IRON networks would be used for IPv4 and IPv6, but perhaps
>> they could be combined in some way if this was desired.
>>
>> IRON does not have a mapping system such as that of LISP or Ivip.
>>
>> A single global network of IRON routers communicate over tunnels,
>> each using their own BGP instance, to form the IRON BGP control
>> plane.  This is unrelated to the DFZ's BGP control plane.  While each
>> IRON router advertises all "edge" prefixes in the routing system of
>> the networks they are based in (of ISPs and large corporations,
>> universities etc.), the current IDs do not call for them to advertise
>> any such prefixes in the DFZ.
> 
> I'm not sure I exactly followed the above. True, the
> IRON routers have to advertise all VPs into the edge
> networks. But, if there is one large prefix from which
> all VPs are derived (e.g., 4::/3), then the IRON routers
> only need to advertise that one prefix. (Similarly, if
> there are only a few large prefixes then only those
> prefixes need to be advertised.)

OK - this may be the case for IPv6 - all "edge" space comes from a
specific short prefix.

For IPv4, the "edge" space is generally going to be cobbled together
from multiple existing prefixes.  This is fine - if there are 50,000
or them or even 100,000 that's OK, provided each one is serving the
needs of multiple (ideally thousands or tens of thousands) of EUNs.


> The IRON BGP control plane on the other hand carries
> all VPs - there should be no more than O(10k) - O(100k)
> of these.

OK.


>> Therefore, as currently described,
>> IRON could only support packets sent by all hosts if it was adopted
>> by all such networks.  However, IRON could easily be adapted to do
>> this by having multiple widely-dispersed IRON routers advertise the
>> complete set of "edge" prefixes in the DFZ.

Do you plan there to be a subset of IRON routers scattered around the
Net, like Ivip's DITRs or LISP's PTRs, advertising "edge" prefixes
the DFZ, to attract all packets to the nearest such IRON router?

If so, then IRON should be able to provide full support for packets
sent by non-upgraded hosts.  I think this is essential for any
proposal to be voluntarily adopted on a wide enough scale to solve
the routing scaling problem.


>> Each IRON router processes packets addressed to "edge" addresses by
>> forwarding them to a particular IRON router which, inside the IRON
>> BGP control plane, advertises a particular Virtual Prefix.  There may
>> be one or more of these VP routers for a given prefix, and the number
>> of VP prefixes for the entire "edge" subset of the global unicast
>> address space would be limited, in part, but the ability of the IRON
>> BGP control plane to handle this number of prefixes.
>>
>> IRON routers peer with topologically nearby IRON routers to be their
>> BGP neighbours.  When the traffic packet arrives at the VP router, it
>> is forwarded (via a tunnel again?) to the IRON router which can
>> deliver the packet to the destination network.
> 
> Yes; via a tunnel.

OK.


>> The VP router also sends a SEAL redirect router to the first IRON
>                                            ^^^^^^
>                                        extraneous word
> 
>> router and thereafter, that first IRON router tunnels the packets
>> directly to the IRON router which connects to the destination
>> end-user network.
>>
>> The VP router's FIB for has more-specific routes for each end-user
>> network prefix which is covered by this VP.
>>
>> There are unresolved scaling questions regarding:
>>
>>   1 - The ability of the initial IRON router to handle in its
>>       FIB the temporarily installed more-specific routes due
>>       to the redirect messages it receives from VP routers.
>>
>>   2 - Likewise, questions of FIB and/or route processor ability
>>       to handle the churn in these, since they will typically
>>       last for seconds or minutes, before having to be withdrawn
>>       and perhaps replaced after a further redirect.
> 
> Neighbor Unreachability Detection (perhaps using the
> SEAL explicit acknowledgement mechanism) can be used to
> detect failures and withdraw information on a more
> timely basis.

OK - I understand the A router regularly requesting acknowledgements
from the D router, but I still don't understand how the D and E
routers securely register themselves via "Router Advertisements" or
how "Neighbor Unreachability Detection" works.  These are IPv6
concepts and I can't yet see how they would work from routers in New
Zealand to the VP router in Seattle, or to multiple VP routers which
could be anywhere.


>>   3 - The number of VP routers - more than one would be necessary
>>       for robustness.
>>
>>   4 - The ability of the VP routers to discern which of the multiple
>>       advertising IRON routers had the highest priority for use
>>       in a multihoming scenario when both were advertising the one
>>       end-user network "edge" prefix.
> 
> Route information carried in the Router Advertisements
> and Redirects would contain metrics.

OK.


>>   5 - The scaling problems inherent in these IRON routers advertising
>>       their collectively millions of end-user "edge" prefixes all
>>       over the IRON network, since the one or more VP routers could
>>       be located anywhere with respect to these advertising IRON
>>       routers.
> 
> The IRON routers only advertise their VPs in the control
> plane. They inform other IRON routers of more-specifics
> in the data plane and only on-demand of actual traffic.

My point 5 was not about the one or more VP routers for each of the
~10k to ~100k VPs advertising their VPs in the IRON BGP control
plane, but the routers such as D and E, which collectively will be
handling millions of EUN "edge" prefixes such as 43.0.56.76 /30.  Do
they use the IRON BGP control plane for this continual process of
"RA" secure registering with multiple VP routers?


> 
>>   6 - The speed with which VP routers can learn of outages detected
>>       by the IRON routers which are capable of delivering packets to
>>       the end-user networks.
> 
> Neighbor Unreachability Detection, e.g., based on SEAL
> explicit acknowledgements.

I thought "Neighbor Unreachability Detection" was a separate system
you borrowed from RFC 4861.


>> IRON is not yet described in sufficient detail for these questions to
>> be answered.  It is not clear how, or if, it would implement load
>> sharing or other forms of inbound TE.
> 
> VET specifies a method for load sharing across
> multiple ETRs using equal-cost multipath.

So that information must be in:

  1 - The "RA" or whatever messages by which routers D and E
      securely register with the one or more VP routers.

  2 - The SEAL redirect message the VP router sends to the A router.

It would be good to explain how this works in the IRON ID.


>> Nor is it clear what approach
>> to mobility the system would adopt, or how this would scale to
>> billions of mobile devices.
> 
> Using IPv6 as an example, mobility is based on a /56
> or larger granularity and not, e.g., a /64 or /128
> smaller granularity. So, IRON takes care of *mobile sites*,
> while mobile end systems are accommodated via other
> mechanisms such as HIP, MIPv6, etc.

OK.


>> There is no current description of the business relationships between
>> the various users and operators of routers - so it is difficult to
>> envisage business arrangements in which costs are generally borne by
>> those who benefit, without unfair burdens being placed on any
>> participants.  Nor is there a description of how IRON could be
>> introduced so as to provide portability, multihoming etc. for all
>> packets received by an adopting network, before all networks have
>> their own IRON routers.
>>
>> IRON is a novel CES architecture in an early stage of its design
>> process.  It can be decentralised in every respect, and uses
>> data-driven "redirect" messages as a form of mapping distribution.
>> However, it is not yet clear how the VP routers learn the mapping for
>> the end-user prefixes in their VP.  If this an be done in a secure,
>> fast and scalable fashion - then IRON may be worth considering as a
>> scalable routing system, at least for providing portability and
>> multihoming to non-mobile end-user networks.
> 
> VET specifies how the end-user prefixes are pushed to
> the IRON routers that own the VPs from which they are
> derived. IRON uses RANGER, VET and SEAL.

OK - I don't have time to read this.  I think the IRON ID needs to
explain all this, because it is impractical to refer people to the
main RANGER, VET and SEAL documents, which have a huge array of
details and options, only a subset of which are important for
IRON-RANGER.


Overall, I think your proposal somewhat resembles LISP-ALT, with
mapping request messages in the form of data packets to be delivered
before the mapping arrives (as is anticipated in ALT, but apparently
not currently used) - with a flattened ALT structure.

  LISP                IRON-RANGER

  ITR                 A router in North Island

  ???                 VP router which gives the map reply to A,
                      and also forwards the packet to the IRON
                      router which can deliver it to the EUN.

  ETR                 One of potentially several IRON routes which
                      can deliver data packets to the EUNs, and which
                      regularly and continually (unlike LISP, I
                      think) register themselves with one or more
                      VP routers, conveying priority information and
                      load-sharing TE instructions.

  Map request         A tunneling data packet to the closest VP
                      router.

  Map reply           VP sending a SEAL redirect to A.  Like LISP,
                      this contains the prefix, the addresses of two
                      or more (assuming multihoming) ETR-like IRON
                      routers, with priorities for which to use if
                      both are reachable and some kind of
                      load-sharing instructions if TE is used.


> Thanks for all of your time and effort on this,

OK.  Thanks for responding to all my questions.  I think it is an
interesting proposal.

I will leave this discussion as a record of my attempt to describe
IRON-RANGER and write a V2 critique.  I need to move on to other
proposals, so you can respond to any further shortcomings in the
critique in your "Rebuttal".

  - Robin