Re: shim6 @ NANOG (forwarded note from John Payne) (fwd)

[marcelo bagnulo braun <[e-mail]>]

El 08/03/2006, a las 10:28, Igor Gashinsky escribió:

> Hi Marcelo,
>
> 	My comments are in-line... sorry for the late reply, but I've been
> traveling too much lately...
>
> :: El 01/03/2006, a las 10:10, Igor Gashinsky escribió:
> :: So the effort for this case imho is putted in enabling the capacity 
> or
> :: establishing new sessions after an outage rather than in preserving
> :: established connections, do you think this makes any sense to you
>
> This makes a lot of sense, provided this happens under the hood of the
> application (ie web-browser in this case). So, right now, for example, 
> if
> a client is pulling down a web page, gets the html, and and in the 
> middle
> of downloading the .gif/jpg his session dies (ie TCP RST), the jpg that
> the client was in the middle of x-fering will get that ugly red "X".
> (most browsers, right now, will not re-try to get the object again, and
> will just show it as unavailable). This issue is deemed important 
> enough
> that most large content providers are spending an inordinate amount of
> money on loadbalancer with active session sync to try to prevent that
> from happening in the even of a loadbalancer fail-over. So, if
> application behavior could be changed to say "if shim6 fail-over is
> possible, and connection just died (for any definition of die), then
> attempt re-establish connection through the shim, and then to re-get 
> the
> failed object", that would go a long way in making this kind of 
> fail-over
> better.
>

This is possible with the shim6 protocol, since it supports unreachable 
ulids when establishing the shim context, so i guess this would be ok. 
Probably a couple of elements that are needed, like an extended API to 
allow the apps to tell this to the shim (you probably want also to 
inform the shim which locator is not working), and the shim needs to 
remember the alternative locators obtained from the DNS even if there 
is not shim context yet, in order to have a clue about which 
alternative address to use (the other other option is to perform a 
reverse lookup for retrieving those... see the threat with Erik for 
more about this point). But in any case, i think all this issues are 
easily solvable

but, i have an additional question about this point. the point is, if 
the application is the one that will determine that there is a problem 
and will ask the shim to establish a context (which is ok and no 
problem here) wouldn't the application be better off simply retrying 
with alternative locators by itself, rather then asking the shim to do 
it?

> The difference with shim6, instead of v4 is that in v4 world, the
> connection wouldn't die, it would just hang for the duration of
> convergence (provided  convergence is fast enough, which normally it 
> is),
> and then continue on it's merry way with new tcp windows. In Shim6, if
> the client[ip1]-server connection goes down, re-establishing to
> client[ip2]-server would not be "hitless" (ie session would die), and 
> to
> solve that problem we are back at either keeping an inordinate amount 
> of
> state on the webservers (which is not very realistic), a shift in the
> way people write applications (which, in my opinion is preferred, but a
> *very* hard problem to solve), or to somehow figure out how to hide 
> this
> in the stack with minimal performance hit (let's say sub 1% memory hit)
> when you have 30k+ simultaneous connections per server...

well if you use the shim approach that you suggest above, the server 
does not have to store any shim state while things are doing fine and 
if a client detects a problem it can trigger the creation of the shim 
context from the client to the server. At this point, the server will 
need some shim state, but only for those connections that have failed 
(of course if one of the links to the server went down, then all the 
clients connecting through that link will attempt to create a shim 
state)

I guess that this could be a reasonable trade-off between state in the 
server and response time when outages occur

>
> :: > 3) While TE has been discussed at length already, but it is 
> something
> :: > which is absolutely required for a content provider to deploy 
> shim6. There
> :: > has been quite a bit of talk about what TE is used for, but it 
> seems that
> :: > few people recognize it as a way of expressing "business/financial
> :: > policies". For example, in the v4 world, the (multi-homed) 
> end-user maybe
> :: > visible via both a *paid* Transit path (say UUNET), and a *free* 
> peering
> :: > link (say Cogent), and I would wager that most content providers 
> would
> :: > choose the free link (even if performance on that link is (not 
> hugely)
> :: > worse). That capability all but disappears in the v6 world if the 
> Client
> :: > ID was sourced from their UUnet ip address (since that's who they 
> chose
> :: > to use for outbound traffic), and the (web) server does not know 
> that
> :: > that locator also corresponds to a Cogent IP (which they can 
> reach for
> :: > free).
> ::
> :: I fail to understand the example the you are presenting here...
> ::
> :: are you considering the case where both the client and the server 
> are both
> :: multihomed to Cognet and UUnet?
> :: something like
> ::
> :: UUnet
> :: /     \
> :: C       S
> :: \     /
> :: Cognet
>
> Yes, but now imagine the the "C" in this case is a client using shim6 
> with
> multiple IP's, and the server is in IPv6 PI space. Also, if it wasn't 
> in
> PI space, the connection to the server *can* be influenced via SRV 
> (although
> that's trying to shoehorn DNS into where perhaps it shouldn't go -- 
> since
> now the DNS server needs to be aware of link-state in the network to
> determine if the UUnet/Cogent connections are even up, and for a
> sufficiently large "S", that could be 10's, or even 100's of links, 
> which
> presents a very interesting scaling problem for DNS.. (even more
> interesting is that most large content providers are actually in the
> 1000's, and that's why they can get PI space -- they are effectively 
> (at
> least) a tier-2 ISP). But, back to the example at hand.. so, for the
> sake of this example, let's say that the UUnet port is $20/Mbps, and
> the Cogent port is a SFI (free) peer. So, the client (with ips of
> IP-uunet and IP-cogent) picks IP-uunet (because they want to use their
> UUnet connection outbound) to initiate a connection to the server, the
> problem now comes from the fact that the server, when replying to the
> client is unaware that IP-cogent IP is associated with the client,
> (since the shim layer has not kicked in on initial connect) and will 
> have
> to send traffic through the very expensive UUnet port.

that i don't follow

suppose that the server has v6 PI addresses, which for very big sites 
makes sense imho

The server can send traffic with destiantion address belonging to UUNet 
through Cognet, right? I mean i am assuming that UUNet and Cognet have 
connectivity that is not through S

I mean, The client can choose to use the IP from UUNet (that is his 
choice and he has the right to do so, because he is paying for it) This 
choice, affects the ISP used to get _to_ the client and it shouldn't 
determine the ISP used to get to the Server

So in this case the traffic would flow:
 From the client to the Internet through UUNet
 From the internet to the Server through Cognet

agree?

Now the problem is when the server also has PA blocks

In this case, the destiantion address selected by the client will 
determine the ISP of the server

Without shim, the server don't have many options, basically what he 
could do is to use the DNS to prioritize the Cognet addresses.
With the shim, the server can rehome any communication that is using 
UUnet addresses to Cognet and start using Cognet locators. This of 
course does not prevent the client to keep on using the UUnet 
destination addresses. In this case, the server can inform the client 
about his preferences using a shim protocol option, but even in this 
case the client can prefer other than what is expressed by S in the 
preferences. In any case, in this model, each can always choose the 
path used to send packets. I guess that in IPv4 is somehow different 
because the decision belongs to the intermediate ASes, which are to 
ones that can select which path to use (note that in this case, is not 
S who is in charge to select the incoming path neither)

>  With v4, on the
> other hand, the router was aware that Client is reachable via both 
> Cogent
> and UUnet, and could have had a localpref configured that would just 
> say
> "anything reachable over cogent, use cogent". One way to fix that 
> would be
> to do a shim6 init in the 3way handshake, but the problem then becomes
> that *every* "S" would have to have a complete routing table, and
> basically, perform the logic that is done in today's routers.

why is that?

I mean if S prefers Cognet, all he has to do is:
- In the PI case, route its outgoing packet through Cognet and do the 
same v4 bgp magic to direct incoming packet through cognet
- In the PA case, always use cognet addresses and try to convince the 
clients to  use the server's IP address of the cognet prefix (through 
SRV and/or shim preferences)

> Obviously
> running Zebra w/ full routes on a server is a non-trivial performance 
> hit,
> and multiplied that out by the number of servers, and it gets
> very expensive, very fast. All to re-gain capabilities we have right 
> now
> in ipv4 for free...
>
> Now, of course, the "so called easy" answer would be "let's introduce a
> routing policy middleware box that would handle that part". That box 
> would
> have the full routing tables, the site policies, and when queried with
> "I'm server X, and this is the client and all his locators, which one 
> do I
> use?" would spit back an answer to that server that would be a fully
> informed decision, and the TE problem becomes mostly solved. I say
>

but there seems to be two different problems here (at least :-)
- one: which are the TE capabilities available with the PA addressing 
model + the shim tool. This is what this can be done in this case.
- second: who is in control of these capabilties and how are they 
managed i.e. who controls the policy and who manages the devices that 
are in control of the policy. Is it possible to have a centralized 
policy management? is it possible to enforce the usage of policy (at 
least within the multihomed site)?

I guess that before we were considering the problem one and now the 
second one...

This server idea that you are considering was presented by Cedric de 
Launois in a work called NAROS a while ago

The other option is what we are discussing below about using a 
DHCP/RAdv option to distribute the policy information among the hosts

The other option is to move to a scheme based on rewriting source 
prefixes

Or a combination of those

> "mostly", because now there are these pesky issues of a) do I trust 
> that
> the server is going to obey by this decision (either hacked, or is a 
> box
> outside of my administrative control, yet is within the scope of my
> network control); b) how do transit ISP's "influence" that decision (at
> some point I cross their network, and they should be able to control 
> how
> the packets are flowing through their network; c) how do I verify that
> their "influencing" doesn't negate mine, and is legitimate; d) how much
> "lag" does it introduce into every session establishment, and is it
> acceptable; d) can this proxy scale to the number of queries fired at 
> it,
> and the real-time computations that would have to happen on each one
> (since we can't precompute the answers); and finally is it *really* 
> more
> cost-effective then doing all this in routers.
>
> So far, I'd rather pay for bigger routers...
>
> :: I mean in this case, the selection of the server provider is 
> determined by
> :: the server's address not by the client address, right?
> :: The server can influence such decision using SRV records in the 
> DNS, but not
> :: sure yet if this is the case you are considering
>
> See above about difficulties of scaling DNS to meet this goal...
>

but the problem with the DNS that you have considered above is about 
having to achieve that the DNS publish information that reflect the 
state of the links.
This seems indeed very dificult, especially becuase of cached 
information and so on. But as far as i know, no one is proposing this. 
The idea is to use SRV records to express policy, and a not very 
dynamic one. I mean, you can express that like 30% of the 
communications needs to use a given address and the others the other 
address and so on, but the idea is not to allow the DNS to reflect the 
state of the network
Actually, it may happen that some of the addresses in the DNS are down. 
In this case, the idea is to let the hosts to detect this an retry 
using alternative addresses. whether this retrial is visible or not to 
the apps, is still an open issue

> :: > This change alone would add millions to the bw bills of said
> :: > content providers, and well, reduce the likelyhood of adoption of 
> the
> :: > protocol by them. Now, if the shim6 init takes place in the 3way
> :: > handshake process, then the servers "somewhat" know what all 
> possible
> :: > paths to reach that locator are, but then would need some sort of 
> a
> :: > policy server telling them who to talk to on what ip, and that's 
> something
> :: > which will not simply scale for 100K+ machines.
> :: >
> ::
> :: I am not sure i understand the scaling problem here
> :: Suppose that you are using a DHCP option for distributing the SHIM6
> :: preferences of the RFC3484 policy table, are you saying that DHCP 
> does not
> :: scale for 100K+ machines? or is there something else other than 
> DHCP that
>
> Well, first, show me a content provider who thinks that dhcp scales 
> for a
> datacenter (other then initial pxeboot/kickstart/jumpstart, whatever), 
> but
> that aside, running zebra/quagga + synchronizing policy updates among
> 100K+ machines simply does not scale (operationally).
>

So, you are considering here the case where policy is changed according 
to the state of the network, right?
So that BGP information is used as feedback to the TE decision, is that 
correct?
Is this possible today? how is it done? could you provide an example of 
how you use this dynamic TE setting?


> :: > 4) As has also been discussed before, the initial connect time 
> has to be
> :: > *very* low. Anything that takes longer then 4-5 seconds the 
> end-users have
> :: > a funny way of clicking "stop" in their browser, deeming that "X 
> is down,
> :: > let me try Y", which is usually not a very acceptable scenario 
> :-) So,
> :: > whatever methodology we use to do the initial set-up has to 
> account for
> :: > that, and be able to get a connection that is actually starting 
> to do
> :: > something in under 2 seconds, along with figuring out which 
> sourceIP and
> :: > destIP pairs actually can talk to each other.
> ::
> :: As i mentioned above, we are working in other mechanisms than the 
> shim6
> :: protocol itself that can be used for establishing new communication 
> through
> :: outages.
> ::
> :: you can find some work in this area in
> ::
> :: 
> ftp://ftp.rfc-editor.org/in-notes/internet-drafts/draft-bagnulo-ipv6-
> :: rfc3484-update-00.txt
>
> It's a fairly good idea of negotiating which SRC and DEST ips to pick, 
> but
> it has to happen *fast* (ie sub 2 seconds), or the end-users will lose
> patience, and declare the site dead. Perhaps racing SYNs?

yes, this is an option and it is nice because you actually get not only 
to detect which ones are actually working but also to pick the fastest 
one. But clearly there is the cost of the additional SYNs you send that 
is basically overhead... would you be willing to pay for this multiple 
SYNs?

>
> Now, I'm not saying that all these problems can't be solved for people 
> to
> consider shim6 a viable solution, but so far, they aren't solved, and
> until they are, I just don't see recommending to my employer to take 
> shim6
> seriously,

I may well agree with you here, but remeber that we are still defining 
the protocol :-)


I guess the point here is how we can manage to provide a solution that 
fits the site's requirements, hence your feedback is very valuable

Regards, marcelo


>  since it seems like all it's going to do is to move the costs
> elsewhere, and quite possibly increase them quite a bit in the 
> process...
>
> -igor