Re: [vwrap] Technical basis for VW client in a web browser?

[Dan Olivares <[e-mail]>]

I think, one thing that we've learned by exploring the leading virtual
world technologies that exist today is that TCP should be used for any
transmission that requires ordered and reliable delivery.   Doing so,
shares the 'flow control' load across the devices on the internet (for
free).

The user level UDP flow control may /seem/ negligible.  With a small
amount of users and high quality connections, it is negligible.
However, as the load increases and the quality of the network links
decreases, there's a considerable spike in CPU time required to
maintain reliable delivery.  Generally, with UDP, flow control, and
bad quality or overloaded connections, the load is felt on the CPU of
the server managing the flow control.   *The quality to CPU usage
curve is precipitous.*  As the quality of the connection decreases a
little, the CPU load increases a LOT.  At a certain point, the server
falls off of the edge of the cliff and can no longer keep up.  Sending
reliable data in any but the lowest quantity, is far worse for the
overall user experience over UDP then TCP as the load increases.
Network congestion, leading to single server CPU overload doesn't just
affect the experience of one user with a low quality connection.  It
affects the quality of the experience for every user connected.
With ISPs de-prioritizing UDP packets, using it for a large amount of
reliable communication isn't a good option.

UDP is of most benefit when there is a significant amount of
unreliable traffic that can be dropped if it comes in out of order.
Real time spacial information, for example, benefits from this
approach.

The flow control coupled with the quantity of traffic that results
from the server relaying reliable updates over UDP leaves less CPU
resources for other things on the server and is one of the reasons for
the 'SecondLife 100 user Limit'.    Intel reached 1000 users by
separating the client manager from the world space managing
server(simulator). (#0)   (There are other reasons for the 100 user
limit, but they're beyond the scope of this document)


TCP has guaranteed delivery of every packet.  In the content space,
TCP is the most effective delivery mechanism for everything except a
small subset of technologies.

That doesn't mean that we should throw UDP out.  TCP has it's own
problems.   With TCP, the load isn't felt on the CPU, it's felt in the
speed of delivery of the content.

As Dahlia said, every packet must arrive before the next one can begin
to be sent.  It isn't effective to send large quantities of quickly
expiring data over TCP.   With multiple types of data requiring
different speeds of delivery, it also isn't effective stuffing
everything into the same connection.

Browsers overcome this problem by making multiple temporary
connections downloading referenced resources immediately after
downloading a full 'index' (html) content page.

Some people might think that if browsers have overcome this problem
then it might just be better to use the same technology for virtual
worlds (HTTP).   There's still an issue with that reasoning.   The
issue is that HTTP was really designed to transmit files and not a
'data stream of multiple objects'.   There is overhead and there are
restrictions, designed with files in mind, built into HTTP that make
it, not quite a perfect fit.  You can overlay 'streaming compatible
files' on top of HTTP but it usually involves establishing several
connections over time for the same content element.

This is where web sockets find their sweet spot.   It's the middle
ground, getting the benefits of TCP while not encountering the
restrictions of HTTP.  You can create multiple TCP connections for
different types of data allowing you to get around the TCP queue bog.
  They still don't really solve the 'mass quantities of quickly
expiring data' problem though, so we still can't throw UDP out the
window.

In conclusion, at the moment, none of the existing underlying
protocols are optimal for the complex nature of the data of virtual
worlds.  However, with some ingenuity and 'out of the box' thinking,
we can utilize what's there to 'almost' get there.

We do need a few things:
1. Web Sockets standard in browsers (we can make due without UDP
support, but we really really want it)
2. Solid, and accepted standard for 3d rendering in the browser.
(Problems: WebGL + Internet Explorer?, Flash + 3d rendering speed +
Closed source renderer?, Unity3d + Plugin distribution + closed source
renderer?)
3. Faster JavaScript (Each WebGL frame, fully processed with
JavaScript is a bit much when you consider the problem space of
thousands of user created objects being displayed)

Regards

Daniel Olivares

References:
(0) : http://nwn.blogs.com/nwn/2010/09/intel-science-sim.html



On Mon, Dec 20, 2010 at 11:44 AM, Morgaine
<[e-mail]> wrote:
> On Mon, Dec 20, 2010 at 12:56 AM, Dahlia Trimble <[e-mail]>
> wrote:
>>
>> If anyone has any evidence of internet pathways that selectively favor TCP
>> over other traffic, I'd be interested in seeing it.
>
>
> I have first-hand knowledge of this.  I worked for several years at the
> Network Operations Centre of a top-tier ISP, and one of my duties was
> looking after service routers and firewalls.  Policy-based routing and
> firewall access lists are configured with rulesets which are processed
> sequentially and eat up router CPU, which is a finite resource.
>
> Under off-peak conditions, packet loss is quite rare in the absence of
> interface or line faults, and router CPUs are scaled to handle the expected
> load so packets are never dropped willfully.  When networks are congested
> however, which unfortunately is not uncommon during peak hours owing to the
> common practice of oversubscribing capacity (or poor scalability planning),
> CPU load often reaches critical levels, and routers are configured to
> prioritize certain payload types in favor of others when this happens.
>
> TCP always gets top priority because it carries HTTP which is most closely
> tied to business revenue.  In contrast, UDP (and also ICMP Echo) are
> normally configured right down the bottom end of the priority list, so under
> peak load when the CPU has to make a choice what to drop to stay within safe
> operating limits, UDP gets the chop first.  This was business as usual at
> the ISP, and that's how the network designers wanted the traffic priorities
> configured.  (I was merely implementing policy, not creating it.)
>
> Gaming fans sometimes complain that ISPs are reducing the quality of service
> of their UDP traffic, and in some sense it's true.  From my experience it's
> not done maliciously nor as an conscious policy of network non-neutrality,
> but simply as a means of protecting the more prized resource of TCP payloads
> when operating conditions mandate that something has to be thrown away.
>
> On the positive side, I've never known UDP packets (nor any other kind) to
> be dropped willfully for the above reason when all equipment is working
> within safe design limits and there is enough capacity to carry them.  If it
> happens off-peak then something is very wrong with network sizing, or the
> equipment is faulty.
>
> Unfortunately, that's not the end of the saga with UDP.  It's just the
> beginning, because there is another big reason for dropped packets, and this
> one occurs even when equipment is working within its designed operating
> limits:  traffic shaping.
>
> IP traffic shaping is performed by packet queuing as a first resort, to slow
> down traffic in the hope that the source notices and adapts.  If the packet
> rate doesn't slow down then the method of last resort is to drop excess
> packets when queue buffers hit their configured highwater marks.  TCP
> implements a lot of things to mitigate packet loss, such as slow start,
> exponential backoff and transmit pacing, with the purpose of adapting
> transmit rate to receipt rate across paths of limited bandwidth so that
> traffic-shaping routers don't enter their drop state.
>
> UDP has no such flow control, so the onus falls upon the UDP application
> endpoints to carry out adaptive flow control themselves.  The likelihood of
> this being done by UDP applications as effectively as it is done in today's
> finely honed TCP stacks is very low.  It may not even be done at all.
>
> As a result, UDP packets can get dropped for being bad network citizens and
> not slowing down in response to packet queueing and transmit pacing.  UDP
> applications may think that they're free of the shackles of TCP flow
> control, but they're not.  They either slow down when given the hint, or
> their traffic gets the chop.  As a result, a UDP application that is
> oblivious of end-to-end timing should expect packet loss when the network
> acts to protect itself against congestion.  See RFC-2309 for more details
> about this issue, in particular "MANAGING AGGRESSIVE FLOWS".  There is no
> free lunch for UDP packets.
>
> This is why the best advice to give prospective users of UDP is "Don't,
> unless your application is tolerant of packet loss, packet duplication,
> packet delay, corrupted packets, and out of order delivery."  To try to work
> around these properties of UDP and make it reliable while simultaneously not
> impacting on the congestion controls of TCP over the shared path is highly
> unlikely to be successful, unless you reimplement the clever control
> features of TCP, and do it compatibly.  Bulldozing your UDP packets through
> a shared network is not a solution, and will fail.
>
> [PS. The problems don't even stop there, as there are further causes of UDP
> packet loss.  One is the effect of TCP flow-control synchronization on UDP
> loss rate over congested paths, which counter-intuitively negates any
> benefit that could result from reducing UDP traffic rates because TCP
> synchronization picks up any bandwidth slack that reducing UDP traffic has
> freed.  As a result, TCP congestion actually increases UDP packet drop
> rates.  (This has been a topic of research.)  Networks protocols have very
> complex behaviors.]
>
>
> Morgaine.
>
>
>
>
> ======================================
>
> On Mon, Dec 20, 2010 at 12:56 AM, Dahlia Trimble <[e-mail]>
> wrote:
>>
>> I have used both TCP and UDP in VW applications. I've found that TCP has
>> acceptable latency and is not really any worse than UDP when either are
>> tried over a clean, highly functional connection. I've not seen any routers
>> which drop UDP packets in favor of TCP, and I've not seen any evidence of
>> better quality TCP connections than UDP in any of my tests. To the contrary,
>> I've seen UDP perform much better when network conditions are less than
>> optimal as small messages can be sent immediately and repeated as needed
>> without waiting for prior message acknowledgement or waiting for a TCP
>> stream to recover in the event of dropped packets.
>>
>> TCP seems to be favorable when latency is not critical as it's generally
>> (but not always) easier to use. UDP seems favorable when latency is critical
>> as it allows the programmer to control network traffic and tailor it to the
>> application requirements.
>>
>> If anyone has any evidence of internet pathways that selectively favor TCP
>> over other traffic, I'd be interested in seeing it.
>>
>>
>> On Sun, Dec 19, 2010 at 3:26 PM, Meadhbh Hamrick <[e-mail]>
>> wrote:
>>>
>>> do we really know that UDP is what we want, even for low latency? if
>>> you're multiplexing messages over a websocket connection, it's highly likely
>>> it'll be an existing connection (i.e.- it's likely one tcp/ip connection
>>> will carry several multiplexed websockets messages.)
>>>
>>> in my tests, UDP doesn't do much better than TCP if you're near the
>>> network rate as it seems a lot of routers tend to dump UDP packets first.
>>>
>>> most modern OSes now have api calls to let you disable TCP slow-start.
>>>
>>> i guess what i'm saying is it might be a good idea to define messages in
>>> a way so they're transport agnostic. that and I would wager that any latency
>>> improvements from UDP are dwarfed by latency introduced by application layer
>>> mechanisms to replace TCP's flow control & resend semantics.
>>>
>>> just my $0.02.
>>>
>>> On Dec 19, 2010 2:34 PM, "SM" <[e-mail]> wrote:
>>>
>>> _______________________________________________
>>> vwrap mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/vwrap
>>>
>>
>>
>> _______________________________________________
>> vwrap mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/vwrap
>>
>
>
> _______________________________________________
> vwrap mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/vwrap
>
>