Re: [vwrap] Technical basis for VW client in a web browser?

[Dahlia Trimble <[e-mail]>]

I believe part of the problem Dan refers to with CPU load is with the
implementation of LLUDP in OpenSimulator ( http://opensimulator.org ). In
this implementation, the scenegraph simulation server communicates with
multiple clients and much (most?) information is transmitted via UDP,
including many assets which could be retrieved from other sources and using
other protocols. In this case the choice of UDP for *all* communications may
have been a poor one, at least from a scalability perspective. Linden Lab
has recently moved some server<->client information transfer to other
processes outside the main simulation server, and via HTTP protocol. I
suspect as the benefits of this are realized there may be more non-realtime
messaging moving out of Linden's simulator process, and the OpenSimulator
implementation of this protocol will follow.

On Mon, Dec 20, 2010 at 11:11 AM, Dan Olivares <[e-mail]> wrote:

> I think, one thing that we've learned by exploring the leading virtual
> world technologies that exist today is that TCP should be used for any
> transmission that requires ordered and reliable delivery.   Doing so,
> shares the 'flow control' load across the devices on the internet (for
> free).
>
> The user level UDP flow control may /seem/ negligible.  With a small
> amount of users and high quality connections, it is negligible.
> However, as the load increases and the quality of the network links
> decreases, there's a considerable spike in CPU time required to
> maintain reliable delivery.  Generally, with UDP, flow control, and
> bad quality or overloaded connections, the load is felt on the CPU of
> the server managing the flow control.   *The quality to CPU usage
> curve is precipitous.*  As the quality of the connection decreases a
> little, the CPU load increases a LOT.  At a certain point, the server
> falls off of the edge of the cliff and can no longer keep up.  Sending
> reliable data in any but the lowest quantity, is far worse for the
> overall user experience over UDP then TCP as the load increases.
> Network congestion, leading to single server CPU overload doesn't just
> affect the experience of one user with a low quality connection.  It
> affects the quality of the experience for every user connected.
> With ISPs de-prioritizing UDP packets, using it for a large amount of
> reliable communication isn't a good option.
>
> UDP is of most benefit when there is a significant amount of
> unreliable traffic that can be dropped if it comes in out of order.
> Real time spacial information, for example, benefits from this
> approach.
>
> The flow control coupled with the quantity of traffic that results
> from the server relaying reliable updates over UDP leaves less CPU
> resources for other things on the server and is one of the reasons for
> the 'SecondLife 100 user Limit'.    Intel reached 1000 users by
> separating the client manager from the world space managing
> server(simulator). (#0)   (There are other reasons for the 100 user
> limit, but they're beyond the scope of this document)
>
>
> TCP has guaranteed delivery of every packet.  In the content space,
> TCP is the most effective delivery mechanism for everything except a
> small subset of technologies.
>
> That doesn't mean that we should throw UDP out.  TCP has it's own
> problems.   With TCP, the load isn't felt on the CPU, it's felt in the
> speed of delivery of the content.
>
> As Dahlia said, every packet must arrive before the next one can begin
> to be sent.  It isn't effective to send large quantities of quickly
> expiring data over TCP.   With multiple types of data requiring
> different speeds of delivery, it also isn't effective stuffing
> everything into the same connection.
>
> Browsers overcome this problem by making multiple temporary
> connections downloading referenced resources immediately after
> downloading a full 'index' (html) content page.
>
> Some people might think that if browsers have overcome this problem
> then it might just be better to use the same technology for virtual
> worlds (HTTP).   There's still an issue with that reasoning.   The
> issue is that HTTP was really designed to transmit files and not a
> 'data stream of multiple objects'.   There is overhead and there are
> restrictions, designed with files in mind, built into HTTP that make
> it, not quite a perfect fit.  You can overlay 'streaming compatible
> files' on top of HTTP but it usually involves establishing several
> connections over time for the same content element.
>
> This is where web sockets find their sweet spot.   It's the middle
> ground, getting the benefits of TCP while not encountering the
> restrictions of HTTP.  You can create multiple TCP connections for
> different types of data allowing you to get around the TCP queue bog.
>  They still don't really solve the 'mass quantities of quickly
> expiring data' problem though, so we still can't throw UDP out the
> window.
>
> In conclusion, at the moment, none of the existing underlying
> protocols are optimal for the complex nature of the data of virtual
> worlds.  However, with some ingenuity and 'out of the box' thinking,
> we can utilize what's there to 'almost' get there.
>
> We do need a few things:
> 1. Web Sockets standard in browsers (we can make due without UDP
> support, but we really really want it)
> 2. Solid, and accepted standard for 3d rendering in the browser.
> (Problems: WebGL + Internet Explorer?, Flash + 3d rendering speed +
> Closed source renderer?, Unity3d + Plugin distribution + closed source
> renderer?)
> 3. Faster JavaScript (Each WebGL frame, fully processed with
> JavaScript is a bit much when you consider the problem space of
> thousands of user created objects being displayed)
>
> Regards
>
> Daniel Olivares
>
> References:
> (0) : http://nwn.blogs.com/nwn/2010/09/intel-science-sim.html
>
>
>
> On Mon, Dec 20, 2010 at 11:44 AM, Morgaine
> <[e-mail]> wrote:
> > On Mon, Dec 20, 2010 at 12:56 AM, Dahlia Trimble <
> [e-mail]>
> > wrote:
> >>
> >> If anyone has any evidence of internet pathways that selectively favor
> TCP
> >> over other traffic, I'd be interested in seeing it.
> >
> >
> > I have first-hand knowledge of this.  I worked for several years at the
> > Network Operations Centre of a top-tier ISP, and one of my duties was
> > looking after service routers and firewalls.  Policy-based routing and
> > firewall access lists are configured with rulesets which are processed
> > sequentially and eat up router CPU, which is a finite resource.
> >
> > Under off-peak conditions, packet loss is quite rare in the absence of
> > interface or line faults, and router CPUs are scaled to handle the
> expected
> > load so packets are never dropped willfully.  When networks are congested
> > however, which unfortunately is not uncommon during peak hours owing to
> the
> > common practice of oversubscribing capacity (or poor scalability
> planning),
> > CPU load often reaches critical levels, and routers are configured to
> > prioritize certain payload types in favor of others when this happens.
> >
> > TCP always gets top priority because it carries HTTP which is most
> closely
> > tied to business revenue.  In contrast, UDP (and also ICMP Echo) are
> > normally configured right down the bottom end of the priority list, so
> under
> > peak load when the CPU has to make a choice what to drop to stay within
> safe
> > operating limits, UDP gets the chop first.  This was business as usual at
> > the ISP, and that's how the network designers wanted the traffic
> priorities
> > configured.  (I was merely implementing policy, not creating it.)
> >
> > Gaming fans sometimes complain that ISPs are reducing the quality of
> service
> > of their UDP traffic, and in some sense it's true.  From my experience
> it's
> > not done maliciously nor as an conscious policy of network
> non-neutrality,
> > but simply as a means of protecting the more prized resource of TCP
> payloads
> > when operating conditions mandate that something has to be thrown away.
> >
> > On the positive side, I've never known UDP packets (nor any other kind)
> to
> > be dropped willfully for the above reason when all equipment is working
> > within safe design limits and there is enough capacity to carry them.  If
> it
> > happens off-peak then something is very wrong with network sizing, or the
> > equipment is faulty.
> >
> > Unfortunately, that's not the end of the saga with UDP.  It's just the
> > beginning, because there is another big reason for dropped packets, and
> this
> > one occurs even when equipment is working within its designed operating
> > limits:  traffic shaping.
> >
> > IP traffic shaping is performed by packet queuing as a first resort, to
> slow
> > down traffic in the hope that the source notices and adapts.  If the
> packet
> > rate doesn't slow down then the method of last resort is to drop excess
> > packets when queue buffers hit their configured highwater marks.  TCP
> > implements a lot of things to mitigate packet loss, such as slow start,
> > exponential backoff and transmit pacing, with the purpose of adapting
> > transmit rate to receipt rate across paths of limited bandwidth so that
> > traffic-shaping routers don't enter their drop state.
> >
> > UDP has no such flow control, so the onus falls upon the UDP application
> > endpoints to carry out adaptive flow control themselves.  The likelihood
> of
> > this being done by UDP applications as effectively as it is done in
> today's
> > finely honed TCP stacks is very low.  It may not even be done at all.
> >
> > As a result, UDP packets can get dropped for being bad network citizens
> and
> > not slowing down in response to packet queueing and transmit pacing.  UDP
> > applications may think that they're free of the shackles of TCP flow
> > control, but they're not.  They either slow down when given the hint, or
> > their traffic gets the chop.  As a result, a UDP application that is
> > oblivious of end-to-end timing should expect packet loss when the network
> > acts to protect itself against congestion.  See RFC-2309 for more details
> > about this issue, in particular "MANAGING AGGRESSIVE FLOWS".  There is no
> > free lunch for UDP packets.
> >
> > This is why the best advice to give prospective users of UDP is "Don't,
> > unless your application is tolerant of packet loss, packet duplication,
> > packet delay, corrupted packets, and out of order delivery."  To try to
> work
> > around these properties of UDP and make it reliable while simultaneously
> not
> > impacting on the congestion controls of TCP over the shared path is
> highly
> > unlikely to be successful, unless you reimplement the clever control
> > features of TCP, and do it compatibly.  Bulldozing your UDP packets
> through
> > a shared network is not a solution, and will fail.
> >
> > [PS. The problems don't even stop there, as there are further causes of
> UDP
> > packet loss.  One is the effect of TCP flow-control synchronization on
> UDP
> > loss rate over congested paths, which counter-intuitively negates any
> > benefit that could result from reducing UDP traffic rates because TCP
> > synchronization picks up any bandwidth slack that reducing UDP traffic
> has
> > freed.  As a result, TCP congestion actually increases UDP packet drop
> > rates.  (This has been a topic of research.)  Networks protocols have
> very
> > complex behaviors.]
> >
> >
> > Morgaine.
> >
> >
> >
> >
> > ======================================
> >
> > On Mon, Dec 20, 2010 at 12:56 AM, Dahlia Trimble <
> [e-mail]>
> > wrote:
> >>
> >> I have used both TCP and UDP in VW applications. I've found that TCP has
> >> acceptable latency and is not really any worse than UDP when either are
> >> tried over a clean, highly functional connection. I've not seen any
> routers
> >> which drop UDP packets in favor of TCP, and I've not seen any evidence
> of
> >> better quality TCP connections than UDP in any of my tests. To the
> contrary,
> >> I've seen UDP perform much better when network conditions are less than
> >> optimal as small messages can be sent immediately and repeated as needed
> >> without waiting for prior message acknowledgement or waiting for a TCP
> >> stream to recover in the event of dropped packets.
> >>
> >> TCP seems to be favorable when latency is not critical as it's generally
> >> (but not always) easier to use. UDP seems favorable when latency is
> critical
> >> as it allows the programmer to control network traffic and tailor it to
> the
> >> application requirements.
> >>
> >> If anyone has any evidence of internet pathways that selectively favor
> TCP
> >> over other traffic, I'd be interested in seeing it.
> >>
> >>
> >> On Sun, Dec 19, 2010 at 3:26 PM, Meadhbh Hamrick <[e-mail]>
> >> wrote:
> >>>
> >>> do we really know that UDP is what we want, even for low latency? if
> >>> you're multiplexing messages over a websocket connection, it's highly
> likely
> >>> it'll be an existing connection (i.e.- it's likely one tcp/ip
> connection
> >>> will carry several multiplexed websockets messages.)
> >>>
> >>> in my tests, UDP doesn't do much better than TCP if you're near the
> >>> network rate as it seems a lot of routers tend to dump UDP packets
> first.
> >>>
> >>> most modern OSes now have api calls to let you disable TCP slow-start.
> >>>
> >>> i guess what i'm saying is it might be a good idea to define messages
> in
> >>> a way so they're transport agnostic. that and I would wager that any
> latency
> >>> improvements from UDP are dwarfed by latency introduced by application
> layer
> >>> mechanisms to replace TCP's flow control & resend semantics.
> >>>
> >>> just my $0.02.
> >>>
> >>> On Dec 19, 2010 2:34 PM, "SM" <[e-mail]> wrote:
> >>>
> >>> _______________________________________________
> >>> vwrap mailing list
> >>> [e-mail]
> >>> https://www.ietf.org/mailman/listinfo/vwrap
> >>>
> >>
> >>
> >> _______________________________________________
> >> vwrap mailing list
> >> [e-mail]
> >> https://www.ietf.org/mailman/listinfo/vwrap
> >>
> >
> >
> > _______________________________________________
> > vwrap mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/vwrap
> >
> >
> _______________________________________________
> vwrap mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/vwrap
>