Re: [vwrap] [wvrap] Simulation consistency

[Morgaine <[e-mail]>]

On Sun, Apr 3, 2011 at 3:42 AM, Israel Alanis <[e-mail]> wrote:

>
> I think what you're saying is: "It costs a lot of money to build a scalable
> asset service, but if assets are spread throughout the internet they don't
> have to be scalable." But that's not quite right. You're opening up every
> asset server to the VW equivalent of being slashdotted, so are you sure
> you're not forcing *every* asset service to be scalable and handle a lot of
> bandwith and network traffic? It's the exact opposite of your intention, but
> I think that's the result, all the same.
>
>

We actually discussed this in the early days of the list, and I was
advocating that part of the duties of a huge commercial service that
supports VW interop needs to be to temporarily cache assets of new arrivals
from other worlds, as an independent service of the large VW, ie. a Caching
Service.

The reason why I believe a separate Caching Service (unrelated to any Asset
Service) is needed is because the commercial service will not be able to
maintain a desired quality of service without it for its users.

Imagine a Truly Awesome Commercial World, able to support media crowd events
of 100,000 avatars.  Lucy The Musical Furries Fan runs a tiny world for her
family with its own midget integrated asset service, and hears that her
adored Musical Furries are going to play at Truly Awesome Commercial World,
so off she teleports to the venue.  Instantly on arrival, 100,000 fans of
the Musical Furries slashdot her asset service into a smoldering slag heap
as they all try to fetch her avatar's assets.

Clearly this is not going to work.  Quite apart from Lucy's problem, it is
the Truly Awesome Commercial World that suffers most, because it now has
100,000 fans who are faced with a broken simulation.  It's no longer very
awesome.

(Not allowing entry to Lucy is not an option because the huge world relies
on 100,000 people like Lucy arriving to support the event, many of whom are
in Lucy's situation.)

Unfortunately I never managed to get anyone interested in the concept of a
Caching Service as an independent entity.  Perhaps it's time to revive the
idea.

Note that the Caching Service would be yet another independent service,
conceptually external and not related administratively with any other entity
in our picture.  It would be optionally engaged by a region that wants to
open its doors to the metaverse but takes pride in offering a better quality
of service than its visitors' asset services can provide by themselves.


Morgaine.



============================


On Sun, Apr 3, 2011 at 3:42 AM, Israel Alanis <[e-mail]> wrote:

>
> > when designing for scalability, the model to bear in mind is ...
>
> Well, there are a lot of different models to keep in mind, and many
> different use cases. One particular use case to keep in mind is: "User
> acquires new outfit, and wants to 'show it off' in a highly populated
> region".
>
> > Both worlds and asset services may include commercial, community, and
> personal services
>
> Yes, yes and yes. I'm particularly concerned about how the model affects
> the ability to host personal asset services.
>
> > a proxying region, which would get slammed for every asset worn by every
> avatar present.
>
> Granted the collection of services that are provided by the region need to
> be scaled to meet the demands of that region. That's all part of capacity
> planning.
>
> > regions run many different CPU-intensive tasks, including physics
> simulation and server-side scripting, and absolutely cannot afford to serve
> assets too
>
> Well... who said the same CPU's have to do proxying, physics simulation and
> server-side scripting? Asset proxying is a different service than physics
> simulation and can be on separate hardware, could make use of geographically
> distributed caching, and in certain deployment patterns, the same caching
> services could be shared by different regions. (Server-side scripting is a
> discussion for another day).
>
> > This is why we have to go parallel...
>
> Totally agree, and a proxying model could and should also take advantage of
> parallelism.
>
> > I think you're wrong that it has to cost much money.
> ?vs?
> > It costs money to host a high performance and scalable asset service and
> a high bandwidth network to handle the traffic.  A *lot* of money.
>
> I think what you're saying is: "It costs a lot of money to build a scalable
> asset service, but if assets are spread throughout the internet they don't
> have to be scalable." But that's not quite right. You're opening up every
> asset server to the VW equivalent of being slashdotted, so are you sure
> you're not forcing *every* asset service to be scalable and handle a lot of
> bandwith and network traffic? It's the exact opposite of your intention, but
> I think that's the result, all the same.
>
> This particular design decision has a big effect on the economics of the VW
> infrastructure. I'd rather the economics to work out such that a region
> provider who wishes to build a region that supports a small population, can
> do so economically. A region that wants to host a *large* population has to
> bear that cost of providing that scalable asset service.
>
> I want the economics of hosting a small asset service to be a non-issue (as
> to best promote creation and creativity). Creating a high bar to provide
> asset services will mean that service will cost money and people shouldn't
> have to pay money just to create or own VW objects (I'm using 'own' here to
> refer to maintaining their existence, I'm not trying to make a
> 'leftist'/'communist' statement about ownership ;)
>
> - Izzy
>
>
> On Apr 2, 2011, at 3:58 PM, Morgaine wrote:
>
> Izzy, when designing for scalability, the model to bear in mind is that of
> seasoned virtual world travelers whose inventories contain assets from many
> different worlds, those assets being served by many different asset
> services.  Both worlds and asset services may include commercial, community,
> and personal services, and as the metaverse grows, that set is highly likely
> to become progressively less clustered and more diverse.
>
> When those seasoned travelers click on an advertised VW link and perform an
> inter-world teleport to one particular world's region to share an
> experience, their "worn" assets (the only ones of interest to the region)
> will contain references to asset services spread widely across the
> Internet.  The fetches by the travelers' clients occur over many parallel
> paths from clients to asset services, so one can reasonably expect reduced
> network contention and reduced asset server loading because they are both
> spread out over however many asset services are being referenced by the
> overall set of assets in the region.
>
> This is very different to the case of a proxying region, which would get
> slammed for every asset worn by every avatar present.  In our current
> architecture, regions run many different CPU-intensive tasks, including
> physics simulation and server-side scripting, and absolutely cannot afford
> to serve assets too unless your scalability requirements are very low
> indeed, ie. just a few dozen avatars of today's kind.  We've hit the ceiling
> already on region scalability done that way.  There is nowhere to go in that
> direction at all beyond improving the code like Intel demonstrated, and that
> work is subject to a law of diminishing returns.
>
> This is why we have to go parallel, and I think you're wrong that it has to
> cost much money.  As we spread the load across more and more asset services,
> we are simply better utilizing all the hardware that's already out there on
> the Internet, at least in respect of community and private resources.  But
> add to the community and private resources the commercial asset services
> that are likely to appear to exploit this opportunity, and not only will the
> number of asset services leap, but the power of each one will rocket too,
> because, after all, these businesses will be heavily optimized for the job.
>
> As to why a world would want clients to access external asset services
> instead of providing its own implementation, that's an easy question.  It
> costs money to host a high performance and scalable asset service and a high
> bandwidth network to handle the traffic.  A *lot* of money.  In contrast,
> it costs a world nothing to let others serve the assets to clients.  And
> that matters to the bottom line.
>
>
> Morgaine.
>
>
>
>
> ======================
>
> On Sat, Apr 2, 2011 at 7:05 PM, Izzy Alanis <[e-mail]> wrote:
>
>> > As always though, it's a trade-off, since the proxied design has very
>> poor scalability compared to the distributed one.
>>
>> I don't agree with that... If a user enters a highly populated region,
>> every other client is going to (could and should be trying to) hit the
>> asset server(s) for the assets that the user is wearing (assuming
>> they're not cached locally).  Every asset server has to be scaled up
>> to the point that it can handle that load from all over...
>>
>> If I'm hosting a region that supports 10s of thousands of simultaneous
>> users (thinking of the future), I already have to scale to meet that
>> demand. If the region is proxying the assets, then, yes the region has
>> to be scaled to meet that asset demand too, but it already has to be
>> scaled to meet other demands of being a region server... and why is
>> scaling those asset proxy services hard?  It's going to cost $, but is
>> not technically challenging. So, if I want to host a region like
>> that... sure it will cost me, but the simulation will be consistent
>> and users will be able to participate equally, regardless of the
>> capabilities of their individual asset services.
>>
>>
>>
>>
>> On Fri, Apr 1, 2011 at 11:55 PM, Morgaine
>> <[e-mail]> wrote:
>> > Every design choice results in a trade-off, Vaughn, improving one thing
>> at
>> > the expense of something else.  If every time we offered a service we
>> had to
>> > inform its users about the downsides of all the trade-offs we have made,
>> > they would have an awful lot to read. ;-)
>> >
>> > The specific trade-off that you are discussing is no different.  A
>> region
>> > that proxies all content has the "benefit" of acquiring control from the
>> > asset servers over the items in the region, so that it can ensure that
>> > everyone in the region not only obtains the items but obtains the same
>> items
>> > as everyone else.  That does indeed provide a greater guarantee of
>> > consistency than a deployment in which the region only passes asset URIs
>> to
>> > clients who then fetch the items from asset services separately.  As
>> always
>> > though, it's a trade-off, since the proxied design has very poor
>> scalability
>> > compared to the distributed one.
>> >
>> > If we're going to warn users of the potential for inconsistency in the
>> > distributed deployment as you suggest, are we also going to warn them of
>> > non-scalability in the proxied one?  I really don't see much merit in
>> the
>> > idea of warning about design choices.  Many such choices are technical,
>> and
>> > the issues are quite likely to be of little interest to non-technical
>> users
>> > anyway.  In any case, the better services are likely to provide such
>> > information in their online documentation, I would guess.
>> >
>> > You mentioned users "voting with their feet" or choosing to accept the
>> risk
>> > of inconsistency.  Well that will happen anyway, when services fail and
>> > users get annoyed.  If some asset services refuse to send the requested
>> > items to some users, those services will get a bad reputation and people
>> > will choose different asset services instead.  Likewise, if a world
>> service
>> > proxies everything and so it can't handle a large number of assets or of
>> > people, users will get annoyed at the lag and will go elsewhere.  This
>> user
>> > evaluation and "voting with their feet" happens already with online
>> services
>> > all over the Internet, and I am sure that this human process will
>> continue
>> > to work when the services are asset and region services.
>> >
>> > Back in September 2010, I wrote this post which proposes that we use in
>> > VWRAP a form of asset addressing that provides massive scalability at
>> the
>> > same time as a very high degree of resilience --
>> > http://www.ietf.org/mail-archive/web/vwrap/current/msg00463.html .  It
>> is
>> > based on the concept of the URI containing a host part and a hash part,
>> > where the hash is generated (once, at the time of storage to the asset
>> > service) using a specified digest algorithm over the content of the
>> asset
>> > being referenced.  You may wish to note that if this design were used,
>> the
>> > failure of an asset service to deliver a requested item would result in
>> a
>> > failover request for the item to one or more backup services, using the
>> same
>> > hash part but with a different host address.
>> >
>> > This can go some way towards overcoming the problem that you think might
>> > occur when assets are fetched by clients from asset services directly.
>> > Although it won't help when the missing item is available from only a
>> single
>> > asset service, it will help in many other cases, and it will compensate
>> for
>> > service failures and network outages automatically at the same time.
>> >
>> > PS. This design for hash-based asset addressing is already being tested
>> by
>> > Mojito Sorbet in her experimental world and client.  It would give
>> > VWRAP-based worlds an improved level of service availability, so I think
>> it
>> > should be a core feature of our protocol.
>> >
>> >
>> > Morgaine.
>> >
>> >
>> >
>> >
>> > ===========================
>> >
>> > On Fri, Apr 1, 2011 at 11:17 PM, Vaughn Deluca <[e-mail]
>> >
>> > wrote:
>> >>
>> >> This is a question i discussed with Morgaine off-list a while ago (I
>> >> intended to send it to the list but pushed the wrong button...) I
>> >> think we need to address this problem, and decide how to deal with it.
>> >>
>> >>  In Davids deployment draft, section 7.3.1.1 an overview is given van
>> >> ways to deliver content to the region. One way is only passing a
>> >> capability that allows access to (part of) the resource:
>> >>
>> >>           7.3.1.1.  Content delivery models
>> >>           A range of possible represenations can be passed to a region
>> for
>> >>           simulation. [...] The other end of the delivery spectrum
>> >> involves passing
>> >>           only a URI or capability used to access the rendering
>> >> information and a
>> >>           collision mesh,and related data for physical simulation.
>> >>           In such a model, the client is responsible for fetching the
>> >> additional
>> >>           information needed to render the item's visual presence from
>> a
>> >> separate
>> >>           service.  This fetch can be done *under the credentials of
>> the
>> >> end user*
>> >>           viewing the material [my emphasis--VD] , and divorces the
>> >> simulation from
>> >>           the trust chain needed to manage content.  Any automation
>> >> is done on a
>> >>           separate host which the content creator or owner trusts,
>> >> interacting with the
>> >>           object through remoted interfaces.
>> >>
>> >>  I can see the need for such a setup, however, i feel we are
>> >> unpleasantly close to a situation were the coherence of the simulation
>> >> falls apart.
>> >> In this deployment pattern the region advertises the presence of the
>> >> asset, and *some* clients will be able to get it as expected, while
>> >> -based on the arbitrary whims of the asset service- others might not.
>> >>
>> >> My hope would be that after the asset server provides the region with
>> >> the capability to get the asset, it gives up control. That would mean
>> >> that if the client finds the inventory server is unwilling to serve
>> >> the content - in spire of the region saying it is present-, the client
>> >> should be able to turn around ask the *region* for the asset, (and get
>> >> is after all).
>> >>
>> >>  If that is not the case, -and there are probably good reasons for the
>> >> deployment pattern as described-  shouldn't we *warn* clients that the
>> >> region might be inconsistent, so the users behind the client can vote
>> >> with their feet, (or take the risk)?
>> >>
>> >> --Vaughn
>> >> _______________________________________________
>> >> vwrap mailing list
>> >> [e-mail]
>> >> https://www.ietf.org/mailman/listinfo/vwrap
>> >
>> >
>> > _______________________________________________
>> > vwrap mailing list
>> > [e-mail]
>> > https://www.ietf.org/mailman/listinfo/vwrap
>> >
>> >
>>
>
>
>