Re: [ogpx] My take on Teleports and protocol resilience

[Vaughn Deluca <[e-mail]>]

On Mon, Oct 26, 2009 at 4:02 PM, Carlo Wood <[e-mail]> wrote:

> That's not how I'd put it Vaughn...
> First of all, for the data proposed (for example, the currently active
>

LOL, I have the strong feeling we agree on almost all counts, I was merely
pointing out some side effects and trying to get a handle on the actual
costs your solution would carry. Your post explains again very clearly the
logic behind your proposal, but that was not my point. I fully agree with
you that it is a very logical and robust solution, my question related to
the efficiency costs.

At the end of your post you come to that point:

>If you'd want to avoid this (minimal!?) RAM usage on the AS
>then you are forced to drop the AS as authority,

First, my question was how much data we are *actually* talking about here.
You use the animation as an example, but there is much more in the Avatars
state than just the current animations. If we serialise the Avatars state to
send it over to the next region, how much data is that, and what if I am a
full size Dragon with a thousand active scripts and hundreds and hundreds of
attachments? And  what can we expect it to be in the future?
I *suspect* its on average not a critical amount, yet, i was hoping somebody
who *knows* would chime in.

Second, as i mentioned in my post i am not very concerned about the storage
space, but more about the *bandwidth*. In your proposal all changes pass
from viewer to AS to region instead of directly from viewer to region. No
matter how you call the info a copy, it is present at *both* locations i and
since you want to keep the AS in the loop, all updates pass trough the AS,
potentially crossing the whole real world. How much traffic is that, and
shopuld we be worried?

>and there is no way you can recover gracefully in case of
>network failure.

Right, that is the consequence of being more efficient.  I am  trying to get
some good estimates for the price of that robustness. But maybe i should
look at the sourcecode and get an estimation myself...

>Worse, in case of a teleport, you need to CHANGE
>who the authority is, which gives heaps and heaps of potential
>problems.

It depends what you are aiming for. You are paying in network traffic and
storage space for robustness and flexibility. In some use cases you will
want to optimise the robustness of the system yet, if i am running an Agent
Domain as a business, i might really not be pleased with he traffic your
solution makes me carry, just in case some region i do not own fails...
 Again, it might be all peanuts, but i would like to *know*

Finally, there might actually be another solution. After all, the *real*
"natural" authority is the viewer.  The viewer can keep a local copy of
avatar state. In case of a network failure the AS could ask the viewer to
send the authoritative info so it can re-create the state and send it over
to the new region. Would that work?

-Vaughn




> animation(s)), the AS *is* the natural authority and origin to begin with.
>
> Allow me to introduce some more theory / jargon. Consider some variable
> that changes over time (lets call it 'data'). The value of the variable
> plays a role in more than one host, and synchronization is done via a
> network. Because networks are not reliable it must be defined who is
> the authority, so that in case of failure, things can be re-synchronized
> from the authority. This means automatically that the authority must
> keep a copy of the data, of course.
>
> Now consider the following:
>
>  Authority             Remote
>  value      -------->  value
>
> I call this a 'downstream' synchronization: the data is always and only
> first changed on the authority, and then propagated away from it.
>
> 'Upstream' messages that would change the value almost garanteed lead
> to problems, simple example: assume some variable has value 'A',
> the authority changes it to 'B', and some downstream host changes it
> at the same time to 'C'. You'd then have:
>
>  Authority                   Remote
>  'B'       ---B-->  <--C---  'C'
>
> ending in
>
>  Authority                   Remote
>  'C'                         'B'
>
> Hence that it is much better to work with requests upstream:
>
>  Authority                           Remote
>  'B'       ---B-->  <--request:C---  'A'
>
>  Authority                           Remote
>  'C'       ---C-->                   'B'
>
>  Authority                           Remote
>  'C'                                 'C'
>
> where there exists a race condition whether or not we end
> up with 'C' or 'B', but at least things stay synchronized.
>
>
> Ok, back to the animation state of an avatar.
> If we state that the AS is the authority, then you can hardly
> speak of 'keeping a copy', it's a trivial thing that both
> the AS as well as the simulator need to know this value.
>
>  AS                       Simulator
>  'animation'    ----->    'animation'
>
> If you'd want to avoid this (minimal!?) RAM usage on the AS
> then you are forced to drop the AS as authority, and there
> is no way you can recover gracefully in case of network
> failure. Worse, in case of a teleport, you need to CHANGE
> who the authority is, which gives heaps and heaps of potential
> problems.
>
> On Sun, Oct 25, 2009 at 11:30:48PM +0100, Vaughn Deluca wrote:
> > > [AS:animation state] --- message stream --> [simulator:animation state
> copy]
> >
> > What you are suggesting here is keeping a copy of the avatar state. We
> would be
> > increasing bandwith and storage space to gain flexibility. I do  not
> think
> > storage space is a real problem. I am not sure about bandwidth.
>
> There would be NO increase in bandwidth imho, because the AS *is* the
> natural authority to begin with: if the user controls the avatar, changes
> it, then the source/origin is the AS (well, the viewer, but it's no
> big deal to use the AS as authority; this DOES mean however that the
> viewer basically only sends requests, and not real state changes).
>
> This picture also shows that the most logical place for HUD scripts
> is the AS as well, not the simulator.
>
> > Is there anybody who can put some real numbers on this? How much extra
> work
> > would the region have to do? Is it still acceptable if the Agent service
> is in
> > Australia and the region in Europe?
> >
> > -Vaughn
> >
> > On Sun, Oct 25, 2009 at 1:15 PM, Carlo Wood <[e-mail]> wrote:
> >
> >     Lets start with some brainstorming...
> >
> >     Things that might be relevant, in no particular order.
> >
> >     * State of avatar
> >     * State of attachments
> >     * Location of avatar
> >     * Time of unresponsiveness of avatar (position) towards viewer
> >     * Time of unresponsiveness of attachments
> (attaching/detaching/scripts)
> >     * Perception of the viewer
> >     * Perception of other viewers
> >     * Region boundary crossing
> >     * Teleport over larger distance
> >     * User configurable parameters
> >     * did I miss something important?
> >
> >
> >     Real message starts here:
> >
> >     Something that protocol (or implementation)- wise leads to problems
> >     are copies of the same data on different hosts; this always leads
> >     to desynchronization of this data with heaps of untrackable problems
> >     (bugs).
> >
> >     The ONLY way to keep a copy of some data reliably synchrononized is
> by
> >     having a stream of state changes being sent from one host to another,
> >     where the messages that contain that state changes always keep
> >     the same order and the stream is never terminated/lost (in which case
> >     a full-resynchronization would be necessary). Note that this means
> >     that the source of the state changes has to be a single point of
> >     origin, which automatically means that we can identify the REAL
> >     (original) copy of the data. Thus:
> >
> >      [original] ---> stream of state changes --> [copy]
> >
> >     Lets call this an "unidirectional state" by lack of a better word.
> >
> >     [PS I'm ignoring actual implementation details here. I'm NOT
> >        saying that currently anyone is using a TCP-stream of
> >        state changes, so don't quote JUST this part with the comment
> >        that this is not how it current works, Infinity :p
> >        Instead, this is an mathematical approach, the abstract
> >        equivalent of any possible deployment case].
> >
> >     Basically, we want to avoid copies of the same data.
> >
> >
> >     The simulator contains a lot of state information that cannot
> >     be moved away from the simulator though: because the state is
> >     needed to calculate the interactions between all objects and
> >     avatars in the region, which would become way too slow if,
> >     for example, several agent domain services would have to be
> >     queried all the time. I can imagine that this also holds for
> >     the attachments on an avatar.
> >
> >     This means that if an avatar moves from one region to another,
> >     all this state information has to be transfered too.
> >
> >     Thus, the origin sends state information to the new region.
> >     If this fails, then we want the user to resume at it's old
> >     location: we need to keep a copy of the state until we know
> >     the teleport was successful. In other words, we will have
> >     a copy of data at two different hosts, temporarily.
> >
> >     One way to make sure that this is not a problem is by
> >     freezing all state; then copy it, and only once the copy
> >     is successful, destroy the old data and resume the simulation
> >     in the other region.
> >
> >     The problem we try to tackle now is the case were the old
> >     region is not responsive...
> >
> >
> >     In this case we have to notice that the AD can ALSO detect
> >     that at least the new region is able to host the avatar:
> >     the teleport can *partly* succeed. This can be detected
> >     independent of copying all the state information.
> >
> >     Secondly, we have to realize that the 'location' of the
> >     avatar is (state) data that does NOT have to be copied
> >     (we're changing it anyway) and therefore is not part of
> >     said state, and does not suffer from the copy-problem.
> >
> >     The same could be said about animations (which is currently
> >     broken in Second Life: you must stop all animations
> >     before teleporting or a desync happens): we could state
> >     that after a teleport no animations are active, and leave
> >     it to the viewer to re-initiate and required animations.
> >     However, animations can be made unidirectional (meaning
> >     that if anyone but the agent service wants to change the
> >     animation, it has to send a request message to the AD,
> >     which then grants it, so that the actual state change always
> >     originates from the agent service (AS)).
> >
> >     As a result, we can think of the animation state as:
> >
> >
> >      [AS:animation state] --- message stream --> [simulator:animation
> state
> >     copy]
> >
> >     which means that we can teleport, keeping the correct
> >     animation(s) without bothering with the source region.
> >
> >
> >     The only clear case where this kind of 'trick' doesn't work
> >     (I think it's not a trick, but the best way to implement this)
> >     is for script states: those change way too frequent to host
> >     them on the AS.
> >
> >
> >     Conclusion
> >
> >     Thus, in the case of an unresponsive source region,
> >     we CAN - without risks of desynchronization - immediately
> >     transfer the avatar (location) and animations, and
> >     any other INFREQUENTLY CHANGING, UNIDIRECTIONAL DATA
> >     that can be stored on the AS (and buffered on the
> >     simulator), like clothing UUIDs, and attachments.
> >
> >     However, the scripts in the attachments will not run
> >     until their state is transfered from the old region.
> >
> >     Imho, this is hardly a problem: after user configurable
> >     timeout we leave it to the user what he wants to do:
> >     logout or reset the scripts :p... Ok, we just reset
> >     the scripts after some time, where the timeout is
> >     determined by the AD with possible input from the user.
> >
> >     --
> >     Carlo Wood <[e-mail]>
> >     _______________________________________________
> >     ogpx mailing list
> >     [e-mail]
> >     https://www.ietf.org/mailman/listinfo/ogpx
> >
> >
>
> --
> Carlo Wood <[e-mail]>
>