Re: [ogpx] Synchronization of gestures (was: Limits to interoperability of scripted objects)

[Han Sontse <[e-mail]>]

My apologies to all for this screwed up thread.   I didn't include the  
mail-host in the thread for the first part of this dialog.

On Dec 4, 2009, at 12:42 PM, Carlo Wood wrote:

> I know I'm not diplomatic, but you'll have to admit
> that in some cases and/or for some people there IS a
> need to wait till the slower ones have the data too.

With all due respect,  Carlo, I am finding that I disagree with your  
analysis.


I have a lot of practical experience in SL getting animations
as well as sound and textures to synchronize under a lot
of different conditions by taking into account the client's existing
cacheing behavior.   I'm drawing on that experience for my assertions.

I can't think of a single reasonable use case that would *require*  
clients
to hold for the slowest link.  The examples you have suggested
don't require it to achieve the outcome you desire.
The only reason that synchronization is a problem today in SL is because
there is no provision for properly notifying the client that two or more
isochronous assets are to be played as part of a sequence.

Signaling the client that some arbitrary bundle of resources are
required to be cached as a suite will handle a vast majority of
the synchronization issues, and a MIDI-like sequence handler can manage
the remainder of specialized cases where complex/strict client side  
isochronous
behavior is desirable.

The outlying cases (the unreasonable ones) are those that really are  
beyond the scope of
what is possible because communication over any appreciable distances  
(> 100km ) is unlikely to
ever be instantaneous in our life times.   While wave fronts in  
electric media do travel
near the speed of light, the switching fabric relaying and repeating  
those signals is
several orders of magnitude slower even under the ideal cases.   Inter- 
client isochronicity to the level
a performing musician might desire is simply out of reach, since the  
ear-brain interface
is sensitive to latencies as small as 2 to 5 ms, and acceptable  
performance almost completely
breaks down at about >50 ms.

So here I'll walk through an example... the hug:

To synchronize a hug all that is needed is for the
hug object to report that it contains bundled isochronous
assets during the normal process of the client caching the
surrounding area.  This bundled sequence might include animation for
the hugger and the huggee, and maybe a sound or two.  It include
possibly a client side script to move the target agents into the
correct relative locations to participate in the hug.

a)	In the simple case the caching of bundled resources has
	occurred.

b)	A permissions request would be sent to the huggee.

c)	The permissions are granted.  (were permission not granted
	a subsequent trigger event would be rejected by the SIM)

d)	The hugger's hug object issues a trigger event which is relayed  
through
	the Sim to all clients within audio/visual range.
	

e)	Each client on receiving the event applies the event to the indicated
	agents using the sequence bundle cached locally

f)	each client views the sequence isochronously as directed by the
	sequence bundle.   Timing for the sequence is local to each client.
	Each client views the hug as a seamless sequence.  Though the exact
	starting time of the event is different for each client.   In all but  
the most
	extreme cases of lag, a user viewing the rendered event will have the  
perception that
	the event occurred simultaneously for all witnessing/participating  
agents.

Ok now for a more complex case where one or more clients have not  
cached the
bundle before the event is triggered:

a) 	permission request is granted.

b)    The hugger's hug object issues a trigger event which is relayed  
through
	the Sim to all clients within audio/visual range.

c)   	As each client receives the trigger event it may find that it must
	request the sequence bundle indicated in the event.  The appropriate
	assets are requested.

d)	The sequence is played as soon as all the assets in the bundle are  
queued.

e)	Each client views the event as a seamless sequence.   In this case  
the delay is likely to
	be longer for some clients than others.  The exact wall clock delay  
does not matter since
	the end user has no idea when the event was actually sent.   Users on  
extremely slow links
	may wish to have their client reject events that have become too  
stale,  but this loss for one client has no
	impact on any other client witnessing the event.


To avoid e) in the second example it might be desirable for a client  
to request sequence bundles
right after caching agent appearances, to avoid experiencing  
excessively stale sequence triggers.



~HS~
---------------
<previous email in thread>
To clarify my position:

Key difference is that there really is no good reason for a client on  
a fast link to wait
for the client on the slowest link before playing isochronous content  
locally.  Ever!

In 'live' performance situations there might be a need to
send trigger events for sequences.  (Trigger_sequence<UUID>) However,  
there is no
really good reason I can see to hold up the show because
some clients are on slow links.

Having clients signal readiness might be a useful *advisory*,
to indicate to a program director that enough clients are fully cached  
to start a performance.
This could be a simple broadcast to all clients requesting caching  
status
on a *particular* bundle of pre-cacheable content.   No automatic  
behavior, optional or
otherwise, is needed for feedback.  A scripted object might make such  
a broadcast and
simply report each responder's findings to the program director.

No client to client synchronization is needed
for isochronous content.   This is the key point I am making.
There's no reason for the complexity you described:
>>> * As soon as all viewers are ready (optionally, a parameter
>>> of the original request includes a timeout after which
>>> the delayed avatar is considered to have said "no"),
>>> the server informs all viewers to start playing the
>>> multi-gesture.


~HS~

On Dec 4, 2009, at 9:42 AM, Carlo Wood wrote:

> Bottomline is, there should be an option (preferably the default
> in the future) to wait a while until all needed assets are
> downloaded, and then there needds to be a way to tell the other
> clients (through the sim) that you are ready.
>
> I don't seem much difference there with your ideas :)
>
> Once all assets are ready (locally cached) then just playing
> them in order as described by the gesture info will work
> good enough: there is only one synchronization point needed:
> when to START.
>
> On Fri, Dec 04, 2009 at 09:01:51AM -0800, Han Sontse wrote:
>> I've been giving this a lot of thought recently.  It seems to me
>> that strict isochronous behavior between clients is not really
>> needed for gestures and synchronous animations (dances and hugs).
>>
>> It doesn't really matter how long it takes a client to acquire the
>> assets needed for a given interaction.  Each client is showing a
>> arbitrarily delayed rendition of the sim state anyway.
>>
>> It seems more useful to me to have a mechanism that describes a
>> isochronous sequence using arbitrary collection of animated content;
>> Audio, MIDI, animation, etc.  A sequence time-line, if you will.
>> The sequence lists the assets that will be played in the time line.
>> These are then requested/located as needed by the client.   Once all
>> are queued, the sequence is played.   From each client's perspective
>> the script plays in local real time, but without any synchronization
>> between clients.    Tighter synchronization between clients could be
>> achieved  simply by having a mechanism for clients to be notified
>> that sequenced objects are present in the local environment.  The
>> client can then choose to pre-cache these sequences subject to user
>> preference.
>>
>> As the gesture mechanism operates today in SL, some content will
>> begin playing as expected, while other content declared in the
>> sequence are still loading.  I do not know if this is as designed,
>> or it reflects some collection of bugs in the gesture handling
>> process.
>>
>> What is missing here is a robust method for notifying clients that
>> isochronous groupings of content are present in the local
>> environment.
>>
>> A more complex issue exists for situations where multiple clients
>> seek to create a live interactive performance.  The current state of
>> the internet as a whole simply cannot support hard isochronous
>> behavior.    However a performing group at a physical location might
>> cause a library of sequences to be cached in the clients, which can
>> then be triggered by a tokenized sequence embedded in a QuickTime
>> Stream.  This choreography would then appear isochronous at each
>> client individually.
>>
>> For an example of how difficult this problem is, try to watch a
>> streamed TV show with someone who is 3K miles away, while you have a
>> voice channel open so that you can comment on the show to each other
>> in real time.  SL or not this takes some insight and discipline to
>> achieve a "just like being there" experience for even for a party of
>> two!   This is even true if the content being viewed is already
>> local to both participants!   For a more typical example: try
>> listening to the same radio stream with a friend and try singing
>> along to it!
>>
>> ~HS~
>>
>>
>>
>> On Nov 10, 2009, at 12:41 PM, Carlo Wood wrote:
>>
>>> On Tue, Nov 10, 2009 at 09:11:57PM +0100, Carlo Wood wrote:
>>>> make animated hugs (or any couple animations) really work (with
>>>> effort).
>>>
>>> without*
>>>
>>>
>>> Here is an idea of how it could work:
>>>
>>> * Viewer A (or object) initiates Multi-gesture Request
>>> to N (other) viewers, this involves a list of the
>>> animations (UUID) to play for each of the avatars
>>> involved, and sounds to play etc, and the starting
>>> position in the world of each avatar.
>>> * Each viewer reacts with: Ok, will try; or Nope, I won't.
>>> If any viewer denies the request (or times out?)
>>> then the server informs all viewers that the multi-
>>> animation was aborted (and why). [ An optional flag
>>> could be to continue anyway, but without the avatar(s)
>>> that don't want to. ]
>>> * Once all avatars that want to play the multi-animation
>>> are known, the server sends them a message: "We have a go!"
>>> * All viewers that have all the needed information to
>>> continue (downloaded and cached the animations, sounds etc)
>>> send a message to the server: "ready!"
>>> * As soon as all viewers are ready (optionally, a parameter
>>> of the original request includes a timeout after which
>>> the delayed avatar is considered to have said "no"),
>>> the server informs all viewers to start playing the
>>> multi-gesture.
>>>
>>> And YAY! finally a completely synchronized couple (and
>>> multi) animation, even synced with sound!
>>>
>>> Options in case of delays (timeouts) should be:
>>> - Play as soon data available locally,
>>> - Wait till at least N, or at least those, avatars
>>> are ready.
>>> - Abort completely if any can't play in time.
>>> - If someone is delayed and the rest already plays
>>> the animation, then the viewer can decide to abort
>>> or play the animations locally anyway, as soon as
>>> they become avaiable, etc etc... All those possibilities
>>> are mere bits in already existing messages, so why not.
>>>
>>> As a side effect of this it should be possible to
>>> play a gesture involving animation and sound by ONE
>>> avatar (as are the gestures in SL now) but *synchronized*:
>>> everyone sees it at the same time, and the animations ARE
>>> synced with the sound once the start to play (unlike now,
>>> where the animation is played and 30 seconds later you
>>> hear the sound).
>>>
>>> -- 
>>> Carlo Wood <[e-mail]>
>>> _______________________________________________
>>> ogpx mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/ogpx
>>
>
> -- 
> Carlo Wood <[e-mail]>

On Fri, Dec 04, 2009 at 10:19:09AM -0800, Han Sontse wrote:
> To clarify my position:
>
> Key difference is that there really is no good reason for a client
> on a fast link to wait
> for the client on the slowest link before playing isochronous
> content locally.  Ever!

Um, not to you - but there is for me, and probably others.
No reason to dictate what is possible and what is not possible.

In my proposal it is possible to NOT wait and just play the
animation whenever you have the needed data AND it is possible
to wait until all or some clients have it too. Not all, not
forever, but configurable. Configurable is always better.

For example, if I'm dancing some romantic slow dance
then I DO want to be full synchronized with my partner,
so that right at the moment I lift him/her up I can type
something that refer to lifting him/her up.

I'd care less about some noob with a slow link at the
other edge of the dance floor - so I'd configure my
client to wait for my partner, and wait at most 4 seconds
for everyone else, and then start it.

> In 'live' performance situations there might be a need to
> send trigger events for sequences.  (Trigger_sequence<UUID>)
> However, there is no
> really good reason I can see to hold up the show because
> some clients are on slow links.
>
> Having clients signal readiness might be a useful *advisory*,
> to indicate to a program director that enough clients are fully
> cached to start a performance.

Well, clearly every viewer by itself is free to do what
they want - so it's automatically advisory in nature.

> This could be a simple broadcast to all clients requesting caching
> status
> on a *particular* bundle of pre-cacheable content.   No automatic
> behavior, optional or
> otherwise, is needed for feedback.  A scripted object might make
> such a broadcast and
> simply report each responder's findings to the program director.
>
> No client to client synchronization is needed
> for isochronous content.   This is the key point I am making.
> There's no reason for the complexity you described:
>>>> * As soon as all viewers are ready (optionally, a parameter
>>>> of the original request includes a timeout after which
>>>> the delayed avatar is considered to have said "no"),
>>>> the server informs all viewers to start playing the
>>>> multi-gesture.

Yes there is (although it's not complex imho).
The argument "we can do it simpler, there is no need for
this complexity" is exactly the reason that currently
all couple animations and all gestures with sound totally
suck.

If you "simplify" the proposed protocol you take away
possibilities, things will not be possible anymore,
things that some people might want or need.

For example, someone wants to jump in the air
shouting "Whoooohoooo!" and right after that run
for the door and bolt out.

Then that person NEEDS to know when everyone is ready,
so that they ALL see the jump and whoohoo at the same
time, otherwise some would see him run to the door
and jump when already being outside (or worse, zap
back inside, jump and then zap outside again).

I know I'm not diplomatic, but you'll have to admit
that in some cases and/or for some people there IS a
need to wait till the slower ones have the data too.

-- 
Carlo Wood <[e-mail]>