Re: [Rum] [EXT] Re: Real-time text in WebRTC is discussed in mmusic - a topic closely telated to rum

["Malloy, Jim" <[e-mail]>]

“line at a time” is a user interface issue – RUM is defining a machine interface.  How it’s presented to the user is out of scope.

--Jim


From: Rum <[e-mail]> On Behalf Of Brian Rosen
Sent: Thursday, August 29, 2019 1:15 PM
To: Kyzivat, Paul <[e-mail]>
Cc: [e-mail]
Subject: [EXT] Re: [Rum] Real-time text in WebRTC is discussed in mmusic - a topic closely telated to rum

“Line at a time” could be an implementation option.  The protocol mechanism is all streams head to a mixer and the mixer sends a single stream to each participant.

Gunnar, we agreed we were using 4103 in RUM.

Brian


On Aug 29, 2019, at 12:05 PM, Paul Kyzivat <[e-mail]<mailto:[e-mail]>> wrote:

I question how well RTT text is suited to multiparty conferences.

If you have messages on your screen from multiple parties, and many of them are updating in real-time, are you going to be able to perceive what is going on?

And while you can have a column per person for two-party and maybe 3-party conversations, that doesn't scale up. With many parties, some typing may scroll off the screen before it is complete.

Perhaps for conferences it is better to just use line-at-a-time chat. If necessary, I presume there could be gateways between RTT and chat. A RUE could have the capability to negotiate down from RTT to chat.

               Thanks,
               Paul

On 8/28/19 2:15 AM, Gunnar Hellström wrote:

Den 2019-08-27 kl. 23:15, skrev Brian Rosen:

At least by centralizing the problem at a “mixer”, Alice and Bob will see the same thing.

You don’t have the problem in Instant Messaging, because you can’t backspace or delete a sent message.  Of course if multiple people are typing simultaneously in such systems, message order will be confusing in that instant.
Right, it is a similar kind of problem that text appears in an unexpected order. There is also at least one instant messaging service that allows modification in already sent message. But I think it has limitations to only accept that in the last message sent. it is convenient anyway.


Anyway, we need to specify the mixer for RTT so it receives each of the RTT streams and produces a single composite stream for each participant.
Yes, right, and there is an effort in that direction in:
http://www.realtimetext.org/sites/default/files/Files_and_Documents/Specifications/multiparty-real-time-text-mixer-2011-04-30.pdf
It is written for conference-unaware user devices.
The goals are specified as follows:
The procedures are intended to make best efforts to present a multi-party text conversation on a terminal that has no awareness of multi-party calls. There are some obvious drawbacks, and a terminal designed with multi-party awareness will be able to present multi-party call contents in a more flexible way. Only two parties at a time will be allowed to display added text in real-time, while the other parties’ produced text will need to be stored in the multi-party server for a moment awaiting a suitable occasion to be displayed. There are also some cases of erasure that will not be performed on the target text but only indicated in another way. Even with these drawbacks, the procedure provides an opportunity to display text from more than two parties in a smooth and readable way.
---------------------------------------------------------------------------------------------------
I see such mixer procedures as a fall-back for cases without conference awareness, but want to see support for conference-aware terminals, where text from more than two parties can be presented in real-time, and the end user or app can have influence over the presentation style - e.g. select between the multiple column view and the one-column-with-labels view.  A mixer for that case would only need to assure that the receiver has the right kind of multi-party awareness and send RTT text with source information attached, and let the receiving terminal sort out the presentation. This is already possible with CSRC and CNAME when using RTP, but we lose that possibility natively when using the WebRTC data channel to transport RTT, and would need to specify a way to include the source also for that case.
------------------------------------------------------
By the way, what is your current view of how to transport RTT for RUM, now when you say that you will use WebRTC transports for media?
Regards
Gunnar



On Aug 27, 2019, at 4:43 PM, Gunnar Hellström <[e-mail]<mailto:[e-mail]> <mailto:[e-mail]>> wrote:

Den 2019-08-27 kl. 21:48, skrev Brian Rosen:

The problem of conference 4103 RTT is high on my list of work I need to get done.  So, I’m motivated to help out.
Thanks, great.

The basic problem is that we’re going to get very inconsistent UI doing it that way, because of how systems will handle backspace of one party that extends beyond responses from other parties:
(well, for me the currently most basic problem is to have a reliable way to append received text to the already presented text of the right participant. And that is getting worse in WebRTC than it was in RFC 4103. But we will sort it out.)


Alice: I waited for you
Bob: I didn’t see you
Alice: sorry

And then Alice types 12 backspaces.

What should happen?

You are right that there are a number of ways to handle the RTT UI. And just as inconsistencies are common with a message oriented UI, where messages show up in a confusing order because two users completed messages in an unexpected time order, it is possible that RTT text gets displayed in a strange order after erasure and retyping. It is better for RTT than for message oriented presentation, and user get used to it in both cases.  With the labelled style in one column you have in the example, I would recommend that first 5 backspaces erase "sorry", next backspace erases the line separator, and pulls down "I waited for you" to be shown last, as an uncompleted text. Then the next 6 backspaces erase so that only "I waited f" is displayed. When Alice adds text and end with a new line, the corrected sentence is allowed to flow up when new text is added from any participant.  That causes a bit strange order, but it is just as manageable as when text in messaging applications appear in an unexpected order so that one message seems to be a respone on something totally else than what was intended.

A sophisticated UI may mark text that is moved and modified.

We want to keep sentences or at least phrases from each participant together in a readable unit. Already that causes a design decision on where to place the completed chunk of text once the user has completed it. The start of the chunk may be older than completed text from other participants which would motivate to move it up a bit in the presentation. But the end of it is at that moment the latest text to present. I think it is best to let the finished text be presented last on the display, but let others' newer text push everything up and be displayed last.


T.140 has information on how to handle erasure:

-------------------From T.140---------------------------

8.2 Erase last character
Purpose: Erase the last character sent from the display at the receiving end.
Code: BS: 0008.
Procedure: On the receiving end: Move the insertion point to the last character and erase it.
Combined characters are erased as a unit, with one BS erasing the whole character even if it is
combined from more than one component.
Control sequences (like CR LF) are erased in one operation.
NOTE – The same action shall be taken on the local display.

------------------------------------------------------------

  /Gunnar



Brian


On Aug 27, 2019, at 9:52 AM, Gunnar Hellström <[e-mail]<mailto:[e-mail]> <mailto:[e-mail]>> wrote:

Hi,

A topic is currently discussed in mmusic that is closely related to rum. it is WebRTC transport of real-time text.

The draft is draft-holmberg-mmusic-t140-usage-data-channel .

A good point to start reading could be:

https://mailarchive.ietf.org/arch/browse/mmusic/?gbt=1&q=draft-holmberg-mmusic-t140-usage-data-channel

Please check if the current state of the discussion suits rum!

The only issue that seems to be remaining is how to transport RTT data to and from a conference server that combines all traffic per media in a meeting in one data stream. That is not very elegantly specified for RFC 4103 transport of RTT in RTP either, so we might want to do a rapid action together to solve the multi-party RTT MCU case in a general and consistent way.

Regards

Gunnar

--
-----------------------------------------
Gunnar Hellström
Omnitor
[e-mail]<mailto:[e-mail]>
[phone]

--
-----------------------------------------
Gunnar Hellström
Omnitor
[e-mail]<mailto:[e-mail]> <mailto:[e-mail]>
[phone]

--
-----------------------------------------
Gunnar Hellström
Omnitor
[e-mail]<mailto:[e-mail]>
[phone]

--
Rum mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/rum