Re: [Rum] [EXT] Re: Real-time text in WebRTC is discussed in mmusic

All,

  $0.02 on RTT:

RTT in General

--------------

ISTM there are a few options for RTT in RUM, assuming the WebRTC media framework is adopted (as seems likely):

1.  Use existing browser support for SCTP as recommended by the some of the WebRTC IETF WG participants. This was discussed as part of the IVC WG and would require and SCTP data channel to be negotiated and to encapsulate the RTP (4103) which then in turn encapsulate T.140 RTT. This solution would require gateways in the VRS provider networks, RTP/RTCP userspace (Javascript) layers in the browser and a T.140 RTT layer/UI also written in userspace in the browser.

2.  Wait for WebRTC to support lower level data channels. This work is being discussed in the WebRTC community, but might be a case of “not holding our breath”. A lower level data channel would allow RTP/RTCP then T.140 to be negotiated in the SDP without the need for SCTP and associated gateway needs.

3.  As a stop-gap and simplification, support SIP MESSAGE as already provided for by the browsers and many endpoints, including WebRTC endpoints

We should not forget that RTCP must be implemented along with RTP.

Conferencing and RTT

--------------------

RTT in a conference is not currently defined and does not work using currently defined standards (I must admit to being being on mmusic so perhaps there are solutions being recommended there). Discussions on this have been ongoing for nearly 10 years (we didn’t solve it then Gunnar and I think the problems are still the same). For RTT to be useful in a conference there needs to be a few problems solved.

1.  Multi-stream support: Brian, whilst you’re right that most conference systems are point-to-point for signalling they are more typically point-to-multi-point for media. The conference bridge usually handles the multiplexing of the streams, but they are usually now multiple streams. RTT conferencing needs to have the same support. We should assume conferencing is supported by way of SFU not MCU – that’s how WebRTC handles conferencing and we should assume the same for RTT.

2.  Participant identification: streams from the conference bridge (MCU/SFU) must be tagged with the sender so the UI can signal to the user who sent that text (and reconstruct the stream correctly).

3.  UI adaptions: Modern WebRTC based multi-party conferences have adapted UIs to display multiple video streams. UI’s are adapted to show the video participants however the user wants them to be displayed (single full screen, tiled, voice activated etc). We should expect no less from RTT conference enabled endpoints. It should be a user decision (if their UI supports it) as to whether they see line-by-line or RTT rendering of the text conversation.

4.  Control of how many simultaneous participants can text and how that is presented to the user needs to be explored. Multiple insertion points in the UI is one approach (though rapidly breaks down as the number of users increases). Line-by-line is another. Facilitated GA is another (the Go Ahead concept has been used for a long time in text conversations and may be appropriate in large RTT conferences to control overload).

And a final FWIW: I think the utility of RTT in a conference quickly breaks down as the number of participants increases. It is completely unacceptable to have a single RTT stream presented to the user and an unmodified UI. You might as well revert to SIP MESSAGE and something like XMPP, and that could also be an option for either the short term or for endpoints that do not support multi-stream RTT.

John

Brian said: we agreed we were using 4103 in RUM.

Good toknow. But is there not a hope to be able to even use browser

technology for the implementation. The browsers only support RTP for

audio and video. Can you make RTP-based RTT work in that environment and

encrypted with the security method specified for WebRTC?

See a bit more below:

Den 2019-08-29 kl. 21:59, skrev Malloy, Jim:

> “line at a time” is a user interface issue – RUM is defining a machine

> interface.  How it’s presented to the user is out of scope.

>

Well, the control and coding must be made so that it is possible to make

a good presentation. Transmission only line-by-line would not suit the

intention of RTT. It is hard to make a good RTT presentation with a

client that is only expecting text from one other participant and a

mixer that has the task to send text from many sources combined in one

stream. The best result is usable but not nice. And it will be aimed at

just one way to present RTT. No freedom to plan the presentation

according to user preferences.

A receiver may, if there is a desire for such functionality, store

received text until a received message is complete. I do not think that

it is to prefer in any situation.

Regards

Gunnar

> --Jim

>

> *From:* Rum <rum-bounces@ietf.org><mailto:rum-bounces@ietf.org&gt>; *On Behalf Of * Brian Rosen

> *Sent:* Thursday, August 29, 2019 1:15 PM

> *To:* Kyzivat, Paul <pkyzivat@alum.mit.edu><mailto:pkyzivat@alum.mit.edu&gt>;

> *Cc:* rum@ietf.org<mailto:rum@ietf.org>

> *Subject:* [EXT] Re: [Rum] Real-time text in WebRTC is discussed in

> mmusic - a topic closely telated to rum

>

> “Line at a time” could be an implementation option.  The protocol

> mechanism is all streams head to a mixer and the mixer sends a single

> stream to each participant.

>

> Gunnar, we agreed we were using 4103 in RUM.

>

> Brian

>

>

>

>     On Aug 29, 2019, at 12:05 PM, Paul Kyzivat <pkyzivat@alum.mit.edu<mailto:pkyzivat@alum.mit.edu>

>     <mailto:pkyzivat@alum.mit.edu>> wrote:

>

>     I question how well RTT text is suited to multiparty conferences.

>

>     If you have messages on your screen from multiple parties, and

>     many of them are updating in real-time, are you going to be able

>     to perceive what is going on?

>

>     And while you can have a column per person for two-party and maybe

>     3-party conversations, that doesn't scale up. With many parties,

>     some typing may scroll off the screen before it is complete.

>

>     Perhaps for conferences it is better to just use line-at-a-time

>     chat. If necessary, I presume there could be gateways between RTT

>     and chat. A RUE could have the capability to negotiate down from

>     RTT to chat.

>

>     Thanks,

>     Paul

>

>     On 8/28/19 2:15 AM, Gunnar Hellström wrote:

>

>         Den 2019-08-27 kl. 23:15, skrev Brian Rosen:

>

>             At least by centralizing the problem at a “mixer”, Alice

>             and Bob will see the same thing.

>

>             You don’t have the problem in Instant Messaging, because

>             you can’t backspace or delete a sent message.  Of course

>             if multiple people are typing simultaneously in such

>             systems, message order will be confusing in that instant.

>

>         Right, it is a similar kind of problem that text appears in an

>         unexpected order. There is also at least one instant messaging

>         service that allows modification in already sent message. But

>         I think it has limitations to only accept that in the last

>         message sent. it is convenient anyway.

>

>

>             Anyway, we need to specify the mixer for RTT so it

>             receives each of the RTT streams and produces a single

>             composite stream for each participant.

>

>         Yes, right, and there is an effort in that direction in:

>         http://www.realtimetext.org/sites/default/files/Files_and_Documents/Specifications/multiparty-real-time-text-mixer-2011-04-30.pdf

>         It is written for conference-unaware user devices.

>         The goals are specified as follows:

>         The procedures are intended to make best efforts to present a

>         multi-party text conversation on a terminal that has no

>         awareness of multi-party calls. There are some obvious

>         drawbacks, and a terminal designed with multi-party awareness

>         will be able to present multi-party call contents in a more

>         flexible way. Only two parties at a time will be allowed to

>         display added text in real-time, while the other parties’

>         produced text will need to be stored in the multi-party server

>         for a moment awaiting a suitable occasion to be displayed.

>         There are also some cases of erasure that will not be

>         performed on the target text but only indicated in another

>         way. Even with these drawbacks, the procedure provides an

>         opportunity to display text from more than two parties in a

>         smooth and readable way.

>         ---------------------------------------------------------------------------------------------------

>         I see such mixer procedures as a fall-back for cases without

>         conference awareness, but want to see support for

>         conference-aware terminals, where text from more than two

>         parties can be presented in real-time, and the end user or app

>         can have influence over the presentation style - e.g. select

>         between the multiple column view and the

>         one-column-with-labels view.  A mixer for that case would only

>         need to assure that the receiver has the right kind of

>         multi-party awareness and send RTT text with source

>         information attached, and let the receiving terminal sort out

>         the presentation. This is already possible with CSRC and CNAME

>         when using RTP, but we lose that possibility natively when

>         using the WebRTC data channel to transport RTT, and would need

>         to specify a way to include the source also for that case.

>         ------------------------------------------------------

>         By the way, what is your current view of how to transport RTT

>         for RUM, now when you say that you will use WebRTC transports

>         for media?

>         Regards

>         Gunnar

>

>

>

>                 On Aug 27, 2019, at 4:43 PM, Gunnar Hellström

>                 <gunnar.hellstrom@omnitor.se<mailto:gunnar.hellstrom@omnitor.se>

>                 <mailto:gunnar.hellstrom@omnitor.se><mailto:gunnar.hellstrom@omnitor.se>>

>                 wrote:

>

>                 Den 2019-08-27 kl. 21:48, skrev Brian Rosen:

>

>                     The problem of conference 4103 RTT is high on my

>                     list of work I need to get done.  So, I’m

>                     motivated to help out.

>

>                 Thanks, great.

>

>                     The basic problem is that we’re going to get very

>                     inconsistent UI doing it that way, because of how

>                     systems will handle backspace of one party that

>                     extends beyond responses from other parties:

>

>                 (well, for me the currently most basic problem is to

>                 have a reliable way to append received text to the

>                 already presented text of the right participant. And

>                 that is getting worse in WebRTC than it was in RFC

>                 4103. But we will sort it out.)

>

>

>                     Alice: I waited for you

>                     Bob: I didn’t see you

>                     Alice: sorry

>

>                     And then Alice types 12 backspaces.

>

>                     What should happen?

>

>

>                 You are right that there are a number of ways to

>                 handle the RTT UI. And just as inconsistencies are

>                 common with a message oriented UI, where messages show

>                 up in a confusing order because two users completed

>                 messages in an unexpected time order, it is possible

>                 that RTT text gets displayed in a strange order after

>                 erasure and retyping. It is better for RTT than for

>                 message oriented presentation, and user get used to it

>                 in both cases.  With the labelled style in one column

>                 you have in the example, I would recommend that first

>                 5 backspaces erase "sorry", next backspace erases the

>                 line separator, and pulls down "I waited for you" to

>                 be shown last, as an uncompleted text. Then the next 6

>                 backspaces erase so that only "I waited f" is

>                 displayed. When Alice adds text and end with a new

>                 line, the corrected sentence is allowed to flow up

>                 when new text is added from any participant.  That

>                 causes a bit strange order, but it is just as

>                 manageable as when text in messaging applications

>                 appear in an unexpected order so that one message

>                 seems to be a respone on something totally else than

>                 what was intended.

>

>                 A sophisticated UI may mark text that is moved and

>                 modified.

>

>                 We want to keep sentences or at least phrases from

>                 each participant together in a readable unit. Already

>                 that causes a design decision on where to place the

>                 completed chunk of text once the user has completed

>                 it. The start of the chunk may be older than completed

>                 text from other participants which would motivate to

>                 move it up a bit in the presentation. But the end of

>                 it is at that moment the latest text to present. I

>                 think it is best to let the finished text be presented

>                 last on the display, but let others' newer text push

>                 everything up and be displayed last.

>

>

>                 T.140 has information on how to handle erasure:

>

>                 -------------------From T.140---------------------------

>

>                 8.2 Erase last character

>                 Purpose: Erase the last character sent from the

>                 display at the receiving end.

>                 Code: BS: 0008.

>                 Procedure: On the receiving end: Move the insertion

>                 point to the last character and erase it.

>                 Combined characters are erased as a unit, with one BS

>                 erasing the whole character even if it is

>                 combined from more than one component.

>                 Control sequences (like CR LF) are erased in one

>                 operation.

>                 NOTE – The same action shall be taken on the local

>                 display.

>

>                 ------------------------------------------------------------

>

>                   /Gunnar

>

>

>

>                     Brian

>

>

>                         On Aug 27, 2019, at 9:52 AM, Gunnar Hellström

>                         <gunnar.hellstrom@omnitor.se<mailto:gunnar.hellstrom@omnitor.se>

>                         <mailto:gunnar.hellstrom@omnitor.se><mailto:gunnar.hellstrom@omnitor.se>>

>                         wrote:

>

>                         Hi,

>

>                         A topic is currently discussed in mmusic that

>                         is closely related to rum. it is WebRTC

>                         transport of real-time text.

>

>                         The draft is

>                         draft-holmberg-mmusic-t140-usage-data-channel .

>

>                         A good point to start reading could be:

>

>                         https://mailarchive.ietf.org/arch/browse/mmusic/?gbt=1&q=draft-holmberg-mmusic-t140-usage-data-channel

>

>                         Please check if the current state of the

>                         discussion suits rum!

>

>                         The only issue that seems to be remaining is

>                         how to transport RTT data to and from a

>                         conference server that combines all traffic

>                         per media in a meeting in one data stream.

>                         That is not very elegantly specified for RFC

>                         4103 transport of RTT in RTP either, so we

>                         might want to do a rapid action together to

>                         solve the multi-party RTT MCU case in a

>                         general and consistent way.

>

>                         Regards

>

>                         Gunnar

>

>                         --

>                         -----------------------------------------

>                         Gunnar Hellström

>                         Omnitor

>                         gunnar.hellstrom@omnitor.se<mailto:gunnar.hellstrom@omnitor.se>

>                         <mailto:gunnar.hellstrom@omnitor.se>

>                         +46 708 204 288

>

>                 --

>                 -----------------------------------------

>                 Gunnar Hellström

>                 Omnitor

>                 gunnar.hellstrom@omnitor.se<mailto:gunnar.hellstrom@omnitor.se>

>                 <mailto:gunnar.hellstrom@omnitor.se><mailto:gunnar.hellstrom@omnitor.se>

>                 +46 708 204 288

>

>         --

>         -----------------------------------------

>         Gunnar Hellström

>         Omnitor

>         gunnar.hellstrom@omnitor.se<mailto:gunnar.hellstrom@omnitor.se> <mailto:gunnar.hellstrom@omnitor.se>

>         +46 708 204 288

>

>

>     --

>     Rum mailing list

>     Rum@ietf.org<mailto:Rum@ietf.org> <mailto:Rum@ietf.org>

>     https://www.ietf.org/mailman/listinfo/rum

>

>

--

-----------------------------------------

Gunnar Hellström

Omnitor

gunnar.hellstrom@omnitor.se<mailto:gunnar.hellstrom@omnitor.se>

+46 708 204 288

  *   [Rum] Real-time text in WebRTC is discussed in ...<https://mailarchive.ietf.org/arch/msg/rum/hzX30Lo4DY4J3RGbtgxoC5wpE1s>  Gunnar Hellström
  *   Re: [Rum] Real-time text in WebRTC is discussed...<https://mailarchive.ietf.org/arch/msg/rum/Lk90w0qStmiAmHoIw_mWFTOE3pY>  Brian Rosen
  *   Re: [Rum] Real-time text in WebRTC is discussed...<https://mailarchive.ietf.org/arch/msg/rum/opt7uhe3jcCAGEXipXCefVN97PE>  Gunnar Hellström
  *   Re: [Rum] Real-time text in WebRTC is discussed...<https://mailarchive.ietf.org/arch/msg/rum/uF83bn9GmEkR2FzE4rvH-DiJn1I>  Brian Rosen
  *   Re: [Rum] Real-time text in WebRTC is discussed...<https://mailarchive.ietf.org/arch/msg/rum/6402q9HzKBNqvkwKajSlo4EYmMU>  Gunnar Hellström
  *   Re: [Rum] Real-time text in WebRTC is discussed...<https://mailarchive.ietf.org/arch/msg/rum/YG02TyKon1tRtctZ1Bwe0BnI7Cw>  Paul Kyzivat
  *   Re: [Rum] [EXT] Re: Real-time text in WebRTC is...<https://mailarchive.ietf.org/arch/msg/rum/UDz0JCzyqQ0ut8RJzWF54KjMUXY>  Malloy, Jim
  *   Re: [Rum] [EXT] Re: Real-time text in WebRTC is...<https://mailarchive.ietf.org/arch/msg/rum/WWekfAbXC81GFmOdUld1YOvV450>  Paul Kyzivat
  *   Re: [Rum] Real-time text in WebRTC is discussed...<https://mailarchive.ietf.org/arch/msg/rum/8N5sWJvqB-J4XgC5xqMCV2nL0gc>  Brian Rosen
  *   Re: [Rum] [EXT] Re: Real-time text in WebRTC is...<https://mailarchive.ietf.org/arch/msg/rum/HoD8Ni1r0f8S11H2K4hp6Ss5NB0>  Brian Rosen
  *   Re: [Rum] [EXT] Re: Real-time text in WebRTC is...<https://mailarchive.ietf.org/arch/msg/rum/eI-dX4HrMUur324GEt-CFmpv4mY>  Paul Kyzivat
  *   Re: [Rum] [EXT] Re: Real-time text in WebRTC is...<https://mailarchive.ietf.org/arch/msg/rum/AaoKog-HhtBUS_a-BKrB5BpwiF4>  Brian Rosen
  *   Re: [Rum] [EXT] Re: Real-time text in WebRTC is...<https://mailarchive.ietf.org/arch/msg/rum/1JojOtitB-FlcmCHQd9LMl4PWM4>  Paul Kyzivat
  *   Re: [Rum] [EXT] Re: Real-time text in WebRTC is...<https://mailarchive.ietf.org/arch/msg/rum/2NDD2kMSGngnh6GyoLD9Th_UYrM>  Brian Rosen
  *   Re: [Rum] [EXT] Re: Real-time text in WebRTC is...<https://mailarchive.ietf.org/arch/msg/rum/UZDyKrfOw-bjkbqjN8E3MHXrseo>  Malloy, Jim
  *   Re: [Rum] [EXT] Re: Real-time text in WebRTC is...<https://mailarchive.ietf.org/arch/msg/rum/avASfCcxCAY18gOFsTGE3P5oTOs>  Gunnar Hellström

--
John Martin
Email: John.Martin@Purple.us<mailto:John.Martin@Purple.us>

Re: [Rum] [EXT] Re: Real-time text in WebRTC is discussed in mmusic - a topic closely telated to rum