Re: [Rum] [EXT] Re: Real-time text in WebRTC is discussed in mmusic - a topic closely telated to rum

[Paul Kyzivat <[e-mail]>]

John,

On 8/30/19 7:32 AM, John Martin wrote:
> All,
> 
>   $0.02 on RTT:

Thanks for joining in! I don't have much knowledge of (and no experience 
using) RTT so I will mostly watch from the sidelines as long as others 
join in.

	Thanks,
	Paul

(BTW, I don't know how you formatted your message, but in my mail reader 
(Thunderbird) none of your lines wrapped, requiring horizontal scrolling 
to read them.)

> RTT in General
> 
> --------------
> 
> ISTM there are a few options for RTT in RUM, assuming the WebRTC media 
> framework is adopted (as seems likely):
> 
> 1.Use existing browser support for SCTP as recommended by the some of 
> the WebRTC IETF WG participants. This was discussed as part of the IVC 
> WG and would require and SCTP data channel to be negotiated and to 
> encapsulate the RTP (4103) which then in turn encapsulate T.140 RTT. 
> This solution would require gateways in the VRS provider networks, 
> RTP/RTCP userspace (Javascript) layers in the browser and a T.140 RTT 
> layer/UI also written in userspace in the browser.
> 
> 2.Wait for WebRTC to support lower level data channels. This work is 
> being discussed in the WebRTC community, but might be a case of “not 
> holding our breath”. A lower level data channel would allow RTP/RTCP 
> then T.140 to be negotiated in the SDP without the need for SCTP and 
> associated gateway needs.
> 
> 3.As a stop-gap and simplification, support SIP MESSAGE as already 
> provided for by the browsers and many endpoints, including WebRTC endpoints
> 
> We should not forget that RTCP must be implemented along with RTP.
> 
> Conferencing and RTT
> 
> --------------------
> 
> RTT in a conference is not currently defined and does not work using 
> currently defined standards (I must admit to being being on mmusic so 
> perhaps there are solutions being recommended there). Discussions on 
> this have been ongoing for nearly 10 years (we didn’t solve it then 
> Gunnar and I think the problems are still the same). For RTT to be 
> useful in a conference there needs to be a few problems solved.
> 
> 1.Multi-stream support: Brian, whilst you’re right that most conference 
> systems are point-to-point for signalling they are more typically 
> point-to-multi-point for media. The conference bridge usually handles 
> the multiplexing of the streams, but they are usually now multiple 
> streams. RTT conferencing needs to have the same support. We should 
> assume conferencing is supported by way of SFU not MCU – that’s how 
> WebRTC handles conferencing and we should assume the same for RTT.
> 
> 2.Participant identification: streams from the conference bridge 
> (MCU/SFU) must be tagged with the sender so the UI can signal to the 
> user who sent that text (and reconstruct the stream correctly).
> 
> 3.UI adaptions: Modern WebRTC based multi-party conferences have adapted 
> UIs to display multiple video streams. UI’s are adapted to show the 
> video participants however the user wants them to be displayed (single 
> full screen, tiled, voice activated etc). We should expect no less from 
> RTT conference enabled endpoints. It should be a user decision (if their 
> UI supports it) as to whether they see line-by-line or RTT rendering of 
> the text conversation.
> 
> 4.Control of how many simultaneous participants can text and how that is 
> presented to the user needs to be explored. Multiple insertion points in 
> the UI is one approach (though rapidly breaks down as the number of 
> users increases). Line-by-line is another. Facilitated GA is another 
> (the Go Ahead concept has been used for a long time in text 
> conversations and may be appropriate in large RTT conferences to control 
> overload).
> 
> And a final FWIW: I think the utility of RTT in a conference quickly 
> breaks down as the number of participants increases. It is completely 
> unacceptable to have a single RTT stream presented to the user and an 
> unmodified UI. You might as well revert to SIP MESSAGE and something 
> like XMPP, and that could also be an option for either the short term or 
> for endpoints that do not support multi-stream RTT.
> 
> John
> 
> Brian said: we agreed we were using 4103 in RUM.
> 
> Good toknow. But is there not a hope to be able to even use browser
> 
> technology for the implementation. The browsers only support RTP for
> 
> audio and video. Can you make RTP-based RTT work in that environment and
> 
> encrypted with the security method specified for WebRTC?
> 
> See a bit more below:
> 
> Den 2019-08-29 kl. 21:59, skrev Malloy, Jim:
> 
>> “line at a time” is a user interface issue – RUM is defining a machine 
> 
>> interface.  How it’s presented to the user is out of scope.
> 
>>
> 
> Well, the control and coding must be made so that it is possible to make
> 
> a good presentation. Transmission only line-by-line would not suit the
> 
> intention of RTT. It is hard to make a good RTT presentation with a
> 
> client that is only expecting text from one other participant and a
> 
> mixer that has the task to send text from many sources combined in one
> 
> stream. The best result is usable but not nice. And it will be aimed at
> 
> just one way to present RTT. No freedom to plan the presentation
> 
> according to user preferences.
> 
> A receiver may, if there is a desire for such functionality, store
> 
> received text until a received message is complete. I do not think that
> 
> it is to prefer in any situation.
> 
> Regards
> 
> Gunnar
> 
>> --Jim
> 
>>
> 
>> *From:* Rum <[e-mail]> <mailto:[e-mail]&gt>; *On Behalf Of * 
> Brian Rosen
> 
>> *Sent:* Thursday, August 29, 2019 1:15 PM
> 
>> *To:* Kyzivat, Paul <[e-mail]> <mailto:[e-mail]&gt>;
> 
>> *Cc:* [e-mail] <mailto:[e-mail]>
> 
>> *Subject:* [EXT] Re: [Rum] Real-time text in WebRTC is discussed in 
> 
>> mmusic - a topic closely telated to rum
> 
>>
> 
>> “Line at a time” could be an implementation option.  The protocol 
> 
>> mechanism is all streams head to a mixer and the mixer sends a single 
> 
>> stream to each participant.
> 
>>
> 
>> Gunnar, we agreed we were using 4103 in RUM.
> 
>>
> 
>> Brian
> 
>>
> 
>>
> 
>>
> 
>>     On Aug 29, 2019, at 12:05 PM, Paul Kyzivat <[e-mail] <mailto:[e-mail]>
> 
>>     <mailto:[e-mail]>> wrote:
> 
>>
> 
>>     I question how well RTT text is suited to multiparty conferences.
> 
>>
> 
>>     If you have messages on your screen from multiple parties, and
> 
>>     many of them are updating in real-time, are you going to be able
> 
>>     to perceive what is going on?
> 
>>
> 
>>     And while you can have a column per person for two-party and maybe
> 
>>     3-party conversations, that doesn't scale up. With many parties,
> 
>>     some typing may scroll off the screen before it is complete.
> 
>>
> 
>>     Perhaps for conferences it is better to just use line-at-a-time
> 
>>     chat. If necessary, I presume there could be gateways between RTT
> 
>>     and chat. A RUE could have the capability to negotiate down from
> 
>>     RTT to chat.
> 
>>
> 
>>     Thanks,
> 
>>     Paul
> 
>>
> 
>>     On 8/28/19 2:15 AM, Gunnar Hellström wrote:
> 
>>
> 
>>         Den 2019-08-27 kl. 23:15, skrev Brian Rosen:
> 
>>
> 
>>             At least by centralizing the problem at a “mixer”, Alice
> 
>>             and Bob will see the same thing.
> 
>>
> 
>>             You don’t have the problem in Instant Messaging, because
> 
>>             you can’t backspace or delete a sent message.  Of course
> 
>>             if multiple people are typing simultaneously in such
> 
>>             systems, message order will be confusing in that instant.
> 
>>
> 
>>         Right, it is a similar kind of problem that text appears in an
> 
>>         unexpected order. There is also at least one instant messaging
> 
>>         service that allows modification in already sent message. But
> 
>>         I think it has limitations to only accept that in the last
> 
>>         message sent. it is convenient anyway.
> 
>>
> 
>>
> 
>>             Anyway, we need to specify the mixer for RTT so it
> 
>>             receives each of the RTT streams and produces a single
> 
>>             composite stream for each participant.
> 
>>
> 
>>         Yes, right, and there is an effort in that direction in:
> 
>>         http://www.realtimetext.org/sites/default/files/Files_and_Documents/Specifications/multiparty-real-time-text-mixer-2011-04-30.pdf
> 
>>         It is written for conference-unaware user devices.
> 
>>         The goals are specified as follows:
> 
>>         The procedures are intended to make best efforts to present a
> 
>>         multi-party text conversation on a terminal that has no
> 
>>         awareness of multi-party calls. There are some obvious
> 
>>         drawbacks, and a terminal designed with multi-party awareness
> 
>>         will be able to present multi-party call contents in a more
> 
>>         flexible way. Only two parties at a time will be allowed to
> 
>>         display added text in real-time, while the other parties’
> 
>>         produced text will need to be stored in the multi-party server
> 
>>         for a moment awaiting a suitable occasion to be displayed.
> 
>>         There are also some cases of erasure that will not be
> 
>>         performed on the target text but only indicated in another
> 
>>         way. Even with these drawbacks, the procedure provides an
> 
>>         opportunity to display text from more than two parties in a
> 
>>         smooth and readable way.
> 
>>         ---------------------------------------------------------------------------------------------------
> 
>>         I see such mixer procedures as a fall-back for cases without
> 
>>         conference awareness, but want to see support for
> 
>>         conference-aware terminals, where text from more than two
> 
>>         parties can be presented in real-time, and the end user or app
> 
>>         can have influence over the presentation style - e.g. select
> 
>>         between the multiple column view and the
> 
>>         one-column-with-labels view.  A mixer for that case would only
> 
>>         need to assure that the receiver has the right kind of
> 
>>         multi-party awareness and send RTT text with source
> 
>>         information attached, and let the receiving terminal sort out
> 
>>         the presentation. This is already possible with CSRC and CNAME
> 
>>         when using RTP, but we lose that possibility natively when
> 
>>         using the WebRTC data channel to transport RTT, and would need
> 
>>         to specify a way to include the source also for that case.
> 
>>         ------------------------------------------------------
> 
>>         By the way, what is your current view of how to transport RTT
> 
>>         for RUM, now when you say that you will use WebRTC transports
> 
>>         for media?
> 
>>         Regards
> 
>>         Gunnar
> 
>>
> 
>>
> 
>>
> 
>>                 On Aug 27, 2019, at 4:43 PM, Gunnar Hellström
> 
>>                 <[e-mail] <mailto:[e-mail]>
> 
>>                 <mailto:[e-mail]><mailto:[e-mail]>>
> 
>>                 wrote:
> 
>>
> 
>>                 Den 2019-08-27 kl. 21:48, skrev Brian Rosen:
> 
>>
> 
>>                     The problem of conference 4103 RTT is high on my
> 
>>                     list of work I need to get done.  So, I’m
> 
>>                     motivated to help out.
> 
>>
> 
>>                 Thanks, great.
> 
>>
> 
>>                     The basic problem is that we’re going to get very
> 
>>                     inconsistent UI doing it that way, because of how
> 
>>                     systems will handle backspace of one party that
> 
>>                     extends beyond responses from other parties:
> 
>>
> 
>>                 (well, for me the currently most basic problem is to
> 
>>                 have a reliable way to append received text to the
> 
>>                 already presented text of the right participant. And
> 
>>                 that is getting worse in WebRTC than it was in RFC
> 
>>                 4103. But we will sort it out.)
> 
>>
> 
>>
> 
>>                     Alice: I waited for you
> 
>>                     Bob: I didn’t see you
> 
>>                     Alice: sorry
> 
>>
> 
>>                     And then Alice types 12 backspaces.
> 
>>
> 
>>                     What should happen?
> 
>>
> 
>>
> 
>>                 You are right that there are a number of ways to
> 
>>                 handle the RTT UI. And just as inconsistencies are
> 
>>                 common with a message oriented UI, where messages show
> 
>>                 up in a confusing order because two users completed
> 
>>                 messages in an unexpected time order, it is possible
> 
>>                 that RTT text gets displayed in a strange order after
> 
>>                 erasure and retyping. It is better for RTT than for
> 
>>                 message oriented presentation, and user get used to it
> 
>>                 in both cases.  With the labelled style in one column
> 
>>                 you have in the example, I would recommend that first
> 
>>                 5 backspaces erase "sorry", next backspace erases the
> 
>>                 line separator, and pulls down "I waited for you" to
> 
>>                 be shown last, as an uncompleted text. Then the next 6
> 
>>                 backspaces erase so that only "I waited f" is
> 
>>                 displayed. When Alice adds text and end with a new
> 
>>                 line, the corrected sentence is allowed to flow up
> 
>>                 when new text is added from any participant.  That
> 
>>                 causes a bit strange order, but it is just as
> 
>>                 manageable as when text in messaging applications
> 
>>                 appear in an unexpected order so that one message
> 
>>                 seems to be a respone on something totally else than
> 
>>                 what was intended.
> 
>>
> 
>>                 A sophisticated UI may mark text that is moved and
> 
>>                 modified.
> 
>>
> 
>>                 We want to keep sentences or at least phrases from
> 
>>                 each participant together in a readable unit. Already
> 
>>                 that causes a design decision on where to place the
> 
>>                 completed chunk of text once the user has completed
> 
>>                 it. The start of the chunk may be older than completed
> 
>>                 text from other participants which would motivate to
> 
>>                 move it up a bit in the presentation. But the end of
> 
>>                 it is at that moment the latest text to present. I
> 
>>                 think it is best to let the finished text be presented
> 
>>                 last on the display, but let others' newer text push
> 
>>                 everything up and be displayed last.
> 
>>
> 
>>
> 
>>                 T.140 has information on how to handle erasure:
> 
>>
> 
>>                 -------------------From T.140---------------------------
> 
>>
> 
>>                 8.2 Erase last character
> 
>>                 Purpose: Erase the last character sent from the
> 
>>                 display at the receiving end.
> 
>>                 Code: BS: 0008.
> 
>>                 Procedure: On the receiving end: Move the insertion
> 
>>                 point to the last character and erase it.
> 
>>                 Combined characters are erased as a unit, with one BS
> 
>>                 erasing the whole character even if it is
> 
>>                 combined from more than one component.
> 
>>                 Control sequences (like CR LF) are erased in one
> 
>>                 operation.
> 
>>                 NOTE – The same action shall be taken on the local
> 
>>                 display.
> 
>>
> 
>>                 ------------------------------------------------------------
> 
>>
> 
>>                   /Gunnar
> 
>>
> 
>>
> 
>>
> 
>>                     Brian
> 
>>
> 
>>
> 
>>                         On Aug 27, 2019, at 9:52 AM, Gunnar Hellström
> 
>>                         <[e-mail] <mailto:[e-mail]>
> 
>>                         <mailto:[e-mail]><mailto:[e-mail]>>
> 
>>                         wrote:
> 
>>
> 
>>                         Hi,
> 
>>
> 
>>                         A topic is currently discussed in mmusic that
> 
>>                         is closely related to rum. it is WebRTC
> 
>>                         transport of real-time text.
> 
>>
> 
>>                         The draft is
> 
>>                         draft-holmberg-mmusic-t140-usage-data-channel .
> 
>>
> 
>>                         A good point to start reading could be:
> 
>>
> 
>>                         https://mailarchive.ietf.org/arch/browse/mmusic/?gbt=1&q=draft-holmberg-mmusic-t140-usage-data-channel
> 
>>
> 
>>                         Please check if the current state of the
> 
>>                         discussion suits rum!
> 
>>
> 
>>                         The only issue that seems to be remaining is
> 
>>                         how to transport RTT data to and from a
> 
>>                         conference server that combines all traffic
> 
>>                         per media in a meeting in one data stream.
> 
>>                         That is not very elegantly specified for RFC
> 
>>                         4103 transport of RTT in RTP either, so we
> 
>>                         might want to do a rapid action together to
> 
>>                         solve the multi-party RTT MCU case in a
> 
>>                         general and consistent way.
> 
>>
> 
>>                         Regards
> 
>>
> 
>>                         Gunnar
> 
>>
> 
>>                         --
> 
>>                         -----------------------------------------
> 
>>                         Gunnar Hellström
> 
>>                         Omnitor
> 
>>                         [e-mail] <mailto:[e-mail]>
> 
>>                         <mailto:[e-mail]>
> 
>>                         [phone]
> 
>>
> 
>>                 --
> 
>>                 -----------------------------------------
> 
>>                 Gunnar Hellström
> 
>>                 Omnitor
> 
>>                 [e-mail] <mailto:[e-mail]>
> 
>>                 <mailto:[e-mail]><mailto:[e-mail]>
> 
>>                 [phone]
> 
>>
> 
>>         --
> 
>>         -----------------------------------------
> 
>>         Gunnar Hellström
> 
>>         Omnitor
> 
>>         [e-mail] <mailto:[e-mail]> 
> <mailto:[e-mail]>
> 
>>         [phone]
> 
>>
> 
>>
> 
>>     --
> 
>>     Rum mailing list
> 
>>     [e-mail] <mailto:[e-mail]> <mailto:[e-mail]>
> 
>>     https://www.ietf.org/mailman/listinfo/rum
> 
>>
> 
>>
> 
> -- 
> 
> -----------------------------------------
> 
> Gunnar Hellström
> 
> Omnitor
> 
> [e-mail] <mailto:[e-mail]>
> 
> [phone]
> 
>   * [Rum] Real-time text in WebRTC is discussed in ...
>     <https://mailarchive.ietf.org/arch/msg/rum/hzX30Lo4DY4J3RGbtgxoC5wpE1s>  Gunnar
>     Hellström
>   * Re: [Rum] Real-time text in WebRTC is discussed...
>     <https://mailarchive.ietf.org/arch/msg/rum/Lk90w0qStmiAmHoIw_mWFTOE3pY>  Brian
>     Rosen
>   * Re: [Rum] Real-time text in WebRTC is discussed...
>     <https://mailarchive.ietf.org/arch/msg/rum/opt7uhe3jcCAGEXipXCefVN97PE>  Gunnar
>     Hellström
>   * Re: [Rum] Real-time text in WebRTC is discussed...
>     <https://mailarchive.ietf.org/arch/msg/rum/uF83bn9GmEkR2FzE4rvH-DiJn1I>  Brian
>     Rosen
>   * Re: [Rum] Real-time text in WebRTC is discussed...
>     <https://mailarchive.ietf.org/arch/msg/rum/6402q9HzKBNqvkwKajSlo4EYmMU>  Gunnar
>     Hellström
>   * Re: [Rum] Real-time text in WebRTC is discussed...
>     <https://mailarchive.ietf.org/arch/msg/rum/YG02TyKon1tRtctZ1Bwe0BnI7Cw>  Paul
>     Kyzivat
>   * Re: [Rum] [EXT] Re: Real-time text in WebRTC is...
>     <https://mailarchive.ietf.org/arch/msg/rum/UDz0JCzyqQ0ut8RJzWF54KjMUXY>  Malloy,
>     Jim
>   * Re: [Rum] [EXT] Re: Real-time text in WebRTC is...
>     <https://mailarchive.ietf.org/arch/msg/rum/WWekfAbXC81GFmOdUld1YOvV450>  Paul
>     Kyzivat
>   * Re: [Rum] Real-time text in WebRTC is discussed...
>     <https://mailarchive.ietf.org/arch/msg/rum/8N5sWJvqB-J4XgC5xqMCV2nL0gc>  Brian
>     Rosen
>   * Re: [Rum] [EXT] Re: Real-time text in WebRTC is...
>     <https://mailarchive.ietf.org/arch/msg/rum/HoD8Ni1r0f8S11H2K4hp6Ss5NB0>  Brian
>     Rosen
>   * Re: [Rum] [EXT] Re: Real-time text in WebRTC is...
>     <https://mailarchive.ietf.org/arch/msg/rum/eI-dX4HrMUur324GEt-CFmpv4mY>  Paul
>     Kyzivat
>   * Re: [Rum] [EXT] Re: Real-time text in WebRTC is...
>     <https://mailarchive.ietf.org/arch/msg/rum/AaoKog-HhtBUS_a-BKrB5BpwiF4>  Brian
>     Rosen
>   * Re: [Rum] [EXT] Re: Real-time text in WebRTC is...
>     <https://mailarchive.ietf.org/arch/msg/rum/1JojOtitB-FlcmCHQd9LMl4PWM4>  Paul
>     Kyzivat
>   * Re: [Rum] [EXT] Re: Real-time text in WebRTC is...
>     <https://mailarchive.ietf.org/arch/msg/rum/2NDD2kMSGngnh6GyoLD9Th_UYrM>  Brian
>     Rosen
>   * Re: [Rum] [EXT] Re: Real-time text in WebRTC is...
>     <https://mailarchive.ietf.org/arch/msg/rum/UZDyKrfOw-bjkbqjN8E3MHXrseo>  Malloy,
>     Jim
>   * Re: [Rum] [EXT] Re: Real-time text in WebRTC is...
>     <https://mailarchive.ietf.org/arch/msg/rum/avASfCcxCAY18gOFsTGE3P5oTOs>  Gunnar
>     Hellström
> 
> -- 
> 
> *John Martin*
> 
> Email: [e-mail] <mailto:[e-mail]>
> 
>