Re: [AVTCORE] Source switching performance in draft-hellstrom-avtcore-multi-party-rtt-source-01.txt

[Gunnar Hellström <[e-mail]>]

Hi,

Since this thread was initially about the source switching performance 
of different mixing method, I should try to express the source switching 
performance also for the method for the multi-party-unaware terminals 
described in Appendix A of draft-hellstrom-mmusic-multi-party-rtt

The switching performance cannot be expressed in seconds delay or 
numbers per second. It can rather be expressed as a cooperative 
switching. As soon as the transmitting party gives a clear indication of 
having a part of an entry ready, a phrase, a sentence or a line, then a 
switch can occur. Then the switch takes 300 ms. But text from other 
parties may have been waiting for long for the transmitting party to 
give turn. It can be 5 - 10 -20 - 50 seconds. It is possible to set a 
time limit after which a switch is forced on next word end so that no 
one can monopolize transmission. (there are more details on that in the 
appendix). The method has a possibility to work well with rapid 300 ms 
switching in a managed conference or in situations where one main typing 
party hands over to another. That is expected to be the case in 
emergency service calls, and therefore, even if I regard it to be a 
method with low functionality, it could still be considered for such cases.


Regards


Gunnar




Den 2020-03-16 kl. 23:12, skrev Gunnar Hellström:
>
> Hi James,
>
>
> Den 2020-03-16 kl. 13:11, skrev James Hamlin:
>>
>> Hi Gunnar
>>
>>
>> Many thanks for taking the time to go through this so thoroughly.
>>
>>
>> I think we have 2 main aspects to this work:-
>>
>>  1. Compatibility with existing implementations
>>  2. Choosing an efficient mechanism for the future
>>
>> For the first of these, it seems to me that the only solution is for 
>> a mixer to be able to do inline participant labeling and buffering to 
>> produce a presentable single text stream. Current implementations of 
>> RFC4103 will simply not understand switching between participants, 
>> nor will they make any visual indication of which text belongs to 
>> which participant, so the mixer needs to do that.
> Yes, it is possible to implement a limited functionality 
> "pseudo-multi-party" text mixer for presenting multi-party rtt on a 
> point-to-point rtt terminal. A procedure is included as Appendix A in 
> this draft:
> https://datatracker.ietf.org/doc/draft-hellstrom-mmusic-multi-party-rtt/
>
> When the mixer has text from more than one source to transmit, it 
> looks for suitable switching moments in the text from the source it is 
> transmitting. When a phrase or a sentence is complete or a line 
> separator issued, or even a long pause, then the mixer decides to 
> switch source and inserts a line separator and a label for next in 
> turn to get its text transmitted, and then the text.  This method can 
> be used but has its limitations. It only displays one source at a time 
> in real-time. Erasure over a switch does not work. It assumes that the 
> receiver accepts to use the chat-style layout in one display area etc.
>
> Each application area should decide what to do with old terminals who 
> do not support multi-party. Implement the switching method above, or 
> upgrade the terminals or refuse to involve the terminals in 
> multi-party sessions.
>
>
>
>
>>
>> For the second we have established that it's possible to: allow the 
>> different redundant blocks in a packet to be for different 
>> participants; or use timestamps to resolve the correct redundant text 
>> to use and to have each packet associated with just one participant. 
>> I can also imagine putting all text generations of each source in the 
>> CSRC list in each packet which adds 12 bytes of header space per CSRC 
>> (assuming 2 redundant generations); I'll write a separate mail about 
>> that.
> Sound interesting.
>> After that, I think we have a fairly complete set of solutions to 
>> choose from.
>
> Thanks,
>
> Gunnar
>
>
>> Some other comments inline.
>>
>> Best regards
>>
>> James
>>
>>
>>
>> James Hamlin
>> Contractor
>> Purple, a Division of ZP Better Together, LLC
>> purplevrs.com
>>
>> The information contained in this e-mail message is intended only for 
>> the personal and confidential use of the recipient(s) named above. If 
>> you have received this communication in error, please notify us 
>> immediately by e-mail, and delete the original message.
>>
>> ------------------------------------------------------------------------
>> *From:* Gunnar Hellström <[e-mail]>
>> *Sent:* 14 March 2020 11:17
>> *To:* James Hamlin; [e-mail]
>> *Subject:* Re: Source switching performance in 
>> draft-hellstrom-avtcore-multi-party-rtt-source-01.txt
>>
>> Hi James,
>>
>> Thanks for an interesting proposal.
>>
>> Let us extend the information about the packet contents of your 
>> example a bit:
>>
>>
>>
>> seq       01  02  03  04  05  06  07  08  09  10  11  12  13  14  15
>>
>> CSRC=source 1   2   3   1   2   3   1   2   3   1   2   3   1   2   3
>>
>> Timestamp   91  92  93  94 95  96  97  98  99 100 101 102 103 104 105
>>
>>
>> R2 t offset  6   6   6   6 6   6   6   6   6   6   6   6   6   6   6
>>
>> R1 t offset  3   3   3   3 3   3   3   3   3   3   3   3   3   3   3
>>
>> R2                          A M   X   B   N  Y   C   O   Z
>>
>> R1                       A M   X   B   N   Y   C   O   Z
>>
>> P            A   M   X   B   N Y   C   O   Z
>>
>>
>> Lost X   X
>>
>>
>> (The timestamps and timestamp offsets ("t offset") are shown in 100 
>> ms, in reality it will be in milliseconds)
>>
>>
>> The SSRC of the packet is always the mixer's SSRC.
>>
>> The source is indicated in the CSRC-list that in this method has only 
>> one member = the SSRC of the source represented in the packet.
>>
>> +jeh: Agreed: My mistake.
>>
>>
>> The timestamp is created by the mixer when sending, and the timestamp 
>> offsets make it possible to calculate the timestamps the redundant 
>> texts had when they were transmitted as originals.
>>
>>
>> The receiver must store essential data from a number of packets. This 
>> data is the sequence number, the source (=CSRC), the Timestamp.
>>
>> So, let us see what happens if both packet 06 and 07 are lost.
>>
>> The receiver must also store for each source, the timestamps for 
>> which text has been recieved (either with real contents or empty).
>>
>>
>> In packets 1 to 5, we have received and put in display areas for 
>> source 1: "AB", for source 2: "MN", for source 3: "X"
>>
>>
>> 08 is received and the gap (07 and 06 ) is detected (07 and 06 with 
>> two redundant elements in both, making a need for retrieval of 4 text 
>> elements ) is remembered so we need to do the recovery analysis..  
>> The source (2) and other essential data is noted. The original 
>> timestamp of R2 is calculated as Timestamp-R2 t offset = 98-6 = 92. 
>> Checking back in the list of received packets we find that we got a 
>> packet with timestamp 92 (and indeed, it contained text from source 
>> 2), so there is no need to recover R2. Then the original timestamp of 
>> R1 is calculated as Timestamp-R1 t offset = 98-3 = 95. Checking back 
>> in the list of received packets we find that we got a packet with 
>> timestamp 95 (and indeed, it contained text from source 2), so there 
>> is no need to recover R1. The Primary text in packet 08 ("O") is 
>> retrieved and put in the display area for source 2. It is noted that 
>> we have got text for timestamp 98 for source 2. The gap is still 4.
>>
>>
>> 09 is received.  The gap (4 elements ) is remembered so we need to do 
>> the recovery analysis.  The source (3) and other essential data is 
>> noted. The original timestamp of R2 is calculated as Timestamp-R2 t 
>> offset = 99-6 = 93. Checking back in the list of received packets we 
>> find that we got a packet with timestamp 93 (and indeed, it contained 
>> text from source 3), so there is no need to recover R2. Then the 
>> original timestamp of R1 is calculated as Timestamp-R1 t offset = 
>> 99-3 = 96. Checking back in the list of received packets we find that 
>> we never got a packet with timestamp 96 , so we recover R1 ("Y") and 
>> insert it in the display area of source 3. The Primary text in packet 
>> 09 ("Z") is retrieved and put in the display area for source 3. It is 
>> noted that we got text from timestamps 96 and 99 for source 3 (the 
>> gap can now be reduced to 3)
>>
>>
>>
>> 10 is received.  The gap (3 ) is remembered so we need to do the 
>> recovery analysis.  The source (1) and other essential data is noted. 
>> The original timestamp of R2 is calculated as Timestamp-R2 t offset = 
>> 100-6 = 94. Checking back in the list of received packets we find 
>> that we got a packet with timestamp 94 (and indeed, it contained text 
>> from source 1), so there is no need to recover R2. Then the original 
>> timestamp of R1 is calculated as Timestamp-R1 t offset = 100-3 = 97. 
>> Checking back in the list of received packets we find that we never 
>> got a packet with timestamp 97 , so we recover R1 ("C") and insert it 
>> in the display area of source 1. The Primary text in packet 10 is 
>> empty so there is nothing to put in the display area for source 1. It 
>> is noted that we got text from timestamps 97 and 100 for source 1 
>> (the gap can now be reduced to 2)
>>
>>
>>
>> 11 is received.  The gap (2 ) is remembered so we need to do the 
>> recovery analysis.  The source (2) and other essential data is noted. 
>> The original timestamp of R2 is calculated as Timestamp-R2 t offset = 
>> 101-6 = 95. Checking back in the list of received packets we find 
>> that we got a packet with timestamp 95 (and indeed, it contained text 
>> from source 2), so there is no need to recover R2. Then the original 
>> timestamp of R1 is calculated as Timestamp-R1 t offset = 101-3 = 98. 
>> Checking back in the list of received packets we find that we got a 
>> packet with timestamp 98 , so we do not need to recover R1. The 
>> Primary text in packet 11 is empty so there is nothing to put in the 
>> display area for source 2. It is noted that we got text from 
>> timestamp 101 for source 2. (we did not recover anything, so the gap 
>> is still 2)
>>
>>
>> 12 is received.  The gap (2 ) is remembered so we need to do the 
>> recovery analysis.  The source (3) and other essential data is noted. 
>> The original timestamp of R2 is calculated as Timestamp-R2 t offset = 
>> 102-6 = 96. Checking back in the list of received packets we find 
>> that we never got a packet with timestamp 96, but from packet 09 we 
>> recovered R1 from timestamp 96. So we shall not recover anything from 
>> R2 here. Then the original timestamp of R1 is calculated as 
>> Timestamp-R1 t offset = 102-3 = 99. Checking back in the list of 
>> received packets we find that we got a packet with timestamp 99, so 
>> we do not need to recover R1. The Primary text in packet 12 is empty 
>> so there is nothing to put in the display area for source 3. It is 
>> noted that we got text for timestamp 102 for source 3 (the gap can 
>> now be reduced to 1)
>>
>>
>> 13 is received.  The gap (1 ) is remembered so we need to do the 
>> recovery analysis.  The source (1) and other essential data is noted. 
>> The original timestamp of R2 is calculated as Timestamp-R2 t offset = 
>> 103-6 = 97. Checking back in the list of received packets we find 
>> that we already recovered text for timestamp 97, so nothing is 
>> recovered and nothing inserted from R2 in the display area of source 
>> 1 . Then the original timestamp of R1 is calculated as Timestamp-R1 t 
>> offset = 103-3 = 100. Checking back in the list of received packets 
>> we find that we got a packet with timestamp 100, so we do not need to 
>> recover R1. The Primary text in packet 13 is empty so there is 
>> nothing to put in the display area for source 1. It is noted that we 
>> got text for timestamp 103 for source 1 (the gap can now be reduced to 0)
>>
>>
>> 14 is received.  The gap is now 0 so we do not need to do any 
>> recovery analysis.   The source (2) and other essential data is 
>> noted. The Primary text in packet 13 is empty so there is nothing to 
>> put in the display area for source 2. It is noted that we got text 
>> for timestamp 104 for source 2.
>>
>>
>> After this, we have in the display areas:  for 1: "ABC" for 2: "MNO", 
>> for 3: "XYZ", so everything is recovered.
>>
>>
>> I am sorry, the narrative above may be hard to follow. It could 
>> probably be converted to some table format if we need to do it again 
>> for other cases.
>>
>>
>> So, yes, this method also works.
>>
>> I see a couple of differences in characteristics between this 
>> "timestamp method" and the "CSRClist method":
>>
>>
>> 1. The recovery time from loss to recovery can with the timestamp 
>> method be 200 times the number of simultaneous sending sources in 
>> milliseconds. Thus with 5 sources: 1 second. With the CSRClist 
>> method, it is steady 200 milliseconds. (assuming a transmission 
>> interval of 100 ms and round robin mixer switching.)
>>
>>
>> 2. The recovery capacity in packets in sequence is 2*the number of 
>> simultaneous sources = 10 packets for 5 sources. With the CSRC method 
>> it is 2 packets. (again assuming round robin mixer switching )
>>
>>
>> 3. The complexity of the procedure is higher but still manageable for 
>> the timestamp method.
>>
>> jeh: Yes. I had thought this would be simpler, but the timestamp 
>> logic gets complicated.
>>
>>
>> 4. The number of packets to store essential information about is 
>> higher for the timestamp method ( I think it is 4*the number of 
>> active sources). For the CSRC list method, it is 4 packets and less 
>> information per packet.
>>
>>
>> You say: "The advantage of this approach is that the format of the 
>> packet doesn't change. The current arrangement where all the text in 
>> a packet is for one participant is preserved."
>>
>>
>> I do not see the CSRClist method as a change in packet format.
>>
>> jeh: Agreed: I should have said that differently: the change is that 
>> the text over the redundant and primary block is no longer 
>> continuous; it's for different participants.
>>
>>
>> The mixer needs in both methods to include a CSRC -list. The 
>> difference is that in the CSRClist method, the list has more members. 
>> It is still within the format description of RTP. The source of the 
>> redundant parts will vary in a packet, but the composition of the 
>> contents of the packet for transmission from the mixer is as usual 
>> for a single sender: Put what was sent next to last in the packet as 
>> R2, put what was sent last as R1, and put the new text chunk as P. 
>> The only addition is the rule that the CSRC list is populated with 
>> the sources in the strict order.
>>
>>
>> Summary: both methods seem possible. It will be interesting to get 
>> more comments.
>>
>>
>> Thanks,
>>
>>
>> Gunnar
>>
>>
>>
>>
>>
>>
>>
>>
>> Den 2020-03-14 kl. 01:45, skrev James Hamlin:
>>> Hi Gunnar
>>>
>>>
>>> I've also been thinking through the possibility of a sender 
>>> switching source without clearing all redundant generations first. 
>>> Clearly, a sender that did this today would cause problems 
>>> for existing receivers. But checking timestamps at the receiver 
>>> should fix this.
>>>
>>>
>>> Consider three senders 1, 2 and 3 which send text "ABC", "MNO" and 
>>> "XYZ". The block below shows this text being 
>>> sent taking round-robin turns with the participants.
>>>
>>>
>>> seq    01  02  03  04  05  06  07  08  09  10  11  12  13  14  15
>>>
>>> part    1   2   3   1   2   3   1   2 3   1   2   3   1   2   3
>>>
>>>                             -
>>>
>>> R2                        A   M   X   B   N  Y   C   O   Z
>>>
>>> R1           A M X   B   N   Y   C   O   Z
>>>
>>> P       A   M   X   B   N   Y   C   O   Z
>>>
>>>
>>> If sequence 06 is lost and the receiver sees sequence 07 then it may 
>>> assume that the lost packet was for participant 1 and use the 
>>> redundant character "B". This would lead to character "B" being 
>>> duplicated in the output for participant 1. But it is possible for a 
>>> receiver to get the correct result; the timestamp for the redundant 
>>> text in packet 07 will not be higher than the most recent timestamp 
>>> previously received and so shouldn't be used. The redundant text in 
>>> the following packet 08 has no applicable redundant text, but the 
>>> timestamp of the R1 in packet 09 will be greater than the most 
>>> recent received and so is usable.
>>>
>>>
>>> The advantage of this approach is that the format of the packet 
>>> doesn't change. The current arrangement where all the text in a 
>>> packet is for one participant is preserved. Tracking the timestamp 
>>> adds some implementation effort but I think 
>>> it's minimal. It does also mean the mixer needs to synchronize 
>>> timestamps across the media sources but it is in a position to do so.
>>>
>>> Best regards
>>>
>>> James
>>>
>>>
>>>
>>> James Hamlin
>>> Contractor
>>> Purple, a Division of ZP Better Together, LLC
>>> purplevrs.com
>>>
>>> The information contained in this e-mail message is intended only 
>>> for the personal and confidential use of the recipient(s) named 
>>> above. If you have received this communication in error, please 
>>> notify us immediately by e-mail, and delete the original message.
>>>
>>> ------------------------------------------------------------------------
>>> *From:* Gunnar Hellström <[e-mail]>
>>> *Sent:* 13 March 2020 16:41
>>> *To:* [e-mail]; James Hamlin
>>> *Subject:* Source switching performance in 
>>> draft-hellstrom-avtcore-multi-party-rtt-source-01.txt
>>>
>>> Hi,
>>>
>>> I want to follow-up on the good discussion on source switching 
>>> performance a couple of days ago, under the subject "[AVTCORE] 
>>> Improved RTP-mixer performance for RFC 2198 and RFC 4103 redundancy 
>>> coding"
>>>
>>> *Two parts in the performance increase solution.*
>>> Two actions are proposed in 
>>> draft-hellstrom-avtcore-multi-party-rtt-source:
>>> a) Reduce the packet transmission interval from 300 to 100 ms.
>>> b) Use a strict relation between members in the CSRC list and the 
>>> parts of the payload that is original text and first generation 
>>> redundancy and second generation redundancy so that the mixer can 
>>> switch source for every new packet and the sources of text recovered 
>>> from redundancy can be assessed by the receiver.
>>>
>>> I think it is worth while to move forward with the complete 
>>> improvement a) and b) proposed in the draft. It will cause less 
>>> complexity, lower delays and lower risk for stalling in case of many 
>>> participants entering new text simultaneously.
>>>
>>> Here is my reasoning:
>>>
>>> I have the following view of the achievable performance improvements 
>>> for different cases:
>>>
>>> 1. With the original source switching with RFC 4103 and an RTP-mixer 
>>> using 300 ms transmission interval and not allowing a mix of sources 
>>> in one packet, there can be one source switching per second by the 
>>> mixer with an introduced delay of up to one second.
>>>
>>>
>>> 2. By just reducing the transmission interval from 300 to 100 ms, it 
>>> will be possible to have three source switches per second with an 
>>> introduced delay of up to one second. (with just two parties sending 
>>> text simultaneously, the delay will be maximum 300 ms. )
>>>
>>> 3. And by applying the proposal from the multi-party-rtt-source 
>>> draft with the CSRC-list as a source list for the redundancy, and 
>>> also using 100 ms transmission interval, there can be switching 
>>> between five source per second with an introduced delay of max 500 
>>> ms. With just two parties typing simultaneously, the delay will be a 
>>> maximum of 100 ms.
>>>
>>> The delays are extreme values from when all sources start to type 
>>> simultaneously.  It was agreed that at least the improvement from 
>>> the reduced transmission interval is needed.
>>>
>>> Case 1 and 2 are a bit complex for the mixer to implement. From the 
>>> moment it has text queued for transmission from another source B 
>>> than the one currently transmitted A, then the mixer needs to stop 
>>> adding new text from A to the packets, but still send two more 
>>> packets with the agreed transmission interval, progressing the 
>>> latest transmitted original text to first generation redundancy and 
>>> then again one more packet with the text as second level redundancy. 
>>> Not until that is done, the mixer is allowed to start taking text 
>>> from the transmission queue from B to transmit. This is the 
>>> background of the 1 s vs 300 ms delays in case 1 and 2.
>>>
>>> In case 3, there is much less complexity. When there is something 
>>> from B in queue for transmission, the mixer can decide to insert 
>>> that in next packet and add the redundancy from earlier 
>>> transmissions from A, because their sources are included in the CSRC 
>>> list in the same packet.
>>>
>>> Therefore I want to move on with the complete solution in case 3.
>>>
>>> -----------------------------------
>>>
>>> *Influence on the multi-party capability negotiation:*
>>> There is an installed park of RTT implementations without 
>>> multi-party awareness. The receiver need to take active part in 
>>> planning the multi-party RTT presentation. Therefore a capability 
>>> negotiation is needed. A simple sdp attribute a=rtt-mix without 
>>> value is proposed in the draft.
>>>
>>> It is important to let this attribute mean capability of the 
>>> complete solution case 3).  If there is a temptation to have 
>>> different levels of implementation, some only implementing the 
>>> shorter transmission interval (2) and some implementing the complete 
>>> solution (3), then threre will be a need for two different 
>>> attributes, or one attribute with a list of parameter values for the 
>>> two cases. That would complicate the evaluation of the negotiation. 
>>> Therefore I would prefer that the attribute can mean capability to 
>>> use the complete mixing solution (3).
>>>
>>> Regards
>>>
>>> Gunnar
>>>
>>>
>>> Den 2020-02-29 kl. 20:13, skrev [e-mail]:
>>>> A New Internet-Draft is available from the on-line Internet-Drafts directories.
>>>>
>>>>
>>>>          Title           : Indicating source of multi-party Real-time text
>>>>          Author          : Gunnar Hellstrom
>>>> 	Filename        : draft-hellstrom-avtcore-multi-party-rtt-source-01.txt
>>>> 	Pages           : 13
>>>> 	Date            : 2020-02-29
>>>>
>>>> Abstract:
>>>>     Real-time text mixers need to identify the source of each transmitted
>>>>     text chunk so that it can be presented in suitable grouping with
>>>>     other text from the same source.  An enhancement for RFC 4103 real-
>>>>     time text is provided, suitable for a centralized conference model
>>>>     that enables source identification, for use by text mixers and
>>>>     conference-enabled participants.  The mechanism builds on use of the
>>>>     CSRC list in the RTP packet.  A capability exchange is specified so
>>>>     that it can be verified that a participant can handle the multi-party
>>>>     coded real-time text stream.  The capability is indicated by an sdp
>>>>     media attribute "rtt-mix".
>>>>
>>>>
>>>> The IETF datatracker status page for this draft is:
>>>> https://datatracker.ietf.org/doc/draft-hellstrom-avtcore-multi-party-rtt-source/
>>>>
>>>> There are also htmlized versions available at:
>>>> https://tools.ietf.org/html/draft-hellstrom-avtcore-multi-party-rtt-source-01
>>>> https://datatracker.ietf.org/doc/html/draft-hellstrom-avtcore-multi-party-rtt-source-01
>>>>
>>>> A diff from the previous version is available at:
>>>> https://www.ietf.org/rfcdiff?url2=draft-hellstrom-avtcore-multi-party-rtt-source-01
>>>>
>>>>
>>>> Please note that it may take a couple of minutes from the time of submission
>>>> until the htmlized version and diff are available at tools.ietf.org.
>>>>
>>>> Internet-Drafts are also available by anonymous FTP at:
>>>> ftp://ftp.ietf.org/internet-drafts/
>>>>
>>>>
>>>> _______________________________________________
>>>> I-D-Announce mailing list
>>>> [e-mail]
>>>> https://www.ietf.org/mailman/listinfo/i-d-announce
>>>> Internet-Draft directories:http://www.ietf.org/shadow.html
>>>> orftp://ftp.ietf.org/ietf/1shadow-sites.txt
>>> -- 
>>>
>>> + + + + + + + + + + + + + +
>>>
>>> Gunnar Hellström
>>> Omnitor
>>> [e-mail]
>>> [phone]
>> -- 
>>
>> + + + + + + + + + + + + + +
>>
>> Gunnar Hellström
>> Omnitor
>> [e-mail]
>> [phone]
> -- 
>
> + + + + + + + + + + + + + +
>
> Gunnar Hellström
> Omnitor
> [e-mail]
> [phone]
>
> _______________________________________________
> Audio/Video Transport Core Maintenance
> [e-mail]
> https://www.ietf.org/mailman/listinfo/avt

-- 
-----------------------------------------
Gunnar Hellström
Omnitor
[e-mail]
[phone]