Re: [ippm] New QUIC Packet Loss Measurement (draft-cfb-ippm-spinbit-measurements)

["Bulgarella Fabio (Guest)" <[e-mail]>]

​

Hello Igor,
thanks for your reply.

This exchange of e-mails is very useful to understand which of the two solutions has the best balance between positive and negative aspects, at least in the measurement scenarios of interest for telecommunication operators. The important thing is that one of the two solution is chosen and a packet loss feature is finally included in the Quic protocol (after the necessary checks in terms of privacy and security). However, we can already say, from the point of view of Telecom Italia and Orange, that the Qbit is a fixed point of this type of measures. It remains to decide which is better between L-bit and R-bit.

First of all, let's remember the reasons that led us to design the Rbit.
We found some weaknesses in the Lbit:

- the LBit logic is strictly tied to the protocol implementation.
For example in LSQUIC there are 6 different hook points where this variable is set. QL performance mechanism logic is intertwined with the protocol logic. So, the concern here is that future changes in the protocol or in new implementations need a careful analysis of the code to avoid different behaviors and different measurement results;

- loss of packets carrying only ACK frames (are detected?).
We saw in the Montreal demo (using Ericsson SpinDump tool), the last July, that if Quic packets only contain ACK frames the Lbit is not set. Actually, during a file download we inserted losses in the upload direction that were not detected by the system. Alexandre confirmed this behavior. Have you changed something since then?

- the loss measurement signal is inaccurate in case of losses:
The Lbit is a simple signaling protocol. If you miss a Lbit marked packet you miss a signal of loss. If losses are not equally distributed between L-marked and not L-marked packets, it's very likely that the produced measurements are wrong or at least inaccurate. The two approaches are completely different. Rbit works exactly as the Qbit. It counts the lost packets, it's not a signaling. So, as the Qbit, the correct percentage of lost packets is detected.

Now some inline answers to your considerations, even though at this point it would be convenient to organize a comparison involving also other people interested in the topic, maybe in a next IETF meeting. We are also available to carry out comparative tests on the same flows (marking simultaneously with both solutions) using for example the Ericsson SpinDump tool which has already been used in the previous hackathons as QUIC performance measurement tool.

Best Regards,
Fabio B.


> Strong reordering exists.  That’s why we are recommending the minimum Q to be 64. This comes from real data, not lab data.  If 5 packets were enough to tell the end of marking periods, we would had recommended minimum Q to be 8.

The length of the period is not chosen only in function of reordering but also to avoid that burst losses completely cancel one or more periods. However, stimulated by Igor's concerns, we thought of an improvement regarding the tolerance of our system against out of sequence. In a nutshell, it would be possible to intercept reordered packets received after the detection of an incoming "Qbit marking period", using them to update the computed average until the current "Rbit marking period" is completed in generation. After the necessary laboratory tests, we could add this new feature to the next revision of our draft.

> R signal is still going to be significantly delayed, especially with slower connections (not every connection is going to be streaming media at a sustained rate of 15Mbps). So if for a 15Mbps average-rate connection you can expect to see R-signal with a delay between 8-RTTs (1:2 ACK) and 38-RTTs (1:10 ACK), the delay increases to 24-RTTs and 114-RTTs for a 5Mbps average-rate connection.

As already said in the previous mail, based on our tests, this is not a big problem because it's still possible to obtain enough measurements to evaluate the packet loss of a session. Moreover, the measurement delay is shortened when packet loss occurs.

> L-bit signal provides a crisp shape of the loss – whether it is random loss, or packets are lost of batches (and how many packets are in a loss batch). By contrast, R-signal only indicates an average loss over 2 to 10 Q-bit marking periods.  The share of the loss is erased. I think for that alone, L-signal is more useful for troubleshooting.

Although we calculate the E2E loss through an average (only if the reflection is on a slower channel than the signal reception channel), we are still able to accurately calculate the loss using a correction factor (see https: //tools.ietf.org/html/draft-cfb-ippm-spinbit-measurements-00#section-7.4).

> This is an important difference, and it seems accurate on its surface.  In reality, every QUIC implementation will keep track of packets and declare them lost for the purposes of data retransmission and/or congestion control.  The timing and details of when packets are declared lost (L-bit signal) will vary.  Likewise, every QUIC implementation will send some ACKs, but their timing and frequency (R-signal) will vary.  R-signal quality will also heavily depend on the “marking period detection” implementation quality and “Q-signal averaging” implementation details.

See first point in the L-bit weeknesses paragraph. Moreover, the two functions "marking period detection" and "Q-signal averaging", really simple in implementation, are completely detached from the rest of the protocol logic and their behaviors are expressly indicated in the draft.

> Correct, as long as both endpoints implement the entire QR scheme (even if this is an unusual setup – measuring path A requires observing path B). Also, accurate accounting with QR scheme requires correlation of both directions, with the R-bit direction significantly lagging the Q-bit direction.  It should be understood how this scheme behaves in cases of changing Connection IDs.

We do not see the problem of implementing the solution on both endpoints since generally the client, unlike the server, does not have scalability problems. So if you implement it on the server it costs nothing to implement it on the client as well.




________________________________
Da: Lubashev, Igor <ilubashe=[e-mail]>
Inviato: giovedì 23 aprile 2020 18:41
A: Bulgarella Fabio (Guest); Ian Swett; Cociglio Mauro
Cc: Riccardo Sisto; [e-mail]; [e-mail]; [e-mail]; IETF IPPM WG ([e-mail])
Oggetto: [EXT] RE: [ippm] New QUIC Packet Loss Measurement (draft-cfb-ippm-spinbit-measurements)

Fabio, thanks for the reply.


  *   “Moreover, with a strong amount of reordering the Qbit (and the SpinBit) would not work either.”

Strong reordering exists.  That’s why we are recommending the minimum Q to be 64.  This comes from real data, not lab data.  If 5 packets were enough to tell the end of marking periods, we would had recommended minimum Q to be 8.



  *   R-bit signal delay.

R signal is still going to be significantly delayed, especially with slower connections (not every connection is going to be streaming media at a sustained rate of 15Mbps).  So if for a 15Mbps average-rate connection you can expect to see R-signal with a delay between 8-RTTs (1:2 ACK) and 38-RTTs (1:10 ACK), the delay increases to 24-RTTs and 114-RTTs for a 5Mbps average-rate connection.


  *   R-bit signal quality.

L-bit signal provides a crisp shape of the loss – whether it is random loss, or packets are lost of batches (and how many packets are in a loss batch).  By contrast, R-signal only indicates an average loss over 2 to 10 Q-bit marking periods.  The share of the loss is erased.

I think for that alone, L-signal is more useful for troubleshooting.



  *   L-bit signal depends on each implementation’s determination of when to “declare a packet lost” (which may differ for each implementation), while R-signal depends only on more generic behaviors.

This is an important difference, and it seems accurate on its surface.  In reality, every QUIC implementation will keep track of packets and declare them lost for the purposes of data retransmission and/or congestion control.  The timing and details of when packets are declared lost (L-bit signal) will vary.  Likewise, every QUIC implementation will send some ACKs, but their timing and frequency (R-signal) will vary.  R-signal quality will also heavily depend on the “marking period detection” implementation quality and “Q-signal averaging” implementation details.



  *   End-to-end loss: “We can also place two independent observers wherever on the two directions and they will still produce coherent E2E measures.”
  *   End-to-end loss: “by observing only one direction we can measure the […] the end-to-end in the opposite direction (Rbit).

Correct, as long as both endpoints implement the entire QR scheme (even if this is an unusual setup – measuring path A requires observing path B).

Also, accurate accounting with QR scheme requires correlation of both directions, with the R-bit direction significantly lagging the Q-bit direction.  It should be understood how this scheme behaves in cases of changing Connection IDs.

Best wishes,


  *   Igor


From: Bulgarella Fabio (Guest) <fabio.bulgarella=[e-mail]>
Sent: Wednesday, April 22, 2020 3:12 PM
To: Lubashev, Igor <[e-mail]>; Ian Swett <[e-mail]>; Cociglio Mauro <[e-mail]>
Cc: Riccardo Sisto <[e-mail]>; [e-mail]; [e-mail]; [e-mail]; IETF IPPM WG ([e-mail]) <[e-mail]>
Subject: Re: [ippm] New QUIC Packet Loss Measurement (draft-cfb-ippm-spinbit-measurements)

Hello Igor,
here some thoughts about your concerns.

1. If big differences in delay are present between two parallel paths on which the same connection is divided, this is certainly an unwanted behavior, because it could create problems to the protocol itself even before damaging the performance measurements. As a network operator, we try to avoid such scenarios. However, our mechanism works correctly in most cases and is still resistant to normal reordering. Moreover, with a strong amount of reordering the Qbit (and the SpinBit) would not work either.


2. It's true that your mechanism allows the measurement of E2E and upstream loss of the observed direction. However, these measurement methodologies are born with the idea of providing information on the performance of the entire connection, as was done with the SpinBit. Actually with our mechanism, when observing a single direction, we still are able to measure the E2E loss of the opposite channel (using the R-bit).


3. The ACK ratio of 1:10 should only occur in the absence of loss. By introducing loss, the ratio between ACK and data is certainly better. So the quality of our signal improves just when it is needed; we don't care if we get delayed measurements when everything is working great. This behavior is confirmed by our laboratory tests where we have seen that loss can be easily detected in an extremely accurate manner even in the case of highly unbalanced flows.

L-Bit has an RTO delay but to evaluate an accurate loss ratio you have still to observe a significant number of packets.


4. First of all, our solution is not dependent on any specific protocol implementation unlike the QL and therefore there shouldn’t be problems related to the quality of the implementation. In other words, our portion of code is always the same, in every protocol implementation.
Secondly, as anticipated in point 2, by observing only one direction we can measure the losses in the operator's domain (Qbit) and the end-to-end in the opposite direction (Rbit). So, our solution does work also for asymmetric path segments. We can also place two indipendent observers wherever on the two directions and they will still produce coherent E2E measures.

The QL solution does not give any information about the opposite direction, it only measures what it observes, that is the current direction. Having two independent measures, one per direction, can be an advantage in some aspects, but a disadvantage for others.


Best regards,
Fabio B..




________________________________
Da: ippm <[e-mail]> per conto di Lubashev, Igor <ilubashe=[e-mail]>
Inviato: martedì 21 aprile 2020 18:31
A: Ian Swett; Cociglio Mauro
Cc: Riccardo Sisto; [e-mail]; [e-mail]; [e-mail]; IETF IPPM WG ([e-mail])
Oggetto: [EXT] Re: [ippm] New QUIC Packet Loss Measurement (draft-cfb-ippm-spinbit-measurements)

I like the QR draft from the perspective that it mostly works (and lots of credit is due for things that work).

I have a few concerns, though.

1. The most trivial one is that QR is actually harder to implement correctly for endpoints, given the tricky processing required to identify when a marking interval has ended.  Experimental data that Akamai & Orange compiled shows that there are networks that do ECMP among links with significantly different latencies.  They will cause a lot more reordering to a burst of packets than the recommended “observe 5 packets (in a row?) of the opposite markings” rule can address.  You might need to potentially defer the decision on where exactly the marking period boundary is until you’ve seen almost half of the next marking period.  Requiring every endpoint to implement this right is tough.

2. Since the most important direction (at least for downloads and media streaming) is server-to-client, to get this signal with QR, you need both endpoints implementing the logic, with the heaviest lift on the clients (see above).  For QL, a client that does not want to implement QL logic but is ok with the loss data reporting can just update the TP and header protection mask – all the lift is on the server side.

3. I am concerned with the quality of the signal you get from QR.  While the upstream loss signal is strong (same Q-bit), the downstream/end-to-end loss signal is weak and is likely to result in very delayed and inconclusive signal.  Google’s experience (as I understand it) and protocol research (https://erg.abdn.ac.uk/~downloads/ackscaling.pdf<https://urldefense.proofpoint.com/v2/url?u=https-3A__erg.abdn.ac.uk_-7Edownloads_ackscaling.pdf&d=DwMF-g&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=u3hS0QnHLn0-P-Lmy8-Yhm7L0G310RjwTQfw2XLvfJk&s=8qb_Zcm3YASCVrMZiZ5Le_3_nPxd8AC7bUbVwXl4L9I&e=>) point to the benefit of ACK-ing only 1:10 packets after the initial slow start (draft-fairhurst-quic-ack-scaling).  Since such behavior has a beneficial impact on networks, mobile devices, and endpoint cpu, and no negative impact on the protocol performance, I expect it to be in common use.  With 1:10 ACK ratio, R-bit signal is significantly delayed as it is lagging the loss by at least 1400*(Q=64)*2*10=1.8MB of data transferred, which is at least 38 RTTs for a 15Mbps stream over a 25ms link (L-bit signal is delayed by an RTO).

4. QR scheme only works for symmetric path segments, when the observer can capture both directions of the flow. You also cannot compute end-to-end loss in any direction, unless both endpoints implement the entire QR algorithm.  In contrast, QL works for single-direction observers, and whichever endpoint implements QL algorithm ensures end-to-end loss signal for the packets it sends, and the quality of that loss signal solely depends on the quality of that endpoint’s implementation (not the quality of the other endpoint’s implementation).  QL seems more deployable.


  *   Igor


From: Ian Swett <[e-mail]>
Sent: Wednesday, April 8, 2020 3:44 PM
To: Cociglio Mauro <mauro.cociglio=[e-mail]>
Cc: [e-mail]; Lubashev, Igor <[e-mail]>; [e-mail]; [e-mail]; Riccardo Sisto <[e-mail]>; IETF IPPM WG ([e-mail]) <[e-mail]>
Subject: Re: [ippm] New QUIC Packet Loss Measurement (draft-cfb-ippm-spinbit-measurements)

Thanks for sharing this.  From my perspective, this is an improvement upon the previous proposal in terms of robustness.  This design feels very similar to the spin bit, and seems trivial to implement, which are also nice properties.

Also, QUIC doesn't really have retransmissions, so the 'R bit' name always made me a bit uncomfortable.

I'm not saying QUIC should adopt this yet, but I'd be interested in seeing a privacy analysis completed for it.

Ian

On Wed, Apr 8, 2020 at 4:25 AM Cociglio Mauro <mauro.cociglio=[e-mail]<mailto:[e-mail]>> wrote:
Dear QUIC WG Members.

We submitted to IPPM WG a draft where we described a new 2 bit packet loss methodology (https://tools.ietf.org/html/draft-cfb-ippm-spinbit-measurements-01<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dcfb-2Dippm-2Dspinbit-2Dmeasurements-2D01&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=e_BQ3UDNSWkXu_ID1rkGv3CCZs89sezu-6Ibkvltf7Y&s=UeTW_fLvxHy_ewVvm0y6kIqZoXUR8PYWgxAOxmQScYw&e=>).
The measurement of packet loss is under discussion in QUIC WG and our draft introduces two alternatives about it.
The first one is a spin-bit dependent signal and uses a single bit. The second one, described in the following linked slides, is a standalone solution based on a two bits loss signal and on alternate marking (RFC8321)..
This last methodology improves, in our opinion, the algorithm proposed by Orange and Akamai described in https://tools.ietf.org/html/draft-ferrieuxhamchaoui-quic-lossbits-03<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dferrieuxhamchaoui-2Dquic-2Dlossbits-2D03&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=e_BQ3UDNSWkXu_ID1rkGv3CCZs89sezu-6Ibkvltf7Y&s=V649Uqu8mfuBWvMRBTUkpEWfpEqlLfufGs67XXzvNFo&e=>.

The 2 bits are the sQuare bit (Q-bit) and the Reflection square bit (R-bit).
The Q-bit doesn’t change from the Ferrieux-Hamchaoui draft but the R-bit substitutes the L-bit.
This avoids the L-bit dependence from an internal protocol variable, a problem raised in the last QUIC interim meeting.

You can find the slides, we prepared for the IPPM interim meeting, at the following link: https://github.com/ietf-ippm/meeting-materials/blob/aa486f7ead28b9f55c5ef499e5c9ed33ab06daf9/ietf107-virtual/Slides/07-spinbit-measurements.pdf<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ietf-2Dippm_meeting-2Dmaterials_blob_aa486f7ead28b9f55c5ef499e5c9ed33ab06daf9_ietf107-2Dvirtual_Slides_07-2Dspinbit-2Dmeasurements.pdf&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=e_BQ3UDNSWkXu_ID1rkGv3CCZs89sezu-6Ibkvltf7Y&s=3l6t3FYQn0iz_xX0sxNFKUnh7bwYFMOpQ0z_zNhXHpg&e=>

Comments and suggestions are always welcome.

Of course we are available to present our proposals in the next QUIC WG meeting or to arrange a Webex side meeting if needed.

Best regards.

Mauro, Fabio, Giuseppe, Massimo and Riccardo


_____________________
Mauro Cociglio
TIM - CT.TA.EI
Via G. Reiss Romoli, 274
10148 - Torino (Italy)
Tel.: [phone]<tel:+39%20011%20228%205028>
Mobile: [phone]<tel:+39%20335%20766%209751>
_____________________


[Image removed by sender.]<https://urldefense.proofpoint.com/v2/url?u=https-3A__on.tim.it_banner-2Dmail-2Ddip&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=e_BQ3UDNSWkXu_ID1rkGv3CCZs89sezu-6Ibkvltf7Y&s=z55QpRg48UYrhUvomUNAlkMCUxdQT20Mz1VhZkNrLQI&e=>

Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie.

This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks...

Rispetta l'ambiente. Non stampare questa mail se non è necessario.

_______________________________________________
ippm mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/ippm<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_ippm&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=e_BQ3UDNSWkXu_ID1rkGv3CCZs89sezu-6Ibkvltf7Y&s=YF4GnFEgwmSakC8Bv_763sIZZR3g8t7wdsuiOl6C8fU&e=>