[tsvwg] links to Canary methods for roll-out of new transport features

[Gorry Fairhurst <[e-mail]>]

So, there's been a change in the way people roll-out new features, that 
maybe we could say more about in the L4S OPS draft. What I write below 
is not specific to L4S, and I'd really welcome other familiar with using 
and evaluating such methods to chime-in and say more, but anyway here is 
starter:

Canarying is a partial and time-limited deployment of a change in a 
service/protocol and its evaluation as a part of the deployment. The 
method is used throughout the roll-out and helps to decide whether or 
not to continue with the rollout. The part of the service that receives 
the change is “the canary,” and the remainder of the service is “the 
control.” The canary deployment is performed on a small subset of the 
networ/users, than the control. Canarying is evaluated as an A/B testing 
process, to check the impact of the (initial) deployment.

See this from google:

https://urldefense.com/v3/__https://sre.google/workbook/canarying-releases/__;!!LpKI!znKgxfRLAd0ogD6r-X5u7fAz0eInch7MTeJDVQB5drmteEU3mhnP0no1TuTKHR8C$ 
[sre[.]google] [sre.google]
<https://urldefense.com/v3/__https:/sre.google/workbook/canarying-releases/__;!!LpKI!x3qiMCrsIjOFTyJVBmvIh9lHMG6KKpySWLDUWq71wo9nz1gKk0cSHRe1loo_i6Op$>

https://urldefense.com/v3/__https://developer.android.com/distribute/best-practices/launch/test-tracks__;!!LpKI!znKgxfRLAd0ogD6r-X5u7fAz0eInch7MTeJDVQB5drmteEU3mhnP0no1TuinRLVe$ 
[developer[.]android[.]com]
[developer.android.com]
<https://urldefense.com/v3/__https:/developer.android.com/distribute/best-practices/launch/test-tracks__;!!LpKI!x3qiMCrsIjOFTyJVBmvIh9lHMG6KKpySWLDUWq71wo9nz1gKk0cSHRe1lkLr2-F-$>

When working with QUIC  people have released an update to only to a 
small subset of the user base, monitor stability or another metric of 
interest, and
decide whether to roll out the update to more users, to wait for more 
data to come in, or to halt the rollout altogether.If one of the metrics 
you’re monitoring is off, or you check the user reviews and  see issues 
or complaints on a specific topic. You don't need to enable a feature
for anyone/network who you might expect to be hurt.

ECN isn't just "automatically" used, the app can decide (or at least the 
app-supplier), this will always be the case for QUIC anyway. The
result of these tests provide the sort of data that has informed QUIC 
(e.g. Chrome Canary), and I expect the basis of what is reported by
google and others in MAPRG. The point is that this allows statistical 
testing without massive impact, and the incremental roll-out.

This says something about akamai's use:

https://urldefense.com/v3/__https://www.akamai.com/uk/en/products/performance/cloudlets/phased-release.jsp__;!!LpKI!znKgxfRLAd0ogD6r-X5u7fAz0eInch7MTeJDVQB5drmteEU3mhnP0no1TvXdVpGA$ 
[akamai[.]com]
[akamai.com]
<https://urldefense.com/v3/__https:/www.akamai.com/uk/en/products/performance/cloudlets/phased-release.jsp__;!!LpKI!x3qiMCrsIjOFTyJVBmvIh9lHMG6KKpySWLDUWq71wo9nz1gKk0cSHRe1ltQQWNXF$>

Cloudflare, etc have used similar approaches:

https://urldefense.com/v3/__https://medium.com/boozt-tech/canary-release-with-cloudflare-workers-84a9b45bac0f__;!!LpKI!znKgxfRLAd0ogD6r-X5u7fAz0eInch7MTeJDVQB5drmteEU3mhnP0no1TozuZjjw$ 
[medium[.]com]
[medium.com]
<https://urldefense.com/v3/__https:/medium.com/boozt-tech/canary-release-with-cloudflare-workers-84a9b45bac0f__;!!LpKI!x3qiMCrsIjOFTyJVBmvIh9lHMG6KKpySWLDUWq71wo9nz1gKk0cSHRe1lu8IFQaL$>


Gorry