RE: Is "Version Greasing" a new benfit or a new obstacle?

[Mikkel Fahnøe Jørgensen <[e-mail]>]

You could add some Magic(™) to Kubernetes ingress controllers and
Kubernetes services of kind Load Balancer.
These already do coordination with mapping internal IP’s to external IP’s
(or ought to do that, since it doesn’t always work, but I guess things will
mature).

For example, you have the name of a Kubernetes secret that is used to
configure the load balancer.

Of course, if you have high performance dedicated hardware that runs for
many customers concurrently, it might be an issue, but surely the ordinary
load balancers that take configurations can observe and alert dedicated
hardware of suspicious 5-tuples.

Outside of Kubernetes, various cloud providers have REST API’s for setting
TLS certificates etc, so I take it, a shared secret could also slip through
here.

I get it, that infrastructure folks don’t want to rewrite their load
balancers, but it is not like it is impossible by any stretch.

Mikkel

On 12 April 2019 at 18.08.09, Praveen Balasubramanian ([e-mail])
wrote:

It being optional helps in cases where the service being deployed and the
DoS protection service are under same administrative domain. So depending
on how DoS protection is done the extension could be turned on or off. The
problem is with the IaaS scenario if the extension is turned on by default
in some major web server. Now DDoS protection cannot be offered as a cloud
service without harming said traffic because it will be rate limited. And
rate limiting is strictly worse than blocking QUIC because performance will
suffer.



We will likely need some text in manageability draft which says that
turning on this extension may break network functions and cause performance
issues so services / implementations should only turn this on after these
considerations.



*From:* QUIC <[e-mail]> *On Behalf Of * Ian Swett
*Sent:* Friday, April 12, 2019 8:57 AM
*To:* Praveen Balasubramanian <pravb=[e-mail]>
*Cc:* Gorry (erg) <[e-mail]>; Roberto Peon <[e-mail]>;
Mirja Kuehlewind <[e-mail]>; Mikkel Fahnøe Jørgensen <
[e-mail]>; [e-mail]; Border, John <[e-mail]>
*Subject:* Re: Is "Version Greasing" a new benfit or a new obstacle?



Possibly I'm missing something, but wouldn't version greasing be an
optional extension?  So if you wanted to do so you could, but those with
load balancing that couldn't handle it would not?



On Fri, Apr 12, 2019 at 11:55 AM Praveen Balasubramanian <pravb=
[e-mail]> wrote:

Sorry for a typo. Plain old UDP not QUIC.



*From:* Praveen Balasubramanian
*Sent:* Friday, April 12, 2019 8:53 AM
*To:* Mikkel Fahnøe Jørgensen <[e-mail]>; Gorry (erg) <
[e-mail]>; Mirja Kuehlewind <[e-mail]>
*Cc:* [e-mail]; Border, John <[e-mail]>; Roberto Peon <
[e-mail]>
*Subject:* RE: Is "Version Greasing" a new benfit or a new obstacle?



Version greasing will create problems for DDoS protection. The DDoS
protection middlebox will have to treat all traffic with unknown version
numbers as plain old QUIC and rate limit it. Any solution that requires per
connection state to track version numbers doesn’t scale. Especially in the
cloud scenario customers bring their own web servers so this makes any
opt-out solution not good enough. So I do think this is an obstacle for
many deployments.



*From:* QUIC <[e-mail]> *On Behalf Of *Mikkel Fahnøe Jørgensen
*Sent:* Thursday, April 11, 2019 1:20 AM
*To:* Gorry (erg) <[e-mail]>; Mirja Kuehlewind <
[e-mail]>
*Cc:* [e-mail]; Border, John <[e-mail]>; Roberto Peon <
[e-mail]>
*Subject:* Re: Is "Version Greasing" a new benfit or a new obstacle?



In a not so distant future, a fallback to TCP would not be a viable
alternative for a number of services. Think WebRTC.

If QUIC and other encrypted protocols become dominant, just blocking
everything would literally block everything.

Of course, http(s) as we know it, it will continue to live on, but many
mobile apps would not.





On 11 April 2019 at 10.03.14, Mirja Kuehlewind (
[e-mail]) wrote:

Hi Mikkel,

I guess it is much more likely that any of those actors would just block
QUIC all together (as there is actually a fallback to TCP).

Mirja



On 10.04.19, 21:57, "QUIC on behalf of Mikkel Fahnøe Jørgensen" <
[e-mail] on behalf of [e-mail]> wrote:


Let’s say China makes a deal with Google to allow their search engine (a
not entirely unreasonably proposition).
The Great Firewall of China (GFC) is reconfigured to accept traffic from a
specific IP range for QUIC protocol v1.



Google later upgrades to v2. GFC is updated. Later, out of ignorance, some
other department within said company decides that its browsers should all
use forced version aliases as a rule. GFC breaks. (Assuming Googles servers
are not located within the demilitarised
zone and users use Chrome).


Alternative 1: Google up front decides to deploy forced version aliasing.
China rejects the deal because they do not want to support random traffic
through GFC.


Alternative 2: Google up front decides to deploy forced version aliasing,
and China does not reject the deal, out of ignorance. Later some department
in China GFC oversight realises the deal and forces Google to remove forced
version aliasing, or shut down
its service.


Alternative 3: Google up front decides to deploy forced version aliasing
and China says, of course, we trust everything from Googles IP range.


Alternative 4: Google realises that forced version aliasing does not work
in the general case, and makes a special case for users within a certain
geographical area.


Disclaimer: I’m not affiliated with Google and this might not at all be how
things would work, but it does highlight some of the challenges.


Of course, to be fair, China is not the only government entity that might
have a vested interest. Let's take a fictional example of US a real estate
investor with heavy ties to the eastern european block who somehow finds
himself elected as president needing
to find leverage on trade negotiations without effectively hurting personal
finances. It turns out that blocking certain digital services from said
eastern european block is the perfect tool in the trade negotiations..
Advisers point out that those services
run heavily encrypted on the dark web with perpetual circulation of IP
ranges so, ignoring any legal concerns, it would not be practically
possible. Eventually someone figures out that these services all usually
are at the forefront of technology and currently
use QUIC v3.2 and no-one else has deployed that version yet, so it would
suffice to ask NSA to tap into the backbone and disrupt specific packets.
Of course, this ends up taking down the Bavarian local government election
process in Germany where they are
the first to use a new digital election system. Not that the is an issue,
since trade negotiations are also running hot in that area, so that is just
an accidental bonus.


Or an endless number of other developments over the next few decades if the
past few years is anything to go by.





On 10 April 2019 at 20.13.59, Gorry (erg) ([e-mail]
<[e-mail]..ac.uk>) wrote:

Thanks Mikkel, I do understand that various actors intentionally drop - but
are you saying these actors would specifically choose to block a new
version of QUIC ... I do not understand that assertion.


Gorry

On 10 Apr 2019, at 18:25, Mikkel Fahnøe Jørgensen <[e-mail]>
wrote:



China blocks as a rule.
Russia is running an experiment to block the rest of the internet.
USA blocks net neutrality.
EU blocks cookies.
GB blocks itself.


So blocking is not limited to what an operator considers best for business.



On 10 April 2019 at 18.59.15, Border, John ([e-mail]) wrote:


I understand why people want to come down on the side of preventing
ossification. But, things have changed. There are a lot of more negative
consequences now when people unnecessarily block things. I think operators
would put a lot of pressure on vendors to
not do it now and to fix it if they did "by accident". Of course, I am only
one operator. It would be nice to hear from others...



John


-----Original Message-----
From: QUIC <[e-mail]> On Behalf Of Roberto Peon
Sent: Wednesday, April 10, 2019 12:52 PM
To: G Fairhurst <[e-mail]>;
[e-mail]
Subject: Re: Is "Version Greasing" a new benfit or a new obstacle?

WARNING: The sender of this email could not be validated and may not match
the person in the "From" field.

CAUTION: This email originated from outside of the organization. Do not
click links or open attachments unless you recognize the sender and know
the content is safe.


You're kinda between a rock-and-a-hard-place either way:

- We've seen how much fun ossification is in TCP and HTTP. If the thing is
observable, it will be ossified seems to be the lesson. A lot of the reason
why QUIC was started in the first place was because of the inability to
improve TCP due to this ossification.
- OTOH, there is the fear of unknown/unobservable which might cause
operators to block things, whether predictably or not.

My opinion is that it is better to start with preventing ossification, and
then if that results in too large a percentage of operators blocking
things, to re-evaluate.

My guesses:
IP+port tuples and traffic patterns are still observable (for better and
worse), which implies operators will still have significant tools for
managing traffic. I believe that these are acted on/matched (ML or not)
regardless of any other data presented. In
other words, I have a doubt that stating the version in an observable way
will prevent the use of such tools.

Most problems I've seen associated with implementations rather than
protocol versions (though when the latter happens it is pretty severe).. If
you believe this assertion, then acting on protocol version is less
interesting than attempting to act based on implementation
fingerprints.
-=R


On 4/10/19, 1:48 AM, "QUIC on behalf of G Fairhurst" <[e-mail]
on behalf of
[e-mail]..uk <[e-mail]>> wrote:

Obscuring the version of a protocol seems like a major design design
decision for wider use cases. So, I'm trying to understand the
motivation for version greasing.

(1) I know there were instances where some early versions of QUIC were
blocked due to an uninitentional matching of the header. Is there
evidence of intentional attempts to block updates to protocols?

(2) Thinking about operating a network that cares about user support and
protection from unwanted traffic, I would expect that there would be
cases where traffic pattern anomolies are found and the appropriate
thing would be to try to determine if a new protocol had been deployed
and monitor it, if not, then the next most obvious thing could be to
block all unexpected traffic, that seems like a decision to hide the
version could increase ossification for new versions in these cases..

(3) Similarly, if a threat is known to impact only a specific (older)
version, it would seem to motivate a drop of that traffic that seeks to
use that version, while still permitting other traffic. Forcing version
detection to use pattern matching/ML will lead to less predictable
outcomes, or blocking based on address, etc.

(4) This obfusticates the most basic piece of reporting information used
for support. It hides the extent of deployment of the current protocol
version and prevlance of old implementations.

(5) On the support, if a problem only emerges when a particular version
is used with a particular address, then this helps pinpoint the issues.
Matching client versions to servers is much more of an issue if the user
community uses a wide range of servers (less so, I expect for major
providers: google, facebook, etc, etc), but significant when there is a
use of a diverse set of external sites and sites with their own load
balancers, etc and a need to manage interactions with L2 services.

I am intersted in knowing if this is likely to benefit or be a new
obstacle?

Gorry