Re: [Ohai] Proxying WebSockets and gRPC

> On Sep 26, 2023, at 2:54 PM, Eric Orth <ericorth@google.com> wrote:
> 
> Maybe it's my unfamiliarity with gRPC, but why wouldn't that simple REST-like API fit over "normal" OHTTP without the need for chunked OHTTP or additional extensive guidance? What is the gRPC layer adding that is both helpful for the interaction and that you can't just send as a simple request and response?

gRPC supports streams of messages, which is why I brought chunked OHTTP into the picture. If you don’t care about streams, then yes, you should in theory just be able to use OHTTP out of the box.

> 
> For the most part, I think we've avoided the need for guidance on adapting specific protocols to OHTTP because it's either obvious or it's not well suited to OHTTP.  If something like gRPC adds a bunch of stuff that doesn't obviously fit into OHTTP, then the answer is probably to not use gRPC and just extract out the requests and responses into a more pure OHTTP interaction.  Just like how OHTTP is not actually compatible with generic HTTP and you can't expect any arbitrary HTTP interaction to cleanly go over OHTTP without massaging it into OHTTP compatibility, we shouldn't expect other HTTP-like protocols to go cleanly over OHTTP either.

Anecdotally, what constitutes “a bunch of stuff that doesn’t obviously fit into OHTTP” is not obvious to everyone, which is partly why I raise this topic. I appreciate that they may be obvious to folks on this list, but I don’t think this list represents the audience of people who may want to actually use these technologies in practice.

Best,
Chris

> 
> On Tue, Sep 26, 2023 at 10:26 AM Christopher Wood <caw@heapingbits.net <mailto:caw@heapingbits.net>> wrote:
>> I agree with the technical points below around privacy, security, and cost, but I feel like the core question was missed. To help sharpen it, let’s ignore WebSockets for the time being and focus on gRPC.
>> 
>> First, I think we all understand that OHTTP targets a constrained set of HTTP use cases — we debated about the name for a while because of this very reason! That constraints are basically what Tommy described: there is no state across requests and there is no notion of a session. My claim is that gRPC can be used to build application APIs with these exact same semantics. Moreover, given that gRPC uses HTTP under the hood, it seems perfectly reasonable to use OHTTP to provide the desired privacy properties under these constraints.
>> 
>> To provide an example, imagine you were building a weather app. The app has one API — give me the forecast for a particular location (where location could be city, lat/lon coordinates, or whatever). We could build this API in a REST-like way by defining a weather resource, and then GETting the weather for a particular location, e.g., 
>> 
>>    Client->Server: GET /weather?location=newyorkcity
>>    Server->Client: 200 OK <weather>
>> 
>> We could also build this API in an RPC-like way by defining a procedure that computes the weather based on a location message and returns the result. Handwaving over the gRPC layer, interacting with this API using HTTP is then something like this:
>> 
>>    Client->Server: POST /weather <body includes protobuf of location message>
>>    Server->Client: 200 OK <body includes protobuf of weather forecast message>
>> 
>> These aren't the world’s greatest APIs, but hopefully they get the point across. Specifically, in both cases, requests to the app are stateless and, ideally, unlinkable. The app should not learn anything about the client’s identity (IP address) when servicing a request. There is no notion of a session.
>> 
>> One might reasonably use OHTTP to mask the client’s IP when interacting with the REST variant. It’s trivial to do and quite cheap. It’s something that has been deployed for a variety of similar use cases today, ranging from DNS to application-specific APIs [1]. I think it’s reasonable to also use OHTTP when interacting with the RPC API. Or, rather, I see no convincing technical reason why you would want to apply MASQUE here instead of OHTTP.
>> 
>> With this as motivating context, my open-ended question was basically how folks think about adding proxy support for these different types of application APIs. Taking a step further back, I think it might be useful if this community produced some sort of guidance on how to design APIs and choose the appropriate proxy technology for developers who write such apps, and for developers who maintain legacy apps. The folks on this list have no doubt thought about and had conversations with people about when and why we might use OHTTP vs MASQUE and how they fit into the application architecture, so perhaps we ought to try and write down some of those considerations for the benefit of those outside of this list.
>> 
>> Thoughts?
>> 
>> Best,
>> Chris
>> 
>> [1] https://flo.health/press-center/anonymous-mode-feature
>> 
>>> On Sep 25, 2023, at 11:29 PM, Tommy Pauly <tpauly=40apple.com@dmarc.ietf.org <mailto:40apple.com@dmarc.ietf.org>> wrote:
>>> 
>>> I agree with Martin that for something that has a generic stream like WebSocket or anything with multiple requests, you’re going to be more stateful, and you’re better off using a MASQUE-flavored proxy.
>>> 
>>> OHTTP is suitable for request-response pairs that benefit from being highly unlinkable, but still aren’t “sessions” where there is back-and-forth state.
>>> 
>>> For cases like the chunked OHTTP variant, we shouldn’t use that as a way to shoehorn in arbitrary streams, but to better accommodate cases where one of the messages is better processed in pieces. Supporting 1XX responses in one case, and the other (which I’m interested in) is when you have responses that may be short or very long, and in the long cases may be slow to generate and process.
>>> 
>>> Thanks,
>>> Tommy
>>> 
>>>> On Sep 25, 2023, at 7:36 PM, Martin Thomson <mt@lowentropy.net <mailto:mt@lowentropy.net>> wrote:
>>>> 
>>>> (Duplicate?)
>>>> 
>>>> Either way, I don't know if something like gRPC or thewebsocketprotocol makes sense to use here.  If you want crazy, then why not also allow the modern HTTP CONNECT variants?  Yes, you could, but the "should" question is probably a more important one.
>>>> 
>>>> Reasons that you might not want to use OHTTP for either of these:
>>>> 
>>>> 1. Replay risk.  OHTTP doesn't protect against replay attacks by a relay (or, if you are crazy enough to send something TLS early data toward the relay, arbitrary network attackers).  It is hard to reason about the replay risk with arbitrary protocols like these.  Maybe we could layer liveness checking on top, but is that really the direction we want to go in?
>>>> 
>>>> 2. Latency savings are marginal when costs can be amortized.  OHTTP does have lower overhead than something like MASQUE, but once the overheads in MASQUE are amortized over a longer-lived session, is this really that valuable?
>>>> 
>>>> 3. Cost is cost and I don't see how a MASQUE proxy is materially different than OHTTP in this case.  Maybe the proxy spends more time on crypto, but the trade-off is the replay attack risk.
>>>> 
>>>> gRPC is a little more complex than a simple one-in-one-out query, which suggests further issues with OHTTP on top of those.  Multiple requests mean that overheads build up and MASQUE starts looking even better for that.
>>>> 
>>>> Do you have something specific in mind?
>>>> 
>>>> On Tue, Sep 26, 2023, at 04:31, Christopher Wood wrote:
>>>>> Hi folks,
>>>>> 
>>>>> Thanks to the contributions of this group and the companion MASQUE 
>>>>> group, we now have a helpful suite of protocols for proxying 
>>>>> application traffic in a variety of scenarios. MASQUE gives us a 
>>>>> generic way to proxy connections to existing servers, and OHTTP gives 
>>>>> us an application-specific way to proxy transactional messages to 
>>>>> supporting applications. These two cover quite a lot of ground as 
>>>>> currently specified, allowing us to proxy browser traffic as well as 
>>>>> most HTTP-based API traffic.
>>>>> 
>>>>> In some of my discussions about "proxying all the things," two use 
>>>>> cases have emerged as somewhat common: proxying traffic for APIs built 
>>>>> on WebSocket and gRPC. This is unsurprising given the popularity of 
>>>>> these two technologies. It seems prudent that we have a reasonable 
>>>>> answer for those who want to support these use cases.
>>>>> 
>>>>> One straightforward answer is to just use MASQUE! This is a perfectly 
>>>>> fine approach if you have a proxy to use. However, perhaps MASQUE is a 
>>>>> bit too expensive -- for some definition of expensive -- for some 
>>>>> deployments, and OHTTP is a more preferred option.
>>>>> 
>>>>> Given that we now have chunked OHTTP as a proposal [1], I wonder what 
>>>>> folks think about using this variant to support WebSockets and gRPC. 
>>>>> gRPC supports transactional and streaming requests and responses, which 
>>>>> fits well with normal and chunked OHTTP, and if you squint at 
>>>>> WebSockets you might view it as a long-standing request and response 
>>>>> stream. (I'm ignoring for the moment that gRPC is heavily coupled to 
>>>>> HTTP/2. Since things like gRPC-web exist perhaps there is a way of 
>>>>> avoiding that coupling.)
>>>>> 
>>>>> Has anyone else encountered these use cases, and if so, what do you 
>>>>> think about the various proxying options available?
>>>>> 
>>>>> Best,
>>>>> Chris
>>>>> 
>>>>> [1] https://datatracker.ietf.org/doc/draft-ohai-chunked-ohttp/
>>>>> -- 
>>>>> Ohai mailing list
>>>>> Ohai@ietf.org <mailto:Ohai@ietf.org>
>>>>> https://www.ietf.org/mailman/listinfo/ohai
>>>> 
>>>> -- 
>>>> Ohai mailing list
>>>> Ohai@ietf.org <mailto:Ohai@ietf.org>
>>>> https://www.ietf.org/mailman/listinfo/ohai
>>> 
>>> -- 
>>> Ohai mailing list
>>> Ohai@ietf.org <mailto:Ohai@ietf.org>
>>> https://www.ietf.org/mailman/listinfo/ohai
>> 
>> -- 
>> Ohai mailing list
>> Ohai@ietf.org <mailto:Ohai@ietf.org>
>> https://www.ietf.org/mailman/listinfo/ohai