Re: Mnot's Pub/Sub for the Web

[Kévin Dunglas <[e-mail]>]

Thanks for bringing this topic to the list again!

On the Mercure side, the spec has stabilized. Several open-source and
proprietary implementations are available (
https://mercure.rocks/spec#implementation-status), and adoption is growing:
2.7K stars on GitHub, dozens of open source projects using it, large
companies publicly declaring use...

Many new use cases have been reported on the bug tracker over the years,
and we improved the spec to cover most of them. Some minor issues still
need to be handled (https://github.com/dunglas/mercure/labels/spec), but
we're very soon to publish the final version of the specification.

As demonstrated by the discussions on Hacker News, Mark's great article,
and by the adoption of Mercure, the community is in demand of a pub/sub
standard for web resources.

Even if most discussions occurred on GitHub, Slack, Twitter, and other
channels instead of on IETF mailing lists, the Mercure spec is now
implemented by production-grade "running code" and has reached "rough
consensus".

Mercure is less ambitious than Braid. Its scope is more limited. It is
focusing on providing a simple pub/sub protocol for web content proved
working with the current web infrastructure (web browsers, proxies,,
firewalls, etc). In its current state, it doesn't require any JS library or
polyfill client-side.

The spec is very similar to the WebSub specification from the W3C, but
mainly targets web browsers instead of servers. As WebSub, Mercure uses a
hub to distribute web resources, which allows implementing the protocol
easily even in legacy applications, with languages not designed to handle
long-living connections (e.g. PHP), and when using modern infrastructure
such as serverless and edge computing platforms (
https://dunglas.fr/2019/07/mercure-real-time-apis-for-serverless-and-beyond/).
Unlike WebSub, Mercure natively supports authorization, end-to-end
encryption, and state reconciliation. Both clients and servers can be
publishers.

Currently, Mercure only allows using SSE as transport, but we'll maybe
allow using other transports such as WebSockets and Web Transports,
probably as extensions to the current spec, to cover use case such as
transmitting non-base64-encoded binary data (
https://github.com/dunglas/mercure/issues/616).

Braid is very interesting and has a much broader scope (state
synchronization, P2P, etc). It also requires more changes to the current
software stack to be natively supported by the web platform. Mercure
overlaps only with the "subscribe" feature of Braid, and I've the feeling
than Braid could use Mercure (and probably WebSub too) for its subscribe
feature, at least in a first iteration.

I wonder how we can move forward regarding the standardization of a pub/sub
protocol for web content and web browsers. Even if Mercure gained traction
outside of the IETF, it hasn't on this group. I was thinking about
proposing the final version of the spec as an independent-track RFC, or to
the W3C as it is very close to WebSub, and is also related to the other
specs published by the Social Web Working Group (ActivityPub, and even
Solid). But as the this topic is discussed again, maybe could we work on a
pub/sub protocol here?



On Sun, Feb 20, 2022 at 10:39 AM Michael Toomim <[e-mail]> wrote:

> Hello, HTTP!
>
> Today Mark Nottingham posted a great articulation of the issues
> programmers face when choosing between using SSE, WebSockets, and
> WebTransports:
>
> https://www.mnot.net/blog/2022/02/20/websockets
>
> I'll attempt to summarize Mark's beautiful insight as: in almost all
> cases, what the programmer *really* wants is a Pub/Sub protocol, not an
> arbitrary socket. And we could standardize a Pub/Sub protocol, and that
> would have great benefits.
>
> These benefits are real and I think could improve performance
> dramatically. CDNs could cache realtime updates, not just static data.
>
> However, I'll take Mnot one further, and propose that when a programmer is
> choosing a Pub/Sub protocol, what he *really* wants is a State
> Synchronization protocol, not an arbitrary Pub/Sub protocol.
>
> He wants to Subscribe specifically to *state updates*. He wants to Publish
> specifically *updates to state*.
>
> What we need is not a general Pub/Sub standard, but specifically a State
> Synchronization standard. State Synchronization is a constrained type of
> general Pub/Sub. And we'll need to constrain Pub/Sub in this way to address
> some of the issues Mark brings up, such as:
>
> > There are also some architectural/philosophical concerns about how
> non-final responses **relate to the state of the resource**.
>
> The relationship between a server's "responses" and the "state of the
> resource" is what a State Synchronization protocol defines. And, in fact,
> we have two proposed solutions to State Synchronization in the IETF!
>
> Braid:
> https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-braid-http
> Mercure:    https://datatracker.ietf.org/doc/draft-dunglas-mercure/
>
> I am seeing a growing awareness that HTTP needs to add State
> Synchronization abilities, as well as excitement about the new fundamental
> power it gives programmers on the web.
>
> These protocols transform HTTP from a State *Transfer* into a State
> *Synchronization* protocol. Whereas a transfer protocol can move a resource
> from server to client in a single request/response, it requires an
> application programmer to take over if the resource ever changes after the
> response completes. That sucks for programmers. A synchronization protocol
> provides a much better programming abstraction. The programmer just says "I
> want state X", and can assume it will be kept up-to-date by the protocol.
>
> If we standardize this, we also get CDNs that automatically cache dynamic
> content (the stuff currently hidden within websockets), just as easily as
> they cache static content today. We get collaborative editing and offline
> modes available in web apps for free. We also take an important step
> towards decentralizing the web, by creating an open standard for the
> trickiest part of decentralized app development — data synchronization —
> that is compatible with P2P CRDT and OT algorithms.
>
> Since this all seems to be coming together, I would like to know what
> HTTPbis as a group thinks. Is there interest in this topic?
>
> If so, what aspects might we want to work on?
>