Re: [dispatch] A State Synchronization Working Group

[Hesham ElBakoury <[e-mail]>]

Hi Mike,
Will this new WG look into the following issues:

Time synchronization protocols. There are protocols that are newer than PTP
which may need to be considered. E.G.  HUYGENS from stanford [
https://www.usenix.org/system/files/conference/nsdi18/nsdi18-geng.pdf]

Consistency protocols. This is an on going area of research. On Oct 23 2023
EFPL researchers published the QuePaxa algorithm which is an asynchronous
consensus protocol that is on par with leader-based protocols [
https://dl.acm.org/doi/10.1145/3600006.3613150]

State replication protocols for things such as high speed packet processing
[https://arxiv.org/pdf/2309.14647]

Do you think we can start with IRTF before we go to IETF for
standardization?

Thanks
Hesham

On Wed, Oct 25, 2023, 3:05 PM Michael Toomim <[e-mail]> wrote:

> I would like to dispatch the idea of forming a general State
> Synchronization Working Group, to coordinate duplicate efforts in defining
> various state synchronization protocols across existing IETF working groups.
>
> Many networking protocols implement some form of state synchronization.
> At the application layer, SCIM, SIP, COAP, ALTO, NETCONF, JMAP, CalDAV, and
> IPP all define state synchronization protocols over HTTP.  Many non-IETF
> protocols also implement sync-over-HTTP for specific uses, such as WebSub
> (to synchronize blogs), ActivityPub (to synchronize social feeds), and
> Matrix (to synchronize chat).  Outside of HTTP, we do state synchronization
> in DNS, IMAP, LDAP, NFS, RSYNC, and Git.  Dropping to the lower-level OSI
> layers, we have protocols to synchronize IP routing tables, network
> reachability, and header compression dictionaries like ROHC, OSPFv2, IS-IS,
> and BGP.  State synchronization also comes up in distributed systems and
> databases, with algorithms such as OT and CRDT, and each system defines its
> own distinct protocol.
>
> We design these as separate protocols because the *general* version of the
> State Synchronization Problem has seemed too challenging to tackle.
> However, the last decade of research has brought breakthroughs in general
> state synchronization algorithms that allow (a) multiple editors, (b)
> editing arbitrary data types, (c) over a distributed network, (d) of
> arbitrary network topology, (e) experiencing arbitrary network partitions
> and delays, (f) with arbitrary merge resolution semantics—all with
> reasonable (and improving) performance.  These general algorithms bring up
> the possibility of designing general *protocols* for state sync.
>
> If IETF working groups could rely on general standards for state sync,
> they could eliminate redundant consensus and specification work, as well as
> implementation work, because implementers could re-use common libraries and
> algorithms instead of writing their own algorithms.  Furthermore, general
> libraries could afford investment in advanced sync features such as
> peer-to-peer networking, offline modes, delta-compression, merge
> resolution, consistency guarantees, and fast-replay; which applications
> might not implement on their own, but now could benefit from for free.
> This would improve networking robustness across the board.  Finally,
> interoperability would improve.  Network sysadmins, for example, could
> inspect and debug the state and history of their routers, emails, web
> applications, email, and file systems with intercompatible tools over a
> general protocol.
>
> This new working group would integrate research with practice by working
> in symbiosis with existing IETF Working Groups.  Our new group would gather
> experts and researchers in state synchronization and interface with
> individual Working Groups.  It would learn the needs of individual working
> groups, and produce general knowledge, protocols, and libraries in return.
>
> We have had initial success with this model in the informal Braid.org
> group, which was roughly modeled on an IETF Working Group, and has produced
> the Braid-HTTP internet draft that is coming up for discussion in HTTPbis.
> The Braid group has met every two weeks for 2.5 years on zoom, with average
> attendance of 4 or 5 people, and has produced a number of research results
> in generalizing performant state synchronization:
>
>   - Diamond-Types <https://github.com/josephg/diamond-types>: World's
> fastest P2P text editing CRDT
>   - List-CRDTs <https://github.com/josephg/reference-crdts>: First text
> editing CRDT with general pluggable merge semantics (video presentation
> <https://braid.org/video/https://invisiblecollege.s3-us-west-1.amazonaws.com/braid-meeting-10.mp4#305>
> )
>   - Antimatter: World's first history-pruning P2P text editing CRDT (draft
> <https://braid.org/antimatter>) (implementation
> <https://www.npmjs.com/package/@braid.org/antimatter>)
>   - Sync9 <https://braid.org/sync9>: First true replace text CRDT
>   - Portals <https://braid.org/meeting-62/portals>: Generalized patch
> semantics for moves, replaces, splices
>   - Time Machines <https://braid.org/drafts/time-machines>:
> Generalization of OT & CRDT theory
>   - Braid-HTTP
> <https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-braid-http>:
> Synchronization for HTTP
>
> We could imagine formalizing the Braid group into this State
> Synchronization group. It would look more broadly at synchronization than
> any individual application, and would produce specs that could be offered
> to other groups for standardization, similar to how the QUIC group seeded
> HTTP/3.
>
> I am very curious what the IETF thinks about this idea for a working
> group, and would be very grateful to hear thoughts and opinions from
> everyone.  Thank you very much!
>
> Michael
>
> [1] Relevant Braid-HTTP draft:
> https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-braid-http
> _______________________________________________
> dispatch mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/dispatch
>