Sync engine & real-time data management #4

New Issue

2026-02-28T17:10:50Z

talksik commented

2026-02-28 17:10:50 +00:00

(Migrated from github.com)

Requirements

Product shows realtime data for a multiplayer experience
Optimistic updates make product interactions feel instantaneous
If the client goes offline, it must either re-bootstrap, or ask for all events since x change
We should keep all data synced real-time on the client, not only particles. For example, billing, networks, members.

Not doing

No offline mode. Client cannot write data without being connected, and there will be no local cache to load data offline on boot.

However, client should receive replay of all transactions from the sync server on reconnect.

Components

There are a few core components for building the sync engine:

Backend services (e.g. particle service) appends changes to the transaction log. No need for pubsub to another worker which then would add to transaction log.
Sync server (web socket) processes new transaction log events and forwards to the appropriate subscribers / clients
Client side object pool
Client side object graph to make it easier for the view layer

Flow

Client fetches snapshot data on load and receives a sequence number of the snapshot
Client connects to sync server via web sockets and passes in the sequence number to receive all missed events during snapshot
Client keeps in-memory data up to date with snapshot data

For writes:

On user interaction, the local store is updated as an optimistic update
API request is sent to orion server
- On failure, revert + toast
- On success, do nothing

Client state management

On the client side, we want to either use MobX or Zustand. All data should reside in a client-side object pool.

And then particular views can construct derived state based on the object pool.

## Requirements - Product shows realtime data for a multiplayer experience - Optimistic updates make product interactions feel instantaneous - If the client goes offline, it must either re-bootstrap, or ask for all events since x change - We should keep all data synced real-time on the client, not only particles. For example, billing, networks, members. ## Not doing No offline mode. Client cannot write data without being connected, and there will be no local cache to load data offline on boot. However, client should receive replay of all transactions from the sync server on reconnect. ## Components There are a few core components for building the sync engine: - Backend services (e.g. particle service) appends changes to the transaction log. No need for pubsub to another worker which then would add to transaction log. - Sync server (web socket) processes new transaction log events and forwards to the appropriate subscribers / clients - Client side object pool - Client side object graph to make it easier for the view layer ## Flow - Client fetches snapshot data on load and receives a sequence number of the snapshot - Client connects to sync server via web sockets and passes in the sequence number to receive all missed events during snapshot - Client keeps in-memory data up to date with snapshot data For writes: - On user interaction, the local store is updated as an optimistic update - API request is sent to orion server - On failure, revert + toast - On success, do nothing ## Client state management On the client side, we want to either use MobX or Zustand. All data should reside in a client-side object pool. And then particular views can construct derived state based on the object pool.

talksik commented

2026-03-12 19:55:38 +00:00

(Migrated from github.com)

Note to self: many of these sync engine abstractions end up being overly complex and create leaky abstractions.

It's easier to build our own system that we own and understand. Start simple and expand over time.

Note to self: many of these sync engine abstractions end up being overly complex and create leaky abstractions. It's easier to build our own system that we own and understand. Start simple and expand over time.

talksik commented

2026-03-14 15:17:43 +00:00

(Migrated from github.com)

As advised here, there are various loopholes with local-first.

We will not be taking on this work for now:

Local first is equivalent to dealing with distributed systems
Migrations are harder because the client and server have to be updated
More complexity pushed onto the client, and we rather write more complexity in our backend
We just don't need offline-first at this stage in the game. We can still be more performant than M365 with a client/server model. Having a lean client and performant server accomplishes the same in terms of performance
We can still do optimistic updates
We barely have that many entities so this abstraction is not warranted
We barely have many mutation endpoints, and they can be maintained easily with a few functions
We can always build it later after we have built a business.

Instead, for now, we will focus on creating a real-time experience that is more conventional similar to the following:

Individual API calls depending on the view and data required for rendering (e.g. structured data and caching with react query)
User takes action
Optimistic update
Mutation API call
Invalid cache OR refetch relevant data to keep state up to date
Websocket connection to hear about multi-player updates and update state
On network offline, the application will be unusable, and on reconnect, we can invalidate all caches to trigger refetches

To keep it simpler, we can even group data together in react query; for example, one react query hook can fetch 3 endpoints but represent one key, and invalidations would refetch entire sets of data. Particular views can reuse this cached data and filter appropriately.

As advised [here](https://rxdb.info/articles/local-first-future.html#challenges-and-limitations-of-local-first), there are various loopholes with local-first. We will not be taking on this work for now: - Local first is equivalent to dealing with distributed systems - Migrations are harder because the client and server have to be updated - More complexity pushed onto the client, and we rather write more complexity in our backend - We just don't need offline-first at this stage in the game. We can still be more performant than M365 with a client/server model. Having a lean client and performant server accomplishes the same in terms of performance - We can still do optimistic updates - We barely have that many entities so this abstraction is not warranted - We barely have many mutation endpoints, and they can be maintained easily with a few functions - _We can always build it later after we have built a business._ Instead, for now, we will focus on creating a real-time experience that is more conventional similar to the following: - Individual API calls depending on the view and data required for rendering (e.g. structured data and caching with react query) - User takes action - Optimistic update - Mutation API call - Invalid cache OR refetch relevant data to keep state up to date - Websocket connection to hear about multi-player updates and update state - On network offline, the application will be unusable, and on reconnect, we can invalidate all caches to trigger refetches To keep it simpler, we can even group data together in react query; for example, one react query hook can fetch 3 endpoints but represent one key, and invalidations would refetch entire sets of data. Particular views can reuse this cached data and filter appropriately.

talksik commented

2026-03-16 19:03:40 +00:00

(Migrated from github.com)

We will setup firestore with our gcp project to launch a real-time multiplayer app sooner.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: FlowyLabs/llink#4