---
title: A comparison of JavaScript CRDTs
status: draft
tags: crdts, umap, sync
display_toc: true
---
This is not done yet, TODO :
- [x] Change the table to be more readable, especially on mobile
- [x] Update the examples with JSON Joy to have a working example
- [x] Change the layout to be easier to understand.
- [ ] Enhance graphics to be easier to understand
- [ ] Part 3: Key takeaways
---
[TOC]
---
Collaboration is one of the most requested features on uMap since a long time. We've added a way to merge same-layers edit, but ideally we would love to make things easier to understand and more fluid for the users.
For this reason, I got more into CRDTs, with the goal of understanding how they work, what are the different libraries out there, and which one would be a good fit for us, if any.
So far, [the way I've though about collaboration features on uMap](https://blog.notmyidea.org/adding-collaboration-on-umap-third-update.html) is by:
- a) catching when changes are done on the interface ;
- b) sending messages to the other party and ;
- c) applying the changes on the receiving client.
This works well in general, but it doesn't take care of conflicts handling, especially when a disconnect can happen.
## Part 1 - What are CRDTs?
Conflict-free Resolution Data Types (CRDTs) are as a specific datatype able to merge its state with other states without generating conflicts. They handle consistency in distributed systems, making them particularly well-suited for collaborative real-time applications.
CRDTs ensure that multiple participants can make changes without strict coordination, and all replicas converge to the same state upon synchronization, without conflicts.
"Append-only sets" are probably one of the most common type of CRDT: if multiple parties add the same element, it will be present only once. It's our old friend `Set`.
### Why using them?
For uMap, CRDTs offer a solution to several challenges:
1. **Simultaneous Editing**: When multiple users interact with the same map, their changes must not only be reflected in real-time but also merged seamlessly without overwriting each other's contributions.
2. **Network Latency and Partition**: uMap operates over networks that can experience delays or temporary outages. CRDTs can handle these conditions gracefully, enabling offline editing and eventual consistency.
3. **Simplified Conflict Resolution**: Traditional methods often require complex algorithms to resolve conflicts, while CRDTs inherently minimize the occurrence of conflicts altogether.
4. **Decentralization**: While uMap currently relies on a central server, adopting CRDTs could pave the way for a more decentralized architecture, increasing resilience and scalability.
### Comparing with traditional data synchronization methods
Traditional data synchronization methods typically rely on a central source of truth, such as a server, to manage and resolve conflicts. When changes are made by different users, these traditional systems require a round-trip to the server for conflict resolution and thus can be slow or inadequate for real-time collaboration.
In contrast, CRDTs leverage mathematical properties (the fact that the datatypes can converge) to ensure that every replica independently reaches the same state, without the need for a central authority, thus minimizing the amount of coordination and communication needed between nodes.
This ability to maintain consistency sets CRDTs apart from conventional synchronization approaches and makes them particularly valuable for the development of collaborative tools like uMap, where real-time updates and reliability are important.
### Last Write Wins Registers
For managing key/value data, I'm leaning onto Last-Write-Wins (LWW) registers within CRDTs. With LWW, the main concern is establishing the sequence of updates. In a single-client scenario or with a central time reference, sequencing is straightforward. However, in a distributed environment, time discrepancies across peers can complicate things, as clocks may drift and lose synchronization.
To address this, CRDTs use vector clocks — a specialized data structure that helps to solve the relative timing of events across distributed systems and pinpoint any inconsistencies.
> A vector clock is a data structure used for determining the partial ordering of events in a distributed system and detecting causality violations.
>
> – [Wikipedia](https://en.wikipedia.org/wiki/Vector_clock)
At first, I found CRDTs somewhat confusing, owing to their role in addressing complex challenges. CRDTs come in various forms, with much of their intricacy tied to resolving content conflicts within textual data or elaborate hierarchical structures. Fortunately for us, our use case is comparatively straightforward.
Note that we could also use a library such as [rxdb](https://github.com/pubkey/rxdb) — to handle the syncing, offline, etc — because we have a master: we use the server, and we can use it to handle the merge conflicts. But by doing so, we also give more responsibility to the server, whereas when using CRDTs it's possible to do the merge only on the clients.
### Different types of CRDTs
While reading the literature, I found that there are two kinds of CRDTs: state-based and operation-based. So, what do we need ?
It turns out most of the CRDTs implementation I looked at are operation-based, and propose an
API to interact with them as you're changing the state, so **it doesn't really matter**.
> The two alternatives are theoretically equivalent, as each can emulate the
> other. However, there are practical differences. State-based CRDTs are
> often simpler to design and to implement; their only requirement from the
> communication substrate is some kind of gossip protocol. **Their drawback is that
> the entire state of every CRDT must be transmitted eventually to every other
> replica, which may be costly**. In contrast, operation-based CRDTs transmit only
> the update operations, which are typically small.
>
> [Wikipedia on CRDTs](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type)
### How the server fits in the picture
While discussing with the automerge team, I understood that I was expecting the server to pass along the messages to the other parties, and that would be the way the synchronisation would be done. It turns out I was mistaken: in this approach, the clients send updates to the server, which merges everything together and only then sends the updates to the other peers.
In order to have peers working with each other, I would need to change the way the provider works, so we can have the server be "brainless" and just relay the messages.
For automerge, it would mean the provider will "just" handle the websocket connection (disconnect and reconnect) and all the peers would be able to talk with each other. The other solution for us would be to have the merge algorithm working on the server side, which comes with upsides (no need to find when the document should be saved by the client to the server) and downsides (it takes some cpu and memory to run the CRDTs on the server)
---
## Part 2: making a demo with different libraries
Now that we're familiar with CRDTs and how they can help us, let's create a map application which syncs marker positions, on different clients.
We'll be comparing three JavaScript libraries: [Y.js](https://yjs.dev/), [Automerge](https://automerge.org/) and [JSON Joy](https://jsonjoy.com), considering:
1. **Efficiency**: Probe the bandwidth when doing edits. What's being transmitted over the wire?
2. **API**: is it easy to use for our use case? What are the challenging parts for us?
3. **Community and Support**: How is the size and activity of the developer community / ecosystem?
4. **Size** of the JavaScript library, because we want to limit the impact on our users browsers.
### A leaflet map
All the demos are made agains the same application, which creates markers when the map is clicked, and move the markers on hover.
Here's the whole code for this:
```js
import L from "leaflet";
import "leaflet/dist/leaflet.css";
// Create a map with a default tilelayer.
const map = L.map("map").setView([48.1173, -1.6778], 13);
L.tileLayer("https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png", {
maxZoom: 19,
attribution: "© OpenStreetMap contributors",
}).addTo(map);
// Features contains a reference to the marker objects, mapped by the uuids
let features = {};
// An upsert function
function upsertMarker({ latlng, uuid, local = false }) {
if (!uuid) uuid = crypto.randomUUID();
let marker;
if (Object.keys(features).includes(uuid)) {
marker = features[uuid];
marker.setLatLng(latlng);
} else {
marker = new L.marker(latlng, {
draggable: true,
autoPan: true,
uuid: uuid,
});
features[uuid] = marker;
marker.addTo(map);
marker.on("dragend", ({ target }) => {
console.log("dragend");
});
}
if (!local) {
console.log()
}
}
// Add new features to the map with a click
map.on("click", ({ latlng }) => {
upsertMarker({ latlng });
});
```
Now, let's add synchronization.
### Y.js
Y.js is the first library I've looked at, because it's the oldest one, and the more commonly referred to.
The API seem to offer what we look for, and provides a way to [observe changes](https://docs.yjs.dev/api/shared-types/y.map#observing-changes-y.mapevent). Here's what I did:
```js
import * as Y from "yjs";
import { WebsocketProvider } from "y-websocket";
// Instanciate a document
const doc = new Y.Doc();
```
To make changes to the CRDT:
```js
let markers = doc.getMap("markers");
markers.set(target.options.uuid, target._latlng);
```
To observe the changes:
```js
markers.observe((event, transaction) => {
if (!transaction.local) {
event.changes.keys.forEach((change, key) => {
let value = markers.get(key);
if (change.action === "add") {
upsertMarker({ latlng: value, uuid: key, local: true });
} else if (change.action === "update") {
upsertMarker({ latlng: value, uuid: key, local: true });
} else if (change.action === "delete") {
console.log(`Property "${key}" was deleted. ".`);
}
});
}
});
```
It comes with multiple "providers", which make it possible to sync with different protocols (there is even a way to sync over the matrix protocol 😇). More usefully for us, there is [an implemented protocol for websockets](https://github.com/yjs/y-websocket).
Using a provider is as easy as:
```js
// Sync clients with the y-websocket provider
const provider = new WebsocketProvider(
"ws://localhost:1234",
"leaflet-sync",
doc
);
```
It's also possible to send "awareness" information (some state you don't want to persist, like the position of the cursor). It contains some useful meta information, such as the number of connected peers.
```js
map.on("mousemove", ({ latlng }) => {
awareness.setLocalStateField("user", {
cursor: latlng,
});
});
```
I made [a quick proof of concept with Y.js](https://gitlab.com/almet/leaflet-sync/-/tree/yjs) in a few hours flawlessly. It handles offline and reconnects, and exposes awareness information.
#### Python bindings
Y.js has been rewritten in rust, with [the Y.rs project](https://github.com/y-crdt/y-crdt), which makes it possible to use with Python (see [Y.py](https://github.com/y-crdt/ypy)) if needed. The project has been implemented quite recently and is currently looking for a maintainer.
#### Library size
Size: Y.js is 4,16 Ko, Y-Websocket is 21,14 Ko
The library is currently used in production for large projects such as AFFiNE and Evernote.
#### The data being transmitted
In the scenario where all clients connect to a central server, which handle the CRDT locally and then transmits back to other parties, I found that adding 20 points on one client, and then 20 points in another generates ~5 ko of data (~16 bytes per edit).
| Pros | Cons |
| -------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| the API was feeling natural to me: it handles plain old JavaScript objects, making it easy to integrate. | It doesn't seem to work well [without a JS bundler](https://github.com/yjs/yjs/issues/325) which could be a problem for us. |
| It seems to be widely used, and the community seems active. | |
| It is [well documented](https://docs.yjs.dev/) | |
| There is awareness support | |
---
### Automerge
[Automerge](https://automerge.org/) is another library to handle CRDTs. Automerge is actually the low level interface, and there is a higher-level interface exposed as [Automerge-repo](https://automerge.org/docs/repositories/). Here is how to use it:
```js
import { Repo } from "@automerge/automerge-repo";
const repo = new Repo();
let handle = repo.create() // or repo.find(name)
```
When you change the document, you actually call `change` which makes it possible to do the changes in a kind of "transaction function".
```js
handle.change((doc) => {
doc[uuid] = cleanLatLng(target._latlng);
});
```
Note that I had to use a `cleanLatLng` function in order to not pass the whole object, otherwise it wouldn't be serialized. It's really just a simple helper taking the properties of interest for us (and letting away all the rest).
You can observe the changes, getting you the whole list of patches:
```js
handle.on("change", ({ doc, patches }) => {
console.log(patches, doc);
patches.forEach(({ action, path }) => {
// We have to know the specifics of the patch operations getting passed
// two items in the path means we're inserting an object
if (path.length == 2 && action === "insert") {
let value = doc[path[0]];
upsertMarker({ latlng: value, uuid: path[0], local: true });
}
});
});
```
#### Python bindings
There is an [automerge.py](https://github.com/automerge/automerge-py) project, but no changes has been made to it since 3 years ago. There are plans to update it, though.
#### Library size
Size: 1,64 mb, total is 1,74 mb. It's relying on Web assembly by default.
The large bundle size is something that the team is aware of, and are working on a solution for. For us, it's important to have something as lightweight as possible, considering CRDTs is only one part of what we're doing, and that mapping can be done in context where connection is not that reliable and fast.
#### The data being transmitted
In the same scenario, I found that adding 20 points on one client, and then 20 points in another generates 90 messages and 24,94 Ko of data transmitted (~12 Ko sent and ~12Ko received), so approximately 75 bytes per edit.
| Pros | Cons |
| -------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| There is an API to [get informed when a conflict occured](https://automerge.org/docs/documents/conflicts/)
| Documentation was a bit hard to understand and to look at. Sometimes, it's easier to go look at the code.
|
| In general, the documentation is low level, which can be a good thing while debuging, or when getting more advanced usage. | The API is more verbose. You can see it as "less magical". |
| The team was responsive and trying to help. | There is no way (at the moment) to tell that a transaction is local or remote (but in practice it wasn't a problem) |
---
### JSON Joy
[Json Joy](https://jsonjoy.com) is the latest to the party.
It takes another stake by providing multiple libraries with a small functional perimeter. It sounds promising, even if still quite new, and would left us with the hands free in order to implement the protocol that would work for us.
```js
import {Model} from 'json-joy/es2020/json-crdt';
import { s } from "json-joy/es6/json-crdt-patch";
// Initiate a model with a custom ID
const rootModel = Model.withLogicalClock(11111111);
// populate it with default data
rootModel.api.root({
markers: {},
});
// Fork it on each client
let userModel = rootModel.fork();
```
Making changes to the model. Here we are changing a constant for another one, by using `s.con`.
```js
userModel.api.obj(["markers"]).set({
[uuid]: s.con(target._latlng),
});
```
Creating a patch, before sending it to the other parties:
```js
import { encode, decode } from "json-joy/es6/json-crdt-patch/codec/verbose";
let patch = userModel.api.flush();
let payload = encode(patch);
```
When receiving a message, decode it and apply it:
```js
let patch = decode(payload);
model.api.apply(patch);
```
We can observe the changes this way. Here, we're having a look at the operation that happened and acting on it. The names of the operations aren't clearly specified by the spec. It seems a bit sketchy, so I'm not sure it's the way to handle this, but it works.
```js
userModel.api.onPatch.listen((patch) => {
patch.ops.forEach((op) => {
if (op.name() === "ins_obj") {
let key = op.data[0][0];
let value = userModel.view().markers[key];
upsertMarker({ latlng: value, uuid: key, local: true });
}
});
});
```
Metrics:
- Size: 143 ko
- Data transmitted for 2 peers and 40 edits: (35 bytes per edit)
| Pros | Cons |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| Low level, so you know what you're doing | It doesn't provide a high level interface for sync
|
| Small atomic libraries, making it easy to use only the parts we need. | It's currently a one-person project, without clear community channels to gather with other interested folks. |
| The interface [proposes to store different type of data](https://jsonjoy.com/libs/json-joy-js/json-crdt/guide/node-types) (constants, values, arrays, etc) | Quite recent, so probably rough spots are to be found |
| Distributed as different type of JS bundles (modules, wasm, etc) |
---
## Part 3: Key takeaways
Y.js:
- **JavaScript objects API**: uses native JS objects, making it easier to integrate with existing applications (but can also transport metadata we don't really want)
- **WebSockets**: a "connection provider" exists for WebSockets, handling offline changes, reconnections, etc. The currently implemented provider considers the server as a central node, which also maintains a CRDT locally.
- **Awareness API**: a protocol defined to send awareness information (who is connected, and for instance the cursor position)
- Versioning and history: Supports selective versioning through snapshots.
- Community and support: Active community with regular updates.
- Library Size / Network usage: Small size with efficient default compression of the data.
Suitability for uMap: Ready-to-use with good documentation and examples, might do more than what we need, some bundler dependencies.
Automerge:
- **Transactional API**: API is transactional (edits should happen in a function), making it obvious, but can also be harder to implement.
- **WebSockets**: Multiple options with providers and storage, including WebSockets. It works similarly to the y.js one, with a copy of the data living on the server. There is currently no detection of offline / retry on disconnect, or awareness API.
- **Conflict Resolution**: [A conflict detection API](https://automerge.org/docs/documents/conflicts/) exists, which make it easy to get the conflicting values, when it happens.
- **Offline Changes Handling**: Requires a more manual approach to handle offline changes.
- Versioning and History: Designed with robust version history tracking.
- **Community and Support**: Strong support with a focus on collaboration.
- **Library Size / Efficiency**: Larger library with dependency on WebAssembly.
JSON Joy:
- **Lower-level API**: A low-level API offers granular control, it's more hands-on, but less magic.
- **WebSockets**: Nothing is provided in this area, and it would need to be implemented by ourselves.
- **Community and Support**: there is only one maintainer as of now, and the community doesn't exist yet.
- **Browser Compatibility**: Flexible bundles for diverse browser support, which might be helpful in our case.
## Extra notes
### YATA and RGA
While researching, I found that the two popular CRDTs implementation out there use different approaches for the virtual counter:
> - **RGA** [used by Automerge] maintains a single globally incremented counter (which can be ordinary integer value), that's updated anytime we detect that remote insert has an id
> with sequence number higher that local counter. Therefore every time, we produce
> a new insert operation, we give it a highest counter value known at the time.
> - **YATA** [used by Yjs] also uses a single integer value, however unlike in case of RGA we
> don't use a single counter shared with other replicas, but rather let each
> peer keep its own, which is incremented monotonically only by that peer. Since
> increments are monotonic, we can also use them to detect missing operations eg.
> updates marked as A:1 and A:3 imply, that there must be another (potentially
> missing) update A:2.
### Resources
- [CRDTs: The Hard Parts](https://www.youtube.com/watch?v=x7drE24geUw), a video by Martin Kleppmann where he explains the current state of the art of CRDTs, and why some problems aren't solved yet.
- [An Interactive Intro to CRDTs](https://jakelazaroff.com/words/an-interactive-intro-to-crdts/) gets you trough different steps to understand what are CRDTs, and how to implement a LWW Register.
- [Bartosz Sypytkowski](https://www.bartoszsypytkowski.com/the-state-of-a-state-based-crdts/) introduction on CRDTs, with practical exemples is very intuitive.
- [CRDT Implementations](https://jzhao.xyz/thoughts/CRDT-Implementations) contains nice information and vocabulary useful when working with CRDTs.