---
title: Adding collaboration on uMap, fourth update
tags: umap, geojson, websockets
status: draft
---

Since last week, the main branch of uMap ships a web socket server, and the
ability to make peers communicate with each other, replicating local changes to
other peers.

Here is a demonstration of the "real-time" import of some data on one peer, with
its replication on another peer.

<video width="100%" controls src="https://files.notmyidea.org/choropleth-sync.webm">
</video>

Our roadmap to having "real-time" collaboration looks like this:

1. ☑︎ Learn about CRDTs, what they are and find how they can be useful;
2. ☑︎ Make structural changes to the codebase; 
3. ☑︎ Allow a replication of a peer state to another connected peer;
4. Sync with peers late to the party ;
5. Handle UX and disconnects.
6. Scale.
## The path behind us

Over the past few months, we did the following changes in the uMap codebase:

First, we [replaced datalayers ids with uuids](https://github.com/umap-project/umap/pull/1630), making it possible to have them generated by the clients in the long term. Then, we [assigned semi-unique ids to each map "features"](https://github.com/umap-project/umap/pull/1649), making it possible to refer to them over different peers. That makes it possible to know which "marker" has moved, for instance.

We then introduced a change in how the data flows, by [separating the UI rendering from data updates](https://github.com/umap-project/umap/pull/1692). With this change, we introduced a way to regenerate only the parts that matters when updating a value. This enabled us to [apply data updates to and from other peers over a web socket connection](https://github.com/umap-project/umap/pull/1754).
## What's still to come

That's a huge step in the right direction, but, that's not the end of the game yet. We still need to do some work in order to finish what's been started. 

The current effort is on applying pre-join changes: with the currently shipped code, you only get the changes while you're connected: changes that happened before you join aren't there. As a result, peers can show diverging maps. We're currently considering using [Hybrid Logical Clocks](https://sergeiturukin.com/2017/06/26/hybrid-logical-clocks.html) (HLC) to reapply the operations in a predictable order on the different peers.

Basically, we'll need to:

- Assign each operation an `id` and store them locally ;
- Find a way for the peers to ask another peer about the missing changes since a specific date ;
- Get these changes and reapply them locally.

HLCs will help us avoid clock drift between the different peers, and ensure the operations are applied in the rough same order.

Once that's covered, we'll still need to handle these:

- **Interface changes**: we have some mockups to integrate, making it obvious other peers are connected and interacting on the map, and handle web socket disconnects.
- **Security**: making peers communicate with each other enables new data flows, and with them new "attack vectors". We want to take the time to think this trough, and cover some of the problems we are already envisioning about handling permissions, and escalation trough a peer with greater privileges.
- **Scaling things up**: uMap is used to serve a quite large number of maps, and probably will need to make sure our changes are able to scale. This will potentially require some structural changes on the web socket server, because not everything might fit in memory anymore.