almet/blog.notmyidea.org
1
0
Fork 0
mirror of https://github.com/almet/notmyidea.git synced 2025-04-28 11:32:39 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

Umap update 3

Browse source
This commit is contained in:
Alexis Métaireau 2024-02-15 11:36:17 +01:00
parent 21ad8deb4b
commit 9b117a8e75
2 changed files with 311 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

16
content/code/2024-02-08-arrow-functions-returns.md Normal file
View file

@ -0,0 +1,16 @@
---
title: Returning objects from an arrow function
tags: Javascript
---
When using an arrow function in JavaScript, I was expecting to be able to return objects, but ended up with returning `undefined` values.
Turns out it's not possible to return directly objects from inside the arrow function because they're confused as statements.
This is [covered by MDN](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Arrow_functions#function_body).
To return an object, I had to **put it inside parenthesis, like this**:
```js
latlngs.map(({ lat, lng }) => ({ lat, lng }))
```

295
content/code/2024-02-12-umap3.md Normal file
View file

@ -0,0 +1,295 @@
---
title: Adding collaboration on uMapn, third update
tags: umap, geojson, websockets
status: draft
---
I've spent the last few weeks working on [uMap](https://umap-project.org), still
with the goal of bringing real-time collaboration to the maps. I'm not there
yet, but I've made some progress that I will relate here.
## JavaSript modules
uMap is an "old" project. It's been there [since 2012](https://github.com/
umap-project/umap/commit/0cce7f9e2a19c83fa76645d7773d39d54f357c43), at a time
when ES6 [wasn't out there yet](https://fr.wikipedia.org/wiki/ECMAScript#ECMAScript_Edition_6_(ES6)
As a result, it wasn't possible to use JavaScript modules, nor modern JavaScript
syntax.
The team has been working hard on bringing modules to the mix, and it
wasn't a piece of cake. But, the result is here: we're [now able to use modern
JavaSript modules](https://github.com/umap-project/umap/pull/1463/files) and we
are now more confident about which features of the languages we can use or not.
---
I then spent ~way too much~ some time trying to integrate existing CRDTs like
Automerge and YJS in our project. These two libs are unfortunately expecting us to
use a bundler, which we aren't currently.
uMap is plain old JavaScript. It's not using react or any other framework. The way
I see this is that it makes it possible for us to have something "close to the
metal", if that means anything when it comes to web development.
So, after making tweaks and learning how "modules", "requires" and "bundling"
was working, I ultimately decided to take a break from this path, to work on the
wiring with uMap. After all, CRDTs might not even be the way forward for us.
## Internals
So, I went for the deep dive into uMap internals. To be frank, I was expecting
this to be hard : there is a lot of code, some legacy, and at first I was the
head under the water, not able to understand how all this code was working together.
After some time, I can now wrap my head around the logic of the project. As one
could expect, it's not the complexity that was overflooding me, but the fact
that it's sometimes not simple to get the big picture. To phrase it otherwise,
it was easy to get lost into the details.
Let me try to summarize what I've learned.
uMap apears to be doing a lot of different things, but in the end it's:
- Using [Leaflet.js](https://leafletjs.com/) to render *features* on the map ;
- Using [Leaflet Editable](https://github.com/Leaflet/Leaflet.Editable) to edit
complex forms, like polylines, polygons, and to draw markers ;
- Using the [Formbuilder](https://github.com/yohanboniface/Leaflet.FormBuilder)
to expose a way for the users to edit the features, and the data of the map
- Serializing the layers to and from [GeoJSON](https://geojson.org/). That's
what's being sent to and received from the server.
- Providing different layer types (marker cluster, chloropleth, etc) to display
the data in different ways.
### Naming matters
There is some naming overlap between the different projects we're using, and
it's important to have these small clarifications in mind:
#### Leaflet layers and uMap features
**In Leaflet, everything is a layer**. What we call *features* in geoJSON are
leaflet layers, and even a (uMap) layer is a layer. We need to be extra careful
what are our inputs and outputs in this context.
We actually have different layers concepts: the *datalayer* and the different
kind of layers (chloropleth, marker cluster, etc). A datalayer, is (as you can
guess) where the data is stored. It's what uMap serializes. It contains the
features (with their properties). But that's the trick: these features are named
*layers* by Leaflet.
#### GeoJSON and Leaflet
We're using GeoJSON to share data with the server, but we're using Laflet
internally. And these two have different way of naming things.
The different geometries are named differently (a leaflet `Marker` is a GeoJSON
`Point`), and their coordinates are stored differently: Leaflet stores `lat,
long` where GeoJSON stores `long, lat`. Not a big deal, but it's a good thing
to know.
Leaflet stores data in `options`, where GeoJSON stores it in `properties`.
### This is not reactive programming
I was expecting the frontend to be organised similarly to Elm apps (or React
apps): a global state and a data flow ([a la redux](https:// react-redux.js.org/
introduction/getting-started)), with events changing the data that will trigger
a rerendering of the interface.
Things work differently for us: different components can write to the map, and
get updated without being centralized. It's just a different paradigm.
## A syncing proof of concept
With that in mind, I started thinking about a simple way to implement syncing.
I let aside all the thinking about how this would relate with CRDTs. It can
be useful, but later. For now, I "just" want to synchronize two maps. I want a
proof of concept to do informed decisions.
### Syncing map properties
I started syncing map properties. Things like the name of the map, the default
color and type of the marker, the description, the default zoom level, etc.
All of these are handled by "the formbuilder". You pass it an object, a list of
properties and a callback to call when an update happens, and it will build for
you form inputs.
Taken from the documentation (and simplified):
```js
var tilelayerFields = [
['name', {handler: 'BlurInput', placeholder: 'display name'}],
['maxZoom', {handler: 'BlurIntInput', placeholder: 'max zoom'}],
['minZoom', {handler: 'BlurIntInput', placeholder: 'min zoom'}],
['attribution', {handler: 'BlurInput', placeholder: 'attribution'}],
['tms', {handler: 'CheckBox', helpText: 'TMS format'}]
];
var builder = new L.FormBuilder(myObject, tilelayerFields, {
callback: myCallback,
callbackContext: this
});
```
In uMap, the formbuilder is used for every form you see on the right panel. Map
properties are stored in the `map` object.
We want two different clients work together. When one changes the value of a
property, the other client needs to be updated, and update its interface.
I've started by creating a mapping of property names to rerender-methods, and
added a method `renderProperties(properties)` which updates the interface,
depending on the properties passed to it.
We now have two important things:
1. Some code getting called each time a property is changed ;
2. A way to refresh the right interface when a property is changed.
In other words, from one client we can send the message to the other client,
which will be able to rerender itself.
Looks like a plan.
## Websockets
We need a way for the data to go from one side to the other. The easiest
way is probably websockets.
Here is a simple code which will relay messages from one websocket to the other
connected clients. It's not the final code, it's just for demo puposes.
A basic way to do this on the server side is to use python's [websockets]
(https://websockets.readthedocs.io/) library.
```python
import asyncio
import websockets
from websockets.server import serve
import json
# Just relay all messages to other connected peers for now
CONNECTIONS = set()
async def join_and_listen(websocket):
CONNECTIONS.add(websocket)
try:
async for message in websocket:
# recompute the peers-list at the time of message-sending.
# doing so beforehand would miss new connections
peers = CONNECTIONS - {websocket}
websockets.broadcast(peers, message)
finally:
CONNECTIONS.remove(websocket)
async def handler(websocket):
message = await websocket.recv()
event = json.loads(message)
# The first event should always be 'join'
assert event["kind"] == "join"
await join_and_listen(websocket)
async def main():
async with serve(handler, "localhost", 8001):
await asyncio.Future() # run forever
asyncio.run(main())
```
On the client side, it's fairly easy as well. I won't even cover it here.
We now have a way to send data from one client to the other.
Let's consider the actions we do as "verbs". For now, we're just updating
properties values, so we just need the `update` verb.
## Code architecture
We need different parts:
- the **transport**, which connects to the websockets, sends and receives messages.
- the **message sender** to relat local messages to the other party.
- the **message receiver** that's being called each time we receive a message.
- the **sync engine** which glues everything together
- Different **updaters**, which knows how to apply received messages, the goal being
to update the interface in the end.
When receiving a message it will be routed to the correct updater, which will
know what to do with it.
In our case, its fairly simple: when updating the `name` property, we send a
message with `name` and `value`. We also need to send along some additional
info: the `subject`.
In our case, it's `map` because we're updating map properties.
When initializing the `map`, we're initializing the `SyncEngine`, like this:
```js
// inside the map
let syncEngine = new umap.SyncEngine(this)
// Then, when we need to send data to the other party
let syncEngine = this.obj.getSyncEngine()
let subject = this.obj.getSyncSubject()
syncEngine.update(subject, field, value)
```
The code on the other side of the wire is simple enough: when you receive the
message, change the data and rerender the properties:
```js
this.updateObjectValue(this.map, key, value)
this.map.renderProperties(key)
```
<video controls width="80%">
<source src="https://files.notmyidea.org/umap-minisync.webm" type="video/webm">
</video>
## Syncing features
At this stage I was able to sync the properties of the map. A
small victory, but not the end of the trip.
The next step was to add syncing for features: markers, polygon and polylines,
alongside their properties.
All of these features have a uMap class representation (which extends Leaflets
ones). All of them share some code in the `FeatureMixin` class.
That seems a good place to do the changes.
I did a few changes:
- Each feature now has an identifier, so clients know they're talking about the same thing. This identifier is also stored in the database, so that persisted GeoJSON don't have to recreate ids.
- I've added an `upsert` verb, because we don't have any way (from the interface) to make a distinction between the creation of a new feature and its modification
The way we intercept the creation of a feature (or its update) is to use Leaflet Editable's `editable:drawing:commit` event. We
just have to listen to it and then send the appropriate messages !
After some giggling around (ah, everybody wants to create a new protocol !) I
went with reusing GeoJSON. It allowed me to have a better understanding of how Leaflet is using latlongs. That's a multi-dimensional array, with variable width, depending on the type of geometry and the number of points in each of these.
Clearly not something I want to redo, so I'm now reusing some Leaflet code, which handles this serialization for me.
I'm now able to sync different kind of features with their properties (the video is just showing points, but it's also working with polygons and polylines)
<video controls width="80%">
<source src="https://files.notmyidea.org/umap-sync-features.webm" type="video/webm">
</video>
## What's next ?
While I'm able to sync map properties, features and their properties, I'm not yet syncing layers. That's the next step! I also plan to make some pull requests with the interesting bits I'm sure will go in the final implementation:
- adding ids to features
- having a way to map properties with how they render the interface
See you for the next update!