mirror of
https://github.com/almet/notmyidea.git
synced 2025-04-28 19:42:37 +02:00
295 lines
No EOL
24 KiB
HTML
295 lines
No EOL
24 KiB
HTML
<!DOCTYPE html>
|
|
<html lang="fr">
|
|
<head>
|
|
<title>
|
|
Adding collaboration on uMap, third update - Alexis Métaireau </title>
|
|
<meta charset="utf-8" />
|
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
|
<link rel="stylesheet"
|
|
href="https://blog.notmyidea.org/theme/css/main.css?v2"
|
|
type="text/css" />
|
|
<link href="https://blog.notmyidea.org/feeds/all.atom.xml"
|
|
type="application/atom+xml"
|
|
rel="alternate"
|
|
title="Alexis Métaireau ATOM Feed" />
|
|
</head>
|
|
<body>
|
|
<div id="content">
|
|
<section id="links">
|
|
<ul>
|
|
<li>
|
|
<a class="main" href="/">Alexis Métaireau</a>
|
|
</li>
|
|
<li>
|
|
<a class=""
|
|
href="https://blog.notmyidea.org/journal/index.html">Journal</a>
|
|
</li>
|
|
<li>
|
|
<a class="selected"
|
|
href="https://blog.notmyidea.org/code/">Code, etc.</a>
|
|
</li>
|
|
<li>
|
|
<a class=""
|
|
href="https://blog.notmyidea.org/weeknotes/">Notes hebdo</a>
|
|
</li>
|
|
<li>
|
|
<a class=""
|
|
href="https://blog.notmyidea.org/lectures/">Lectures</a>
|
|
</li>
|
|
<li>
|
|
<a class=""
|
|
href="https://blog.notmyidea.org/projets.html">Projets</a>
|
|
</li>
|
|
</ul>
|
|
</section>
|
|
|
|
<header>
|
|
<h1 class="post-title">Adding collaboration on uMap, third update</h1>
|
|
<time datetime="2024-02-12T00:00:00+01:00">12 février 2024</time>
|
|
|
|
|
|
</header>
|
|
<article>
|
|
<p>I’ve spent the last few weeks working on <a href="https://umap-project.org">uMap</a>, still
|
|
with the goal of bringing real-time collaboration to the maps. I’m not there
|
|
yet, but I’ve made some progress that I will relate here.</p>
|
|
<h2 id="javascript-modules">JavaScript modules</h2>
|
|
<p>uMap has been there <a href="https://github.com/
|
|
umap-project/umap/commit/0cce7f9e2a19c83fa76645d7773d39d54f357c43">since 2012</a>, at a time
|
|
when <span class="caps">ES6</span> <a href="https://fr.wikipedia.org/wiki/ECMAScript">wasn’t out there yet</a></p>
|
|
<p>At the time, it wasn’t possible to use JavaScript modules, nor modern JavaScript
|
|
syntax. The project stayed with these requirements for a long time, in order to support
|
|
people with old browsers. But as time goes on, we can now have access to more features.</p>
|
|
<p>The team has been working hard on bringing modules to the mix, and it
|
|
wasn’t a piece of cake. But, the result is here: we’re <a href="https://github.com/umap-project/umap/pull/1463/files">now able to use modern
|
|
JavaSript modules</a> and we
|
|
are now more confident <a href="https://github.com/umap-project/umap/commit/65f1cdd6b4569657ef5e219d9b377fec85c41958">about which features of the languages we can use or not</a></p>
|
|
<hr>
|
|
<p>I then spent ~way too much~ some time trying to integrate existing CRDTs like
|
|
Automerge and <span class="caps">YJS</span> in our project. These two libs are unfortunately expecting us to
|
|
use a bundler, which we aren’t currently.</p>
|
|
<p>uMap is plain old JavaScript. It’s not using react or any other framework. The way
|
|
I see this is that it makes it possible for us to have something “close to the
|
|
metal”, if that means anything when it comes to web development. We’re not tied
|
|
to the development pace of these frameworks, and have more control on what we
|
|
do. It’s easier to debug.</p>
|
|
<p>So, after making tweaks and learning how “modules”, “requires” and “bundling”
|
|
was working, I ultimately decided to take a break from this path, to work on the
|
|
wiring with uMap. After all, CRDTs might not even be the way forward for us.</p>
|
|
<h2 id="internals">Internals</h2>
|
|
<p>I was not expecting this to be easy and was a bit afraid. Mostly because I’m out of my
|
|
comfort zone. After some time with the head under the water, I’m now able to better
|
|
understand the big picture, and I’m not getting lost in the details like I was at first.</p>
|
|
<p>Let me try to summarize what I’ve learned.</p>
|
|
<p>uMap appears to be doing a lot of different things, but in the end it’s:</p>
|
|
<ul>
|
|
<li>Using <a href="https://leafletjs.com/">Leaflet.js</a> to render <em>features</em> on the map ;</li>
|
|
<li>Using <a href="https://github.com/Leaflet/Leaflet.Editable">Leaflet Editable</a> to edit
|
|
complex shapes, like polylines, polygons, and to draw markers ;</li>
|
|
<li>Using the <a href="https://github.com/yohanboniface/Leaflet.FormBuilder">Formbuilder</a>
|
|
to expose a way for the users to edit the features, and the data of the map</li>
|
|
<li>Serializing the layers to and from <a href="https://geojson.org/">GeoJSON</a>. That’s
|
|
what’s being sent to and received from the server.</li>
|
|
<li>Providing different layer types (marker cluster, chloropleth, etc) to display
|
|
the data in different ways.</li>
|
|
</ul>
|
|
<h3 id="naming-matters">Naming matters</h3>
|
|
<p>There is some naming overlap between the different projects we’re using, and
|
|
it’s important to have these small clarifications in mind:</p>
|
|
<h4 id="leaflet-layers-and-umap-features">Leaflet layers and uMap features</h4>
|
|
<p><strong>In Leaflet, everything is a layer</strong>. What we call <em>features</em> in geoJSON are
|
|
leaflet layers, and even a (uMap) layer is a layer. We need to be extra careful
|
|
what are our inputs and outputs in this context.</p>
|
|
<p>We actually have different layers concepts: the <em>datalayer</em> and the different
|
|
kind of layers (chloropleth, marker cluster, etc). A datalayer, is (as you can
|
|
guess) where the data is stored. It’s what uMap serializes. It contains the
|
|
features (with their properties). But that’s the trick: these features are named
|
|
<em>layers</em> by Leaflet.</p>
|
|
<h4 id="geojson-and-leaflet">GeoJSON and Leaflet</h4>
|
|
<p>We’re using GeoJSON to share data with the server, but we’re using Leaflet
|
|
internally. And these two have different way of naming things.</p>
|
|
<p>The different geometries are named differently (a leaflet <code>Marker</code> is a GeoJSON
|
|
<code>Point</code>), and their coordinates are stored differently: Leaflet stores <code>lat,
|
|
long</code> where GeoJSON stores <code>long, lat</code>. Not a big deal, but it’s a good thing
|
|
to know.</p>
|
|
<p>Leaflet stores data in <code>options</code>, where GeoJSON stores it in <code>properties</code>.</p>
|
|
<h3 id="this-is-not-reactive-programming">This is not reactive programming</h3>
|
|
<p>I was expecting the frontend to be organised similarly to Elm apps (or React
|
|
apps): a global state and a data flow (<a href="https:// react-redux.js.org/
|
|
introduction/getting-started">a la redux</a>), with events changing the data that will trigger
|
|
a rerendering of the interface.</p>
|
|
<p>Things work differently for us: different components can write to the map, and
|
|
get updated without being centralized. It’s just a different paradigm.</p>
|
|
<h2 id="a-syncing-proof-of-concept">A syncing proof of concept</h2>
|
|
<p>With that in mind, I started thinking about a simple way to implement syncing. </p>
|
|
<p>I left aside all the thinking about how this would relate with CRDTs. It can
|
|
be useful, but later. For now, I “just” want to synchronize two maps. I want a
|
|
proof of concept to do informed decisions.</p>
|
|
<h3 id="syncing-map-properties">Syncing map properties</h3>
|
|
<p>I started syncing map properties. Things like the name of the map, the default
|
|
color and type of the marker, the description, the default zoom level, etc.</p>
|
|
<p>All of these are handled by “the formbuilder”. You pass it an object, a list of
|
|
properties and a callback to call when an update happens, and it will build for
|
|
you form inputs.</p>
|
|
<p>Taken from the documentation (and simplified):</p>
|
|
<div class="highlight"><pre><span></span><code><span class="kd">var</span><span class="w"> </span><span class="nx">tilelayerFields</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span>
|
|
<span class="w"> </span><span class="p">[</span><span class="s1">'name'</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="nx">handler</span><span class="o">:</span><span class="w"> </span><span class="s1">'BlurInput'</span><span class="p">,</span><span class="w"> </span><span class="nx">placeholder</span><span class="o">:</span><span class="w"> </span><span class="s1">'display name'</span><span class="p">}],</span>
|
|
<span class="w"> </span><span class="p">[</span><span class="s1">'maxZoom'</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="nx">handler</span><span class="o">:</span><span class="w"> </span><span class="s1">'BlurIntInput'</span><span class="p">,</span><span class="w"> </span><span class="nx">placeholder</span><span class="o">:</span><span class="w"> </span><span class="s1">'max zoom'</span><span class="p">}],</span>
|
|
<span class="w"> </span><span class="p">[</span><span class="s1">'minZoom'</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="nx">handler</span><span class="o">:</span><span class="w"> </span><span class="s1">'BlurIntInput'</span><span class="p">,</span><span class="w"> </span><span class="nx">placeholder</span><span class="o">:</span><span class="w"> </span><span class="s1">'min zoom'</span><span class="p">}],</span>
|
|
<span class="w"> </span><span class="p">[</span><span class="s1">'attribution'</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="nx">handler</span><span class="o">:</span><span class="w"> </span><span class="s1">'BlurInput'</span><span class="p">,</span><span class="w"> </span><span class="nx">placeholder</span><span class="o">:</span><span class="w"> </span><span class="s1">'attribution'</span><span class="p">}],</span>
|
|
<span class="w"> </span><span class="p">[</span><span class="s1">'tms'</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="nx">handler</span><span class="o">:</span><span class="w"> </span><span class="s1">'CheckBox'</span><span class="p">,</span><span class="w"> </span><span class="nx">helpText</span><span class="o">:</span><span class="w"> </span><span class="s1">'TMS format'</span><span class="p">}]</span>
|
|
<span class="p">];</span>
|
|
<span class="kd">var</span><span class="w"> </span><span class="nx">builder</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">FormBuilder</span><span class="p">(</span><span class="nx">myObject</span><span class="p">,</span><span class="w"> </span><span class="nx">tilelayerFields</span><span class="p">,</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">callback</span><span class="o">:</span><span class="w"> </span><span class="nx">myCallback</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="nx">callbackContext</span><span class="o">:</span><span class="w"> </span><span class="k">this</span>
|
|
<span class="p">});</span>
|
|
</code></pre></div>
|
|
|
|
<p>In uMap, the formbuilder is used for every form you see on the right panel. Map
|
|
properties are stored in the <code>map</code> object.</p>
|
|
<p>We want two different clients work together. When one changes the value of a
|
|
property, the other client needs to be updated, and update its interface.</p>
|
|
<p>I’ve started by creating a mapping of property names to rerender-methods, and
|
|
added a method <code>renderProperties(properties)</code> which updates the interface,
|
|
depending on the properties passed to it.</p>
|
|
<p>We now have two important things:</p>
|
|
<ol>
|
|
<li>Some code getting called each time a property is changed ;</li>
|
|
<li>A way to refresh the right interface when a property is changed.</li>
|
|
</ol>
|
|
<p>In other words, from one client we can send the message to the other client,
|
|
which will be able to rerender itself.</p>
|
|
<p>Looks like a plan.</p>
|
|
<h2 id="websockets">Websockets</h2>
|
|
<p>We need a way for the data to go from one side to the other. The easiest
|
|
way is probably websockets.</p>
|
|
<p>Here is a simple code which will relay messages from one websocket to the other
|
|
connected clients. It’s not the final code, it’s just for demo puposes.</p>
|
|
<p>A basic way to do this on the server side is to use python’s
|
|
<a href="https://websockets.readthedocs.io/">websockets</a> library.</p>
|
|
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">asyncio</span>
|
|
<span class="kn">import</span> <span class="nn">websockets</span>
|
|
<span class="kn">from</span> <span class="nn">websockets.server</span> <span class="kn">import</span> <span class="n">serve</span>
|
|
<span class="kn">import</span> <span class="nn">json</span>
|
|
|
|
<span class="c1"># Just relay all messages to other connected peers for now</span>
|
|
|
|
<span class="n">CONNECTIONS</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span>
|
|
|
|
<span class="k">async</span> <span class="k">def</span> <span class="nf">join_and_listen</span><span class="p">(</span><span class="n">websocket</span><span class="p">):</span>
|
|
<span class="n">CONNECTIONS</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">websocket</span><span class="p">)</span>
|
|
<span class="k">try</span><span class="p">:</span>
|
|
<span class="k">async</span> <span class="k">for</span> <span class="n">message</span> <span class="ow">in</span> <span class="n">websocket</span><span class="p">:</span>
|
|
<span class="c1"># recompute the peers-list at the time of message-sending.</span>
|
|
<span class="c1"># doing so beforehand would miss new connections</span>
|
|
<span class="n">peers</span> <span class="o">=</span> <span class="n">CONNECTIONS</span> <span class="o">-</span> <span class="p">{</span><span class="n">websocket</span><span class="p">}</span>
|
|
<span class="n">websockets</span><span class="o">.</span><span class="n">broadcast</span><span class="p">(</span><span class="n">peers</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
|
|
<span class="k">finally</span><span class="p">:</span>
|
|
<span class="n">CONNECTIONS</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="n">websocket</span><span class="p">)</span>
|
|
|
|
|
|
<span class="k">async</span> <span class="k">def</span> <span class="nf">handler</span><span class="p">(</span><span class="n">websocket</span><span class="p">):</span>
|
|
<span class="n">message</span> <span class="o">=</span> <span class="k">await</span> <span class="n">websocket</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
|
|
<span class="n">event</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">message</span><span class="p">)</span>
|
|
|
|
<span class="c1"># The first event should always be 'join'</span>
|
|
<span class="k">assert</span> <span class="n">event</span><span class="p">[</span><span class="s2">"kind"</span><span class="p">]</span> <span class="o">==</span> <span class="s2">"join"</span>
|
|
<span class="k">await</span> <span class="n">join_and_listen</span><span class="p">(</span><span class="n">websocket</span><span class="p">)</span>
|
|
|
|
<span class="k">async</span> <span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
|
|
<span class="k">async</span> <span class="k">with</span> <span class="n">serve</span><span class="p">(</span><span class="n">handler</span><span class="p">,</span> <span class="s2">"localhost"</span><span class="p">,</span> <span class="mi">8001</span><span class="p">):</span>
|
|
<span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">Future</span><span class="p">()</span> <span class="c1"># run forever</span>
|
|
|
|
<span class="n">asyncio</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">main</span><span class="p">())</span>
|
|
</code></pre></div>
|
|
|
|
<p>On the client side, it’s fairly easy as well. I won’t even cover it here.</p>
|
|
<p>We now have a way to send data from one client to the other.
|
|
Let’s consider the actions we do as “verbs”. For now, we’re just updating
|
|
properties values, so we just need the <code>update</code> verb.</p>
|
|
<h2 id="code-architecture">Code architecture</h2>
|
|
<p>We need different parts:</p>
|
|
<ul>
|
|
<li>the <strong>transport</strong>, which connects to the websockets, sends and receives messages.</li>
|
|
<li>the <strong>message sender</strong> to relat local messages to the other party.</li>
|
|
<li>the <strong>message receiver</strong> that’s being called each time we receive a message.</li>
|
|
<li>the <strong>sync engine</strong> which glues everything together</li>
|
|
<li>Different <strong>updaters</strong>, which knows how to apply received messages, the goal being
|
|
to update the interface in the end.</li>
|
|
</ul>
|
|
<p>When receiving a message it will be routed to the correct updater, which will
|
|
know what to do with it.</p>
|
|
<p>In our case, its fairly simple: when updating the <code>name</code> property, we send a
|
|
message with <code>name</code> and <code>value</code>. We also need to send along some additional
|
|
info: the <code>subject</code>.</p>
|
|
<p>In our case, it’s <code>map</code> because we’re updating map properties.</p>
|
|
<p>When initializing the <code>map</code>, we’re initializing the <code>SyncEngine</code>, like this:</p>
|
|
<div class="highlight"><pre><span></span><code><span class="c1">// inside the map</span>
|
|
<span class="kd">let</span><span class="w"> </span><span class="nx">syncEngine</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">umap</span><span class="p">.</span><span class="nx">SyncEngine</span><span class="p">(</span><span class="k">this</span><span class="p">)</span>
|
|
|
|
<span class="c1">// Then, when we need to send data to the other party</span>
|
|
<span class="kd">let</span><span class="w"> </span><span class="nx">syncEngine</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">obj</span><span class="p">.</span><span class="nx">getSyncEngine</span><span class="p">()</span>
|
|
<span class="kd">let</span><span class="w"> </span><span class="nx">subject</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">obj</span><span class="p">.</span><span class="nx">getSyncSubject</span><span class="p">()</span>
|
|
|
|
<span class="nx">syncEngine</span><span class="p">.</span><span class="nx">update</span><span class="p">(</span><span class="nx">subject</span><span class="p">,</span><span class="w"> </span><span class="nx">field</span><span class="p">,</span><span class="w"> </span><span class="nx">value</span><span class="p">)</span>
|
|
</code></pre></div>
|
|
|
|
<p>The code on the other side of the wire is simple enough: when you receive the
|
|
message, change the data and rerender the properties:</p>
|
|
<div class="highlight"><pre><span></span><code><span class="k">this</span><span class="p">.</span><span class="nx">updateObjectValue</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">map</span><span class="p">,</span><span class="w"> </span><span class="nx">key</span><span class="p">,</span><span class="w"> </span><span class="nx">value</span><span class="p">)</span>
|
|
<span class="k">this</span><span class="p">.</span><span class="nx">map</span><span class="p">.</span><span class="nx">renderProperties</span><span class="p">(</span><span class="nx">key</span><span class="p">)</span>
|
|
</code></pre></div>
|
|
|
|
<h2 id="syncing-features">Syncing features</h2>
|
|
<p>At this stage I was able to sync the properties of the map. A
|
|
small victory, but not the end of the trip.</p>
|
|
<p>The next step was to add syncing for features: markers, polygon and polylines,
|
|
alongside their properties.</p>
|
|
<p>All of these features have a uMap class representation (which extends Leaflets
|
|
ones). All of them share some code in the <code>FeatureMixin</code> class.</p>
|
|
<p>That seems a good place to do the changes.</p>
|
|
<p>I did a few changes:</p>
|
|
<ul>
|
|
<li>Each feature now has an identifier, so clients know they’re talking about the
|
|
same thing. This identifier is also stored in the database when saved.</li>
|
|
<li>I’ve added an <code>upsert</code> verb, because we don’t have any way (from the
|
|
interface) to make a distinction between the creation of a new feature and
|
|
its modification. The way we intercept the creation of a feature (or its
|
|
update) is to use Leaflet Editable’s <code>editable:drawing:commit</code> event. We just
|
|
have to listen to it and then send the appropriate messages !</li>
|
|
</ul>
|
|
<p>After some giggling around (ah, everybody wants to create a new protocol !) I
|
|
went with reusing GeoJSON. It allowed me to have a better understanding of how
|
|
Leaflet is using latlongs. That’s a multi-dimensional array, with variable
|
|
width, depending on the type of geometry and the number of shapes in each of these.</p>
|
|
<p>Clearly not something I want to redo, so I’m now reusing some Leaflet code, which handles this serialization for me.</p>
|
|
<p>I’m now able to sync different types of features with their properties.</p>
|
|
<video controls width="80%">
|
|
<source src="/images/umap/sync-features.webm" type="video/webm">
|
|
</video>
|
|
|
|
<p>Point properties are also editable, using the already-existing table editor. I
|
|
was expecting this to require some work but it’s just working without more changes.</p>
|
|
<h2 id="whats-next">What’s next ?</h2>
|
|
<p>I’m able to sync map properties, features and their properties, but I’m not
|
|
yet syncing layers. That’s the next step! I also plan to make some pull
|
|
requests with the interesting bits I’m sure will go in the final implementation:</p>
|
|
<ul>
|
|
<li>Adding ids to features, so we have a way to refer to them.</li>
|
|
<li>Having a way to map properties with how they render the interface, the <code>renderProperties</code> bits.</li>
|
|
</ul>
|
|
<p>When this demo will be working, I’ll probably spend some time updating it with the latest changes (umap is moving a lot these weeks).
|
|
I will probably focus on how to integrate websockets in the server side, and then will see how to leverage (maybe) some magic from CRDTs, if we need it.</p>
|
|
<p>See you for the next update!</p>
|
|
<p>
|
|
<a href="https://blog.notmyidea.org/tag/umap.html">#umap</a>, <a href="https://blog.notmyidea.org/tag/geojson.html">#geojson</a>, <a href="https://blog.notmyidea.org/tag/websockets.html">#websockets</a> - Posté dans la catégorie <a href="https://blog.notmyidea.org/code/">code</a>
|
|
</p>
|
|
</article>
|
|
<footer>
|
|
<a id="feed" href="/feeds/all.atom.xml">
|
|
<img alt="RSS Logo" src="/theme/rss.svg" />
|
|
</a>
|
|
</footer>
|
|
</div>
|
|
</body>
|
|
</html> |