mirror of
https://github.com/almet/notmyidea.git
synced 2025-04-28 11:32:39 +02:00
A bunch of updates :-)
This commit is contained in:
parent
436c5e7602
commit
6be91639f6
20 changed files with 825 additions and 481 deletions
0
content/Sans titre.md
Normal file
0
content/Sans titre.md
Normal file
|
|
@ -1,432 +0,0 @@
|
|||
---
|
||||
title: A comparison of JavaScript CRDTs
|
||||
status: draft
|
||||
tags: crdts, umap, sync
|
||||
display_toc: true
|
||||
---
|
||||
|
||||
This is not done yet, TODO :
|
||||
|
||||
- [x] Change the table to be more readable, especially on mobile
|
||||
- [x] Update the examples with JSON Joy to have a working example
|
||||
- [x] Change the layout to be easier to understand.
|
||||
- [ ] Enhance graphics to be easier to understand
|
||||
- [ ] Part 3: Key takeaways
|
||||
---
|
||||
|
||||
[TOC]
|
||||
|
||||
---
|
||||
|
||||
Collaboration is one of the most requested features on uMap since a long time. We've added a way to merge same-layers edit, but ideally we would love to make things easier to understand and more fluid for the users.
|
||||
|
||||
For this reason, I got more into CRDTs, with the goal of understanding how they work, what are the different libraries out there, and which one would be a good fit for us, if any.
|
||||
|
||||
So far, [the way I've though about collaboration features on uMap](https://blog.notmyidea.org/adding-collaboration-on-umap-third-update.html) is by:
|
||||
|
||||
- a) catching when changes are done on the interface ;
|
||||
- b) sending messages to the other party and ;
|
||||
- c) applying the changes on the receiving client.
|
||||
|
||||
This works well in general, but it doesn't take care of conflicts handling, especially when a disconnect can happen.
|
||||
## Part 1 - What are CRDTs?
|
||||
|
||||
Conflict-free Resolution Data Types (CRDTs) are as a specific datatype able to merge its state with other states without generating conflicts. They handle consistency in distributed systems, making them particularly well-suited for collaborative real-time applications.
|
||||
|
||||
CRDTs ensure that multiple participants can make changes without strict coordination, and all replicas converge to the same state upon synchronization, without conflicts.
|
||||
|
||||
"Append-only sets" are probably one of the most common type of CRDT: if multiple parties add the same element, it will be present only once. It's our old friend `Set`.
|
||||
### Why using them?
|
||||
|
||||
For uMap, CRDTs offer a solution to several challenges:
|
||||
|
||||
1. **Simultaneous Editing**: When multiple users interact with the same map, their changes must not only be reflected in real-time but also merged seamlessly without overwriting each other's contributions.
|
||||
|
||||
2. **Network Latency and Partition**: uMap operates over networks that can experience delays or temporary outages. CRDTs can handle these conditions gracefully, enabling offline editing and eventual consistency.
|
||||
|
||||
3. **Simplified Conflict Resolution**: Traditional methods often require complex algorithms to resolve conflicts, while CRDTs inherently minimize the occurrence of conflicts altogether.
|
||||
|
||||
4. **Decentralization**: While uMap currently relies on a central server, adopting CRDTs could pave the way for a more decentralized architecture, increasing resilience and scalability.
|
||||
|
||||
### Comparing with traditional data synchronization methods
|
||||
|
||||
Traditional data synchronization methods typically rely on a central source of truth, such as a server, to manage and resolve conflicts. When changes are made by different users, these traditional systems require a round-trip to the server for conflict resolution and thus can be slow or inadequate for real-time collaboration.
|
||||
|
||||
In contrast, CRDTs leverage mathematical properties (the fact that the datatypes can converge) to ensure that every replica independently reaches the same state, without the need for a central authority, thus minimizing the amount of coordination and communication needed between nodes.
|
||||
|
||||
This ability to maintain consistency sets CRDTs apart from conventional synchronization approaches and makes them particularly valuable for the development of collaborative tools like uMap, where real-time updates and reliability are important.
|
||||
|
||||
### Last Write Wins Registers
|
||||
|
||||
For managing key/value data, I'm leaning onto Last-Write-Wins (LWW) registers within CRDTs. With LWW, the main concern is establishing the sequence of updates. In a single-client scenario or with a central time reference, sequencing is straightforward. However, in a distributed environment, time discrepancies across peers can complicate things, as clocks may drift and lose synchronization.
|
||||
|
||||
To address this, CRDTs use vector clocks — a specialized data structure that helps to solve the relative timing of events across distributed systems and pinpoint any inconsistencies.
|
||||
|
||||
> A vector clock is a data structure used for determining the partial ordering of events in a distributed system and detecting causality violations.
|
||||
>
|
||||
> – [Wikipedia](https://en.wikipedia.org/wiki/Vector_clock)
|
||||
|
||||
At first, I found CRDTs somewhat confusing, owing to their role in addressing complex challenges. CRDTs come in various forms, with much of their intricacy tied to resolving content conflicts within textual data or elaborate hierarchical structures. Fortunately for us, our use case is comparatively straightforward.
|
||||
|
||||
|
||||
|
||||
Note that we could also use a library such as [rxdb](https://github.com/pubkey/rxdb) — to handle the syncing, offline, etc — because we have a master: we use the server, and we can use it to handle the merge conflicts. But by doing so, we also give more responsibility to the server, whereas when using CRDTs it's possible to do the merge only on the clients.
|
||||
|
||||
### Different types of CRDTs
|
||||
|
||||
While reading the literature, I found that there are two kinds of CRDTs: state-based and operation-based. So, what do we need ?
|
||||
|
||||
It turns out most of the CRDTs implementation I looked at are operation-based, and propose an
|
||||
API to interact with them as you're changing the state, so **it doesn't really matter**.
|
||||
|
||||
> The two alternatives are theoretically equivalent, as each can emulate the
|
||||
> other. However, there are practical differences. State-based CRDTs are
|
||||
> often simpler to design and to implement; their only requirement from the
|
||||
> communication substrate is some kind of gossip protocol. **Their drawback is that
|
||||
> the entire state of every CRDT must be transmitted eventually to every other
|
||||
> replica, which may be costly**. In contrast, operation-based CRDTs transmit only
|
||||
> the update operations, which are typically small.
|
||||
>
|
||||
> [Wikipedia on CRDTs](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type)
|
||||
|
||||
### How the server fits in the picture
|
||||
|
||||
While discussing with the automerge team, I understood that I was expecting the server to pass along the messages to the other parties, and that would be the way the synchronisation would be done. It turns out I was mistaken: in this approach, the clients send updates to the server, which merges everything together and only then sends the updates to the other peers.
|
||||
|
||||
In order to have peers working with each other, I would need to change the way the provider works, so we can have the server be "brainless" and just relay the messages.
|
||||
|
||||
For automerge, it would mean the provider will "just" handle the websocket connection (disconnect and reconnect) and all the peers would be able to talk with each other. The other solution for us would be to have the merge algorithm working on the server side, which comes with upsides (no need to find when the document should be saved by the client to the server) and downsides (it takes some cpu and memory to run the CRDTs on the server)
|
||||
|
||||
---
|
||||
## Part 2: making a demo with different libraries
|
||||
|
||||
Now that we're familiar with CRDTs and how they can help us, let's create a map application which syncs marker positions, on different clients.
|
||||
|
||||
We'll be comparing three JavaScript libraries: [Y.js](https://yjs.dev/), [Automerge](https://automerge.org/) and [JSON Joy](https://jsonjoy.com), considering:
|
||||
|
||||
1. **Efficiency**: Probe the bandwidth when doing edits. What's being transmitted over the wire?
|
||||
2. **API**: is it easy to use for our use case? What are the challenging parts for us?
|
||||
3. **Community and Support**: How is the size and activity of the developer community / ecosystem?
|
||||
4. **Size** of the JavaScript library, because we want to limit the impact on our users browsers.
|
||||
|
||||
### A leaflet map
|
||||
|
||||
All the demos are made agains the same application, which creates markers when the map is clicked, and move the markers on hover.
|
||||
|
||||
Here's the whole code for this:
|
||||
|
||||
```js
|
||||
import L from "leaflet";
|
||||
import "leaflet/dist/leaflet.css";
|
||||
|
||||
// Create a map with a default tilelayer.
|
||||
const map = L.map("map").setView([48.1173, -1.6778], 13);
|
||||
L.tileLayer("https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png", {
|
||||
maxZoom: 19,
|
||||
attribution: "© OpenStreetMap contributors",
|
||||
}).addTo(map);
|
||||
|
||||
// Features contains a reference to the marker objects, mapped by the uuids
|
||||
let features = {};
|
||||
|
||||
// An upsert function
|
||||
function upsertMarker({ latlng, uuid, local = false }) {
|
||||
if (!uuid) uuid = crypto.randomUUID();
|
||||
let marker;
|
||||
|
||||
if (Object.keys(features).includes(uuid)) {
|
||||
marker = features[uuid];
|
||||
marker.setLatLng(latlng);
|
||||
} else {
|
||||
marker = new L.marker(latlng, {
|
||||
draggable: true,
|
||||
autoPan: true,
|
||||
uuid: uuid,
|
||||
});
|
||||
|
||||
features[uuid] = marker;
|
||||
marker.addTo(map);
|
||||
marker.on("dragend", ({ target }) => {
|
||||
console.log("dragend");
|
||||
});
|
||||
}
|
||||
|
||||
if (!local) {
|
||||
console.log()
|
||||
}
|
||||
}
|
||||
|
||||
// Add new features to the map with a click
|
||||
map.on("click", ({ latlng }) => {
|
||||
upsertMarker({ latlng });
|
||||
});
|
||||
```
|
||||
|
||||
Now, let's add synchronization.
|
||||
### Y.js
|
||||
|
||||
Y.js is the first library I've looked at, because it's the oldest one, and the more commonly referred to.
|
||||
|
||||
The API seem to offer what we look for, and provides a way to [observe changes](https://docs.yjs.dev/api/shared-types/y.map#observing-changes-y.mapevent). Here's what I did:
|
||||
|
||||
```js
|
||||
import * as Y from "yjs";
|
||||
import { WebsocketProvider } from "y-websocket";
|
||||
|
||||
// Instanciate a document
|
||||
const doc = new Y.Doc();
|
||||
```
|
||||
|
||||
To make changes to the CRDT:
|
||||
|
||||
```js
|
||||
let markers = doc.getMap("markers");
|
||||
markers.set(target.options.uuid, target._latlng);
|
||||
```
|
||||
|
||||
To observe the changes:
|
||||
|
||||
```js
|
||||
markers.observe((event, transaction) => {
|
||||
if (!transaction.local) {
|
||||
event.changes.keys.forEach((change, key) => {
|
||||
let value = markers.get(key);
|
||||
if (change.action === "add") {
|
||||
upsertMarker({ latlng: value, uuid: key, local: true });
|
||||
} else if (change.action === "update") {
|
||||
upsertMarker({ latlng: value, uuid: key, local: true });
|
||||
} else if (change.action === "delete") {
|
||||
console.log(`Property "${key}" was deleted. ".`);
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
It comes with multiple "providers", which make it possible to sync with different protocols (there is even a way to sync over the matrix protocol 😇). More usefully for us, there is [an implemented protocol for websockets](https://github.com/yjs/y-websocket).
|
||||
|
||||
Using a provider is as easy as:
|
||||
|
||||
```js
|
||||
// Sync clients with the y-websocket provider
|
||||
const provider = new WebsocketProvider(
|
||||
"ws://localhost:1234",
|
||||
"leaflet-sync",
|
||||
doc
|
||||
);
|
||||
```
|
||||
|
||||
It's also possible to send "awareness" information (some state you don't want to persist, like the position of the cursor). It contains some useful meta information, such as the number of connected peers.
|
||||
|
||||
```js
|
||||
map.on("mousemove", ({ latlng }) => {
|
||||
awareness.setLocalStateField("user", {
|
||||
cursor: latlng,
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
I made [a quick proof of concept with Y.js](https://gitlab.com/almet/leaflet-sync/-/tree/yjs) in a few hours flawlessly. It handles offline and reconnects, and exposes awareness information.
|
||||
#### Python bindings
|
||||
|
||||
Y.js has been rewritten in rust, with [the Y.rs project](https://github.com/y-crdt/y-crdt), which makes it possible to use with Python (see [Y.py](https://github.com/y-crdt/ypy)) if needed. The project has been implemented quite recently and is currently looking for a maintainer.
|
||||
#### Library size
|
||||
|
||||
Size: Y.js is 4,16 Ko, Y-Websocket is 21,14 Ko
|
||||
The library is currently used in production for large projects such as AFFiNE and Evernote.
|
||||
#### The data being transmitted
|
||||
|
||||
In the scenario where all clients connect to a central server, which handle the CRDT locally and then transmits back to other parties, I found that adding 20 points on one client, and then 20 points in another generates ~5 ko of data (~16 bytes per edit).
|
||||
|
||||
| Pros | Cons |
|
||||
| -------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
|
||||
| the API was feeling natural to me: it handles plain old JavaScript objects, making it easy to integrate. | It doesn't seem to work well [without a JS bundler](https://github.com/yjs/yjs/issues/325) which could be a problem for us. |
|
||||
| It seems to be widely used, and the community seems active. | |
|
||||
| It is [well documented](https://docs.yjs.dev/) | |
|
||||
| There is awareness support | |
|
||||
|
||||
---
|
||||
### Automerge
|
||||
|
||||
[Automerge](https://automerge.org/) is another library to handle CRDTs. Automerge is actually the low level interface, and there is a higher-level interface exposed as [Automerge-repo](https://automerge.org/docs/repositories/). Here is how to use it:
|
||||
|
||||
```js
|
||||
import { Repo } from "@automerge/automerge-repo";
|
||||
|
||||
const repo = new Repo();
|
||||
let handle = repo.create() // or repo.find(name)
|
||||
```
|
||||
|
||||
When you change the document, you actually call `change` which makes it possible to do the changes in a kind of "transaction function".
|
||||
|
||||
```js
|
||||
handle.change((doc) => {
|
||||
doc[uuid] = cleanLatLng(target._latlng);
|
||||
});
|
||||
```
|
||||
|
||||
Note that I had to use a `cleanLatLng` function in order to not pass the whole object, otherwise it wouldn't be serialized. It's really just a simple helper taking the properties of interest for us (and letting away all the rest).
|
||||
|
||||
You can observe the changes, getting you the whole list of patches:
|
||||
|
||||
```js
|
||||
handle.on("change", ({ doc, patches }) => {
|
||||
console.log(patches, doc);
|
||||
patches.forEach(({ action, path }) => {
|
||||
// We have to know the specifics of the patch operations getting passed
|
||||
// two items in the path means we're inserting an object
|
||||
if (path.length == 2 && action === "insert") {
|
||||
let value = doc[path[0]];
|
||||
upsertMarker({ latlng: value, uuid: path[0], local: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
#### Python bindings
|
||||
|
||||
There is an [automerge.py](https://github.com/automerge/automerge-py) project, but no changes has been made to it since 3 years ago. There are plans to update it, though.
|
||||
|
||||
#### Library size
|
||||
|
||||
Size: 1,64 mb, total is 1,74 mb. It's relying on Web assembly by default.
|
||||
|
||||
The large bundle size is something that the team is aware of, and are working on a solution for. For us, it's important to have something as lightweight as possible, considering CRDTs is only one part of what we're doing, and that mapping can be done in context where connection is not that reliable and fast.
|
||||
|
||||
#### The data being transmitted
|
||||
|
||||
In the same scenario, I found that adding 20 points on one client, and then 20 points in another generates 90 messages and 24,94 Ko of data transmitted (~12 Ko sent and ~12Ko received), so approximately 75 bytes per edit.
|
||||
|
||||
| Pros | Cons |
|
||||
| -------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
|
||||
| There is an API to [get informed when a conflict occured](https://automerge.org/docs/documents/conflicts/)<br><br> | Documentation was a bit hard to understand and to look at. Sometimes, it's easier to go look at the code.<br><br> |
|
||||
| In general, the documentation is low level, which can be a good thing while debuging, or when getting more advanced usage. | The API is more verbose. You can see it as "less magical". |
|
||||
| The team was responsive and trying to help. | There is no way (at the moment) to tell that a transaction is local or remote (but in practice it wasn't a problem) |
|
||||
|
||||
---
|
||||
### JSON Joy
|
||||
|
||||
[Json Joy](https://jsonjoy.com) is the latest to the party.
|
||||
|
||||
It takes another stake by providing multiple libraries with a small functional perimeter. It sounds promising, even if still quite new, and would left us with the hands free in order to implement the protocol that would work for us.
|
||||
|
||||
```js
|
||||
import {Model} from 'json-joy/es2020/json-crdt';
|
||||
import { s } from "json-joy/es6/json-crdt-patch";
|
||||
|
||||
// Initiate a model with a custom ID
|
||||
|
||||
const rootModel = Model.withLogicalClock(11111111);
|
||||
|
||||
// populate it with default data
|
||||
rootModel.api.root({
|
||||
markers: {},
|
||||
});
|
||||
|
||||
// Fork it on each client
|
||||
let userModel = rootModel.fork();
|
||||
```
|
||||
|
||||
Making changes to the model. Here we are changing a constant for another one, by using `s.con`.
|
||||
|
||||
```js
|
||||
userModel.api.obj(["markers"]).set({
|
||||
[uuid]: s.con(target._latlng),
|
||||
});
|
||||
```
|
||||
|
||||
Creating a patch, before sending it to the other parties:
|
||||
|
||||
```js
|
||||
import { encode, decode } from "json-joy/es6/json-crdt-patch/codec/verbose";
|
||||
|
||||
let patch = userModel.api.flush();
|
||||
let payload = encode(patch);
|
||||
```
|
||||
|
||||
When receiving a message, decode it and apply it:
|
||||
|
||||
```js
|
||||
let patch = decode(payload);
|
||||
model.api.apply(patch);
|
||||
```
|
||||
|
||||
We can observe the changes this way. Here, we're having a look at the operation that happened and acting on it. The names of the operations aren't clearly specified by the spec. It seems a bit sketchy, so I'm not sure it's the way to handle this, but it works.
|
||||
|
||||
```js
|
||||
userModel.api.onPatch.listen((patch) => {
|
||||
patch.ops.forEach((op) => {
|
||||
if (op.name() === "ins_obj") {
|
||||
let key = op.data[0][0];
|
||||
let value = userModel.view().markers[key];
|
||||
upsertMarker({ latlng: value, uuid: key, local: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
Metrics:
|
||||
|
||||
- Size: 143 ko
|
||||
- Data transmitted for 2 peers and 40 edits: (35 bytes per edit)
|
||||
|
||||
| Pros | Cons |
|
||||
| ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
|
||||
| Low level, so you know what you're doing | It doesn't provide a high level interface for sync<br><br> |
|
||||
| Small atomic libraries, making it easy to use only the parts we need. | It's currently a one-person project, without clear community channels to gather with other interested folks. |
|
||||
| The interface [proposes to store different type of data](https://jsonjoy.com/libs/json-joy-js/json-crdt/guide/node-types) (constants, values, arrays, etc) | Quite recent, so probably rough spots are to be found |
|
||||
| Distributed as different type of JS bundles (modules, wasm, etc) |
|
||||
|
||||
---
|
||||
## Part 3: Key takeaways
|
||||
|
||||
Y.js:
|
||||
|
||||
- **JavaScript objects API**: uses native JS objects, making it easier to integrate with existing applications (but can also transport metadata we don't really want)
|
||||
- **WebSockets**: a "connection provider" exists for WebSockets, handling offline changes, reconnections, etc. The currently implemented provider considers the server as a central node, which also maintains a CRDT locally.
|
||||
- **Awareness API**: a protocol defined to send awareness information (who is connected, and for instance the cursor position)
|
||||
- Versioning and history: Supports selective versioning through snapshots.
|
||||
- Community and support: Active community with regular updates.
|
||||
- Library Size / Network usage: Small size with efficient default compression of the data.
|
||||
|
||||
Suitability for uMap: Ready-to-use with good documentation and examples, might do more than what we need, some bundler dependencies.
|
||||
|
||||
Automerge:
|
||||
|
||||
- **Transactional API**: API is transactional (edits should happen in a function), making it obvious, but can also be harder to implement.
|
||||
- **WebSockets**: Multiple options with providers and storage, including WebSockets. It works similarly to the y.js one, with a copy of the data living on the server. There is currently no detection of offline / retry on disconnect, or awareness API.
|
||||
- **Conflict Resolution**: [A conflict detection API](https://automerge.org/docs/documents/conflicts/) exists, which make it easy to get the conflicting values, when it happens.
|
||||
- **Offline Changes Handling**: Requires a more manual approach to handle offline changes.
|
||||
- Versioning and History: Designed with robust version history tracking.
|
||||
- **Community and Support**: Strong support with a focus on collaboration.
|
||||
- **Library Size / Efficiency**: Larger library with dependency on WebAssembly.
|
||||
|
||||
JSON Joy:
|
||||
|
||||
- **Lower-level API**: A low-level API offers granular control, it's more hands-on, but less magic.
|
||||
- **WebSockets**: Nothing is provided in this area, and it would need to be implemented by ourselves.
|
||||
- **Community and Support**: there is only one maintainer as of now, and the community doesn't exist yet.
|
||||
- **Browser Compatibility**: Flexible bundles for diverse browser support, which might be helpful in our case.
|
||||
|
||||
## Extra notes
|
||||
|
||||
### YATA and RGA
|
||||
|
||||
While researching, I found that the two popular CRDTs implementation out there use different approaches for the virtual counter:
|
||||
|
||||
> - **RGA** [used by Automerge] maintains a single globally incremented counter (which can be ordinary integer value), that's updated anytime we detect that remote insert has an id
|
||||
> with sequence number higher that local counter. Therefore every time, we produce
|
||||
> a new insert operation, we give it a highest counter value known at the time.
|
||||
> - **YATA** [used by Yjs] also uses a single integer value, however unlike in case of RGA we
|
||||
> don't use a single counter shared with other replicas, but rather let each
|
||||
> peer keep its own, which is incremented monotonically only by that peer. Since
|
||||
> increments are monotonic, we can also use them to detect missing operations eg.
|
||||
> updates marked as A:1 and A:3 imply, that there must be another (potentially
|
||||
> missing) update A:2.
|
||||
|
||||
### Resources
|
||||
|
||||
- [CRDTs: The Hard Parts](https://www.youtube.com/watch?v=x7drE24geUw), a video by Martin Kleppmann where he explains the current state of the art of CRDTs, and why some problems aren't solved yet.
|
||||
- [An Interactive Intro to CRDTs](https://jakelazaroff.com/words/an-interactive-intro-to-crdts/) gets you trough different steps to understand what are CRDTs, and how to implement a LWW Register.
|
||||
- [Bartosz Sypytkowski](https://www.bartoszsypytkowski.com/the-state-of-a-state-based-crdts/) introduction on CRDTs, with practical exemples is very intuitive.
|
||||
- [CRDT Implementations](https://jzhao.xyz/thoughts/CRDT-Implementations) contains nice information and vocabulary useful when working with CRDTs.
|
||||
493
content/code/2024-03-18-crdts.md
Normal file
493
content/code/2024-03-18-crdts.md
Normal file
|
|
@ -0,0 +1,493 @@
|
|||
---
|
||||
title: A comparison of JavaScript CRDTs
|
||||
tags: crdts, umap, sync
|
||||
display_toc: true
|
||||
---
|
||||
|
||||
Collaboration is one of the most requested features on [uMap](https://umap-project.org).
|
||||
I've talked [in previous articles](https://blog.notmyidea.org/tag/umap.html) how
|
||||
we could add real-time features "the simple way", by:
|
||||
|
||||
- a) catching when changes are done on the interface ;
|
||||
- b) sending messages to the other parties and ;
|
||||
- c) applying the changes on the receiving client.
|
||||
|
||||
This works well in general, but it doesn't take care of conflicts handling, especially when a disconnect can happen.
|
||||
|
||||
For this reason, I got more into "Conflict-free Resolution Data Types" (CRDTs), with the goal of understanding what they are, how they work, what are the different libraries out there, and which one would be a good fit for us, if any.
|
||||
|
||||
As things are changing quickly in this field, note that this article was written in March 2024.
|
||||
|
||||
---
|
||||
|
||||
[TOC]
|
||||
|
||||
---
|
||||
|
||||
## Part 1 - What are CRDTs?
|
||||
|
||||
Conflict-free Resolution Data Types are a family of data types able to merge their states with other states without generating conflicts. They handle consistency in distributed systems, making them particularly well-suited for collaborative real-time applications.
|
||||
|
||||
CRDTs ensure that multiple participants can make changes without strict coordination, and all replicas converge to the same state upon synchronization, without conflicts.
|
||||
|
||||
"Append-only sets" are probably one of the most common type of CRDT: you can add the same element again and again, it will only be present once in the set. It's our old friend `Set`, as we can find in many programming languages.
|
||||
|
||||
### Why using CRDTs?
|
||||
|
||||
For uMap, CRDTs offer a solution to several challenges:
|
||||
|
||||
1. **Simultaneous Editing**: When multiple users interact with the same map, their changes must not only be reflected in real-time but also merged seamlessly without overwriting each other's contributions. We need all the replicas to converge to the same state.
|
||||
|
||||
2. **Network Latency and Partition**: uMap operates over networks that can experience delays or temporary outages (think editing on the ground). CRDTs can handle these conditions gracefully, enabling offline editing and eventual consistency.
|
||||
|
||||
3. **Simplified Conflict Resolution**: Traditional methods often require complex algorithms to resolve conflicts, while CRDTs inherently minimize the occurrence of conflicts altogether.
|
||||
|
||||
4. **Server load**: uMap currently relies on central servers (one per instance). Adopting CRDTs could help lower the work done on the server, increasing resilience and scalability.
|
||||
|
||||
### Traditional data synchronization methods
|
||||
|
||||
Traditional data synchronization methods typically rely on a central source of truth (the server) to manage and resolve conflicts. When changes are made by different users, these traditional systems require a round-trip to the server for conflict resolution and thus can be slow or inadequate for real-time collaboration.
|
||||
|
||||
In contrast, CRDTs leverage mathematical properties (the fact that the data types can converge) to ensure that every replica independently reaches the same state, without the need for a central authority, thus minimizing the amount of coordination and communication needed between nodes.
|
||||
|
||||
This ability to maintain consistency sets CRDTs apart from conventional synchronization approaches and makes them particularly valuable for the development of collaborative tools like uMap, where real-time updates and reliability are important.
|
||||
|
||||
### Solving complex cases
|
||||
|
||||
At first, I found CRDTs somewhat confusing, owing to their role in addressing complex challenges. CRDTs come in various forms, with much of their intricacy tied to resolving content conflicts within textual data or elaborate hierarchical structures.
|
||||
|
||||
Fortunately for us, our use case is comparatively straightforward, and we probably only need LWW registers.
|
||||
|
||||
### Last Write Wins Registers
|
||||
|
||||
As you might have guessed from the name, a LWW register is a specific type of CRDT which "just" replaces the value with the last write. The main concern is establishing the sequence of updates, to order them together (who is the last one?).
|
||||
|
||||
In a single-client scenario or with a central time reference, sequencing is straightforward. However, in a distributed environment, time discrepancies across peers can complicate things, as clocks may drift and lose synchronization.
|
||||
|
||||
To address this, CRDTs use vector clocks — a specialized data structure that helps to solve the relative timing of events across distributed systems and pinpoint any inconsistencies.
|
||||
|
||||
> A vector clock is a data structure used for determining the partial ordering of events in a distributed system and detecting causality violations.
|
||||
>
|
||||
> – [Wikipedia](https://en.wikipedia.org/wiki/Vector_clock)
|
||||
|
||||
<div class="align-center">
|
||||
<img alt="CRDTs converging to the same state" src="/images/umap/crdt-converge.png" />
|
||||
</div>
|
||||
|
||||
Note that we could also use a library such as [rxdb](https://github.com/pubkey/rxdb) — to handle the syncing, offline, etc. — because we have a master: we use the server, and we can use it to handle the merge conflicts.
|
||||
But by doing so, we would give more responsibility to the server, whereas when using CRDTs it's possible to do the merge only on the clients (enabling no-master replications).
|
||||
|
||||
### State-based vs Operation based
|
||||
|
||||
While reading the literature, I found that there are two kinds of CRDTs: state-based and operation-based. It turns out most of the CRDTs implementation I looked at are operation-based, and propose an API to interact with them as you're changing the state, so **it doesn't really matter** in practice.
|
||||
|
||||
> The two alternatives are theoretically equivalent, as each can emulate the
|
||||
> other. However, there are practical differences. State-based CRDTs are
|
||||
> often simpler to design and to implement; their only requirement from the
|
||||
> communication substrate is some kind of gossip protocol. **Their drawback is that
|
||||
> the entire state of every CRDT must be transmitted eventually to every other
|
||||
> replica, which may be costly**. In contrast, operation-based CRDTs transmit only
|
||||
> the update operations, which are typically small.
|
||||
>
|
||||
> [Wikipedia on CRDTs](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type)
|
||||
|
||||
### How the server fits in the picture
|
||||
|
||||
While discussing with the automerge team, I understood that I was expecting the server to pass along the messages to the other parties, and that would be the way the synchronization would be done. It turns out I was mistaken: in this approach, the clients send updates to the server, which merges everything together and only then sends the updates to the other peers. It makes it easy for the server to send back the needed information to the clients (for new peers, or if the peers didn't cache the data locally).
|
||||
|
||||
In order to have peers working with each other, I would need to change the way the provider works, so we can have the server be "brainless" and just relay the messages.
|
||||
|
||||
For automerge, it would mean the provider will "just" handle the websocket connection (disconnect and reconnect) and all the peers would be able to talk with each other. The other solution for us would be to have the merge algorithm working on the server side, which comes with upsides (no need to find when the document should be saved by the client to the server) and downsides (it takes some cpu and memory to run the CRDTs on the server)
|
||||
|
||||
### How offline is handled
|
||||
|
||||
I was curious about how offline editing might work, and what would happen when going back online. Changes can happen both online and offline, making no difference for the "reconciliation" step. When going back online, a "patch" is computed by the newly reconnected peer, and sent to the other peers.
|
||||
|
||||
---
|
||||
|
||||
## Part 2: JavaScript CRDTs
|
||||
|
||||
Now that we're familiar with CRDTs and how they can help us, let's create a map application which syncs marker positions, on different browsers.
|
||||
|
||||
We'll be comparing three JavaScript libraries: [Y.js](https://yjs.dev/), [Automerge](https://automerge.org/) and [JSON Joy](https://jsonjoy.com), considering:
|
||||
|
||||
1. **Their external API**: is it easy to use in our case? What are the challenging parts?
|
||||
2. **Community and Support**: What is the size and activity of the developer community / ecosystem?
|
||||
3. **Size of the JS library**, because we want to limit the impact on our users browsers.
|
||||
4. **Efficiency**: Probe the bandwidth when doing edits. What's being transmitted over the wire?
|
||||
|
||||
I setup a demo application for each of the libraries. Everything is available [in a git repository](https://gitlab.com/umap-project/leaflet-sync) if you want to try it out yourself.
|
||||
|
||||
### The demo application
|
||||
|
||||
All the demos are made against the same set of features. It
|
||||
|
||||
- Creates a marker when the map is clicked
|
||||
- Moves the markers on hover.
|
||||
|
||||
This should probably be enough for us to try out.
|
||||
|
||||
Here's the whole code for this, using [Leaflet - a JavaScript library for interactive maps](https://leafletjs.com/).
|
||||
|
||||
|
||||
```js
|
||||
import L from "leaflet";
|
||||
import "leaflet/dist/leaflet.css";
|
||||
|
||||
// Create a map with a default tilelayer.
|
||||
const map = L.map("map").setView([48.1173, -1.6778], 13);
|
||||
L.tileLayer("https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png", {
|
||||
maxZoom: 19,
|
||||
attribution: "© OpenStreetMap contributors",
|
||||
}).addTo(map);
|
||||
|
||||
// Features contains a reference to the marker objects, mapped by the uuids
|
||||
let features = {};
|
||||
|
||||
// An upsert function, creating a marker at the passed latlng.
|
||||
// If an uuid is provided, it changes the coordinates at the given address
|
||||
function upsertMarker({ latlng, uuid }) {
|
||||
if (!uuid) uuid = crypto.randomUUID();
|
||||
let marker;
|
||||
|
||||
if (Object.keys(features).includes(uuid)) {
|
||||
marker = features[uuid];
|
||||
marker.setLatLng(latlng);
|
||||
} else {
|
||||
marker = new L.marker(latlng, {
|
||||
draggable: true,
|
||||
autoPan: true,
|
||||
uuid: uuid,
|
||||
});
|
||||
|
||||
features[uuid] = marker;
|
||||
marker.addTo(map);
|
||||
marker.on("dragend", ({ target }) => {
|
||||
console.log("dragend");
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Add new features to the map with a click
|
||||
map.on("click", ({ latlng }) => {
|
||||
upsertMarker({ latlng });
|
||||
});
|
||||
```
|
||||
|
||||
It does the following:
|
||||
|
||||
- Creates a map zoomed on Rennes, France
|
||||
- Maintains a `features` object, referencing the added markers
|
||||
- Provides a `upsertMarker` function, creating a marker on the map at the given latitude and longitude, and updating its latitude and longitude if it already exists.
|
||||
- It listens to the `click` event on the map, calling `upsertMarker` with the appropriate arguments.
|
||||
|
||||
Note that the data is not "reactive" (in the sense of React apps): there is no central state that's updated and triggers the rendering of the user interface.
|
||||
|
||||
### Y.js
|
||||
|
||||
Y.js is the first library I've looked at, probably because it's the oldest one, and the more commonly referred to.
|
||||
|
||||
The API seem to offer what we look for, and provides a way to [observe changes](https://docs.yjs.dev/api/shared-types/y.map#observing-changes-y.mapevent). Here's how it works:
|
||||
|
||||
```js
|
||||
import * as Y from "yjs";
|
||||
import { WebsocketProvider } from "y-websocket";
|
||||
|
||||
// Instanciate a document
|
||||
const doc = new Y.Doc();
|
||||
```
|
||||
|
||||
When we add a new marker, we update the CRDT (`markers.set`).
|
||||
|
||||
```js
|
||||
let markers = doc.getMap("markers");
|
||||
markers.set(target.options.uuid, target._latlng);
|
||||
```
|
||||
|
||||
Another connected peer can observe the changes, like this:
|
||||
|
||||
```js
|
||||
markers.observe((event, transaction) => {
|
||||
if (!transaction.local) {
|
||||
event.changes.keys.forEach((change, key) => {
|
||||
let value = markers.get(key);
|
||||
switch(change.action){
|
||||
case 'add':
|
||||
case 'update':
|
||||
upsertMarker({ latlng: value, uuid: key, local: true });
|
||||
break;
|
||||
case 'delete':
|
||||
console.log(`Property "${key}" was deleted. ".`);
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
We cycle on the received changes, and then apply them on our map. In the case of an offline peer coming back online after some time, the `observe` event will be called only once.
|
||||
|
||||
I'm not dealing with the case of marker deletions here, but deleted items are also taken into account. The data isn't really deleted in this case, but a "tombstone" is used, making it possible to resolve conflicts (for instance, if one people deleted a marker and some other updated it during the same time).
|
||||
|
||||
Y.js comes with multiple "connection providers", which make it possible to sync with different protocols (there is even [a way to sync over the matrix protocol](https://github.com/yousefED/matrix-crdt) 😇).
|
||||
|
||||
More usefully for us, there is [an implemented protocol for WebSockets](https://github.com/yjs/y-websocket). Here is how to use one of these providers:
|
||||
|
||||
```js
|
||||
// Sync clients with the y-websocket provider
|
||||
const provider = new WebsocketProvider(
|
||||
"ws://localhost:1234",
|
||||
"leaflet-sync",
|
||||
doc
|
||||
);
|
||||
```
|
||||
|
||||
This code setups a WebSocket connection with a server that will maintain a local copy of the CRDT, as explained above.
|
||||
|
||||
It's also possible to send "awareness" information (some state you don't want to persist, like the position of the cursor). It contains some useful meta information, such as the number of connected peers, if they're connected, etc.
|
||||
|
||||
```js
|
||||
map.on("mousemove", ({ latlng }) => {
|
||||
awareness.setLocalStateField("user", {
|
||||
cursor: latlng,
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
I made [a quick proof of concept with Y.js](https://gitlab.com/umap-project/leaflet-sync/-/tree/yjs) in a few hours flawlessly. It handles offline and reconnects, and exposes awareness information.
|
||||
|
||||
#### Python bindings
|
||||
|
||||
Y.js has been ported to rust with [the Y.rs project](https://github.com/y-crdt/y-crdt), making it possible to have binding in other languages, like ruby and python (see [Y.py](https://github.com/y-crdt/ypy)). The python bindings are currently looking for a maintainer.
|
||||
|
||||
#### Library size
|
||||
|
||||
- Y.js is 4,16 Ko
|
||||
- Y-Websocket is 21,14 Ko
|
||||
|
||||
#### The data being transmitted
|
||||
|
||||
In a scenario where all clients connect to a central server, which handles the CRDT locally and then transmits back to other parties, adding 20 points on one client, then 20 points in another generates ~5 Ko of data (so approximately 16 bytes per edit).
|
||||
|
||||
Pros:
|
||||
|
||||
- The API was feeling natural to me: it handles plain old JavaScript objects, making it easy to integrate.
|
||||
- It seems to be widely used, and the community seems active.
|
||||
- It is [well documented](https://docs.yjs.dev/)
|
||||
- There is awareness support
|
||||
|
||||
Cons:
|
||||
|
||||
- It doesn't seem to work well [without a JS bundler](https://github.com/yjs/yjs/issues/325) which could be a problem for us.
|
||||
- The Websocket connection provider doesn't do what I was expecting it to, as it requires the server to run a CRDT locally.
|
||||
|
||||
---
|
||||
### Automerge
|
||||
|
||||
[Automerge](https://automerge.org/) is another CRDT library started by the folks at [Ink & Switch](https://www.inkandswitch.com/) with Martin Kleppmann. Automerge is actually the low level interface. There is a higher-level interface named [Automerge-repo](https://automerge.org/docs/repositories/). Here is how to use it:
|
||||
|
||||
```js
|
||||
import { Repo } from "@automerge/automerge-repo";
|
||||
|
||||
const repo = new Repo();
|
||||
let handle = repo.create()
|
||||
// or repo.find(name)
|
||||
```
|
||||
|
||||
To change the document, call `handle.change` and pass it a function that will make changes to the document.
|
||||
|
||||
Here, when a new marker is added:
|
||||
|
||||
```js
|
||||
handle.change((doc) => {
|
||||
doc[uuid] = cleanLatLng(target._latlng);
|
||||
});
|
||||
```
|
||||
|
||||
I had to use a `cleanLatLng` function in order to not pass the whole object, otherwise it wouldn't serialize. It's really just a simple helper taking the properties of interest for us (and letting away all the rest).
|
||||
|
||||
Another peer can observe the changes, like this:
|
||||
|
||||
```js
|
||||
handle.on("change", ({ doc, patches }) => {
|
||||
patches.forEach(({ action, path }) => {
|
||||
if (path.length == 2 && action === "insert") {
|
||||
let value = doc[path[0]];
|
||||
upsertMarker({ latlng: value, uuid: path[0], local: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
The "patch" interface is a bit less verbose than the one from Y.js. It wasn't well documented, so I had to play around with the messages to understand exactly what the possible actions were. In the end, it gets me what I'm looking for, the changes that occurred to my data.
|
||||
|
||||
Here I'm using `path.length == 2 && action === 'insert'` to detect that a change occurred to the marker. Since we don't make a difference between the creation of a marker and its update (when being moved), it works well for us.
|
||||
|
||||
#### Python bindings
|
||||
|
||||
There are [python bindings](https://github.com/automerge/automerge-py) for automerge "core". (It doesn't — yet — provide ways to interact with "repos").
|
||||
|
||||
#### Library size
|
||||
|
||||
- Size of the automerge + automerge repo: 1,64 mb
|
||||
- Size of the WebSocket provider: 0,10 mb
|
||||
|
||||
This is quite a large bundle size, and the team behind automerge is aware of it and working on a solution.
|
||||
|
||||
#### The data being transmitted
|
||||
|
||||
In the same scenario, adding 20 points on one client, then 20 points in another generates 90 messages and 24,94 Ko of data transmitted (~12 Ko sent and ~12Ko received), so approximately 75 bytes per edit.
|
||||
|
||||
Pros:
|
||||
|
||||
- There is an API to [get informed when a conflict occurred](https://automerge.org/docs/documents/conflicts/)
|
||||
- Python bindings are currently being worked on, soon to reach a stable version
|
||||
- The team was overall very responsive and trying to help.
|
||||
|
||||
Cons:
|
||||
|
||||
- The JavaScript is currently generated via Web Assembly, which could make it harder to debug.
|
||||
- The large bundle size of the generated files.
|
||||
|
||||
---
|
||||
|
||||
### JSON Joy
|
||||
|
||||
[JSON Joy](https://jsonjoy.com) is the latest to the party.
|
||||
|
||||
It takes another stake by providing multiple libraries with a small functional perimeter. It sounds promising, even if still quite new, and would leave us with the hands free in order to implement the protocol that would work for us.
|
||||
|
||||
Here is how to use it. On the different peers you start with different forks of the same document:
|
||||
|
||||
```js
|
||||
import {Model} from 'json-joy/es2020/json-crdt';
|
||||
import { s } from "json-joy/es6/json-crdt-patch";
|
||||
|
||||
// Initiate a model with a custom ID
|
||||
|
||||
const rootModel = Model.withLogicalClock(11111111);
|
||||
|
||||
// populate it with default data
|
||||
rootModel.api.root({
|
||||
markers: {},
|
||||
});
|
||||
|
||||
// Fork it on each client
|
||||
let userModel = rootModel.fork();
|
||||
```
|
||||
|
||||
When adding a new marker, we can define a new constant, by using `s.con`…
|
||||
|
||||
```js
|
||||
userModel.api.obj(["markers"]).set({
|
||||
[uuid]: s.con(target._latlng),
|
||||
});
|
||||
```
|
||||
|
||||
… and then create a patch and send it to the other peers:
|
||||
|
||||
```js
|
||||
import { encode, decode } from "json-joy/es6/json-crdt-patch/codec/verbose";
|
||||
|
||||
let patch = userModel.api.flush();
|
||||
let payload = encode(patch);
|
||||
```
|
||||
|
||||
On the other peers, when we receive a patch message, decode it and apply it:
|
||||
|
||||
```js
|
||||
let patch = decode(payload);
|
||||
model.api.apply(patch);
|
||||
```
|
||||
|
||||
We can observe the changes this way:
|
||||
|
||||
```js
|
||||
userModel.api.onPatch.listen((patch) => {
|
||||
patch.ops.forEach((op) => {
|
||||
if (op.name() === "ins_obj") {
|
||||
let key = op.data[0][0];
|
||||
let value = userModel.view().markers[key];
|
||||
upsertMarker({ latlng: value, uuid: key, local: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
Similarly to what we did with automerge, we're having a look into the patch, and filter on the operations of interest for us (`ins_obj`). The names of the operations aren't clearly specified by the spec.
|
||||
|
||||
Metrics:
|
||||
|
||||
- Size: 143 ko
|
||||
- Data transmitted for 2 peers and 40 edits: (35 bytes per edit)
|
||||
|
||||
Pros:
|
||||
|
||||
- Small atomic libraries, making it easy to use only the parts we need.
|
||||
- The interface [proposes to store different type of data](https://jsonjoy.com/libs/json-joy-js/json-crdt/guide/node-types) (constants, values, arrays, etc.)
|
||||
- Distributed as different type of JS bundles (modules, wasm, etc.)
|
||||
- Low level, so you know what you're doing
|
||||
|
||||
Cons:
|
||||
|
||||
- It doesn't provide a high level interface for sync
|
||||
- It's currently a one-person project, without clear community channels to gather with other interested folks.
|
||||
- Quite recent, so probably rough spots are to be found
|
||||
|
||||
---
|
||||
|
||||
## Part 3: Comparison table
|
||||
|
||||
I found Y.js and Automerge quite similar for my use case, while JSON Joy was taking a different (less "all-included") approach. Here is a summary table to help read the differences I found.
|
||||
|
||||
| | Y.js | Automerge | JSON Joy |
|
||||
| ---------------------- | ------------------------------------ | -------------------------------------------------- | ------------------------------------------------------- |
|
||||
| Python bindings | [Yes](https://github.com/y-crdt/ypy) | [Yes](https://github.com/automerge/automerge-py) | No |
|
||||
| Syncing | Native JS structures | Transactional functions | Specific types (bonus points for handling constants) |
|
||||
| Coded in | JavaScript / Rust | TypeScript / Rust | Typescript |
|
||||
| Awareness protocol | Yes, with presence | Yes, without presence | No |
|
||||
| Conflict-detection API | No | Yes | No |
|
||||
| Library size | **24.3Ko** §<br> | **1,74 mb** § | **143 ko** |
|
||||
|
||||
§ size of the connectors included
|
||||
|
||||
### Working with patches
|
||||
|
||||
In order to observe the changes, we need to inspect the given patches and work on what we find. I found the different libraries expose different sets of APIs. All of these APIs were quite a bit hard to find, and it's not clear if they are public or private.
|
||||
|
||||
One thing to keep in mind is that these "patch" events happen only once per patch received. You can see it as a "diff" of the state between the current and the incoming states.
|
||||
|
||||
- Y.js exposes a utility which is able to tell you what the action on the key is ("delete", "update" and "add")
|
||||
- Automerge and JSON Joy, on the other hand, don't provide such utility functions, meaning you would need to find that out yourself.
|
||||
|
||||
### Conclusion
|
||||
|
||||
The goal here is not to tell which one of these libraries is the best one. They're all great and have their strenghs. None of them implement the high-level API I was expecting, where the clients talk with each other and the server just relays messages, but maybe it's because it's better in general to have the server have the representation of the data, saving a roundtrip for the clients.
|
||||
|
||||
I wasn't expecting to have a look at patches to understand what changed at the low level. The way it's currently implemented is very suitable for "reactive" applications, but require more involvement to sync between the CRDTs state and the application state.
|
||||
|
||||
In the end, adding CRDTs is made very simple, probably due to the fact all we really need is a sort of distributed key/value store.
|
||||
|
||||
I'm not sure yet which library we will end up using for uMap (if any), but my understanding is clearer than it was when I started. I guess that's what progress looks like 😎
|
||||
|
||||
## Extra notes
|
||||
|
||||
### YATA and RGA
|
||||
|
||||
While researching, I found that the two popular CRDTs implementation out there use different approaches for the virtual counter:
|
||||
|
||||
> - **RGA** [used by Automerge] maintains a single globally incremented counter (which can be ordinary integer value), that's updated anytime we detect that remote insert has an id
|
||||
> with sequence number higher that local counter. Therefore, every time we produce
|
||||
> a new insert operation, we give it a highest counter value known at the time.
|
||||
> - **YATA** [used by Yjs] also uses a single integer value, however unlike in case of RGA we
|
||||
> don't use a single counter shared with other replicas, but rather let each
|
||||
> peer keep its own, which is incremented monotonically only by that peer. Since
|
||||
> increments are monotonic, we can also use them to detect missing operations eg.
|
||||
> updates marked as A:1 and A:3 imply, that there must be another (potentially
|
||||
> missing) update A:2.
|
||||
|
||||
### Resources
|
||||
|
||||
- [CRDTs: The Hard Parts](https://www.youtube.com/watch?v=x7drE24geUw), a video by Martin Kleppmann where he explains the current state of the art of CRDTs, and why some problems aren't solved yet.
|
||||
- [An Interactive Intro to CRDTs](https://jakelazaroff.com/words/an-interactive-intro-to-crdts/) gets you trough different steps to understand what are CRDTs, and how to implement a LWW Register.
|
||||
- [Bartosz Sypytkowski](https://www.bartoszsypytkowski.com/the-state-of-a-state-based-crdts/) introduction on CRDTs, with practical examples is very intuitive.
|
||||
- [CRDT Implementations](https://jzhao.xyz/thoughts/CRDT-Implementations) are notes by Jacky which were useful to me when understanding CRDTs.
|
||||
|
|
@ -1,20 +1,20 @@
|
|||
User-agent: ChatGPT-User
|
||||
Disallow: /
|
||||
user-agent: ChatGPT-User
|
||||
disallow: /
|
||||
|
||||
User-agent: GPTBot
|
||||
Disallow: /
|
||||
user-agent: GPTBot
|
||||
disallow: /
|
||||
|
||||
User-agent: Twitterbot
|
||||
Disallow: /
|
||||
user-agent: Twitterbot
|
||||
disallow: /
|
||||
|
||||
User-agent: CCBot
|
||||
Disallow: /
|
||||
user-agent: CCBot
|
||||
disallow: /
|
||||
|
||||
User-agent: Google-Extended
|
||||
Disallow: /
|
||||
user-agent: Google-Extended
|
||||
disallow: /
|
||||
|
||||
User-agent: Omgilibot
|
||||
Disallow: /
|
||||
user-agent: Omgilibot
|
||||
disallow: /
|
||||
|
||||
User-agent: FacebookBot
|
||||
Disallow: /
|
||||
user-agent: FacebookBot
|
||||
disallow: /
|
||||
|
|
|
|||
Binary file not shown.
|
Before Width: | Height: | Size: 93 KiB After Width: | Height: | Size: 33 KiB |
71
content/pages/resume.md
Normal file
71
content/pages/resume.md
Normal file
|
|
@ -0,0 +1,71 @@
|
|||
---
|
||||
save_as: resume/index.html
|
||||
slug: resume
|
||||
title: Alexis Métaireau / Resume
|
||||
---
|
||||
# Alexis Métaireau - Resume
|
||||
|
||||
---
|
||||
|
||||
| **Email** | [alexis@notmyidea.org](mailto:alexis@notmyidea.org) |
|
||||
| ----------- | ------------------------------------------------------------------------------------------------------------- |
|
||||
| **Code** | [https://github.com/almet](https://github.com/almet) and [https://gitlab.com/almet](https://gitlab.com/almet) |
|
||||
| **Website** | [https://notmyidea.org](https://notmyidea.org) |
|
||||
| **Country** | France |
|
||||
|
||||
---
|
||||
## Core Values and Interests
|
||||
|
||||
- **Free Software**: Active engagement with open-source software for years, enjoying the community spirit and culture of openness.
|
||||
- **Cooperation and Mediation**: Enjoys collaborating with others and ensuring healthy interactions are as important as the work itself.
|
||||
- **Pragmatism**: Strives for good practices without getting bogged down in unnecessary details, embracing the principle that "perfect can be the enemy of good."
|
||||
- **Positive Mindset**: Approaches challenges with optimism, valuing a balanced and enjoyable working environment.
|
||||
|
||||
## Projects
|
||||
|
||||
- #### 🗺️ [uMap](https://umap-project.org) (2023-2024)
|
||||
Collaborative web-based map creation tool. Working on real-time collaborative features.
|
||||
|
||||
- #### 🚨 [Argos](https://framasoft.frama.io/framaspace/argos/) (2023-2024)
|
||||
Web supervision software and status board created for the [Framasoft Association](https://framasoft.org).
|
||||
|
||||
- #### 🧶 [Jacquard Pattern Generator](https://bekeko.notmyidea.org/) (2022)
|
||||
Tool for designing knitting patterns. Simplifies color and pattern choices for hand-crafted projects. ([Coded in Elm](https://github.com/almet/bekeko))
|
||||
|
||||
- #### 🙌 [Copanier](https://github.com/spiral-project/copanier) (2019-2024)
|
||||
Web software to facilitate group purchases, modified for a co-op. ([Coded in Python](https://github.com/almet/copanier))
|
||||
|
||||
- #### 💸 [I Hate Money](http://ihatemoney.org) (2011-2023)
|
||||
Expense management web service for groups. Maintains an open instance at [ihatemoney.org](https://ihatemoney.org). ([Coded in Python+Flask](https://github.com/spiral-project/ihatemoney))
|
||||
|
||||
- #### 🔄 [Kinto](https://github.com/kinto/kinto) (2012-2015)
|
||||
Generic backend for web applications, initiated at Mozilla. Used for data synchronization in Firefox. ([Coded in Python+Pyramid](https://github.com/Kinto/kinto))
|
||||
|
||||
- #### ✍️ [Pelican](http://getpelican.com) (2010-2017)
|
||||
Written a static site generator for transforming Markdown files into easily hosted websites. Used by projects like the [Linux Kernel](https://www.kernel.org/pelican.html) and [Debian](https://bits.debian.org/pages/about.html). ([Coded in Python](https://github.com/getpelican/pelican))
|
||||
|
||||
## Professional Experience
|
||||
|
||||
- #### Independent Developer (Since 2023)
|
||||
Contractor for different projects. [Blog](https://blog.notmyidea.org)
|
||||
|
||||
- #### Co-founder, [Brasserie du Vieux Singe](https://www.vieuxsinge.com/) (2017 - 2023)
|
||||
Established an organic craft brewery promoting cooperation. Transitioned out in the summer of 2023 to return to development.
|
||||
|
||||
- #### Engineer, [Mozilla](https://mozilla.org) (2011 - 2016)
|
||||
Part of the "Services" team working on user data, encryption, scaling sites like addons.mozilla.com, and data synchronization.
|
||||
|
||||
- #### [Le Grappe](https://www.reseaugrappe.org/) (2007 - 2012)
|
||||
Co-created and engaged in managing a network of environmental project organizations during student years, fostering collectivist values.
|
||||
|
||||
## Technical Skills
|
||||
|
||||
- **Python**: Regularly used, with familiarity in its ecosystem and community.
|
||||
- **Linux**: Comfortable in operating system and server management, I use [Arch Linux](https://archlinux.org/) and [Debian](https://debian.org).
|
||||
- **Frontend**: Enjoyed using [Elm](https://elm-lang.org/) for robust frontend development. I also use HTML, CSS, and JavaScript.
|
||||
- **System Administration**: Experience with managing services for self and others, includes server maintenance.
|
||||
|
||||
## Education
|
||||
|
||||
- **Master's Degree in Software Engineering**, University of Oxford Brookes.
|
||||
- **BTS Informatique et Réseau**, Angers.
|
||||
|
|
@ -4,7 +4,11 @@ save_as: argos/index.html
|
|||
total_days: 8
|
||||
template: worklog
|
||||
---
|
||||
## Jeudi 1 Février 2023 (0h, 1h bénévoles, 4/5)
|
||||
|
||||
## Vendredi 5 Avril 2024 (0h, 3h bénévoles, 5/5)
|
||||
|
||||
Je suis repassé sur le code, changé la manière dont le packaging était fait (on passe maintenant par hatch) puis documenté le processus de release.
|
||||
## Jeudi 1 Février 2024 (0h, 1h bénévoles, 4/5)
|
||||
|
||||
Petite session de pair programming avec Luc, dans laquelle on discute de comment ajouter des tests avec pytest.
|
||||
On en profite pour faire un tour rapide des quelques requêtes de fusion en attente, et de tester que les migrations fonctionnent bien chez lui.
|
||||
|
|
|
|||
41
content/pages/worklog/flash.md
Normal file
41
content/pages/worklog/flash.md
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
---
|
||||
title: Flash Messenger
|
||||
save_as: flash/index.html
|
||||
template: worklog
|
||||
---
|
||||
|
||||
Todo:
|
||||
|
||||
- [ ] Ecrire la roadmap quelque part
|
||||
|
||||
## Jeudi 28 Mars 2024 (0h, 3h bénévoles, 5/5)
|
||||
|
||||
Je reprends le travail du flux d'autentification, pour [le mettre dans un readme](https://gitlab.com/flashmessenger/flashmessenger-server/#authentication-flow) avec un graphique client/serveur.
|
||||
Je continue d'avancer sur le client en python, en me rendant compte au milieu que les primitives de chiffrement en curves elliptiques "NIST" [ne sont pas considérées sécures](https://safecurves.cr.yp.to/) pour du chiffrement de données.
|
||||
|
||||
Je bifurque donc vers l'utilisation de chiffrement symétrique (juste une clé privée) plutôt qu'assimétrique (clé privée + clé publique) pour la partie chiffrement. J'en profite pour regarder un peu ce qu'il se fait en terme de bibliothèques (libsodium, cryptography, [noise](https://noiseprotocol.org/noise.html)) et je réussis à mieux les mettre dans différentes cases. Noise me semble proposer une approche plus haut niveau de comment les primitives de crypto sont utilisées ensemble, dans un protocole qui a du sens du point de vue sécurité, en rendant les vecteurs d'attaque clairs.
|
||||
|
||||
J'avance sur le client en python, et je commence à faire l'intégration avec le serveur en rust pour [la vérification des signatures](https://docs.rs/ring/latest/ring/signature/index.html)
|
||||
|
||||
## Mercredi 27 Mars 2024 (0h, 9h bénévoles, 4/5)
|
||||
|
||||
Le matin on se fait un point avec tout le monde, pour prévoir une roadmap pour le futur, qu'il faudrait retransmettre quelque part.
|
||||
Je me mets en quête de compiler le serveur rust vers une plateform `x86_64 Linux` depuis une architecture `arm64`. J'apprends au passage comment fonctionne la toolchain de rust.
|
||||
Le serveur tourne, donc, mais ne veux pas se connecter au redis. Je ne trouve pas pourquoi et je passe à autre chose.
|
||||
|
||||
Je passe sur la rédaction d'un client en python qui génère des clés, et qui les utilise pour créer un canal de discussion. C'est à la fois plus dans ma zone de confort de code, mais également sur des problématiques crypto que je ne maitrise pas tout à fait.
|
||||
|
||||
Au passage, je jette un coup d'œil au travail en cours de D. qui arrive à cross compiler du rust vers du Java pour Android, en utilisant des JNI et la bibliothèque rust qui va bien.
|
||||
|
||||
On se pose aussi avec T. et D. pour réfléchir aux flux d'authentification et de chiffrement des données, parce que je n'arrivais pas à y voir clair seul. Il se trouve qu'il fallait signer les messages avec la clé d'authentification privée, pour pouvoir s'authentifier auprès du serveur pour ce canal.
|
||||
|
||||
## Mardi 26 Mars 2024 (0h, 12h bénévoles, 4/5)
|
||||
|
||||
Une matinée parler à échanger sur ce qu'on veut faire, et comment on veut se séparer le travail.
|
||||
|
||||
L'équipe est naissante, on n'a pas encore l'habitude de travailler ensemble. Je mesure un écart entre les enjeux techniques et les enjeux de simplicité d'utilisation (ça me rappelle [le triangle de Zooko](https://fr.wikipedia.org/wiki/Triangle_de_Zooko)). On se dit qu'avant de partir sur une version plus compliquée, on a envie de creuser aussi sur les aspects techniques.
|
||||
Je pars sur un prototype de serveur en rust. On s'y mets à plusieurs et on réussis à avoir en fin de soirée un code qui est capable de parler à une base de données redis (ou [redict](https://codeberg.org/redict/redict/)).
|
||||
|
||||
J'ai beaucoup aimé un passage ou on était à trois à essayer de faire fonctionner un bout de code rust, en tergiversant et en essayant de comprendre les concepts. C'est vraiment ma manière d'apprendre, en faisant.
|
||||
|
||||
|
||||
|
|
@ -4,6 +4,10 @@ save_as: notmyidea/index.html
|
|||
template: worklog
|
||||
---
|
||||
|
||||
## Lundi 29 Mars 2024 (0h, 2h bénévoles, 4/5)
|
||||
|
||||
J'ai pris du temps pour faire un CV et une lettre de motivation pour la freedom of the press foundation, qui recherche quelqu'un pour travailler sur [Danger Zone](https://dangerzone.rocks/). Je me dis que ce serait l'occasion d'avancer sur des outils qui servent à la liberté de la presse, et ça me donne envie. Je me demande bien sur comment ça pourrait fonctionner avec uMap. A suivre.
|
||||
|
||||
## Mercredi 23 Novembre 2023 (0h, 5h bénévoles, 5/5)
|
||||
|
||||
J'ai passé du temps à coder un système qui me permet de faire le suivi de mes heures, par projet.make
|
||||
J'ai passé du temps à coder un système qui me permet de faire le suivi de mes heures, par projet.
|
||||
|
|
|
|||
|
|
@ -2,8 +2,63 @@
|
|||
title: uMap
|
||||
save_as: umap/index.html
|
||||
template: worklog
|
||||
total_days: 25
|
||||
total_days: 90
|
||||
---
|
||||
## Mardi 05 Avril 2024 (4h)
|
||||
|
||||
J'ai fait passer les tests sur la Pull Request en cours. Les tests ajoutés sur cette PR m'ont permis de detecter des bugs que j'avais introduit lors de la refactorisation des `utils`, content de les trouver 😅.
|
||||
|
||||
Je fais un tour des différents outils qui permettent l'édition collaborative et je note les parcours qui y sont présents.
|
||||
|
||||
Je continue ma réflexion autour de la propagation des changements locaux vers d'autres pairs: actuellement ces changements ne sont pas liés au format GeoJSON, et je me demande si cela serait possible. Ça aurait l'avantage de s'intégrer facilement avec d'autres outils qui utilisent ce format, et ça permettrait de résoudre le problème du chargement initial: ce ne serait plus les clients qui enverraient leur dernière version courante, mais le serveur qui compacterait les opérations en attente.
|
||||
|
||||
## Lundi 01 Avril 2024 (5h, 5/5)
|
||||
|
||||
J'ai relu, modifié puis envoyé la proposition pour les tuiles vectorielles pour uMap. J'ai ensuite discuté avec Vadims (de JSON Joy) de notre cas d'utilisation. Il semble ressortir qu'il serait quand même plus simple d'avoir un serveur qui est capable d'avoir une representation de l'état du document.
|
||||
|
||||
Le serveur pourrait stocker les opérations (indéxées) qui lui sont envoyées, avec une vue de l'état du document, qui serait compacté de temps en temps.
|
||||
|
||||
On a évoqué le fait que ce serait aussi peut-être plus simple pour nous d'utiliser des Hybrid Logical Clocks (ts + logical time + userId), et de recoder un CRDT nous même. Il m'a parlé de museapp ([Metamuse podcast — Muse](https://museapp.com/podcast/)) qui semble avoir fait ça et qui en ont parlé dans un podcast.
|
||||
|
||||
J'ai ensuite mergé les deux PR en attente sur les changement dans la suite de test, et rajouté quelques entrées dans le schema, qui ne prenait pas en compte les données à l'intérieur des layers (choropleth, etc.).
|
||||
## Jeudi 29 Mars 2024 (5h, 4/5)
|
||||
|
||||
J'ai travaillé sur deux propositions de financement: une pour NLNet pour la quelle on propose d'ajouter les fonctionnalitézs de tuiles vectorielles, et l'autre pour Google Season of Docs ou on aimerait bien avoir quelqu'un qui nous aide à améliorer la documentation technique.
|
||||
|
||||
J'ai aussi avancé sur la séparation des tests unitaires JS avec le reste, et ça passe !
|
||||
|
||||
## Lundi 25 Mars 2024 (9h, 4/5)
|
||||
|
||||
Le matin je travaille à faire passer les tests. J'ai pas mal bloqué sur le JSDom (encore), et la manière d'intégrer tout ça dans le contexte des tests. C'est plus clair maintenant, mais j'aurai aimé que ce soit plus simple dès le début.
|
||||
|
||||
L'après midi à été utilisée à comprendre ce qui s'est passé lors de la mise en prod, le passage aux UUIDs étant moins simple que prévu, avec un cas limite qui arrive parce que la liste des fichiers à purger (les anciennes versions) mettait en tête de liste les nouveaux fichiers, qui utilisaient les UUIDs (en tout cas, dans certains cas). On a mis quelques heures a trouver ce qui se passait, à priori ça devrait être réparé.
|
||||
|
||||
## Vendredi 22 Mars 2024 (4h, 3/5)
|
||||
|
||||
On a fait un point avec Virgile autour du google season of docs, on se dit que ça pourrait être chouette de faire une proposition sur la partie documentation technique.
|
||||
J'enchaine avec un point sur une prochaine session NLNet, ou on aimerait proposer de faire des vector tiles. Je comprends mieux de quoi il s'agit, et je vois les futurs que ça ouvre pour uMap, entre autres avec le lien possible avec les données OSM.
|
||||
|
||||
J'aimerai bien que ce soit une étape dans l'idée d'avoir un jour des cartes plus facilement accessibles hors ligne, et synchronisables. J'ai enchainé sur le fait de séparer les tests unitaires actuels, qui tournent dans un navigateur, du reste des tests. L'idée étant de les faire tourner dans un contexte de ligne de commande, pour s'intégrer avec le CI, entre autres.
|
||||
|
||||
## Jeudi 21 Mars 2024 (6h, 3/5)
|
||||
|
||||
J'ai passé une journée à écrire des tests, à la fois pour playwright (j'en ai profité pour découvrir qu'il était possible d'enregistrer sa session, et que le code soit produit pour moi derrière), et pour des tests unitaires JS. Je tire un peu la langue, c'est long et fastidieux, et je n'ai pas encore terminé.
|
||||
|
||||
## Lundi 18 Mars 2024 (6h, 5/5)
|
||||
|
||||
Je commence à creuser sur l'intégration des websockets avec Django Channels, l'implication technique que ça pourrait avoir pour les personnes qui déploient, pour finalement changer d'approche en fin d'après-midi suite à une discussion avec David et Yohan, ce sera surement plus simple d'ajouter un serveur de manière séparée (au moins pour le moment) pour les personnes qui ont envie d'ajouter de la synchro.
|
||||
|
||||
On s'est fait un moment de rétrospective, avec le format du conseil de famille, proposé par David. Je me suis senti faire équipe avec le reste des participant·es.
|
||||
|
||||
Puis, un moment pour planifier les prochains développement. On discute de comment nommer les jalons dans notre outil de gestion des fonctionnalités.
|
||||
|
||||
## Dimanche 17 Mars 2024 (2h, 5/5)
|
||||
|
||||
J'ai continué à améliorer l'article sur les CRDTs.
|
||||
|
||||
## Samedi 16 Mars 2024 (2h, 5/5)
|
||||
|
||||
J'ai refais une passe sur l'article sur les CRDTs, en changeant sa structure et en clarifiant certains aspects. On est pas loin d'un article prêt, j'aimerai bien refaire une dernière passe dessus pour que les "key takeaways" soient plus clairs (et moins nombreux).
|
||||
|
||||
## Vendredi 15 Mars 2024 (4h, 5/5)
|
||||
|
||||
|
|
|
|||
51
content/weeknotes/21-22.md
Normal file
51
content/weeknotes/21-22.md
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
---
|
||||
date: 2024-03-19
|
||||
headline: notes hebdo de la semaine
|
||||
projects: umap
|
||||
---
|
||||
# Notes hebdo #21-22
|
||||
|
||||
## Ce qui s'est passé
|
||||
|
||||
**🗺️ [uMap](https://umap-project.org)**
|
||||
: J'ai avancé sur des parties un peu souterraines, qui permettent de choisir ce qui va être *rendu* à l'écran lorsque les données sont elles mêmes modifiées.
|
||||
: J'ai publié l'article sur la comparaison des technologies de gestion des conflits pour permettre une édition décentralisée et avec des connections intermitentes (« CRDT »)
|
||||
: Une rétrospective avec tout le monde, c'était un chouette moment ou je nous ai senti faire équipe
|
||||
: Rajouté des tests (pour m'assurer que tout marche bien comme prévu), mais aussi à faire en sorte que la suite de tests actuels soit utilisable en ligne de commande. C'est plus long que prévu, mais ça avance !
|
||||
: Une démonstration de l'utilisation de certaines bibliothéques dans le cadre d'une application de carto, et pu entammer des discussions avec des personnes de chez automerge, qui aimeraient pousser l'utilisation de cartes, notamment dans le cadre hors-ligne.
|
||||
|
||||
Notmyidea:
|
||||
: J'ai donné un coup de jeune a mon site web. J'ai l'impression de gagner en clarté.
|
||||
|
||||
## Des joies 🤗
|
||||
|
||||
- Réussir à tenir le cadre du travail alors que j'étais dans un cadre plutôt orienté « vacances ». J'ai vraiment fait plusieurs bonnes sessions de travail !
|
||||
- Fier de me sentir plus efficace sur la base de code actuelle de umap. Je sens certains points de tension se débloquer.
|
||||
- Avoir du temps à donner à mes parents pour leur prêter main forte dans un voyage un peu long et fatiguant sinon.
|
||||
- Recevoir un moment pour discuter facilitation avec Frank. Ça me touche que quelqu'un dont c'est le métier prenne le temps d'en discuter avec moi sans plus d'attentes.
|
||||
- Aller observer les oiseaux au delta de l'Ebre. Mes premiers Ibis Falsinele, des Talève sutanes par centaines, beaucoup de flamands roses. Incroyable.
|
||||
|
||||
## Des peines 😬
|
||||
|
||||
- J'ai mal au dos, et je ne comprends pas bien pourquoi. Un passage chez l'ostéo ne m'a pas tout à fait réparé. J'aimerai vraiment me remettre en mouvement, le changement de la brasserie au dev se fait sentir dans mon corps (moins de port de charge, mais aussi moins d'activité physique).
|
||||
- Je me suis trouvé pendu à mon téléphone en itinérance. J'aimerai réussir à m'en défaire, pour donner plus de place aux rencontres.
|
||||
- Je n'ai pas suivi mon intuition qui me disait de remettre une discussion à plus tard. La discussion à donc eu lieu au mauvais moment. On s'en est sorti, mais j'aimerai saisir les opportunités de me taire.
|
||||
- Je me suis senti mal à l'aise et je n'ai pas osé entamer une discussion pour chercher des alliés. J'aimerai me l'autoriser, c'était possible et utile.
|
||||
- J'ai décalé une session « Autodéfense numérique » parce que certaines personnes ne pouvaient plus venir, faisant peser leur décision sur les autres. J'aimerai mieux m'organiser pour la prochaine.
|
||||
|
||||
## Vu, Lu, etc
|
||||
|
||||
- 📖 Lu [Gestion de(s) conflit(s)](https://www.cdgai.be/publications/gestion-des-conflits/) de Rose-Marie Dethier. Un résumé assez concis (86 pages) de pas mal de choses que j'ai pu lire sur le sujet. J'aime bien le fait que beaucoup de choses y soient. Envie de le trouver au format papier.
|
||||
- 🎧 Écouté [Les Couilles sur la table: (Une) Histoire des masculinités juives, première partie](https://www.binge.audio/podcast/les-couilles-sur-la-table/(une)-histoire-des-masculinites-juives-(1-2)) [et deuxième](https://www.binge.audio/podcast/les-couilles-sur-la-table/une-histoire-des-masculinites-juives-2-2) qui vient alimenter une discussion familiale, c'est précieux d'avoir ce genre d'émissions.
|
||||
- 🎧 Écouté plein d'épisodes de « Programme B »:
|
||||
- [La mytho des fachos, l'Adrénochrome](https://www.binge.audio/podcast/programme-b/la-mytho-des-fachos-ladrenochrome) je trouve ça dingue que ce mythe vienne d'un livre. C'est toujours chouette de lire les liens entre le milieu conspi et l'extrême droite.
|
||||
- Un [épisode sur les tradwives](https://www.binge.audio/podcast/programme-b/la-mytho-des-fachos-les-tradwives) ou les liens entre les milieux masculinistes et traditionalistes.
|
||||
- [Aux origines de la discrimination positive](https://www.binge.audio/podcast/programme-b/aux-origines-de-la-discrimination-positive) qui revient sur l'histoire de la discrimination positive aux états unis (je ne savais pas que la cour suprème avait décidé d'y mettre un terme en 2023)
|
||||
- La série « ils ont fait la télé d'Hanouna », avec [un épisode sur Ardisson et l'autre sur Cauet](https://www.binge.audio/podcast/programme-b/ils-ont-fait-la-tele-dhanouna)
|
||||
- 🎧 Écouté le podcast « Encore heureux »
|
||||
- Des épisodes [sur la santé mentale au travail](https://www.binge.audio/podcast/encoreheureux/bien-etre-mental-au-travail-1-5) (et [un second](https://www.binge.audio/podcast/encoreheureux/bien-etre-mental-au-travail-2-5)) qui ont fait écho avec des situations vécues. J'aurai aimé avoir cette resource il y a deux ans, même si je reste quand même un peu surpris du format « sponsorisé par » avec un épisode ou l'invitée travaille chez le sponsor.
|
||||
- Un épisode [sur le rapport au sport](https://www.binge.audio/podcast/encoreheureux/se-reconcilier-avec-le-sport-pourquoi-faire) dans lequel j'ai beaucoup aimé l'approche de « se mettre en mouvement », pour bouger sans se mettre dans une quête de compétition.
|
||||
- Un autre [sur le rapport à la créativité](https://www.binge.audio/podcast/encoreheureux/creer-oui-mais-pourquoi) dans lequel j'ai beaucoup aimé le concept des jeux finis (ou l'objectif est de s'amuser) VS les jeux finis (ou il faut gagner).
|
||||
- [What Makes Disney Villains so Gay? - YouTube](https://www.youtube.com/watch?v=GsWpUSEKSbk). J'ai appris entre autre comment les états-unis ont mis en place [le code Hays](https://fr.wikipedia.org/wiki/Code_Hays) en 1934, qui empêchait les films d'avoir des représentations homosexuelles (et de manquer de respect à la police !). Ce code nécessitait également que les « péchés » soient uniquement représentés par des méchants/antagonistes, qui étaient punis à la fin du film.
|
||||
- [Le syndrome Magneto : Pourquoi sont-ils si méchants ? - YouTube](https://www.youtube.com/watch?v=VXNcemkm2zY&t=2s)
|
||||
- [Sainte-Soline, Autopsie d'un carnage - YouTube](https://www.youtube.com/watch?v=3ymjnILRclQ)
|
||||
51
content/weeknotes/23.md
Normal file
51
content/weeknotes/23.md
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
---
|
||||
date: 2024-04-01
|
||||
headline: notes hebdo de la semaine
|
||||
projects: umap
|
||||
---
|
||||
# Notes hebdo #23
|
||||
|
||||
## Ce qui s'est passé
|
||||
|
||||
**🗺️ [uMap](https://umap-project.org)**
|
||||
: Des modifications sur la manière dont les tests sont lancés, pour faire en sorte qu'il soit plus facile de détecter des régressions. C'est utile dans le contexte actuel ou on bouge beaucoup la base de code, et ça s'insère dans notre intégration continue.
|
||||
: Une mise en production qui se passe mal et qui [nous fait perdre des données](https://forum.openstreetmap.fr/t/mise-en-production-umap-v2-1-x-compliquee-perte-des-dernieres-donnees-sauvegardees-sur-52-cartes/22336). On n'avait pas prévu que le passage des identifiants séquentiels vers des identifiants uniques (uuid) allait rentrer en conflit avec le cache qui permet de garder d'anciennes versions accessibles. Ça nous aura pris du temps pour comprendre, et pour boucher les trous pour éviter que la navire prenne l'eau.
|
||||
: J'ai travaillé sur une demande de financement pour NLNet autour du support des tuiles vectorielles. Affaire à suivre.
|
||||
|
||||
⚡️ **Flash Messenger**
|
||||
: Deux jours de travail (en mode *hackaton*) sur un projet de messagerie chiffrée qui efface les messages de sa mémoire une fois lus. Mes premiers pas en rust, et l'occasion de refaire un peu de chiffrement de données en python.
|
||||
|
||||
## Des joies 🤗
|
||||
|
||||
- Apprendre avec d'autres personnes. C'était grisant d'échanger autour du code, de comprendre des concepts en faisant.
|
||||
- Relâcher la pression pour avancer sur uMap, ça fait du bien de travailler quand je sens que ça va avancer, et de ne pas le faire quand je sens que j'ai la tête ailleurs.
|
||||
## Des peines 😬
|
||||
|
||||
- J'étais complètement décalé et en gros manque de sommeil. J'aimerai m'en rendre compte plus tôt pour changer de schéma.
|
||||
- Je suis triste de voir que parfois je prends une posture de « sachant » qui pourrait être intéressante mais en pratique bloque la conversation. J'aimerai m'en défaire, que les discussions s'ouvrent.
|
||||
|
||||
## Vu, Lu, etc
|
||||
|
||||
- J'ai commencé la lecture de « La fille feu follet » de Ursula K Leguin.
|
||||
- [How to change a system (18 ways) – Changeology](https://www.enablingchange.com.au/blog/strategy-2/how-to-change-a-system/) qui fait un travail de fond sur les différentes manières de faire évoluer un système. Beaucoup de pistes, je me demande comment c'est utilisable en pratique.
|
||||
|
||||
## Notes
|
||||
|
||||
### Cryptographie
|
||||
|
||||
En avançant sur Flash Messenger, j'ai découvert que les primitives en curves elliptiques "NIST" [ne sont pas considérées sécures](https://safecurves.cr.yp.to/) pour du chiffrement de données. Ce sont pourtant celles qui sont le plus utilisées à l'heure actuelle, il me semble. J'aimerai mieux comprendre en quoi (et comment) ça pourrait être un problème.
|
||||
|
||||
Je bifurque vers l'utilisation de chiffrement symétrique (juste une clé privée) plutôt qu'asymétrique (clé privée + clé publique) pour la partie chiffrement. J'en profite pour regarder un peu ce qu'il se fait en terme de bibliothèques ([libsodium](https://pynacl.readthedocs.io/en/latest/), [cryptography](https://cryptography.io/), [noise](https://noiseprotocol.org/noise.html)) et je réussis à mieux les mettre dans différentes cases.
|
||||
|
||||
Noise me semble proposer une approche plus haut niveau de comment les primitives de crypto sont utilisées ensemble, dans un protocole qui a du sens du point de vue sécurité, en rendant les vecteurs d'attaque clairs.
|
||||
|
||||
---
|
||||
### Travail
|
||||
|
||||
> Le management moderne cherche à responsabiliser le·la salarié·e par rapport au maintien de son propre emploi et de son propre salaire, ce qui n’a rien d’original et s’étend à l’ensemble du monde du travail. En droit du travail, il relève pourtant de la responsabilité de l’employeur de fournir un travail et de le rémunérer. La responsabilité du salarié est alors d’exécuter le travail prévu dans le contrat de travail en se subordonnant aux directives, au contrôle et aux possibles sanctions de son employeur. **Le discours selon lequel le salarié devrait être compétitif, flexible, dévoué pour maintenir son entreprise et son emploi révèle encore un abus de l’employeur qui se décharge sur ses salarié·es de sa propre responsabilité**. Là encore, un rappel des fonctions et des responsabilités peut permettre de distinguer ce qui relève de la pression extérieure et d’une méthode de management.
|
||||
> [Le syndrome du patron de gauche (Arthur Brault-Moreau) — Zéro Janvier](https://blog.zerojanvier.fr/le-syndrome-du-patron-de-gauche-arthur-brault-moreau?pk_campaign=rss-feed)
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
|
@ -515,14 +515,10 @@ dd {
|
|||
padding-bottom: 0.5em;
|
||||
}
|
||||
|
||||
.page-title {
|
||||
flex: 0;
|
||||
}
|
||||
|
||||
.item>time {
|
||||
flex: 1;
|
||||
text-align: right;
|
||||
color: #acabab;
|
||||
color: #797878;
|
||||
padding-left: 1em;
|
||||
}
|
||||
|
||||
|
|
|
|||
BIN
mnmlist/static/fonts/Luciole-Regular.ttf
Normal file
BIN
mnmlist/static/fonts/Luciole-Regular.ttf
Normal file
Binary file not shown.
|
|
@ -29,9 +29,9 @@
|
|||
<a href="{{ SITEURL }}/{{ article.next_article.url }}"
|
||||
title="{{ article.next_article.title }}">Suivant →</a>
|
||||
</li>
|
||||
{% endif %}
|
||||
</ul>
|
||||
</nav>
|
||||
{% endif %}
|
||||
{% if article.headline and article.category != "weeknotes" %}
|
||||
<p>
|
||||
<em>{{ article.headline }}</em>
|
||||
|
|
|
|||
|
|
@ -23,7 +23,7 @@
|
|||
{% for article in articles[0:20] %}
|
||||
<li class="item">
|
||||
{% set category_description = CATEGORIES_DESCRIPTION.get(article.category)[0] %}
|
||||
<a href="{{ SITEURL }}/{{ article.url }}" id="page-title">{{ category_description }}: {{ article.title.replace(category_description, "") }}</a>
|
||||
<a href="{{ SITEURL }}/{{ article.url }}" class="page-title">{{ category_description }}: {{ article.title.replace(category_description, "") }}</a>
|
||||
<time datetime="{{ article.date.isoformat() }}">{{ article.date.strftime("%Y-%m-%d") }}</time>
|
||||
</li>
|
||||
{% endfor %}
|
||||
|
|
|
|||
|
|
@ -31,16 +31,16 @@
|
|||
<thead>
|
||||
<tr>
|
||||
<td>Mois</td>
|
||||
<td>Heures</td>
|
||||
<td>Jours</td>
|
||||
<td>Jours rémunérés</td>
|
||||
<td>Jours bénévoles</td>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for month, amount in page.metadata.worklog.payed_monthly.items() %}
|
||||
{% for month, hours in page.metadata.worklog.monthly_hours.items() %}
|
||||
<tr>
|
||||
<td>{{ month }}</td>
|
||||
<td>{{ amount }}</td>
|
||||
<td>{{ (amount / 7.0) | round(1) }}</td>
|
||||
<td>{{ (hours['payed'] / 7.0) | round(1) }}</td>
|
||||
<td>{{ (hours['volunteered'] / 7.0) | round(1) }}</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
PATH = "content"
|
||||
AUTHOR = "Alexis"
|
||||
SITENAME = "Alexis Métaireau"
|
||||
AUTHOR = ""
|
||||
THEME = "mnmlist"
|
||||
STATIC_PATHS = ["images", "audio", "extra", "docs"]
|
||||
STYLESHEET_URL = "/theme/css/main.css"
|
||||
|
|
|
|||
|
|
@ -65,5 +65,7 @@ def neighbors(generator):
|
|||
set_neighbors(articles, next_name, prev_name)
|
||||
|
||||
|
||||
|
||||
|
||||
def register():
|
||||
signals.article_generator_finalized.connect(neighbors)
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ from datetime import datetime
|
|||
from pathlib import Path
|
||||
|
||||
from markdown.preprocessors import Preprocessor
|
||||
|
||||
from pelican import signals
|
||||
from pelican.readers import Markdown, MarkdownReader, pelican_open
|
||||
from pelican.utils import get_date, slugify
|
||||
|
|
@ -19,26 +20,26 @@ except Exception:
|
|||
class WorklogPreprocessor(Preprocessor):
|
||||
pattern = re.compile(
|
||||
r"""
|
||||
(?:(\w+)\s+)? # Day name
|
||||
(\d{1,2})\s+ # Day number
|
||||
([\wéû]+)\s+ # Month name
|
||||
(\d{4})\s+ # Year
|
||||
(?:(\w+)\s+)? # Day name
|
||||
(\d{1,2})\s+ # Day number
|
||||
([\wéû]+)\s+ # Month name
|
||||
(\d{4})\s+ # Year
|
||||
\(
|
||||
(\d{1,2})h # Hours (mandatory)
|
||||
(?:\s+facturées)? # Optionally 'facturées', if not present, assume hours are 'facturées'
|
||||
(\d{1,2})h # Hours (mandatory)
|
||||
(?:\s+facturées)? # Optionally 'facturées', if not present, assume hours are 'facturées'
|
||||
(?:,\s*(\d{1,2})h\s*bénévoles)? # Optionally 'volunteer hours 'bénévoles'
|
||||
,? # An optional comma
|
||||
\s* # Optional whitespace
|
||||
(?:fun\s+)? # Optionally 'fun' (text) followed by whitespace
|
||||
(\d)/5 # Happiness rating (mandatory, always present)
|
||||
\) # Closing parenthesis
|
||||
,? # An optional comma
|
||||
\s* # Optional whitespace
|
||||
(?:fun\s+)? # Optionally 'fun' (text) followed by whitespace
|
||||
(\d)/5 # Happiness rating (mandatory, always present)
|
||||
\) # Closing parenthesis
|
||||
""",
|
||||
re.VERBOSE | re.UNICODE,
|
||||
)
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
self.data = {}
|
||||
self.payed_monthly = defaultdict(int)
|
||||
self.monthly_hours = defaultdict(lambda: defaultdict(int))
|
||||
super().__init__(*args, **kwargs)
|
||||
|
||||
def run(self, lines):
|
||||
|
|
@ -58,7 +59,8 @@ class WorklogPreprocessor(Preprocessor):
|
|||
happiness,
|
||||
) = match.groups()
|
||||
|
||||
volunteer_hours = int(volunteer_hours) if volunteer_hours else 0
|
||||
volunteer_hours = int(
|
||||
volunteer_hours) if volunteer_hours else 0
|
||||
payed_hours = int(payed_hours)
|
||||
happiness = int(happiness)
|
||||
date = datetime.strptime(f"{day} {month} {year}", "%d %B %Y")
|
||||
|
|
@ -67,7 +69,9 @@ class WorklogPreprocessor(Preprocessor):
|
|||
"volunteer_hours": volunteer_hours,
|
||||
"happiness": happiness,
|
||||
}
|
||||
self.payed_monthly[date.strftime("%Y/%m")] += payed_hours
|
||||
current_date = date.strftime("%Y/%m")
|
||||
self.monthly_hours[current_date]['payed'] += payed_hours
|
||||
self.monthly_hours[current_date]['volunteered'] += volunteer_hours
|
||||
displayed_date = date.strftime("%A %d %B %Y")
|
||||
|
||||
# Replace the line with just the date
|
||||
|
|
@ -82,13 +86,14 @@ class WorklogPreprocessor(Preprocessor):
|
|||
This is run once, after everything has been parsed
|
||||
"""
|
||||
payed_hours = sum([item["payed_hours"] for item in self.data.values()])
|
||||
volunteer_hours = sum([item["volunteer_hours"] for item in self.data.values()])
|
||||
volunteer_hours = sum([item["volunteer_hours"]
|
||||
for item in self.data.values()])
|
||||
|
||||
data = dict(
|
||||
data=self.data,
|
||||
payed_hours=payed_hours,
|
||||
volunteer_hours=volunteer_hours,
|
||||
payed_monthly=self.payed_monthly,
|
||||
monthly_hours=self.monthly_hours,
|
||||
template="worklog",
|
||||
)
|
||||
if "total_days" in metadata:
|
||||
|
|
@ -110,8 +115,10 @@ class SimpleReader(MarkdownReader):
|
|||
|
||||
def __init__(self, *args, **kwargs):
|
||||
super(SimpleReader, self).__init__(*args, **kwargs)
|
||||
self.settings["MARKDOWN"]["extensions"].append("markdown.extensions.toc")
|
||||
self.settings["MARKDOWN"]["extension_configs"].update({'markdown.extensions.toc': {'toc_depth': 3}})
|
||||
self.settings["MARKDOWN"]["extensions"].append(
|
||||
"markdown.extensions.toc")
|
||||
self.settings["MARKDOWN"]["extension_configs"].update(
|
||||
{'markdown.extensions.toc': {'toc_depth': 3}})
|
||||
|
||||
def read(self, source_path):
|
||||
self._source_path = source_path
|
||||
|
|
@ -154,7 +161,8 @@ class SimpleReader(MarkdownReader):
|
|||
|
||||
if "slug" not in metadata:
|
||||
metadata["slug"] = slugify(
|
||||
metadata["title"], self.settings.get("SLUG_REGEX_SUBSTITUTIONS", [])
|
||||
metadata["title"], self.settings.get(
|
||||
"SLUG_REGEX_SUBSTITUTIONS", [])
|
||||
)
|
||||
|
||||
category = os.path.basename(
|
||||
|
|
|
|||
Loading…
Reference in a new issue