mirror of
https://github.com/almet/notmyidea.git
synced 2025-04-28 19:42:37 +02:00
5492 lines
No EOL
511 KiB
XML
5492 lines
No EOL
511 KiB
XML
<?xml version="1.0" encoding="utf-8"?>
|
||
<feed xmlns="http://www.w3.org/2005/Atom"><title>Alexis - Carnets en ligne - Technologie</title><link href="https://blog.notmyidea.org/" rel="alternate"></link><link href="https://blog.notmyidea.org/feeds/technologie.atom.xml" rel="self"></link><id>https://blog.notmyidea.org/</id><updated>2018-03-03T00:00:00+01:00</updated><entry><title>Groupement d'achats & partage d'expérience</title><link href="https://blog.notmyidea.org/groupement-dachats-partage-dexperience.html" rel="alternate"></link><published>2018-03-03T00:00:00+01:00</published><updated>2018-03-03T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2018-03-03:/groupement-dachats-partage-dexperience.html</id><summary type="html"><p>Il y a quelques années, on s'est motivé entre copains copines pour créer un groupement d'achat.</p>
|
||
<p>L'idée est simple:</p>
|
||
<ul>
|
||
<li>commander en gros, pour faire baisser les prix</li>
|
||
<li>se passer d'intermédiaires et favoriser les circuits courts</li>
|
||
<li>aller à la rencontre des producteurs locaux et échanger</li>
|
||
</ul>
|
||
<p>Notre groupement dessert actuellement 18 foyers …</p></summary><content type="html"><p>Il y a quelques années, on s'est motivé entre copains copines pour créer un groupement d'achat.</p>
|
||
<p>L'idée est simple:</p>
|
||
<ul>
|
||
<li>commander en gros, pour faire baisser les prix</li>
|
||
<li>se passer d'intermédiaires et favoriser les circuits courts</li>
|
||
<li>aller à la rencontre des producteurs locaux et échanger</li>
|
||
</ul>
|
||
<p>Notre groupement dessert actuellement 18 foyers et une 60aine de personnes.</p>
|
||
<p>Au fur et à mesure de la vie du groupement, on a développé quelques outils pour se simplifier la vie. Voici un retour d'expérience et quelques astuces / outils, au cas où l'envie vous prenne à vous aussi :)</p>
|
||
<h1 id="organisation">Organisation</h1>
|
||
<p>On organise environs trois ou quatre distributions par an. Le <em>modus operandi</em> est le suivant:</p>
|
||
<ul>
|
||
<li>chaque product·eur·rice à un·e référent·e, qui s'occupe de faire le lien;</li>
|
||
<li>une personne est désignée pour coordonner la distribution;</li>
|
||
<li>4 semaines avant la distribution, les référent·e·s mettent à jour les prix / produits dans le tableau de commandes;·e·</li>
|
||
<li>3 semaines avant la distribution, les commandes sont ouvertes;</li>
|
||
<li>2 semaines avant la distribution, les commandes sont closes;</li>
|
||
<li>Les référent·e·s ont ensuite deux semaines pour récupérer les commandes pour la distribution</li>
|
||
</ul>
|
||
<h1 id="quels-produits">Quels produits ?</h1>
|
||
<p>On essaye d'avoir uniquement des produits qui se conservent (on a également quelques autres produits plus frais, mais avec d'autres modalités).</p>
|
||
<p>Entre autres: bières, légumes secs, conserves, jus, miel, pâtes, semoule, café, vinaigres, pommes de terre, oignons, huiles, farines.</p>
|
||
<p>On essaye de faire du local puis du bio au plus proche plutôt que de trouver nécessairement les prix les plus bas. C'est une discussion qui revient assez
|
||
souvent, et donc un point à évoquer lors de la création pour avoir une posture
|
||
claire sur le sujet (tout le monde n'est pas animé par la même éthique !).</p>
|
||
<h1 id="paiements">Paiements</h1>
|
||
<p>Pour les paiements, on utilise autant que possible des chèques. Chaque référent·e paye la·le product·rice·eur en son nom, et lui demande d'attendre la date de la distribution pour l'encaissement. La plupart des producteurs acceptent d'être payés sous quinzaine.</p>
|
||
<p>Le jour de la distribution, tout le monde apporte son chéquier. Nous avons mis
|
||
en place une moulinette qui s'occupe de faire la répartition des chèques automatiquement, chaque membre se retrouve à faire en moyenne un ou deux chèques.</p>
|
||
<p>Chaque référent·e est ainsi remboursé·e de la somme avancée, et chaque
|
||
membre du groupement d'achat paye ce qu'il doit payer. Nous n'avons
|
||
volontairement pas de structure juridique et pas de compte en banque. Les
|
||
paiements s'effectuent directement entre nous.</p>
|
||
<h1 id="transports">Transports</h1>
|
||
<p>Chaque référent·e commande les produits, puis ensuite s'occupe de les rapatrier. À Rennes, on a la chance d'avoir pas mal de producteurs aux alentours, donc c'est assez simple.</p>
|
||
<p>Le mieux est de ramener les produits juste un peu avant la distribution au lieu de distribution, ça permet d'éviter de les stocker trop longtemps, et d'éviter aux producteurs d'attendre trop longtemps avant d'encaisser les chèques.</p>
|
||
<p>Pour les grosses commandes, les voitures se remplissent bien, mais ma petite Clio suffit, que ce soit dit !</p>
|
||
<h1 id="la-distribution">La distribution</h1>
|
||
<p>Un peu en amont de la distribution, il faut organiser l'espace. Des tas par membre sont constitués pour faciliter les choses le jour de la distribution.</p>
|
||
<p>Le jour même, on se retrouve, on charge ses marchandises, on échange quelques chèques et on papote ! On en profite pour:</p>
|
||
<ul>
|
||
<li>discuter de la date de la prochaine distribution;</li>
|
||
<li>trouver une nouvelle personne pour la coordonner;</li>
|
||
<li>discuter de nouveaux produits;</li>
|
||
<li>refaire le monde;</li>
|
||
<li>changer de référents pour les producteurs.</li>
|
||
</ul>
|
||
<p>Et c'est reparti pour un tour ;)</p>
|
||
<h1 id="nos-outils">Nos outils</h1>
|
||
<p>On utilise un tableur en ligne pour partager les prix et prendre les commandes. On a essayé d'utiliser <em>ethercalc</em> au début mais ça ne fonctionnait pas pour nous à l'époque (trop de petits bugs). On a donc préféré utiliser Google docs (ouch).</p>
|
||
<p>Il est d'ailleurs possible d'y intégrer de nouvelles fonctionnalités assez facilement, du coup Fred et Rémy ont planché sur un moyen d'automatiser la répartition des chèques (qu'on faisait dans un premier temps à la main - assez péniblement).</p>
|
||
<p>Le système n'est pas parfait mais fonctionne quand même assez bien !</p>
|
||
<p>Quelques ressources, donc:</p>
|
||
<ul>
|
||
<li><a href="https://gist.github.com/almet/8c77fafc9e487c02ded852ec4a91ae16">le code pour faire la répartition des chèques</a></li>
|
||
<li><a href="https://docs.google.com/spreadsheets/d/1bnPRSvf2Q2RDxKerWnEqUyJjuCFePnVMq6pWo8LeA_k/edit?usp=sharing">une version « à remplir » de notre tableau de commandes</a> (le mieux est d'en faire une copie !).</li>
|
||
</ul>
|
||
<p>Bon groupement d'achat ;)</p></content></entry><entry><title>Webnotes</title><link href="https://blog.notmyidea.org/webnotes.html" rel="alternate"></link><published>2018-02-25T00:00:00+01:00</published><updated>2018-02-25T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2018-02-25:/webnotes.html</id><summary type="html"><p>Quand je navigue en ligne, j'aime bien prendre des notes sur ce que je lis. C'est utile pour les retrouver plus tard. Il existe quelques outils pour ce genre de cas, mais j'ai vraiment eu du mal à trouver un outil qui faisais ce que je voulais, de la manière …</p></summary><content type="html"><p>Quand je navigue en ligne, j'aime bien prendre des notes sur ce que je lis. C'est utile pour les retrouver plus tard. Il existe quelques outils pour ce genre de cas, mais j'ai vraiment eu du mal à trouver un outil qui faisais ce que je voulais, de la manière que je voulais, c'est à dire:</p>
|
||
<ul>
|
||
<li>enregistrer une sélection de texte ainsi que son contexte: heure, site web.</li>
|
||
<li>fonctionner sur Firefox;</li>
|
||
<li>stocker mes notes à un endroit que je contrôle (ce sont mes données, après tout !)</li>
|
||
<li>rester en dehors de mon chemin: je suis en train de lire, pas en train d'organiser mes notes.</li>
|
||
<li>automatiquement partager les notes sur une page web.</li>
|
||
</ul>
|
||
<p>J'ai donc pris un peu de temps pour fabriquer mon outil de prises de notes, que j'ai baptisé « Webnotes ». C'est <a href="https://addons.mozilla.org/en-US/firefox/addon/wwebnotes/">une extension Firefox</a>, qui se configure assez simplement, et qui stocke les données dans une instance de <a href="http://kinto-storage.org/">Kinto</a>.</p>
|
||
<p><img src="https://github.com/almet/webnotes/blob/master/webnotes.gif?raw=true" /></p>
|
||
<p>C'est aussi simple que sélectionner du texte, faire « clic droit » puis « save as webnote », entrer un tag et le tour est joué !</p>
|
||
<p>Mes notes sont disponibles <a href="https://notes.notmyidea.org">sur notes.notmyidea.org</a>, et voici <a href="https://github.com/almet/webnotes">le lien vers les sources</a>, si ça vous intéresse de regarder comment ça fonctionne !</p></content></entry><entry><title>Comment est-ce que vous générez vos formulaires ?</title><link href="https://blog.notmyidea.org/comment-est-ce-que-vous-generez-vos-formulaires.html" rel="alternate"></link><published>2016-05-31T00:00:00+02:00</published><updated>2016-05-31T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2016-05-31:/comment-est-ce-que-vous-generez-vos-formulaires.html</id><summary type="html"><p>TL; DR: Je viens à peine de <em>releaser</em> la première version d'un service de génération de formulaires.
|
||
Allez jeter un coup d'œil sur <a href="https://www.fourmilieres.net">https://www.fourmilieres.net</a></p>
|
||
<p><em>En février 2012, je parlais ici <a href="https://blog.notmyidea.org/carto-forms.html">d'un service de génération de formulaires</a>.
|
||
Depuis, pas mal d'eau à coulé sous les ponts, on est …</em></p></summary><content type="html"><p>TL; DR: Je viens à peine de <em>releaser</em> la première version d'un service de génération de formulaires.
|
||
Allez jeter un coup d'œil sur <a href="https://www.fourmilieres.net">https://www.fourmilieres.net</a></p>
|
||
<p><em>En février 2012, je parlais ici <a href="https://blog.notmyidea.org/carto-forms.html">d'un service de génération de formulaires</a>.
|
||
Depuis, pas mal d'eau à coulé sous les ponts, on est passé par pas mal d'étapes pour
|
||
finalement arriver à une première version de ce service de génération de
|
||
formulaires (à la </em>google forms<em>).</em></p>
|
||
<p>En tant qu'organisateurs d'évènements (petits et gros), je me retrouve souvent
|
||
dans une situation ou je dois créer des formulaires pour recueillir des
|
||
informations. Actuellement, la meilleure solution disponible est <em>Google Forms</em>,
|
||
mais celle ci à plusieurs problèmes, à commencer par le fait que le code n'est
|
||
pas libre et que les données sont stockées chez Google.</p>
|
||
<p>La plupart du temps, le besoin est assez simple: je veux spécifier quelques
|
||
questions, et donner un lien à mes amis pour qu'ils puissent y répondre.
|
||
Je reviens ensuite plus tard pour voir la liste des réponses apportées.</p>
|
||
<p><img alt="Capture de l'interface de création du formulaire" src="{filename}/static/formbuilder-build.png"></p>
|
||
<h2 id="fonctionnalites">Fonctionnalités</h2>
|
||
<p>Il existe pas mal de solutions techniques qui essayent de répondre à la même
|
||
problématique, mais la plupart d'entre elles sont assez souvent compliquées,
|
||
nécessitent de se créer un compte, et/ou ne vous laisse pas la main libre sur
|
||
les données générées, voire le code est assez difficile à faire évoluer ou à
|
||
déployer.</p>
|
||
<p>Je voulais donc quelque chose de simple à utiliser <em>et</em> pour les créateurs de
|
||
formulaires <em>et</em> pour les utilisateurs finaux. Pas de chichis, juste quelques
|
||
vues, et des URLs à sauvegarder une fois l'opération terminée.</p>
|
||
<p><img alt="Capture de l'écran avec les URLs générées" src="{filename}/static/formbuilder-created.png">
|
||
<img alt="Capture d'écran d'un exemple de formulaire" src="{filename}/static/formbuilder-form.png"></p>
|
||
<h3 id="pas-de-compte">Pas de compte</h3>
|
||
<p>Vous n'avez pas besoin d'avoir un compte sur le site pour commencer à l'utiliser.
|
||
Vous créez simplement un nouveau formulaire puis envoyez le lien à vos amis pour
|
||
qu'eux puissent à leur tour le remplir.</p>
|
||
<p><img alt="Capture de la page d'accueil, ou aucun compte n'est requis" src="{filename}/static/formbuilder-welcome.png"></p>
|
||
<h3 id="gardez-la-main-sur-vos-donnees">Gardez la main sur vos données</h3>
|
||
<p>Une fois que vous avez récupéré les réponses à vos questions, vous pouvez
|
||
récupérer les données sur votre machines dans un fichier <code>.csv</code>.</p>
|
||
<p><img alt="Capture de la page de resultats, il est possible de télécharger en CSV." src="{filename}/static/formbuilder-results.png"></p>
|
||
<h3 id="api">API</h3>
|
||
<p>L'ensemble des données sont en fait stockées dans <a href="https://kinto.readthedocs.org">Kinto</a>
|
||
qui est interrogeable très facilement en HTTP. Ce qui fait qu'il est très facile de
|
||
réutiliser les formulaires que vous avez construits (ou leurs réponses) depuis
|
||
d'autres outils.</p>
|
||
<h3 id="auto-hebergeable">Auto-hébergeable</h3>
|
||
<p>Un des objectifs de ce projet est de vous redonner la main sur vos données.
|
||
Bien sur, vous pouvez utiliser l'instance qui est mise à votre disposition sur
|
||
<a href="https://www.fourmilieres.net">wwww.fourmilieres.net</a>, mais vous pouvez
|
||
également l'héberger vous même très
|
||
simplement, et vous êtes d'ailleurs fortement encouragés à le faire ! Notre
|
||
objectif n'est pas de stocker l'ensemble des formulaires du monde, mais de
|
||
(re)donner le contrôle aux utilisateurs !</p>
|
||
<h2 id="on-commence-petit">On commence petit…</h2>
|
||
<p>Cette <em>release</em> n'est (bien sur) pas parfaite, et il reste encore pas mal de
|
||
travail sur cet outil, mais je pense qu'il s'agit d'une base de travail
|
||
intéressante pour un futur où Google n'a pas la main sur toutes nos données.</p>
|
||
<p>La liste des champs supportés est pour l'instant assez faible (Texte court,
|
||
Texte long, Oui/Non, choix dans une liste) mais elle à vocation à s'étendre, en
|
||
fonction des besoins de chacun.</p>
|
||
<p>J'ai d'ailleurs créé <a href="https://www.fourmilieres.net/#/form/cfd878264cec4ed2">un formulaire pour que vous puissiez me faire part de vos
|
||
retours</a>, n'hésitez pas !</p>
|
||
<h2 id="et-euh-comment-ca-marche">Et, euh, comment ça marche ?</h2>
|
||
<p>Le <em>formbuilder</em>, comme j'aime l'appeler se compose en fin de compte de deux
|
||
parties distinctes:</p>
|
||
<ul>
|
||
<li><a href="https://kinto.readthedocs.org">Kinto</a>, un service qui stocke
|
||
des données coté serveur et qui les expose via des <strong>APIs HTTP</strong></li>
|
||
<li><a href="https://github.com/kinto/formbuilder">Le formbuilder</a>, une application
|
||
JavaScript qui ne tourne que coté client (dans votre navigateur) qui permet
|
||
de construire les formulaires et d'envoyer les données sur les <em>APIs</em> coté
|
||
serveur.</li>
|
||
</ul>
|
||
<p>Au niveau de la <em>stack</em> technique, le <strong>formbuilder</strong> est codé en ReactJS. Un
|
||
des points techniques intéressants du projet est qu'il génère en fin de compte du
|
||
<a href="http://jsonschema.net/">JSON Schema</a>, un format de validation de données <em>JSON</em>.</p>
|
||
<p>Donc, reprenons! Vous arrivez sur la page d'accueil puis cliquez sur
|
||
"Create a new form", puis vous vous retrouvez face à une interface ou vous pouvez
|
||
ajouter des champs de formulaire. Une fois ce travail effectué, vous appuyez sur
|
||
"Create the form".</p>
|
||
<ul>
|
||
<li>Le JSON Schema est alors envoyé au serveur Kinto, qui l'utilisera pour valider
|
||
les données qu'il recevra par la suite.</li>
|
||
<li>Ce JSON Schema sera aussi utilisé lors de l'affichage du formulaire aux
|
||
personnes qui le remplissent.</li>
|
||
<li>Un jeton d'accès est généré et ajouté à l'URL, il s'agit de l'identifiant du
|
||
formulaire.</li>
|
||
<li>Un second jeton d'accès administrateur et généré, il vous faut le garder de
|
||
coté pour avoir accès aux réponses.</li>
|
||
</ul>
|
||
<p>Bref, en espérant que ça vous serve ! Un petit pas dans la direction des données
|
||
rendues à leurs utilisateurs !</p></content></entry><entry><title>Avez vous confiance en SSL?</title><link href="https://blog.notmyidea.org/avez-vous-confiance-en-ssl.html" rel="alternate"></link><published>2016-03-25T00:00:00+01:00</published><updated>2016-03-25T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2016-03-25:/avez-vous-confiance-en-ssl.html</id><summary type="html"><p>Dans le cadre <a href="http://autodefense-numerique.readthedocs.org/en/latest/">des ateliers d'autodéfense numérique</a>,
|
||
j'ai passé un peu de temps à creuser sur l'utilisation de SSL puisque
|
||
contrairement à ce que la plupart des personnes ont encore tendance à croire,
|
||
le petit cadenas (qui prouve qu'une connexion SSL est en cours) n'est
|
||
<strong>absolument</strong> pas suffisant.</p>
|
||
<p>Allez hop …</p></summary><content type="html"><p>Dans le cadre <a href="http://autodefense-numerique.readthedocs.org/en/latest/">des ateliers d'autodéfense numérique</a>,
|
||
j'ai passé un peu de temps à creuser sur l'utilisation de SSL puisque
|
||
contrairement à ce que la plupart des personnes ont encore tendance à croire,
|
||
le petit cadenas (qui prouve qu'une connexion SSL est en cours) n'est
|
||
<strong>absolument</strong> pas suffisant.</p>
|
||
<p>Allez hop, c'est parti pour:</p>
|
||
<ul>
|
||
<li>un tour d'horizon du fonctionnement de SSl</li>
|
||
<li>quelques moyens contourner cette "protection" en faisant une attaque en pratique</li>
|
||
<li>un tour des solutions existantes actuellement et de pourquoi je ne les trouve
|
||
pas vraiment satisfaisantes.</li>
|
||
</ul>
|
||
<h2 id="comment-fonctionne-ssl">Comment fonctionne SSL?</h2>
|
||
<p>Pour expliquer les problèmes de SSL, j'ai d'abord besoin d'expliquer comment
|
||
tout ça fonctionne.</p>
|
||
<p>SSL repose sur l'utilisation de certificats, qui sont générés par des autorités
|
||
de certification (<em>Certificate Authority</em> que je nomme <em>CA</em> dans la suite de
|
||
l'article).</p>
|
||
<p>Les certificats SSL permettent deux choses:</p>
|
||
<ul>
|
||
<li>De garantir que les communications entre les navigateurs (vous) et les sites
|
||
Web ne sont connues que du détenteur du certificat du site et de vous même.</li>
|
||
<li>De garantir que le site sur lequel vous vous connectez est bien celui que
|
||
vous imaginez.</li>
|
||
</ul>
|
||
<p>Le navigateur, lors d'une visite d'un site, va télécharger le certificat
|
||
associé puis vérifier que le certificat en question a bien été généré par un
|
||
des <em>CA</em> en qui il a confiance.</p>
|
||
<p>Imaginons maintenant qu'une des <em>CA</em> essaye de savoir ce qui s'échange entre
|
||
mon navigateur et le site de ma banque (protégé par SSL). Comment cela se
|
||
passerait il ?</p>
|
||
<p>N'importe quel <em>CA</em> peut donc générer des certificats pour n'importe quel site,
|
||
et le navigateur vérifierait, lui, que le certificat a bien été généré par une
|
||
<em>CA</em>.</p>
|
||
<p>Tout cela ne poserait pas de soucis si les <em>CA</em> étaient gérés de manière fiable,
|
||
mais il s'agit d'un travail compliqué, et certains <em>CA</em> ont par le passé montré
|
||
des faiblesses.</p>
|
||
<p>Par exemple, <a href="https://en.wikipedia.org/wiki/DigiNotar">DigiNotar</a> (un <em>CA</em> des Pays-Bas)
|
||
a été compromise et les attaquant.e.s ont pu générer des certificats SSL
|
||
frauduleux, ce qui leur a permis d'attaquer des sites tels que Facebook ou GMail.</p>
|
||
<p>Vous pouvez retrouver une liste des risques et menaces autour des <em>CA</em> <a href="http://wiki.cacert.org/Risk/History">sur le
|
||
wiki de CACert</a>.</p>
|
||
<h2 id="attaque-de-lhomme-du-milieu-avec-ssl">Attaque de l'homme du milieu avec SSL</h2>
|
||
<p>A force de dire que c'était très facile à faire, j'ai eu envie d'essayer
|
||
d'espionner des connections protégées par SSL, et effectivement c'est
|
||
carrément flippant tellement c'est simple.</p>
|
||
<p>En l'espace de quelques minutes, il est possible de faire une <em>attaque de
|
||
l'homme du milieu</em> en utilisant par exemple un outil nommé <a href="http://docs.mitmproxy.org/en/stable">mitm-proxy</a>.</p>
|
||
<p>Pour déchiffrer l'ensemble du trafic SSL, j'ai simplement eu à lancer quelques
|
||
commandes et avoir un <em>CA</em> dans lequel le navigateur de la victime a confiance.
|
||
Je l'ai ajouté dans le navigateur cible pour simuler que je l'avais déjà
|
||
(c'est le cas si un des 1200 CA se fait pirater, ce qui me semble une surface
|
||
d'attaque assez large).</p>
|
||
<p>Je les colle ici si ça vous intéresse:</p>
|
||
<div class="highlight"><pre><span></span>$ sudo aptitude install mitmproxy
|
||
$ mitm-proxy -T --host
|
||
</pre></div>
|
||
|
||
|
||
<p>Il faut faire croire à votre victime que vous êtes la passerelle vers
|
||
l'extérieur et à la passerelle que vous êtes la victime:</p>
|
||
<div class="highlight"><pre><span></span>arpspoof -i wlan0 -t victime gateway
|
||
arpspoof -i wlan0 -t gateway victime
|
||
</pre></div>
|
||
|
||
|
||
<p>Puis dire à notre fausse passerelle de rediriger le trafic des ports 80 et 443
|
||
vers notre proxy:</p>
|
||
<div class="highlight"><pre><span></span>sudo sysctl -w net.ipv4.ip_forward<span class="o">=</span><span class="m">1</span>
|
||
sudo iptables -t nat -A PREROUTING -i wlan0 -p tcp --dport <span class="m">443</span> -j REDIRECT --to-port <span class="m">4443</span>
|
||
sudo iptables -t nat -A PREROUTING -i wlan0 -p tcp --dport <span class="m">80</span> -j REDIRECT --to-port <span class="m">4443</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>Et paf, <strong>on voit tout ce qui passe entre la machine et le serveur SSL</strong>. On peut
|
||
d'ailleurs même imaginer faire tourner ces quelques commandes sur un
|
||
raspberry pi, pour aller encore plus vite…</p>
|
||
<h3 id="key-pinning-dans-les-navigateurs">Key-pinning dans les navigateurs</h3>
|
||
<p>Actuellement, n'importe quel <em>CA</em> peut générer des certificats pour
|
||
n'importe quel site, et c'est en grande partie ce qui pose souci. Une des
|
||
manières de faire évoluer la situation est d'épingler les certificats de
|
||
certains sites directement dans les navigateurs.</p>
|
||
<p>Cette approche a le mérite de fonctionner très bien <a href="https://dxr.mozilla.org/mozilla-central/source/security/manager/ssl/StaticHPKPins.h?from=StaticHPKPins.h">pour un petit nombre de
|
||
sites critiques (Google, Facebook, etc)</a>.</p>
|
||
<h3 id="http-public-key-pinning-hpkp">HTTP Public Key Pinning (HPKP)</h3>
|
||
<p><a href="https://developer.mozilla.org/en/docs/Web/Security/Public_Key_Pinning"><em>HTTP Public Key Pinning</em></a>
|
||
est également une solution de <em>pinning</em> qui permet d'établir une confiance lors
|
||
de la première connexion avec le site. C'est ce qu'on appelle du <em>Trust on First
|
||
Use</em> ou <em>TOFU</em>.</p>
|
||
<p>Le navigateur va alors mettre ces informations dans un cache et vérifiera que
|
||
les certificats correspondent bien lors des prochaines visites.</p>
|
||
<p><em>HPKP</em> est disponible dans Firefox depuis Janvier 2015 et dans Chrome
|
||
depuis Octobre 2015.</p>
|
||
<h3 id="certificate-transparency-des-journaux-auditables">Certificate transparency: des journaux auditables</h3>
|
||
<p>Une autre approche est celle proposée par <em>certificate transparency</em>:</p>
|
||
<blockquote>
|
||
<p>Certificate Transparency aims to remedy these certificate-based threats by
|
||
making the issuance and existence of SSL certificates open to scrutiny by
|
||
domain owners, CAs, and domain users.</p>
|
||
<p>-- <a href="https://www.certificate-transparency.org/what-is-ct">Certificate Transparency</a></p>
|
||
</blockquote>
|
||
<p>Autrement dit, avec ce système les <em>CA</em> doivent rendre public le fait qu'ils
|
||
aient signé de nouveaux certificats intermédiaires. La signature est ajoutée à
|
||
un journal sur lequel il n'est possible que d'écrire.</p>
|
||
<p>Les navigateurs vont alors vérifier que les certificats utilisés sont bien des
|
||
certificats qui ont été ajoutés au journal.</p>
|
||
<p>Ici, toute l'intelligence est dans la vérification de ces journaux, qui
|
||
permettent donc de valider/invalider des certificats racines ou intermédiaires.</p>
|
||
<p>Il me semble donc qu'il serait possible d'ajouter un certificat frauduleux le
|
||
temps d'une attaque (et celui ci serait détecté et supprimé ensuite).</p>
|
||
<p><em>Certificate-Transparency</em> n'est donc pas une solution contre une écoute
|
||
globale mise en place par les gouvernements par exemple.</p>
|
||
<p>Si vous lisez bien l'anglais, je vous invite à aller lire
|
||
<a href="http://security.stackexchange.com/a/52838">cette description du problème et de la solution</a>
|
||
que je trouve très bien écrite.</p>
|
||
<h3 id="dane-dnssec">DANE + DNSSEC</h3>
|
||
<blockquote>
|
||
<p>The DANE working group has developed a framework for securely
|
||
retrieving keying information from the DNS [RFC6698]. This
|
||
framework allows secure storing and looking up server public key
|
||
information in the DNS. This provides a binding between a domain
|
||
name providing a particular service and the key that can be used
|
||
to establish encrypted connection to that service.</p>
|
||
<p>-- <a href="https://datatracker.ietf.org/wg/dane/charter/">Dane WG</a></p>
|
||
</blockquote>
|
||
<p>Une autre solution est appelée "DANE" et repose par dessus le protocole
|
||
<em>DNSSEC</em>.</p>
|
||
<p>Je connais assez mal <em>DNSSEC</em> donc j'ai passé un peu de temps à lire des
|
||
documents. L'impression finale que ça me laisse est que le problème est
|
||
exactement le même que pour SSL: un certain nombre de personnes détiennent les
|
||
clés et toute la sécurité repose sur cette confiance. Or il est possible que
|
||
ces clés soient détenues par des personnes non dignes de confiance.</p>
|
||
<blockquote>
|
||
<p>Secure DNS (DNSSEC) uses cryptographic digital signatures signed with a
|
||
trusted public key certificate to determine the authenticity of data.
|
||
-- https://en.wikipedia.org/wiki/DNS_spoofing</p>
|
||
</blockquote>
|
||
<p>Et aussi:</p>
|
||
<blockquote>
|
||
<p>It is widely believed[1] that securing the DNS is critically important for
|
||
securing the Internet as a whole, but deployment of DNSSEC specifically has
|
||
been hampered (As of 22 January 2010) by several difficulties:</p>
|
||
<ul>
|
||
<li>The need to design a backward-compatible standard that can scale to the
|
||
size of the Internet</li>
|
||
<li>Prevention of "zone enumeration" (see below) where desired</li>
|
||
<li>Deployment of DNSSEC implementations across a wide variety of DNS servers
|
||
and resolvers (clients)</li>
|
||
<li>Disagreement among implementers over who should own the top-level domain
|
||
root keys Overcoming the perceived complexity of DNSSEC and DNSSEC
|
||
deployment</li>
|
||
</ul>
|
||
</blockquote>
|
||
<h2 id="solutions-basees-sur-la-blockchain">Solutions basées sur la blockchain</h2>
|
||
<p>Une dernière piste semble être l'utilisation de la <em>blockchain</em> pour distribuer
|
||
des clés par site.</p>
|
||
<p>La solution <em>DNSChain</em> me paraissait tout d'abord un bon point de départ mais
|
||
la lecture de <a href="https://www.indolering.com/okturtles-dnschain-unblock-us">quelques critiques</a>
|
||
et interventions du développeur du projet m'ont fait changer d'avis.</p>
|
||
<p>Reste encore la piste de <em>Namecoin Control</em> que je n'ai pas encore creusée.
|
||
Peut-être pour un prochain billet. Toute piste de réflexion est bien sur la
|
||
bienvenue sur ces sujets!</p></content></entry><entry><title>Retours sur un atelier ZeroNet</title><link href="https://blog.notmyidea.org/retours-sur-un-atelier-zeronet.html" rel="alternate"></link><published>2016-03-17T00:00:00+01:00</published><updated>2016-03-17T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2016-03-17:/retours-sur-un-atelier-zeronet.html</id><summary type="html"><p>Mardi dernier se tenait <a href="http://biblio.insa-rennes.fr/crypto">une <em>cryptoparty</em></a> dans les locaux de l'INSA de Rennes.</p>
|
||
<p>L'évènement s'étant rempli au delà de toutes les espérances, on m'a proposé de
|
||
venir y tenir un atelier, que j'ai proposé sur <a href="https://zeronet.io">ZeroNet</a>, un
|
||
petit projet fort sympathique qui pourrait devenir une nouvelle manière de
|
||
distribuer le …</p></summary><content type="html"><p>Mardi dernier se tenait <a href="http://biblio.insa-rennes.fr/crypto">une <em>cryptoparty</em></a> dans les locaux de l'INSA de Rennes.</p>
|
||
<p>L'évènement s'étant rempli au delà de toutes les espérances, on m'a proposé de
|
||
venir y tenir un atelier, que j'ai proposé sur <a href="https://zeronet.io">ZeroNet</a>, un
|
||
petit projet fort sympathique qui pourrait devenir une nouvelle manière de
|
||
distribuer le Web, permettant notamment d'éviter la censure.</p>
|
||
<p>Avant toute autre chose, merci énormément à l'équipe de la bibliothèque de
|
||
l'INSA pour l'organisation de cet évènement qui à une réelle portée politique.</p>
|
||
<h2 id="un-peu-dhistoire">Un peu d'histoire</h2>
|
||
<p>Il me semble que Tim Bernes Lee (l'inventeur du Web) avait prévu le Web comme un
|
||
protocole décentralisé. Chacun hébergerait ses données et les servirait aux
|
||
autres, qui pourraient alors y accéder.</p>
|
||
<p>Avec ce fonctionnement, impossible alors d'accéder à des sites si leur auteur
|
||
n'est pas en ligne. Qu'à cela ne tienne, on s'est mis à avoir des machines qui
|
||
restent connectées au réseau 24 heures par jour. Et puis une machine ne
|
||
suffisant plus, on a eu des fermes de machines dans des <em>data centers</em> etc afin
|
||
de supporter les milliers d'utilisateurs des sites.</p>
|
||
<h2 id="un-web-decentralise">Un Web décentralisé</h2>
|
||
<p>ZeroNet permet (entre autres) de répondre à ce problème en proposant une manière alternative de <strong>distribuer le Web</strong>, en pair à pair. Lors d'une visite d'un site:</p>
|
||
<ol>
|
||
<li>Vous contactez un <em>tracker</em> BitTorrent pour connaitre la liste des autres
|
||
visiteurs du site (les <em>pairs</em>).</li>
|
||
<li>Vous demandez aux <em>pairs</em> de vous donner les fichiers du site.</li>
|
||
<li>Vous validez que les fichiers servis sont bien les bons (en vérifiant la
|
||
signature attachée).</li>
|
||
</ol>
|
||
<p>N'importe quel visiteur devient alors un <em>pair</em>, qui sert le site aux autres
|
||
visiteurs.</p>
|
||
<p>Parmi les nombreux avantages de cette approche, je note particulièrement que:</p>
|
||
<ul>
|
||
<li>Il est très difficile de censurer un site — Il est sur l'ensemble des machines
|
||
des visiteurs.</li>
|
||
<li>Les attaques par <em>fingerprinting</em> sont impossibles: le navigateur Web se
|
||
connecte à un serveur <em>proxy</em> local.</li>
|
||
<li>Vous détenez directement vos données et (par design) ne les donnez pas à des
|
||
silos (Facebook, Google, etc.)</li>
|
||
</ul>
|
||
<p>Si vous êtes interessés par une démonstration rapide, j'ai enregistré une vidéo
|
||
de 10 minutes où je parle en anglais avec une voix très grave.</p>
|
||
<video controls="" src="http://alexis.notmyidea.org/zeronet.webm" width=800></video>
|
||
|
||
<h2 id="atelier">Atelier</h2>
|
||
<p>Pour l'atelier, j'ai choisi de faire une présentation rapide du projet (<a href="{filename}/static/zeronet-presentation-fr.pdf">j'ai
|
||
traduit les slides</a> anglais
|
||
pour l'occasion — <a href="https://docs.google.com/presentation/d/158C_-V1ueNaaKHMBMBgGOVhunb9xrXzB3hC_g1N53c0/edit?usp=sharing">accès aux sources</a>)
|
||
avant d'installer ZeroNet sur les machines et de l'utiliser pour publier un
|
||
site.</p>
|
||
<h3 id="partager-sur-le-reseau-local">Partager sur le réseau local</h3>
|
||
<p>Nous avons eu des soucis à cause du réseau (un peu congestionné) sur lequel
|
||
les ports utilisés pour la discussion entre <em>pairs</em> étaient fermés. Il est bien
|
||
sur possible de faire tourner le tout de manière indépendante du reste du réseau,
|
||
mais je n'avais pas prévu le coup.</p>
|
||
<p>Voici donc comment faire pour contourner le souci:</p>
|
||
<ol>
|
||
<li>Installer et lancer un <em>tracker</em> BitTorrent (De manière surprenante,
|
||
<a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=685575">rien n'est packagé pour debian pour l'instant</a>)
|
||
J'ai choisi d'installer <a href="http://erdgeist.org/arts/software/opentracker/#build-instructions">OpenTracker</a></li>
|
||
<li>Ensuite lancer ZeroNet avec des options spécifiques.</li>
|
||
</ol>
|
||
<div class="highlight"><pre><span></span>$ python zeronet.py --trackers udp://localhost:6969 --ip_external <span class="m">192</span>.168.43.207
|
||
$ python zeronet.py --trackers udp://192.168.43.207:6969 --ip_external <span class="m">192</span>.168.43.172
|
||
</pre></div>
|
||
|
||
|
||
<p>Il est nécessaire de spécifier l'adresse IP externe que chaque nœud expose pour
|
||
éviter qu'elle n'essaye d'aller la trouver par elle même: nous voulons l'adresse
|
||
du réseau local, et non pas l'adresse internet.</p>
|
||
<p>La prochaine fois je tenterais de venir avec un HotSpot Wifi et un tracker
|
||
BitTorrent dans la poche!</p>
|
||
<h2 id="questions-reponses">Questions / Réponses</h2>
|
||
<p>Il y avait quelques questions intéressantes auxquelles je n'ai pas toujours su
|
||
répondre sur le moment. Après quelques recherches, je rajoute des détails ici.</p>
|
||
<h3 id="torrent-tor-breche-de-secu">Torrent + Tor = brèche de sécu ?</h3>
|
||
<p>Il me semblait avoir entendu parler de problèmes de <em>dé-anonymisation</em>
|
||
<a href="https://hal.inria.fr/file/index/docid/471556/filename/TorBT.pdf">lors de l'utilisation de BitTorrent par dessus Tor</a>.</p>
|
||
<blockquote>
|
||
<p>Dans certains cas, certains clients torrents (uTorrent, BitSpirit, etc)
|
||
écrivent directement votre adresse IP dans l'information qui est envoyée
|
||
au tracker et/ou aux autres pairs.
|
||
— https://blog.torproject.org/blog/bittorrent-over-tor-isnt-good-idea</p>
|
||
</blockquote>
|
||
<p><a href="https://github.com/HelloZeroNet/ZeroNet/issues/274">Ce n'est pas le cas de ZeroNet</a>, ce qui évacue le souci.</p>
|
||
<h3 id="zeromail-cest-lent-non">ZeroMail, c'est lent non ?</h3>
|
||
<p>Une des applications de démo, <em>ZeroMail</em>, propose un mécanisme qui permet de
|
||
s'envoyer des messages chiffrés sur un réseau pair à pair. L'approche choisie
|
||
est de chiffrer les messages avec la clé du destinataire et de le mettre dans
|
||
un <em>pot commun</em>. Tout le monde essaye de déchiffrer tous les messages, mais ne
|
||
peut déchiffrer que les siens.</p>
|
||
<p>Cela permet de ne <strong>pas</strong> fuiter de méta-données, <a href="{filename}../crypto/2015.05.pgp-problemes.rst">à l'inverse de PGP</a>.</p>
|
||
<p>Je n'ai en fait pas de réponse claire à donner à cette question: l'auteur de
|
||
ZeroNet me disait que 10MB (la limite de taille d'un site, par défaut)
|
||
correspondait à beaucoup de place pour stocker des messages, et qu'il était
|
||
possible de supprimer les anciens messages une fois qu'ils sont lus par exemple.</p>
|
||
<p>Une autre solution à laquelle je pensait était de créer un <em>ZeroSite</em> pour
|
||
chaque récipient, mais on connait à ce moment là le nombre de messages qu'un
|
||
utilisateur peut recevoir.</p>
|
||
<p>Je vois plusieurs problèmes avec le design actuel de ZeroMail (il me semble
|
||
assez facile d'y faire un déni de service par exemple). A creuser.</p>
|
||
<h3 id="comment-heberger-des-tres-gros-sites">Comment héberger des très gros sites ?</h3>
|
||
<p>Par exemple, comment faire pour héberger Wikipedia ?</p>
|
||
<p>Il semble que la meilleure manière de faire serait de séparer Wikipedia en
|
||
un tas de petites ressources (par catégorie par ex.). Les gros médias pourraient
|
||
être considérés optionnels (et donc téléchargés uniquement à la demande)</p>
|
||
<h3 id="est-ce-quon-a-vraiment-besoin-dun-tracker">Est-ce qu'on à vraiment besoin d'un tracker ?</h3>
|
||
<p>Le support d'une DHT <a href="https://github.com/HelloZeroNet/ZeroNet/issues/57">est souhaité</a>,
|
||
mais pour l'instant pas encore implémenté. L'utilisation de la DHT BitTorrent
|
||
n'est pas une option puisque <a href="https://github.com/HelloZeroNet/ZeroNet/issues/57">Tor ne supporte pas UDP</a>.</p></content></entry><entry><title>Let's Encrypt + HAProxy</title><link href="https://blog.notmyidea.org/lets-encrypt-haproxy.html" rel="alternate"></link><published>2016-02-11T00:00:00+01:00</published><updated>2016-02-11T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2016-02-11:/lets-encrypt-haproxy.html</id><summary type="html"><p><em>Note : Cet article n'est plus à jour. Il est maintenant (2018) possible d'installer des certificats SSL Let's Encrypt d'une manière beaucoup plus simple, en utilisant certbot (et le plugin nginx <code>certbot --nginx</code>).</em></p>
|
||
<blockquote>
|
||
<p>It’s time for the Web to take a big step forward in terms of security
|
||
and privacy …</p></blockquote></summary><content type="html"><p><em>Note : Cet article n'est plus à jour. Il est maintenant (2018) possible d'installer des certificats SSL Let's Encrypt d'une manière beaucoup plus simple, en utilisant certbot (et le plugin nginx <code>certbot --nginx</code>).</em></p>
|
||
<blockquote>
|
||
<p>It’s time for the Web to take a big step forward in terms of security
|
||
and privacy. We want to see HTTPS become the default. Let’s Encrypt
|
||
was built to enable that by making it as easy as possible to get and
|
||
manage certificates.</p>
|
||
<p>-- <a href="https://letsencrypt.org/">Let's Encrypt</a></p>
|
||
</blockquote>
|
||
<p>Depuis début Décembre, la nouvelle <em>autorité de certification</em> Let's
|
||
Encrypt est passée en version <em>Beta</em>. Les certificats SSL sont un moyen
|
||
de 1. chiffrer la communication entre votre navigateur et le serveur et
|
||
2. un moyen d'être sur que le site Web auquel vous accédez est celui
|
||
auquel vous pensez vous connecter (pour éviter des <a href="https://fr.wikipedia.org/wiki/Attaque_de_l'homme_du_milieu">attaques de l'homme
|
||
du milieu</a>).</p>
|
||
<p>Jusqu'à maintenant, il était nécessaire de payer une entreprise pour
|
||
faire en sorte d'avoir des certificats qui évitent d'avoir ce genre
|
||
d'erreurs dans vos navigateurs:</p>
|
||
<p><img alt="Message de firefox lorsque une connexion n'est pas
|
||
sécurisée." src="%7Bfilename%7D/static/unsecure-connection.png"></p>
|
||
<p>Maintenant, grâce à Let's Encrypt il est possible d'avoir des
|
||
certificats SSL <strong>gratuits</strong>, ce qui représente un grand pas en avant
|
||
pour la sécurité de nos communications.</p>
|
||
<p>Je viens de mettre en place un procédé (assez simple) qui permet de
|
||
configurer votre serveur pour générer des certificats SSL valides avec
|
||
Let's Encrypt et le répartiteur de charge
|
||
<a href="http://www.haproxy.org/">HAProxy</a>.</p>
|
||
<p>Je me suis basé pour cet article sur
|
||
d'<a href="https://blog.infomee.fr/p/letsencrypt-haproxy">autres</a>
|
||
<a href="http://blog.victor-hery.com/article22/utiliser-let-s-encrypt-avec-haproxy">articles</a>,
|
||
dont je vous recommande la lecture pour un complément d'information.</p>
|
||
<h2 id="validation-des-domaines-par-lets-encrypt">Validation des domaines par Let's Encrypt</h2>
|
||
<p>Je vous passe les détails d'installation du client de Let's Encrypt, qui
|
||
sont <a href="https://github.com/letsencrypt/letsencrypt#installation">très bien expliqués sur leur
|
||
documentation</a>.</p>
|
||
<p>Une fois installé, vous allez taper une commande qui va ressembler à:</p>
|
||
<div class="highlight"><pre><span></span><span class="n">letsencrypt</span><span class="o">-</span><span class="n">auto</span> <span class="n">certonly</span> <span class="c1">--renew-by-default</span>
|
||
<span class="c1">--webroot -w /home/www/letsencrypt-requests/ \</span>
|
||
<span class="o">-</span><span class="n">d</span> <span class="n">hurl</span><span class="p">.</span><span class="n">kinto</span><span class="o">-</span><span class="k">storage</span><span class="p">.</span><span class="n">org</span> <span class="err">\</span>
|
||
<span class="o">-</span><span class="n">d</span> <span class="n">forums</span><span class="p">.</span><span class="n">kinto</span><span class="o">-</span><span class="k">storage</span><span class="p">.</span><span class="n">org</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>Le <em>webroot</em> est l'endroit ou les preuves de détention du domaine vont
|
||
être déposées.</p>
|
||
<p>Lorsque les serveurs de Let's Encrypt vont vouloir vérifier que vous
|
||
êtes bien à l'origine des demandes de certificats, ils vont envoyer une
|
||
requête HTTP sur <code>http://domaine.org/.well-known/acme-challenge</code>, ou il
|
||
voudra trouver des informations qu'il aura généré via la commande
|
||
<code>letsencrypt-auto</code>.</p>
|
||
<p>J'ai choisi de faire une règle dans haproxy pour diriger toutes les
|
||
requêtes avec le chemin <code>.well-known/acme-challenge</code> vers un <em>backend</em>
|
||
nginx qui sert des fichiers statiques (ceux contenus dans
|
||
<code>/home/www/letsencrypt-requests/</code>).</p>
|
||
<p>Voici la section de la configuration de HAProxy (et <a href="https://github.com/almet/infra/blob/master/haproxy/haproxy.cfg#L63-L72">la configuration
|
||
complete</a>
|
||
si ça peut être utile):</p>
|
||
<div class="highlight"><pre><span></span><span class="nv">frontend</span> <span class="nv">http</span>
|
||
<span class="nv">bind</span> <span class="mi">0</span>.<span class="mi">0</span>.<span class="mi">0</span>.<span class="mi">0</span>:<span class="mi">80</span>
|
||
<span class="nv">mode</span> <span class="nv">http</span>
|
||
<span class="nv">default_backend</span> <span class="nv">nginx_server</span>
|
||
|
||
<span class="nv">acl</span> <span class="nv">letsencrypt_check</span> <span class="nv">path_beg</span> <span class="o">/</span>.<span class="nv">well</span><span class="o">-</span><span class="nv">known</span><span class="o">/</span><span class="nv">acme</span><span class="o">-</span><span class="nv">challenge</span>
|
||
<span class="nv">use_backend</span> <span class="nv">letsencrypt_backend</span> <span class="k">if</span> <span class="nv">letsencrypt_check</span>
|
||
|
||
<span class="nv">redirect</span> <span class="nv">scheme</span> <span class="nv">https</span> <span class="nv">code</span> <span class="mi">301</span> <span class="k">if</span> <span class="o">!</span>{ <span class="nv">ssl_fc</span> } <span class="o">!</span><span class="nv">letsencrypt_check</span>
|
||
|
||
<span class="nv">backend</span> <span class="nv">letsencrypt_backend</span>
|
||
<span class="nv">http</span><span class="o">-</span><span class="nv">request</span> <span class="nv">set</span><span class="o">-</span><span class="nv">header</span> <span class="nv">Host</span> <span class="nv">letsencrypt</span>.<span class="nv">requests</span>
|
||
<span class="nv">dispatch</span> <span class="mi">127</span>.<span class="mi">0</span>.<span class="mi">0</span>.<span class="mi">1</span>:<span class="mi">8000</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>Et celle de NGINX:</p>
|
||
<div class="highlight"><pre><span></span><span class="n">server</span> <span class="err">{</span>
|
||
<span class="k">listen</span> <span class="mi">8000</span><span class="p">;</span>
|
||
<span class="k">server_name</span> <span class="n">letsencrypt</span><span class="p">.</span><span class="n">requests</span><span class="p">;</span>
|
||
<span class="n">root</span> <span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">www</span><span class="o">/</span><span class="n">letsencrypt</span><span class="o">-</span><span class="n">requests</span><span class="p">;</span>
|
||
<span class="err">}</span>
|
||
</pre></div>
|
||
|
||
|
||
<h2 id="installation-des-certificats-dans-haproxy">Installation des certificats dans HAProxy</h2>
|
||
<p>Vos certificats SSL devraient être générés dans <code>/etc/letsencrypt/live</code>,
|
||
mais ils ne sont pas au format attendu par haproxy. Rien de grave, la
|
||
commande suivant convertit l'ensemble des certificats en une version
|
||
compatible avec
|
||
HAProxy:</p>
|
||
<div class="highlight"><pre><span></span><span class="n">cat</span> <span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">letsencrypt</span><span class="o">/</span><span class="n">live</span><span class="o">/</span><span class="n">domaine</span><span class="p">.</span><span class="n">org</span><span class="o">/</span><span class="n">privkey</span><span class="p">.</span><span class="n">pem</span> <span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">letsencrypt</span><span class="o">/</span><span class="n">live</span><span class="o">/</span><span class="n">domaine</span><span class="p">.</span><span class="n">org</span><span class="o">/</span><span class="n">fullchain</span><span class="p">.</span><span class="n">pem</span> <span class="o">&gt;</span> <span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">ssl</span><span class="o">/</span><span class="n">letsencrypt</span><span class="o">/</span><span class="n">domaine</span><span class="p">.</span><span class="n">org</span><span class="p">.</span><span class="n">pem</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>Et ensuite dans la configuration de haproxy, pour le (nouveau)
|
||
<em>frontend</em> https:</p>
|
||
<div class="highlight"><pre><span></span><span class="n">bind</span> <span class="mi">0</span><span class="p">.</span><span class="mi">0</span><span class="p">.</span><span class="mi">0</span><span class="p">.</span><span class="mi">0</span><span class="p">:</span><span class="mi">443</span> <span class="n">ssl</span> <span class="k">no</span><span class="o">-</span><span class="n">sslv3</span> <span class="n">crt</span> <span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">ssl</span><span class="o">/</span><span class="n">letsencrypt</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>Faites bien attention à avoir un <em>frontend</em> https pour tous vos sites en
|
||
HTTPS. <a href="https://github.com/almet/infra/blob/master/haproxy/haproxy.cfg#L38-L60">Pour moi cela ressemble à
|
||
ça</a>.</p>
|
||
<p>Une fois tout ceci fait, redémarrez votre service haproxy et zou !</p>
|
||
<h2 id="automatisation">Automatisation</h2>
|
||
<p>Pour automatiser un peu tout ça, j'ai choisi de faire ça comme suit:</p>
|
||
<ul>
|
||
<li>Un fichier domaine dans <code>letsencrypt/domains/domain.org</code> qui
|
||
contient le script <code>letsencrypt</code>.</li>
|
||
<li>Un fichier d'installation de certificats dans
|
||
<code>letsencrypt/install-certs.sh</code> qui s'occupe d'installer les
|
||
certificats déjà générés.</li>
|
||
</ul>
|
||
<p>Et voila ! <a href="https://github.com/almet/infra/">Le tout est dans un dépot
|
||
github</a>, si jamais ça peut vous servir,
|
||
tant mieux !</p></content></entry><entry><title>Ateliers d'autodéfense numérique</title><link href="https://blog.notmyidea.org/ateliers-dautodefense-numerique.html" rel="alternate"></link><published>2016-01-14T00:00:00+01:00</published><updated>2016-01-14T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2016-01-14:/ateliers-dautodefense-numerique.html</id><summary type="html"><p>Il y a huit mois, je me rendais compte de l'importance du choix des
|
||
outils pour faire face à la surveillance généralisée, et notamment en
|
||
rapport au chiffrement des données. Une de mes envies de l'époque était
|
||
l'animation d'ateliers.</p>
|
||
<blockquote>
|
||
<p>Je compte donc:</p>
|
||
<ul>
|
||
<li>Organiser des ateliers de sensibilisation aux outils de …</li></ul></blockquote></summary><content type="html"><p>Il y a huit mois, je me rendais compte de l'importance du choix des
|
||
outils pour faire face à la surveillance généralisée, et notamment en
|
||
rapport au chiffrement des données. Une de mes envies de l'époque était
|
||
l'animation d'ateliers.</p>
|
||
<blockquote>
|
||
<p>Je compte donc:</p>
|
||
<ul>
|
||
<li>Organiser des ateliers de sensibilisation aux outils de
|
||
communication, envers mes proches;</li>
|
||
<li>Utiliser la communication chiffrée le plus souvent possible, au
|
||
moins pour rendre le déchiffrement des messages plus longue,
|
||
"noyer le poisson".</li>
|
||
</ul>
|
||
<p>-- <a href="http://blog.notmyidea.org/chiffrement.html">Chiffrement</a></p>
|
||
</blockquote>
|
||
<p>J'ai mis un peu de temps à mettre le pied à l'étrier, mais je ressors
|
||
finalement du premier atelier que j'ai co-animé avec geb, auprès d'un
|
||
public de journalistes.</p>
|
||
<p>Pour cette première édition l'idée était à la fois d'aller à la
|
||
rencontre d'un public que je connais mal, de leur donner des outils pour
|
||
solutionner les problèmes auxquels ils font parfois face, et de me faire
|
||
une idée de ce que pouvait être un atelier sur l'autodéfense numérique.</p>
|
||
<p>L'objectif pour ce premier atelier était de:</p>
|
||
<ol>
|
||
<li>Échanger autour des besoins et <strong>faire ressortir des histoires</strong> ou
|
||
le manque d'outillage / connaissances à posé problème, dans des
|
||
situations concrètes;</li>
|
||
<li>Se rendre compte des "conduites à risque", <strong>faire peur</strong> aux
|
||
personnes formées pour qu'elles se rendent compte de l'état actuel
|
||
des choses;</li>
|
||
<li><strong>Proposer des solutions concrètes</strong> aux problèmes soulevés, ainsi
|
||
que le minimum de connaissance théorique pour les appréhender.</li>
|
||
</ol>
|
||
<h2 id="146-faire-ressortir-les-problemes">1. Faire ressortir les problèmes</h2>
|
||
<p>Afin de faire ressortir les problèmes, nous avons choisi de constituer
|
||
des petits groupes de discussion, afin de faire des "Groupes d'Interview
|
||
Mutuels", ou "GIM":</p>
|
||
<blockquote>
|
||
<p>l’animateur invite les participants à se regrouper par trois, avec des
|
||
personnes qu’on connaît moins puis invite chacun à livrer une
|
||
expérience vécue en lien avec le thème de la réunion et les deux
|
||
autres à poser des questions leur permettant de bien saisir ce qui a
|
||
été vécu.</p>
|
||
<p>-- «<a href="http://www.scoplepave.org/pour-s-ecouter">Pour s'écouter</a>», SCOP
|
||
Le Pavé.</p>
|
||
</blockquote>
|
||
<p>De ces <em>GIMs</em> nous avons pu ressortir quelques histoires, gravitant
|
||
autour de:</p>
|
||
<ul>
|
||
<li><strong>La protection des sources (d'information)</strong>: Comment faire pour
|
||
aider quelqu'un à faire "fuiter" des données depuis l'intérieur
|
||
d'une entreprise ?</li>
|
||
<li><strong>Le chiffrement de ses données</strong>: Comment éviter de faire "fuiter"
|
||
des données importantes lors d'une perquisition de matériel ?</li>
|
||
</ul>
|
||
<h2 id="246-faire-peur">2. Faire peur</h2>
|
||
<p>Un des premiers objectifs est de faire peur, afin que tout le monde se
|
||
rende compte à quel point il est facile d'accéder à certaines données.
|
||
<a href="http://blog.barbayellow.com/">Grégoire</a> m'avait conseillé quelques
|
||
petites accroches qui ont ma foi bien marché:</p>
|
||
<p>J'ai demandé aux présent.e.s de:</p>
|
||
<ul>
|
||
<li>donner leur mot de passe à voix haute devant les autres: a priori
|
||
personne ne le fera;</li>
|
||
<li>venir se connecter à leur compte email depuis mon ordinateur. J'ai
|
||
piégé une personne, qui est venu pour taper son mot de passe.</li>
|
||
</ul>
|
||
<p>Cela à été un bon moyen de parler de l'importance des traces que l'on
|
||
peut laisser sur un ordinateur, et de la confiance qu'il faut avoir dans
|
||
le matériel que l'on utilise, à fortiori si ce ne sont pas les vôtres.</p>
|
||
<p>Pour continuer à leur faire peur, après une brève explication de ce
|
||
qu'est SSL nous avons montré comment il était facile de scruter le
|
||
réseau à la recherche de mots de passe en clair.</p>
|
||
<h2 id="346-proposer-des-solutions-concretes">3. Proposer des solutions concrêtes</h2>
|
||
<p>Une fois que tout le monde avait pleinement pris sonscience des
|
||
problématiques et n'osait plus utiliser son ordinateur ou son
|
||
téléphone, on à commencé à parler de quelques solutions. Plusieurs
|
||
approches étaient possibles ici, nous avons choisi de présenter quelques
|
||
outils qui nous semblaient répondre aux attentes:</p>
|
||
<ul>
|
||
<li>On a expliqué ce qu'était <a href="https://tails.boum.org">Tails</a>, et
|
||
comment l'utiliser et le dupliquer.</li>
|
||
<li>On a pu faire un tour des outils existants sur Tails, notamment
|
||
autour de l'<em>anonymisation</em> de fichiers et la suppression effective
|
||
de contenus.</li>
|
||
<li>Certaines personnes ont pu créer une clé tails avec la persistance
|
||
de configurée.</li>
|
||
<li>Nous nous sommes connectés au réseau
|
||
<a href="https://www.torproject.org">Tor</a> et testé que nos adresses IP
|
||
changeaient bien à la demande.</li>
|
||
<li>Nous avons utilisé <a href="https://crypto.cat">CryptoCat</a> par dessus Tor,
|
||
afin de voir comment avoir une conversation confidentielle dans
|
||
laquelle il est possible d'échanger des fichiers.</li>
|
||
</ul>
|
||
<h2 id="retours">Retours</h2>
|
||
<p>D'une manière générale, pour une formation de trois heures et demi, je
|
||
suis assez content de l'exercice, et de l'ensemble des sujets que nous
|
||
avons pu couvrir. Il y a beaucoup de place pour l'amélioration,
|
||
notamment en amont (j'avais par exemple oublié d'amener avec moi
|
||
suffisamment de clés USB pour utiliser Tails).</p>
|
||
<p>La plupart des retours qu'on a pu avoir jusqu'à maintenant sont
|
||
positifs, et il y a l'envie d'aller plus loin sur l'ensemble de ces
|
||
sujets.</p>
|
||
<h2 id="la-suite">La suite</h2>
|
||
<p>Il y a beaucoup de sujets que nous n'avons pas abordés, ou uniquement
|
||
survolés, à cause du manque de temps disponible. Idéalement, il faudrait
|
||
au moins une journée entière pour couvrir quelques sujets plus en détail
|
||
(on peut imaginer avoir une partie théorique le matin et une partie
|
||
pratique l'après-midi par exemple).</p>
|
||
<p>J'ai choisi volontairement de ne pas aborder le chiffrement des messages
|
||
via PGP parce que <a href="%7Bfilename%7D2015.05.pgp-problemes.rst">je pense que la protection que ce média propose n'est
|
||
pas suffisante</a>, mais je suis
|
||
en train de revenir sur ma décision: il pourrait être utile de présenter
|
||
l'outil, à minima, en insistant sur certaines de ses faiblesses.</p>
|
||
<p>Un compte twitter à été créé recemment autour des crypto-party à Rennes,
|
||
si vous êtes interessés, <a href="https://twitter.com/CryptoPartyRNS">allez jeter un coup
|
||
d'œil</a>!</p>
|
||
<p>Je n'ai pas trouvé de ressources disponibles par rapport à des plans de
|
||
formation sur le sujet, j'ai donc décidé de publier les nôtres, afin de
|
||
co-construire avec d'autres des plans de formation.</p>
|
||
<p>Ils sont pour l'instant disponibles <a href="http://autodefense-numerique.readthedocs.org/en/latest/">sur Read The
|
||
Docs</a>. Tous les
|
||
retours sont évidemment les bienvenus !</p></content></entry><entry><title>Le mail doit-il mourir ?</title><link href="https://blog.notmyidea.org/le-mail-doit-il-mourir.html" rel="alternate"></link><published>2015-11-24T00:00:00+01:00</published><updated>2015-11-24T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2015-11-24:/le-mail-doit-il-mourir.html</id><summary type="html"><p>J'utilise quotidiennement le protocole email, tant bien que mal, tout en sachant que l'ensemble de mes messages passent en clair sur le réseau pour la plupart de mes conversations, puisque trop peu de monde utilise le chiffrement des messages.</p>
|
||
<p>Et même si j'arrive à convaincre certains de mes proches à …</p></summary><content type="html"><p>J'utilise quotidiennement le protocole email, tant bien que mal, tout en sachant que l'ensemble de mes messages passent en clair sur le réseau pour la plupart de mes conversations, puisque trop peu de monde utilise le chiffrement des messages.</p>
|
||
<p>Et même si j'arrive à convaincre certains de mes proches à installer PGP, je ne suis pas satisfait du résultat: les méta-données (qui contacte qui à quel
|
||
moment, et pour lui dire quoi) transitent de toute manière, elles, en clair, à la vue de tous.</p>
|
||
<p>Ce problème est lié directement au protocole email: il est <em>necessaire</em> de faire fuiter ces meta-données (au moins le destinataire) pour avoir un protocole
|
||
mail fonctionnel.</p>
|
||
<p>Le mail répond à un besoin de communication asynchrone qui permet des conversations plus réfléchies qu'un simple chat (miaou). Il est tout à fait possible d'utiliser certaines technologies existantes afin de construire le futur de l'email, pour lequel:</p>
|
||
<ul>
|
||
<li>Les méta-données seraient chiffrées — Il n'est pas possible de savoir qui
|
||
communique avec qui, et quand;</li>
|
||
<li>Le chiffrement serait fort (et protégé d'une phrase de passe ?);</li>
|
||
<li>La fuite d'une clé de chiffrement utilisée dans un échange ne permette pas de
|
||
déchiffrer l'ensemble des échanges (forward secrecy);</li>
|
||
<li>Il ne soit pas possible de réutiliser les données comme preuve pour
|
||
incriminer l'emmeteur du message (deniability);</li>
|
||
</ul>
|
||
<p>Avec au moins ces besoins en tête, il semble qu'une revue de l'ensemble des projets existants pointe du doigt vers <a href="https://github.com/agl/pond">pond</a>, ou vers <a href="https://www.whispersystems.org">Signal</a>.</p>
|
||
<p>Malheureusement, Pond est le projet d'une seule personne, qui veut plutôt utiliser ce code comme démonstration du concept en question.</p></content></entry><entry><title>Web distribution signing</title><link href="https://blog.notmyidea.org/web-distribution-signing.html" rel="alternate"></link><published>2015-10-12T00:00:00+02:00</published><updated>2015-10-12T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2015-10-12:/web-distribution-signing.html</id><summary type="html"><p><em>I'm not a crypto expert, nor pretend to be one. These are thoughts I
|
||
want to share with the crypto community to actually see if any solution
|
||
exists to solve this particular problem.</em></p>
|
||
<p>One <a href="http://www.tonyarcieri.com/whats-wrong-with-webcrypto">often pointed</a> flaw in
|
||
web-based cryptographic applications is the fact that there is no way to …</p></summary><content type="html"><p><em>I'm not a crypto expert, nor pretend to be one. These are thoughts I
|
||
want to share with the crypto community to actually see if any solution
|
||
exists to solve this particular problem.</em></p>
|
||
<p>One <a href="http://www.tonyarcieri.com/whats-wrong-with-webcrypto">often pointed</a> flaw in
|
||
web-based cryptographic applications is the fact that there is no way to
|
||
trust online software distributions. Put differently, you don't actually
|
||
trust the software authors but are rather trusting the software
|
||
distributors and certificate authorities (CAs).</p>
|
||
<p>I've been talking with a few folks in the past months about that and
|
||
they suggested me to publish something to discuss the matter. So here I
|
||
come!</p>
|
||
<h2 id="the-problem-attack-vectors">The problem (Attack vectors)</h2>
|
||
<p>Let's try to describe a few potential attacks:</p>
|
||
<p><em>Application Authors</em> just released a new version of their open source
|
||
web crypto messaging application. An <em>Indie Hoster</em> installs it on their
|
||
servers so a wide audience can actually use it.</p>
|
||
<p>Someone alters the files on <em>Indie Hoster</em> servers, effectively
|
||
replacing them with other <em>altered files</em> with less security properties
|
||
/ a backdoor. This someone could either be an <em>Evil Attacker</em> which
|
||
found its way trough, the <em>Indie Hoster</em> or a CDN which delivers the
|
||
files,</p>
|
||
<p>Trusted <em>Certificate Authorities</em> ("governments" or "hacking team") can
|
||
also trick the User Agents (i.e. Firefox) into thinking they're talking
|
||
to <em>Indie Hoster</em> even though they're actually talking to a different
|
||
server.</p>
|
||
<p><strong>Altered files</strong> are then being served to the User Agents, and <em>Evil
|
||
Attacker</em> now has a way to actually attack the end users.</p>
|
||
<h2 id="problem-mitigation">Problem Mitigation</h2>
|
||
<p>Part of the problem is solved by the recently introduced <a href="https://w3c.github.io/webappsec/specs/subresourceintegrity/">Sub Resource
|
||
Integrity</a>
|
||
(SRI). To quote them: "[it] defines a mechanism by which user agents
|
||
may verify that a fetched resource has been delivered without unexpected
|
||
manipulation.".</p>
|
||
<p>SRI is a good start, but isn't enough: it ensures the assets (JavaScript
|
||
files, mainly) loaded from a specific HTML page are the ones the author
|
||
of the HTML page intends. However, SRI doesn't allow the User Agent to
|
||
ensure the HTML page is the one he wants.</p>
|
||
<p>In other words, we miss a way to create trust between <em>Application
|
||
Authors</em> and <em>User Agents</em>. The User-Agent currently has to trust the
|
||
<em>Certificate Authorities</em> and the delivery (<em>Indie Hoster</em>).</p>
|
||
<p>For desktop software distribution: <em>Crypto Experts</em> audit the software,
|
||
sign it somehow and then this signature can be checked locally during
|
||
installation or runtime. It's not automated, but at least it's possible.</p>
|
||
<p>For web applications, we don't have such a mechanism, but it should be
|
||
possible. Consider the following:</p>
|
||
<ul>
|
||
<li><em>App Authors</em> publish a new version of their software; They provide
|
||
a hash of each of their distributed files (including the HTML
|
||
files);</li>
|
||
<li><em>Crypto Experts</em> audit these files and sign the hashes somehow;</li>
|
||
<li><em>User Agents</em> can chose to trust some specific <em>Crypto Experts</em>;</li>
|
||
<li>When a <em>User Agent</em> downloads files, it checks if they're signed by
|
||
a trusted party.</li>
|
||
</ul>
|
||
<h2 id="chosing-who-you-trust">Chosing who you trust</h2>
|
||
<p>In terms of user experience, handling certificates is hard, and that's
|
||
where the community matters. Distributions such as
|
||
<a href="https://tails.boom.org">Tails</a> could chose who they trust to verify the
|
||
files, and issue warnings / refuse to run the application in case files
|
||
aren't verified.</p>
|
||
<p>But, as highligted earlier, CAs are hard to trust. A new instance of the
|
||
same CA system wouldn't make that much differences, expect the fact that
|
||
distributions could ship with a set of trusted authorities (for which
|
||
revocation would still need to be taken care of).</p>
|
||
<blockquote>
|
||
<p>[...] users are vulnerable to MitM attacks by the authority, which
|
||
can vouch for, or be coerced to vouch for, false keys. This weakness
|
||
has been highlighted by recent CA scandals. Both schemes can also be
|
||
attacked if the authority does not verify keys before vouching for
|
||
them.</p>
|
||
<p>-- <a href="http://cacr.uwaterloo.ca/techreports/2015/cacr2015-02.pdf">SoK : Secure
|
||
Messaging</a>;</p>
|
||
</blockquote>
|
||
<p>It seems that some other systems could allow for something more
|
||
reliable:</p>
|
||
<blockquote>
|
||
<p>Melara et al proposed CONIKS, using a series of chained commitments to
|
||
Merkle prefix trees to build a key directory [...] for which
|
||
individual users can efficiently verify the consistency of their own
|
||
entry in the directory without relying on a third party.</p>
|
||
<p>This “self- auditing log” approach makes the system partially have no
|
||
auditing required (as general auditing of non-equivocation is still
|
||
required) and also enables the system to be privacy preserving as the
|
||
entries in the directory need not be made public. This comes at a mild
|
||
bandwidth cost not reflected in our table, estimated to be about 10
|
||
kilobytes per client per day for self-auditing.</p>
|
||
<p>-- <a href="http://cacr.uwaterloo.ca/techreports/2015/cacr2015-02.pdf">SoK : Secure
|
||
Messaging</a>;</p>
|
||
</blockquote>
|
||
<p>Now, I honestly have no idea if this thing solves the whole problem, and
|
||
I'm pretty sure this design has many security problems attached to it.</p>
|
||
<p>However, that's a problem I would really like to see solved one day, so
|
||
here the start of the discussion, don't hesitate to <a href="/pages/about.html">get in
|
||
touch</a>!</p>
|
||
<h2 id="addendum">Addendum</h2>
|
||
<p>It seems possible to increase the level a user has in a Web Application
|
||
by adding indicators in the User-Agent. For instance, when using an
|
||
application that's actually signed by someone considered trustful by the
|
||
User-Agent (or the distributor of the User-Agent), a little green icon
|
||
could be presented to the User, so they know that they can be confident
|
||
about this.</p>
|
||
<p>A bit like User-Agents do for SSL, but for the actual signature of the
|
||
files being viewed.</p></content></entry><entry><title>Service de nuages : Pourquoi avons-nous fait Cliquet ?</title><link href="https://blog.notmyidea.org/pourquoi-cliquet" rel="alternate"></link><published>2015-07-14T00:00:00+02:00</published><updated>2015-07-14T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2015-07-14:/pourquoi-cliquet</id><summary type="html"><p class="first last">Basé sur Pyramid, Cliquet est un projet qui permet de se concentrer sur l'essentiel
|
||
lors de la conception d'APIs.</p>
|
||
</summary><content type="html"><p><em>Cet article est repris depuis le blog « Service de Nuages » de mon équipe à Mozilla</em></p>
|
||
<p><strong>tldr; Cliquet est un toolkit Python pour construire des APIs, qui implémente
|
||
les bonnes pratiques en terme de mise en production et de protocole HTTP.</strong></p>
|
||
<div class="section" id="les-origines">
|
||
<h2>Les origines</h2>
|
||
<p>L'objectif pour le premier trimestre 2015 était de construire un service de
|
||
stockage et de <a class="reference external" href="{filename}2015.04.service-de-nuages.rst">synchronisation de listes de lecture</a>.</p>
|
||
<p>Au démarrage du projet, nous avons tenté de rassembler toutes les bonnes pratiques
|
||
et recommandations, venant de différentes équipes et surtout des derniers projets déployés.</p>
|
||
<p>De même, nous voulions tirer parti du protocole de <em>Firefox Sync</em>, robuste et éprouvé,
|
||
pour la synchronisation des données «offline».</p>
|
||
<p>Plutôt qu'écrire un <a class="reference external" href="http://blog.octo.com/en/design-a-rest-api/">énième</a>
|
||
<a class="reference external" href="http://www.vinaysahni.com/best-practices-for-a-pragmatic-restful-api">article</a> de blog,
|
||
nous avons préféré les rassembler dans ce qu'on a appellé «un protocole».</p>
|
||
<p>Comme pour l'architecture envisagée nous avions deux projets à construire, qui
|
||
devaient obéir globalement à ces mêmes règles, nous avons décidé de mettre en
|
||
commun l'implémentation de ce protocole et de ces bonnes pratiques dans un
|
||
«toolkit».</p>
|
||
<p><em>Cliquet</em> est né.</p>
|
||
<img alt="Cliquet logo" class="align-center" src="{filename}/images/cliquet-logo.png" />
|
||
<div class="section" id="les-intentions">
|
||
<h3>Les intentions</h3>
|
||
<blockquote class="epigraph">
|
||
Quelle structure JSON pour mon API ? Quelle syntaxe pour filtrer la liste
|
||
via la querystring ? Comment gérer les écritures concurrentes ?
|
||
Et synchroniser les données dans mon application cliente ?</blockquote>
|
||
<p>Désormais, quand un projet souhaite bénéficier d'une API REST pour stocker et consommer
|
||
des données, il est possible d'utiliser le <strong>protocole HTTP</strong> proposé
|
||
et de se concentrer sur l'essentiel. Cela vaut aussi pour les clients, où
|
||
la majorité du code d'interaction avec le serveur est réutilisable.</p>
|
||
<blockquote class="epigraph">
|
||
Comment pouvons-nous vérifier que le service est opérationnel ? Quels indicateurs StatsD ?
|
||
Est-ce que Sentry est bien configuré ? Comment déployer une nouvelle version
|
||
sans casser les applications clientes ?</blockquote>
|
||
<p>Comme <em>Cliquet</em> fournit tout ce qui est nécessaire pour être conforme avec les
|
||
exigences de la <strong>mise en production</strong>, le passage du prototype au service opérationnel
|
||
est très rapide ! De base le service répondra aux attentes en terme supervision, configuration,
|
||
déploiement et dépréciation de version. Et si celles-ci évoluent, il suffira
|
||
de faire évoluer le toolkit.</p>
|
||
<blockquote class="epigraph">
|
||
Quel backend de stockage pour des documents JSON ? Comment faire si l'équipe
|
||
de production impose PostgreSQL ? Et si on voulait passer à Redis ou en
|
||
mémoire pour lancer les tests ?</blockquote>
|
||
<p>En terme d'implémentation, nous avons choisi de <strong>fournir des abstractions</strong>.
|
||
En effet, nous avions deux services dont le coeur consistait
|
||
à exposer un <em>CRUD</em> en <em>REST</em>, persistant des données JSON dans un backend.
|
||
Comme <em>Pyramid</em> et <em>Cornice</em> ne fournissent rien de tout prêt pour ça,
|
||
nous avons voulu introduire des classes de bases pour abstraire les notions
|
||
de resource REST et de backend de stockage.</p>
|
||
<p>Dans le but de tout rendre optionnel et «pluggable», <strong>tout est configurable</strong>
|
||
depuis le fichier <tt class="docutils literal">.ini</tt> de l'application. Ainsi tous les projets qui utilisent
|
||
le toolkit se déploieront de la même manière : seuls quelques éléments de configuration
|
||
les distingueront.</p>
|
||
<img alt="Une réunion à Paris..." class="align-center" src="{filename}/images/cliquet-notes-whiteboard.jpg" />
|
||
</div>
|
||
</div>
|
||
<div class="section" id="le-protocole">
|
||
<h2>Le protocole</h2>
|
||
<blockquote class="epigraph">
|
||
Est-ce suffisant de parler d'«API REST» ? Est-ce bien nécessaire de
|
||
relire la spec HTTP à chaque fois ? Pourquoi réinventer un protocole complet
|
||
à chaque fois ?</blockquote>
|
||
<p>Quand nous développons un (micro)service Web, nous dépensons généralement beaucoup
|
||
trop d'énergie à (re)faire des choix (arbitraires).</p>
|
||
<p>Nul besoin de lister ici tout ce qui concerne la dimension
|
||
de la spécification HTTP pure, qui nous impose le format des headers,
|
||
le support de CORS, la négocation de contenus (types mime), la différence entre
|
||
authentification et autorisation, la cohérence des code status...</p>
|
||
<p>Les choix principaux du protocole concernent surtout :</p>
|
||
<ul class="simple">
|
||
<li><strong>Les resources REST</strong> : Les deux URLs d'une resource (pour la collection
|
||
et les enregistrements) acceptent des verbes et des headers précis.</li>
|
||
<li><strong>Les formats</strong> : le format et la structure JSON des réponses est imposé, ainsi
|
||
que la <a class="reference external" href="{filename}/2015.05.continuation-token.rst">pagination des listes</a>
|
||
ou la syntaxe pour filtrer/trier les resources via la <a class="reference external" href="https://en.wikipedia.org/wiki/Query_string">querystring</a>.</li>
|
||
<li><strong>Les timestamps</strong> : un numéro de révision qui s'incrémente à chaque opération
|
||
d'écriture sur une collection d'enregistrements.</li>
|
||
<li><strong>La synchronisation</strong> : une série de leviers pour récupérer et renvoyer des
|
||
changements sur les données, sans perte ni collision, en utilisant les timestamps.</li>
|
||
<li><strong>Les permissions</strong> : les droits d'un utilisateur sur une collection ou un enregistrement
|
||
(<em>encore frais et sur le point d'être documenté</em>) <a class="footnote-reference" href="#id3" id="id1">[1]</a>.</li>
|
||
<li><strong>Opérations par lot</strong>: une URL qui permet d'envoyer une série de requêtes
|
||
décrites en JSON et d'obtenir les réponses respectives.</li>
|
||
</ul>
|
||
<p>Dans la dimension opérationnelle du protocole, on trouve :</p>
|
||
<ul class="simple">
|
||
<li><strong>La gestion de version</strong> : cohabitation de plusieurs versions en production,
|
||
avec alertes dans les entêtes pour la fin de vie des anciennes versions.</li>
|
||
<li><strong>Le report des requêtes</strong> : entêtes interprétées par les clients, activées en cas de
|
||
maintenance ou de surchage, pour ménager le serveur.</li>
|
||
<li><strong>Le canal d'erreurs</strong> : toutes les erreurs renvoyées par le serveur ont le même
|
||
format JSON et ont un numéro précis.</li>
|
||
<li><strong>Les utilitaires</strong> : URLs diverses pour répondre aux besoins exprimés par
|
||
l'équipe d'administrateurs (monitoring, metadonnées, paramètres publiques).</li>
|
||
</ul>
|
||
<p>Ce protocole est une compilation des bonnes pratiques pour les APIs HTTP (<em>c'est notre métier !</em>),
|
||
des conseils des administrateurs système dont c'est le métier de mettre à disposition des services
|
||
pour des millions d'utilisateurs et des retours d'expérience de l'équipe
|
||
de <em>Firefox Sync</em> pour la gestion de la concurrence et de l'«offline-first».</p>
|
||
<p>Il est <a class="reference external" href="http://cliquet.readthedocs.org/en/latest/api/index.html">documenté en détail</a>.</p>
|
||
<p>Dans un monde idéal, ce protocole serait versionné, et formalisé dans une RFC.
|
||
En rêve, il existerait même plusieurs implémentations avec des technologies différentes
|
||
(Python, Go, Node, etc.). <a class="footnote-reference" href="#id4" id="id2">[2]</a></p>
|
||
<table class="docutils footnote" frame="void" id="id3" rules="none">
|
||
<colgroup><col class="label" /><col /></colgroup>
|
||
<tbody valign="top">
|
||
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>Voir notre <a class="reference external" href="{filename}/2015.05.cliquet-permissions.rst">article dédié sur les permissions</a></td></tr>
|
||
</tbody>
|
||
</table>
|
||
<table class="docutils footnote" frame="void" id="id4" rules="none">
|
||
<colgroup><col class="label" /><col /></colgroup>
|
||
<tbody valign="top">
|
||
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>Rappel: nous sommes une toute petite équipe !</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
<div class="section" id="le-toolkit">
|
||
<h2>Le toolkit</h2>
|
||
<div class="section" id="choix-techniques">
|
||
<h3>Choix techniques</h3>
|
||
<p><em>Cliquet</em> implémente le protocole en Python (<em>2.7, 3.4+, pypy</em>), avec <a class="reference external" href="http://trypyramid.com/">Pyramid</a> <a class="footnote-reference" href="#id6" id="id5">[3]</a>.</p>
|
||
<p><strong>Pyramid</strong> est un framework Web qui va prendre en charge tout la partie HTTP,
|
||
et qui s'avère pertinent aussi bien pour des petits projets que des plus
|
||
ambitieux.</p>
|
||
<p><strong>Cornice</strong> est une extension de <em>Pyramid</em>, écrite en partie par Alexis et Tarek,
|
||
qui permet d'éviter d'écrire tout le code <em>boilerplate</em> quand on construit une
|
||
API REST avec Pyramid.</p>
|
||
<p>Avec <em>Cornice</em>, on évite de réécrire à chaque fois le code qui va
|
||
cabler les verbes HTTP aux méthodes, valider les entêtes, choisir le sérialiseur
|
||
en fonction des entêtes de négociation de contenus, renvoyer les codes HTTP
|
||
rigoureux, gérer les entêtes CORS, fournir la validation JSON à partir de schémas...</p>
|
||
<p><strong>Cliquet</strong> utilise les deux précédents pour implémenter le protocole et fournir
|
||
des abstractions, mais on a toujours <em>Pyramid</em> et <em>Cornice</em> sous la main pour
|
||
aller au delà de ce qui est proposé !</p>
|
||
<table class="docutils footnote" frame="void" id="id6" rules="none">
|
||
<colgroup><col class="label" /><col /></colgroup>
|
||
<tbody valign="top">
|
||
<tr><td class="label"><a class="fn-backref" href="#id5">[3]</a></td><td>Au tout début nous avons commencé une implémentation avec <em>Python-Eve</em>
|
||
(Flask), mais n'étions pas satisfaits de l'approche pour la configuration
|
||
de l'API. En particulier du côté magique.</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
<div class="section" id="concepts">
|
||
<h3>Concepts</h3>
|
||
<p>Bien évidemment, les concepts du toolkit reflètent ceux du protocole mais il y
|
||
a des éléments supplémentaires:</p>
|
||
<ul class="simple">
|
||
<li><strong>Les backends</strong> : abstractions pour le stockage, le cache et les permissions
|
||
(<em>ex. PostgreSQL, Redis, en-mémoire, ...</em>)</li>
|
||
<li><strong>La supervision</strong> : logging JSON et indicateurs temps-réel (<em>StatsD</em>) pour suivre les
|
||
performances et la santé du service.</li>
|
||
<li><strong>La configuration</strong> : chargement de la configuration depuis les variables
|
||
d'environnement et le fichier <tt class="docutils literal">.ini</tt></li>
|
||
<li><strong>La flexibilité</strong> : dés/activation ou substitution de la majorité des composants
|
||
depuis la configuration.</li>
|
||
<li><strong>Le profiling</strong> : utilitaires de développement pour trouver les <a class="reference external" href="https://fr.wiktionary.org/wiki/goulet_d%E2%80%99%C3%A9tranglement">goulets
|
||
d'étranglement</a>.</li>
|
||
</ul>
|
||
<img alt="Cliquet concepts" class="align-center" src="{filename}/images/cliquet-concepts.png" />
|
||
<p>Proportionnellement, l'implémentation du protocole pour les resources REST est
|
||
la plus volumineuse dans le code source de <em>Cliquet</em>.
|
||
Cependant, comme nous l'avons décrit plus haut, <em>Cliquet</em> fournit tout un
|
||
ensemble d'outillage et de bonnes pratiques, et reste
|
||
donc tout à fait pertinent pour n'importe quel type d'API, même sans
|
||
manipulation de données !</p>
|
||
<p>L'objectif de la boîte à outils est de faire en sorte qu'un développeur puisse constuire
|
||
une application simplement, en étant sûr qu'elle réponde aux exigeances de la
|
||
mise en production, tout en ayant la possibilité de remplacer certaines parties
|
||
au fur et à mesure que ses besoins se précisent.</p>
|
||
<p>Par exemple, la persistence fournie par défault est <em>schemaless</em> (e.g <em>JSONB</em>),
|
||
mais rien n'empêcherait d'implémenter le stockage dans un modèle relationnel.</p>
|
||
<p>Comme les composants peuvent être remplacés depuis la configuration, il est
|
||
tout à fait possible d'étendre <em>Cliquet</em> avec des notions métiers ou des
|
||
technologies exotiques ! Nous avons posé quelques idées dans <a class="reference external" href="http://cliquet.readthedocs.org/en/latest/ecosystem.html">la documentation
|
||
de l'éco-système</a>.</p>
|
||
<p>Dans les prochaines semaines, nous allons introduire la notion d'«évènements» (ou signaux),
|
||
qui permettraient aux extensions de s'interfacer beaucoup plus proprement.</p>
|
||
<p>Nous attachons beaucoup d'importance à la clareté du code, la pertinence des
|
||
<em>patterns</em>, des tests et de la documentation. Si vous avez des commentaires,
|
||
des critiques ou des interrogations, n'hésitez pas à <a class="reference external" href="https://github.com/mozilla-services/cliquet/issues">nous en faire part</a> !</p>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="cliquet-a-l-action">
|
||
<h2>Cliquet, à l'action.</h2>
|
||
<p>Nous avons écrit un <a class="reference external" href="http://cliquet.readthedocs.org/en/latest/quickstart.html">guide de démarrage</a>,
|
||
qui n'exige pas de connaître <em>Pyramid</em>.</p>
|
||
<p>Pour illustrer la simplicité et les concepts, voici quelques extraits !</p>
|
||
<div class="section" id="etape-1">
|
||
<h3>Étape 1</h3>
|
||
<p>Activer <em>Cliquet</em>:</p>
|
||
<div class="highlight"><pre><span></span><span class="hll"><span class="kn">import</span> <span class="nn">cliquet</span>
|
||
</span><span class="kn">from</span> <span class="nn">pyramid.config</span> <span class="kn">import</span> <span class="n">Configurator</span>
|
||
|
||
<span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">global_config</span><span class="p">,</span> <span class="o">**</span><span class="n">settings</span><span class="p">):</span>
|
||
<span class="n">config</span> <span class="o">=</span> <span class="n">Configurator</span><span class="p">(</span><span class="n">settings</span><span class="o">=</span><span class="n">settings</span><span class="p">)</span>
|
||
|
||
<span class="hll"> <span class="n">cliquet</span><span class="o">.</span><span class="n">initialize</span><span class="p">(</span><span class="n">config</span><span class="p">,</span> <span class="s1">&#39;1.0&#39;</span><span class="p">)</span>
|
||
</span> <span class="k">return</span> <span class="n">config</span><span class="o">.</span><span class="n">make_wsgi_app</span><span class="p">()</span>
|
||
</pre></div>
|
||
<p>À partir de là, la plupart des outils de <em>Cliquet</em> sont activés et accessibles.</p>
|
||
<p>Par exemple, les URLs <em>hello</em> (<tt class="docutils literal">/v1/</tt>) ou <em>supervision</em> (<tt class="docutils literal">/v1/__heartbeat__</tt>).
|
||
Mais aussi les backends de stockage, de cache, etc.
|
||
qu'il est possible d'utiliser dans des vues classiques <em>Pyramid</em> ou <em>Cornice</em>.</p>
|
||
</div>
|
||
<div class="section" id="etape-2">
|
||
<h3>Étape 2</h3>
|
||
<p>Ajouter des vues:</p>
|
||
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">global_config</span><span class="p">,</span> <span class="o">**</span><span class="n">settings</span><span class="p">):</span>
|
||
<span class="n">config</span> <span class="o">=</span> <span class="n">Configurator</span><span class="p">(</span><span class="n">settings</span><span class="o">=</span><span class="n">settings</span><span class="p">)</span>
|
||
|
||
<span class="n">cliquet</span><span class="o">.</span><span class="n">initialize</span><span class="p">(</span><span class="n">config</span><span class="p">,</span> <span class="s1">&#39;1.0&#39;</span><span class="p">)</span>
|
||
<span class="hll"> <span class="n">config</span><span class="o">.</span><span class="n">scan</span><span class="p">(</span><span class="s2">&quot;myproject.views&quot;</span><span class="p">)</span>
|
||
</span> <span class="k">return</span> <span class="n">config</span><span class="o">.</span><span class="n">make_wsgi_app</span><span class="p">()</span>
|
||
</pre></div>
|
||
<p>Pour définir des resources CRUD, il faut commencer par définir un schéma,
|
||
avec <em>Colander</em>, et ensuite déclarer une resource:</p>
|
||
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">cliquet</span> <span class="kn">import</span> <span class="n">resource</span><span class="p">,</span> <span class="n">schema</span>
|
||
|
||
<span class="k">class</span> <span class="nc">BookmarkSchema</span><span class="p">(</span><span class="n">schema</span><span class="o">.</span><span class="n">ResourceSchema</span><span class="p">):</span>
|
||
<span class="n">url</span> <span class="o">=</span> <span class="n">schema</span><span class="o">.</span><span class="n">URL</span><span class="p">()</span>
|
||
|
||
<span class="hll"><span class="nd">@resource.register</span><span class="p">()</span>
|
||
</span><span class="hll"><span class="k">class</span> <span class="nc">Bookmark</span><span class="p">(</span><span class="n">resource</span><span class="o">.</span><span class="n">BaseResource</span><span class="p">):</span>
|
||
</span><span class="hll"> <span class="n">mapping</span> <span class="o">=</span> <span class="n">BookmarkSchema</span><span class="p">()</span>
|
||
</span></pre></div>
|
||
<p>Désormais, la resource CRUD est disponible sur <tt class="docutils literal">/v1/bookmarks</tt>, avec toutes
|
||
les fonctionnalités de synchronisation, filtrage, tri, pagination, timestamp, etc.
|
||
De base les enregistrements sont privés, par utilisateur.</p>
|
||
<div class="highlight"><pre><span></span><span class="err">$</span> <span class="err">http</span> <span class="err">GET</span> <span class="s2">&quot;http://localhost:8000/v1/bookmarks&quot;</span>
|
||
<span class="err">HTTP/</span><span class="mf">1.1</span> <span class="mi">200</span> <span class="err">OK</span>
|
||
<span class="err">...</span>
|
||
<span class="p">{</span>
|
||
<span class="nt">&quot;data&quot;</span><span class="p">:</span> <span class="p">[</span>
|
||
<span class="p">{</span>
|
||
<span class="nt">&quot;url&quot;</span><span class="p">:</span> <span class="s2">&quot;http://cliquet.readthedocs.org&quot;</span><span class="p">,</span>
|
||
<span class="nt">&quot;id&quot;</span><span class="p">:</span> <span class="s2">&quot;cc103eb5-0c80-40ec-b6f5-dad12e7d975e&quot;</span><span class="p">,</span>
|
||
<span class="nt">&quot;last_modified&quot;</span><span class="p">:</span> <span class="mi">1437034418940</span><span class="p">,</span>
|
||
<span class="p">}</span>
|
||
<span class="p">]</span>
|
||
<span class="p">}</span>
|
||
</pre></div>
|
||
</div>
|
||
<div class="section" id="etape-3">
|
||
<h3>Étape 3</h3>
|
||
<p>Évidemment, il est possible choisir les URLS, les verbes HTTP supportés, de modifier
|
||
des champs avant l'enregistrement, etc.</p>
|
||
<div class="highlight"><pre><span></span><span class="hll"><span class="nd">@resource.register</span><span class="p">(</span><span class="n">collection_path</span><span class="o">=</span><span class="s1">&#39;/user/bookmarks&#39;</span><span class="p">,</span>
|
||
</span><span class="hll"> <span class="n">record_path</span><span class="o">=</span><span class="s1">&#39;/user/bookmarks/{{id}}&#39;</span><span class="p">,</span>
|
||
</span><span class="hll"> <span class="n">collection_methods</span><span class="o">=</span><span class="p">(</span><span class="s1">&#39;GET&#39;</span><span class="p">,))</span>
|
||
</span><span class="k">class</span> <span class="nc">Bookmark</span><span class="p">(</span><span class="n">resource</span><span class="o">.</span><span class="n">BaseResource</span><span class="p">):</span>
|
||
<span class="n">mapping</span> <span class="o">=</span> <span class="n">BookmarkSchema</span><span class="p">()</span>
|
||
|
||
<span class="hll"> <span class="k">def</span> <span class="nf">process_record</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">new</span><span class="p">,</span> <span class="n">old</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
|
||
</span><span class="hll"> <span class="k">if</span> <span class="n">old</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span> <span class="ow">and</span> <span class="n">new</span><span class="p">[</span><span class="s1">&#39;device&#39;</span><span class="p">]</span> <span class="o">!=</span> <span class="n">old</span><span class="p">[</span><span class="s1">&#39;device&#39;</span><span class="p">]:</span>
|
||
</span><span class="hll"> <span class="n">device</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">headers</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">&#39;User-Agent&#39;</span><span class="p">)</span>
|
||
</span><span class="hll"> <span class="n">new</span><span class="p">[</span><span class="s1">&#39;device&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">device</span>
|
||
</span><span class="hll"> <span class="k">return</span> <span class="n">new</span>
|
||
</span></pre></div>
|
||
<p><a class="reference external" href="http://cliquet.readthedocs.org/en/latest/reference/resource.html">Plus d'infos dans la documentation dédiée</a> !</p>
|
||
<div class="admonition note">
|
||
<p class="first admonition-title">Note</p>
|
||
<p class="last">Il est possible de définir des resources sans validation de schema.
|
||
<a class="reference external" href="https://github.com/mozilla-services/kinto/blob/master/kinto/views/records.py">Voir le code source de Kinto</a>.</p>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="etape-4-optionelle">
|
||
<h3>Étape 4 (optionelle)</h3>
|
||
<p>Utiliser les abstractions de <em>Cliquet</em> dans une vue <em>Cornice</em>.</p>
|
||
<p>Par exemple, une vue qui utilise le backend de stockage:</p>
|
||
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">cliquet</span> <span class="kn">import</span> <span class="n">Service</span>
|
||
|
||
<span class="n">score</span> <span class="o">=</span> <span class="n">Service</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s2">&quot;score&quot;</span><span class="p">,</span>
|
||
<span class="n">path</span><span class="o">=</span><span class="s1">&#39;/score/{game}&#39;</span><span class="p">,</span>
|
||
<span class="n">description</span><span class="o">=</span><span class="s2">&quot;Store game score&quot;</span><span class="p">)</span>
|
||
|
||
<span class="nd">@score.post</span><span class="p">(</span><span class="n">schema</span><span class="o">=</span><span class="n">ScoreSchema</span><span class="p">)</span>
|
||
<span class="k">def</span> <span class="nf">post_score</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
|
||
<span class="n">collection_id</span> <span class="o">=</span> <span class="s1">&#39;scores-&#39;</span> <span class="o">+</span> <span class="n">request</span><span class="o">.</span><span class="n">match_dict</span><span class="p">[</span><span class="s1">&#39;game&#39;</span><span class="p">]</span>
|
||
<span class="n">user_id</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">authenticated_userid</span>
|
||
<span class="n">value</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">validated</span> <span class="c1"># c.f. Cornice.</span>
|
||
|
||
<span class="hll"> <span class="n">storage</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">registry</span><span class="o">.</span><span class="n">storage</span>
|
||
</span><span class="hll"> <span class="n">record</span> <span class="o">=</span> <span class="n">storage</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="n">collection_id</span><span class="p">,</span> <span class="n">user_id</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
|
||
</span> <span class="k">return</span> <span class="n">record</span>
|
||
</pre></div>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="vos-retours">
|
||
<h2>Vos retours</h2>
|
||
<p>N'hésitez pas à nous faire part de vos retours ! Cela vous a donné envie
|
||
d'essayer ? Vous connaissez un outil similaire ?
|
||
Y-a-t-il des points qui ne sont pas clairs ? Manque de cas d'utilisation concrets ?
|
||
Certains aspects mal pensés ? Trop contraignants ? Trop de magie ? Overkill ?</p>
|
||
<p>Nous prenons tout.</p>
|
||
<div class="section" id="points-faibles">
|
||
<h3>Points faibles</h3>
|
||
<p>Nous sommes très fiers de ce que nous avons construit, en relativement peu
|
||
de temps. Et comme nous l'exposions dans <a class="reference external" href="{filename/2015.07.whistler-use-cases.rst}">l'article précédent</a>, il y a du potentiel !</p>
|
||
<p>Cependant, nous sommes conscients d'un certain nombre de points
|
||
qui peuvent être vus comme des faiblesses.</p>
|
||
<ul class="simple">
|
||
<li><strong>La documentation d'API</strong> : actuellement, nous n'avons pas de solution pour qu'un
|
||
projet qui utilise <em>Cliquet</em> puisse intégrer facilement toute
|
||
<a class="reference external" href="http://cliquet.readthedocs.org/en/latest/api/index.html">la documentation de l'API</a>
|
||
obtenue.</li>
|
||
<li><strong>La documentation</strong> : il est très difficile d'organiser la documentation, surtout
|
||
quand le public visé est aussi bien débutant qu'expérimenté. Nous sommes probablement
|
||
victimes du «<a class="reference external" href="https://en.wikipedia.org/wiki/Curse_of_knowledge">curse of knowledge</a>».</li>
|
||
<li><strong>Le protocole</strong> : on sent bien qu'on va devoir versionner le protocole. Au
|
||
moins pour le désolidariser des versions de <em>Cliquet</em>, si on veut aller au
|
||
bout de la philosophie et de l'éco-système.</li>
|
||
<li><strong>Le conservatisme</strong> : Nous aimons la stabilité et la robustesse. Mais surtout
|
||
nous ne sommes pas tout seuls et devons nous plier aux contraintes de la mise
|
||
en production ! Cependant, nous avons très envie de faire de l'async avec Python 3 !</li>
|
||
<li><strong>Publication de versions</strong> : le revers de la médaille de la factorisation. Il
|
||
arrive qu'on préfère faire évoluer le toolkit (e.g. ajouter une option) pour
|
||
un point précis d'un projet. En conséquence, on doit souvent releaser les
|
||
projets en cascade.</li>
|
||
</ul>
|
||
</div>
|
||
<div class="section" id="quelques-questions-courantes">
|
||
<h3>Quelques questions courantes</h3>
|
||
<blockquote>
|
||
Pourquoi Python ?</blockquote>
|
||
<p>On prend beaucoup de plaisir à écrire du Python, et le calendrier annoncé
|
||
initialement était très serré: pas question de tituber avec une technologie
|
||
mal maitrisée !</p>
|
||
<p>Et puis, après avoir passé près d'un an sur un projet Node.js, l'équipe avait
|
||
bien envie de refaire du Python.</p>
|
||
<blockquote>
|
||
Pourquoi pas Django ?</blockquote>
|
||
<p>On y a pensé, surtout parce qu'il y a plusieurs fans de <em>Django REST Framework</em>
|
||
dans l'équipe.</p>
|
||
<p>On l'a écarté principalement au profit de la légèreté et la modularité de
|
||
<em>Pyramid</em>.</p>
|
||
<blockquote>
|
||
Pourquoi pas avec un framework asynchrone en Python 3+ ?</blockquote>
|
||
<p>Pour l'instant nos administrateurs système nous imposent des déploiements en
|
||
Python 2.7, à notre grand désarroi /o\</p>
|
||
<p>Pour <em>Reading List</em>, nous <a class="reference external" href="https://github.com/mozilla-services/readinglist/blob/1.7.0/readinglist/__init__.py#L19-L26">avions activé</a>
|
||
<em>gevent</em>.</p>
|
||
<p>Puisque l'approche consiste à implémenter un protocole bien déterminé, nous n'excluons
|
||
pas un jour d'écrire un <em>Cliquet</em> en <em>aiohttp</em> ou <em>Go</em> si cela s'avèrerait pertinent.</p>
|
||
<blockquote>
|
||
Pourquoi pas JSON-API ?</blockquote>
|
||
<p>Comme nous l'expliquions <a class="reference external" href="{filename}/2015.05.retour-apidays.rst">au retour des APIdays</a>,
|
||
JSON-API est une spécification qui rejoint plusieurs de nos intentions.</p>
|
||
<p>Quand nous avons commencé le protocole, nous ne connaissions pas JSON-API.
|
||
Pour l'instant, comme notre proposition est beaucoup plus minimaliste, le
|
||
rapprochement n'a <a class="reference external" href="https://github.com/mozilla-services/cliquet/issues/254">pas dépassé le stade de la discussion</a>.</p>
|
||
<blockquote>
|
||
Est-ce que Cliquet est un framework REST pour Pyramid ?</blockquote>
|
||
<p>Non.</p>
|
||
<p>Au delà des classes de resources CRUD de Cliquet, qui implémentent un
|
||
protocole bien précis, il faut utiliser Cornice ou Pyramid directement.</p>
|
||
<blockquote>
|
||
Est-ce que Cliquet est suffisamment générique pour des projets hors Mozilla ?</blockquote>
|
||
<p>Premièrement, nous faisons en sorte que tout soit contrôlable depuis la
|
||
configuration <tt class="docutils literal">.ini</tt> pour permettre la dés/activation ou substitution des
|
||
composants.</p>
|
||
<p>Si le protocole HTTP/JSON des resources CRUD vous satisfait,
|
||
alors Cliquet est probablement le plus court chemin pour construire une
|
||
application qui tient la route.</p>
|
||
<p>Mais l'utilisation des resources CRUD est facultative, donc Cliquet reste pertinent
|
||
si les bonnes pratiques en terme de mise en production ou les abstractions fournies
|
||
vous paraissent valables !</p>
|
||
<p>Cliquet reste un moyen simple d'aller très vite pour mettre sur pied
|
||
une application Pyramid/Cornice.</p>
|
||
<blockquote>
|
||
Est-ce que les resources JSON supporte les modèles relationnels complexes ?</blockquote>
|
||
<p>La couche de persistence fournie est très simple, et devrait
|
||
répondre à la majorité des cas d'utilisation où les données n'ont pas de
|
||
relations.</p>
|
||
<p>En revanche, il est tout à fait possible de bénéficier de tous les aspects
|
||
du protocole en utilisant une classe <tt class="docutils literal">Collection</tt> maison, qui se chargerait
|
||
elle de manipuler les relations.</p>
|
||
<p>Le besoin de relations pourrait être un bon prétexte pour implémenter le
|
||
protocole avec Django REST Framework :)</p>
|
||
<blockquote>
|
||
Est-il possible de faire ci ou ça avec Cliquet ?</blockquote>
|
||
<p>Nous aimerions collecter des besoins pour écrire un ensemble de «recettes/tutoriels». Mais
|
||
pour ne pas travailler dans le vide, nous aimerions <a class="reference external" href="https://github.com/mozilla-services/cliquet/issues">connaitre vos idées</a> !
|
||
(<em>ex. brancher l'authentification Github, changer le format du logging JSON, stocker des
|
||
données cartographiques, ...</em>)</p>
|
||
<blockquote>
|
||
Est-ce que Cliquet peut manipuler des fichiers ?</blockquote>
|
||
<p><a class="reference external" href="https://github.com/mozilla-services/cliquet/issues/236">Nous l'envisageons</a>,
|
||
mais pour l'instant nous attendons que le besoin survienne en interne pour se
|
||
lancer.</p>
|
||
<p>Si c'est le cas, le protocole utilisé sera <a class="reference external" href="http://remotestorage.io/">Remote Storage</a>,
|
||
afin notamment de s'intégrer dans l'éco-système grandissant.</p>
|
||
<blockquote>
|
||
Est-ce que la fonctionnalité X va être implémentée ?</blockquote>
|
||
<p><em>Cliquet</em> est déjà bien garni. Plutôt qu'implémenter la fonctionnalité X,
|
||
il y a de grandes chances que nous agissions pour s'assurer que les abstractions
|
||
et les mécanismes d'extension fournis permettent de l'implémenter sous forme
|
||
d'extension.</p>
|
||
</div>
|
||
</div>
|
||
</content></entry><entry><title>Service de nuages : Perspectives pour l'été</title><link href="https://blog.notmyidea.org/service-de-nuages-perspectives-pour-lete-fr.html" rel="alternate"></link><published>2015-07-07T00:00:00+02:00</published><updated>2015-07-07T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2015-07-07:/service-de-nuages-perspectives-pour-lete-fr.html</id><summary type="html"><p class="first last">Le travail en cours et les fonctionnalités à venir pour les prochains mois.</p>
|
||
</summary><content type="html"><p><em>Cet article est repris depuis le blog « Service de Nuages » de mon équipe à Mozilla</em></p>
|
||
<p>Mozilla a pour coutume d'organiser régulièrement des semaines de travail où tous les employés
|
||
sont réunis physiquement. Pour cette dernière édition, nous avons pu retrouver
|
||
nos collègues du monde entier à <a class="reference external" href="http://www.openstreetmap.org/node/268148288#map=4/50.12/-122.95">Whistler, en Colombie Britannique au Canada</a> !</p>
|
||
<img alt="«All Hands» talk about Lego, by &#64;davidcrob - CC0" class="align-center" src="{filename}/images/whistler-talks.jpg" />
|
||
<p>Ce fût l'occasion pour notre équipe de se retrouver, et surtout de partager notre
|
||
vision et nos idées dans le domaine du stockage, afin de collecter des cas d'utilisation pour
|
||
notre solution <a class="reference external" href="https://kinto.readthedocs.org">Kinto</a>.</p>
|
||
<p>Dans cet article, nous passons en revue les pistes que nous avons pour
|
||
les prochains mois.</p>
|
||
<div class="section" id="ateliers-et-promotion">
|
||
<h2>Ateliers et promotion</h2>
|
||
<p>Nicolas a présenté <a class="reference external" href="https://github.com/mozilla-services/kinto.js">Kinto.js</a> dans un atelier dédié, avec comme support de
|
||
présentation le <a class="reference external" href="http://kintojs.readthedocs.org/en/latest/tutorial/">tutorial d'introduction</a>.</p>
|
||
<p>L'application résultante, pourtant toute simple, permet d'appréhender les
|
||
concepts de synchronisation de Kinto. Le tout sans installation prélable,
|
||
puisque Rémy a mis en place un <a class="reference external" href="https://kinto.dev.mozaws.net/v1/">serveur de dev effacé tous les jours</a>.</p>
|
||
<p>Nous avions mis un point d'honneur à faire du Vanilla.JS, déjà pour éviter les
|
||
combats de clochers autour des frameworks, mais aussi pour mettre en évidence qu'avec
|
||
HTML5 et ES6, on n'était plus aussi démunis qu'il y a quelques années.</p>
|
||
<p>Ce petit atelier nous a permis de nous rendre compte qu'on avait encore de
|
||
grosses lacunes en terme de documentation, surtout en ce qui concerne
|
||
l'éco-système et la vision globale des projets (Kinto, Kinto.js, Cliquet, ...).
|
||
Nous allons donc faire de notre mieux pour combler ce manque.</p>
|
||
<img alt="Kinto.js workshop - CC0" class="align-center" src="{filename}/images/whistler-workshop.jpg" />
|
||
</div>
|
||
<div class="section" id="mozilla-payments">
|
||
<h2>Mozilla Payments</h2>
|
||
<p>Comme <a class="reference external" href="http://www.servicedenuages.fr/la-gestion-des-permissions">décrit précédemment</a>, nous avons mis en place un système de permissions pour répondre aux besoins de suivi des paiements et abonnements.</p>
|
||
<p>Pour ce projet, Kinto sera utilisé depuis une application Django, via un client Python.</p>
|
||
<p>Maintenant que les développements ont été livrés, il faut transformer l'essai, réussir l'intégration, l'hébergement et la montée en puissance. La solution doit être livrée à la fin de l'année.</p>
|
||
<div class="section" id="a-venir">
|
||
<h3>À venir</h3>
|
||
<p>Nous aimerions en profiter pour implémenter une fonctionnalité qui nous tient à coeur : la construction de la liste des enregistrements accessibles en lecture sur une collection partagée.</p>
|
||
<img alt="Whistler Alta Lake - CC0" class="align-center" src="{filename}/images/whistler-lake.jpg" />
|
||
</div>
|
||
</div>
|
||
<div class="section" id="firefox-os-et-stockage">
|
||
<h2>Firefox OS et stockage</h2>
|
||
<p>Nous avons eu beaucoup d'échanges avec l'équipe de Firefox OS, avec qui nous avions
|
||
déjà eu l'occasion de collaborer, pour le <a class="reference external" href="https://github.com/mozilla-services/msisdn-gateway">serveur d'identification BrowserID par SMS</a> et pour <a class="reference external" href="https://github.com/mozilla-services/loop-server">Firefox Hello</a>.</p>
|
||
<div class="section" id="in-app-sync">
|
||
<h3>In-App sync</h3>
|
||
<p>Kinto, la solution simple promue pour la synchronisation de données dans les applications
|
||
Firefox OS ? La classe ! C'est ce qu'on avait en tête depuis longtemps, déjà à
|
||
l'époque avec <a class="reference external" href="http://daybed.readthedocs.org/">Daybed</a>. Voici donc une belle opportunité à saisir !</p>
|
||
<p>Il va falloir expliciter les limitations et hypothèses simplificatrices de notre
|
||
solution, surtout en termes de gestion de la concurrence. Nous sommes persuadés
|
||
que ça colle avec la plupart des besoins, mais il ne faudrait pas décevoir :)</p>
|
||
<p>Le fait que <a class="reference external" href="https://github.com/daleharvey">Dale</a>, un des auteurs de <a class="reference external" href="http://pouchdb.com/">PouchDB</a> et <a class="reference external" href="https://github.com/michielbdejong">Michiel de Jong</a>, un des auteurs de <a class="reference external" href="http://remotestorage.io/">Remote Storage</a>, nous aient encouragés sur nos premiers pas nous a bien motivé !</p>
|
||
</div>
|
||
<div class="section" id="cut-the-rope">
|
||
<h3>Cut the Rope</h3>
|
||
<p>Kinto devrait être mis à profit pour synchroniser les paramètres et les scores
|
||
du <a class="reference external" href="http://mozilla.cuttherope.net/">jeu</a>. Un premier exercice et une première vitrine sympas !</p>
|
||
</div>
|
||
<div class="section" id="syncto">
|
||
<h3>« SyncTo »</h3>
|
||
<p><a class="reference external" href="https://docs.services.mozilla.com/storage/apis-1.5.html">Firefox Sync</a> est la solution qui permet de synchroniser les données de Firefox (favoris, extensions, historique, complétion des formulaires, mots de passe, ...) entre plusieurs périphériques, de manière chiffrée.</p>
|
||
<p>L'implémentation du client en JavaScript est relativement complexe et date un peu maintenant.
|
||
Le code existant n'est pas vraiment portable dans <em>Firefox OS</em> et les tentatives de réécriture
|
||
n'ont pas abouti.</p>
|
||
<p>Nous souhaitons implémenter un pont entre <em>Kinto</em> et <em>Firefox Sync</em>, de manière
|
||
à pouvoir utiliser le client <em>Kinto.js</em>, plus simple et plus moderne, pour récupérer
|
||
les contenus et les stocker dans IndexedDB. Le delta à implémenter côté serveur est faible car nous nous étions
|
||
inspirés du protocole déjà éprouvé de Sync. Côté client, il s'agira surtout de
|
||
câbler l'authentification BrowserId et la Crypto.</p>
|
||
<p>Alexis a sauté sur l'occasion pour commencer l'écriture d'<a class="reference external" href="https://github.com/mozilla-services/syncclient">un client python pour Firefox Sync</a>, qui servira de brique de base pour l'écriture du service.</p>
|
||
</div>
|
||
<div class="section" id="cloud-storage">
|
||
<h3>Cloud Storage</h3>
|
||
<p>Eden Chuang et Sean Lee ont présenté les avancées sur l'intégration de services de stockages
|
||
distants (<em>DropBox, Baidu Yun</em>) dans <em>Firefox OS</em>. Actuellement, leur preuve de
|
||
concept repose sur <a class="reference external" href="https://fr.wikipedia.org/wiki/Filesystem_in_Userspace">FUSE</a>.</p>
|
||
<p>Nous avons évidemment en tête d'introduire la notion de fichiers attachés dans
|
||
<em>Kinto</em>, en implémentant la specification
|
||
<a class="reference external" href="https://tools.ietf.org/html/draft-dejong-remotestorage-05">*Remote Storage*</a>,
|
||
mais pour l'instant les cas d'utilisations ne se sont pas encore présentés officiellement.</p>
|
||
</div>
|
||
<div class="section" id="id2">
|
||
<h3>À venir</h3>
|
||
<p>Nous serons probablement amenés à introduire la gestion de la concurrence dans
|
||
le client JS, en complément de ce qui a été fait sur le serveur, pour permettre
|
||
les écritures simultanées et synchronisation en tâche de fond.</p>
|
||
<p>Nous sommes par ailleurs perpétuellement preneurs de vos retours — et bien
|
||
entendu de vos contributions — tant sur le code <a class="reference external" href="https://github.com/mozilla-services/kinto/">serveur</a>
|
||
que <a class="reference external" href="https://github.com/mozilla-services/kinto.js/">client</a> !</p>
|
||
<img alt="Firefox OS Cloud Storage Presentation - CC0" class="align-center" src="{filename}/images/whistler-cloud-storage.jpg" />
|
||
</div>
|
||
</div>
|
||
<div class="section" id="contenus-applicatifs-de-firefox">
|
||
<h2>Contenus applicatifs de Firefox</h2>
|
||
<p>Aujourd'hui Firefox a un cycle de release de six semaines. Un des objectifs
|
||
consiste à désolidariser certains contenus applicatifs de ces cycles
|
||
relativement longs (ex. <em>règles de securité, dictionnaires, traductions, ...</em>) <a class="footnote-reference" href="#id4" id="id3">[1]</a>.</p>
|
||
<p>Il s'agit de données JSON et binaire qui doivent être versionnées et synchronisées par
|
||
les navigateurs (<em>lecture seule</em>).</p>
|
||
<p>Il y a plusieurs outils officiels qui existent pour gérer ça (<em>Balrog</em>, <em>Shavar</em>, ...),
|
||
et pour l'instant, aucun choix n'a été fait. Mais lors des conversations avec
|
||
l'équipe en charge du projet, ce fût vraiment motivant de voir que même pour
|
||
ce genre de besoins internes, <em>Kinto</em> est tout aussi pertinent !</p>
|
||
<table class="docutils footnote" frame="void" id="id4" rules="none">
|
||
<colgroup><col class="label" /><col /></colgroup>
|
||
<tbody valign="top">
|
||
<tr><td class="label"><a class="fn-backref" href="#id3">[1]</a></td><td>La bonne nouvelle c'est que toutes les fonctionnalités <em>third-party</em> qui ont
|
||
été intégrées récemment vont redevenir des <em>add-ons</em> \o/.</td></tr>
|
||
</tbody>
|
||
</table>
|
||
<img alt="Landscape - CC0" class="align-center" src="{filename}/images/whistler-landscape.jpg" />
|
||
</div>
|
||
<div class="section" id="awesome-bar">
|
||
<h2>Awesome bar</h2>
|
||
<p>L'équipe <em>Firefox Labs</em>, le laboratoire qui élève des pandas roux en éprouvette,
|
||
serait vraiment intéressé par notre solution, notamment pour abreuver en données
|
||
un prototype pour améliorer <em>Awesome bar</em>, qui fusionnerait URL, historique et recherche.</p>
|
||
<p>Nous ne pouvons pas en dire beaucoup plus pour l'instant, mais les fonctionnalités
|
||
de collections d'enregistrements partagées entre utilisateurs de <em>Kinto</em>
|
||
correspondent parfaitement à ce qui est envisagé pour le futur du navigateur :)</p>
|
||
<div class="section" id="id5">
|
||
<h3>À venir</h3>
|
||
<p>Nous serons donc probablement amenés, avant de la fin de l'année, à introduire des
|
||
fonctionnalités d'indexation et de recherche <em>full-text</em> (comprendre <em>ElasticSearch</em>).
|
||
Cela rejoint nos plans précédents, puisque c'est quelque chose que nous avions dans
|
||
<em>Daybed</em>, et qui figurait sur notre feuille de route !</p>
|
||
<img alt="Firefox Labs Meeting - CC0" class="align-center" src="{filename}/images/whistler-labs.jpg" />
|
||
</div>
|
||
</div>
|
||
<div class="section" id="browser-html">
|
||
<h2>Browser.html</h2>
|
||
<p>L'équipe <em>Recherche</em> explore les notions de plateforme, et travaille notamment
|
||
sur l'implémentation d'un navigateur en JS/HTML avec <em>React</em>:
|
||
<a class="reference external" href="https://github.com/mozilla/browser.html">browser.html</a></p>
|
||
<p><em>Kinto</em> correspond parfaitement aux attentes
|
||
de l'équipe pour synchroniser les données associées à un utilisateur.</p>
|
||
<p>Il pourrait s'agir de données de navigation (comme Sync), mais aussi de collections
|
||
d'enregistrements diverses, comme par exemple les préférences du navigateur
|
||
ou un équivalent à <em>Alexa.com Top 500</em> pour fournir la complétion d'URL sans
|
||
interroger le moteur de recherche.</p>
|
||
<p>L'exercice pourrait être poussé jusqu'à la synchronisation d'états <em>React</em>
|
||
entre périphériques (par exemple pour les onglets).</p>
|
||
<div class="section" id="id7">
|
||
<h3>À venir</h3>
|
||
<p>Si <em>browser.html</em> doit stocker des données de navigation, il faudra ajouter
|
||
des fonctionnalités de chiffrement sur le client JS. Ça tombe bien, c'est un
|
||
sujet passionant, et <a class="reference external" href="http://www.w3.org/TR/WebCryptoAPI/">il y a plusieurs standards</a> !</p>
|
||
<p>Pour éviter d'interroger le serveur à intervalle régulier afin de synchroniser les
|
||
changements, l'introduction des <a class="reference external" href="https://w3c.github.io/push-api/">*push notifications*</a> semble assez naturelle.
|
||
Il s'agirait alors de la dernière pierre qui manque à l'édifice pour obtenir
|
||
un «<em>Mobile/Web backend as a service</em>» complet.</p>
|
||
<img alt="Roadmap - CC0" class="align-center" src="{filename}/images/whistler-roadmap.jpg" />
|
||
</div>
|
||
</div>
|
||
<div class="section" id="conclusion">
|
||
<h2>Conclusion</h2>
|
||
<p>Nous sommes dans une situation idéale, puisque ce que nous avions imaginé
|
||
sur <a class="reference external" href="https://github.com/mozilla-services/kinto/wiki/Roadmap">notre feuille de route</a> correspond à ce qui nous est demandé par les
|
||
différentes équipes.</p>
|
||
<p>L'enjeu consiste maintenant à se coordonner avec tout le monde, ne pas décevoir,
|
||
tenir la charge, continuer à améliorer et à faire la promotion du produit, se concentrer
|
||
sur les prochaines étapes et embarquer quelques contributeurs à nos cotés pour
|
||
construire une solution libre, générique, simple et auto-hébergeable pour le stockage
|
||
de données sur le Web :)</p>
|
||
<img alt="Friday Night Party - CC0" class="align-center" src="{filename}/images/whistler-top-roof.jpg" />
|
||
</div>
|
||
</content></entry><entry><title>Service de nuages : Stocker et interroger les permissions avec Kinto</title><link href="https://blog.notmyidea.org/service-de-nuages-stocker-et-interroger-les-permissions-avec-kinto-fr.html" rel="alternate"></link><published>2015-05-26T00:00:00+02:00</published><updated>2015-05-26T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2015-05-26:/service-de-nuages-stocker-et-interroger-les-permissions-avec-kinto-fr.html</id><summary type="html"><p class="first last">Comment faire pour stocker et interroger la base de données au sujet des permissions avec Kinto ?</p>
|
||
</summary><content type="html"><p><em>Cet article est repris depuis le blog « Service de Nuages » de mon équipe à Mozilla</em></p>
|
||
<p><strong>tl;dr: On a maintenant un super système de permission mais comment faire pour stocker et interroger ces permissions de manière efficace ?</strong></p>
|
||
<div class="section" id="la-problematique">
|
||
<h2>La problématique</h2>
|
||
<p>Maintenant que nous avons défini un modèle de gestion des permissions
|
||
sur les objets qui nous satisfait, le problème est de stocker ces
|
||
permissions de manière efficace afin de pouvoir autoriser ou interdire
|
||
l'accès à un objet pour la personne qui fait la requête.</p>
|
||
<p>Chaque requête sur notre API va générer une ou plusieurs demandes
|
||
d'accès, il faut donc que la réponse soit très rapide sous peine
|
||
d'impacter la vélocité du service.</p>
|
||
</div>
|
||
<div class="section" id="obtenir-la-liste-des-principals-d-un-utilisateur">
|
||
<h2>Obtenir la liste des &quot;principals&quot; d'un utilisateur</h2>
|
||
<p>Les <em>principals</em> de l'utilisateur correspondent à son <tt class="docutils literal">user_id</tt>
|
||
ainsi qu'à la liste des identifiants des groupes dans lesquels il a
|
||
été ajouté.</p>
|
||
<p>Pour éviter de recalculer les <em>principals</em> de l'utilisateur à chaque
|
||
requête, le mieux reste de maintenir une liste des <em>principals</em> par
|
||
utilisateur.</p>
|
||
<p>Ainsi lorsqu'on ajoute un utilisateur à un groupe, il faut bien penser
|
||
à ajouter le groupe à la liste des <em>principals</em> de l'utilisateur.</p>
|
||
<p>Ça se complexifie lorsqu'on ajoute un groupe à un groupe.</p>
|
||
<p>Dans un premier temps interdire l'ajout d'un groupe à un groupe est
|
||
une limitation qu'on est prêts à accepter pour simplifier le
|
||
modèle.</p>
|
||
<p>L'avantage de maintenir la liste des <em>principals</em> d'un utilisateur
|
||
lors de la modification de cette liste c'est qu'elle est déjà
|
||
construite lors des lectures, qui sont dans notre cas plus fréquentes
|
||
que les écritures.</p>
|
||
<p>Cela nécessite de donner un identifiant unique aux groupes pour tous
|
||
les <em>buckets</em>.</p>
|
||
<p>Nous proposons de de les nommer avec leur URI:
|
||
<tt class="docutils literal">/buckets/blog/groups/moderators</tt></p>
|
||
</div>
|
||
<div class="section" id="obtenir-la-liste-des-principals-d-un-ace">
|
||
<h2>Obtenir la liste des &quot;principals&quot; d'un ACE</h2>
|
||
<blockquote>
|
||
Rappel, un &quot;ACE&quot; est un <em>Access Control Entry</em>, un des éléments
|
||
d'une ACL (e.g. <em>modifier un enregistrement</em>).</blockquote>
|
||
<p>Avec le <a class="reference external" href="{filename}/2015.05.cliquet-permissions.rst">système de permissions choisi</a>, les permissions d'un
|
||
objet héritent de celle de l'objet parent.</p>
|
||
<p>Par exemple, avoir le droit d'écriture sur un <em>bucket</em> permet la
|
||
création des permissions et la modification de tous ses records.</p>
|
||
<p>Ce qui veut dire que pour obtenir la liste complète des <em>principals</em>
|
||
ayant une permission sur un objet, il faut regarder à plusieurs
|
||
endroits.</p>
|
||
<p>Rémy a <a class="reference external" href="https://gist.github.com/Natim/77c8f61c1d42e476cef8#file-permission-py-L9-L52">décrit dans un gist la liste d'héritage de chaque permission</a>.</p>
|
||
<p>Prenons l'exemple de l'ajout d'un record dans une collection.</p>
|
||
<p>Le droit <tt class="docutils literal">records:create</tt> est obtenu si l'on a l'un des droits suivants:</p>
|
||
<ul class="simple">
|
||
<li><tt class="docutils literal">bucket:write</tt></li>
|
||
<li><tt class="docutils literal">collection:write</tt></li>
|
||
<li><tt class="docutils literal">records:create</tt></li>
|
||
</ul>
|
||
<p>Notre première idée était de stocker les permissions sur chaque objet
|
||
et de maintenir la liste exhaustive des permissions lors d'une
|
||
modification d'ACL. Cependant cela nécessitait de construire cette
|
||
liste lors de l'ajout d'un objet et de mettre à jour tout l'arbre lors
|
||
de sa suppression. (<em>Je vous laisse imaginer le nombre d'opérations
|
||
nécessaires pour ajouter un administrateur sur un *bucket</em> contenant
|
||
1000 collections avec 100000 records chacune.*)</p>
|
||
<p>La solution que nous avons désormais adoptée consiste à stocker les
|
||
<em>principals</em> de chaque <em>ACE</em> (<em>qui</em> a le droit de faire telle action
|
||
sur l'objet), et de faire l'union des <em>ACE</em> hérités, afin de les
|
||
croiser avec les <em>principals</em> de l'utilisateur :</p>
|
||
<blockquote>
|
||
(ACE(object, permission) ∪ inherited_ACE) ∩ PRINCIPALS(user)</blockquote>
|
||
<p>Par exemple l'ACE: <tt class="docutils literal">/buckets/blog/collections/article:records:create</tt> hérite de
|
||
l'ACE <tt class="docutils literal">/buckets/blog/collections/article:write</tt> et de <tt class="docutils literal">/buckets/blog:write</tt> :</p>
|
||
<blockquote>
|
||
(ACE(/buckets/blog/collections/article:records:create) ∪ ACE(/buckets/blog/collections/article:write) ∪ ACE(/buckets/blog:write)) ∩ PRINCIPALS('fxa:alexis')</blockquote>
|
||
</div>
|
||
<div class="section" id="recuperer-les-donnees-de-l-utilisateur">
|
||
<h2>Récupérer les données de l'utilisateur</h2>
|
||
<p>La situation se corse lorsqu'on souhaite limiter la liste des
|
||
<em>records</em> d'une collection à ceux accessibles pour l'utilisateur, car
|
||
on doit faire cette intersection pour tous les <em>records</em>.</p>
|
||
<p>Une première solution est de regarder si l'utilisateur est mentionné
|
||
dans les <em>ACL*s du *bucket</em> ou de la <em>collection</em>:</p>
|
||
<p>Ensuite, si ce n'est pas le cas, alors on filtre les <em>records</em> pour
|
||
lesquels les <em>principals</em> correspondent à ceux de l'utilisateur.</p>
|
||
<div class="highlight"><pre><span></span><span class="n">principals</span> <span class="o">=</span> <span class="n">get_user_principals</span><span class="p">(</span><span class="n">user_id</span><span class="p">)</span>
|
||
<span class="n">can_read_all</span> <span class="o">=</span> <span class="n">has_read_perms</span><span class="p">(</span><span class="n">bucket_id</span><span class="p">,</span> <span class="n">collection_id</span><span class="p">,</span>
|
||
<span class="n">principals</span><span class="p">)</span>
|
||
<span class="k">if</span> <span class="n">can_read_all</span><span class="p">:</span>
|
||
<span class="n">records</span> <span class="o">=</span> <span class="n">get_all_records</span><span class="p">(</span><span class="n">bucket_id</span><span class="p">,</span> <span class="n">collection_id</span><span class="p">,</span>
|
||
<span class="n">filters</span><span class="o">=</span><span class="p">[</span><span class="o">...</span><span class="p">])</span>
|
||
<span class="k">else</span><span class="p">:</span>
|
||
<span class="n">records</span> <span class="o">=</span> <span class="n">filter_read_records</span><span class="p">(</span><span class="n">bucket_id</span><span class="p">,</span> <span class="n">collection_id</span><span class="p">,</span>
|
||
<span class="n">principals</span><span class="o">=</span><span class="n">principals</span><span class="p">,</span>
|
||
<span class="n">filters</span><span class="o">=</span><span class="p">[</span><span class="o">...</span><span class="p">])</span>
|
||
</pre></div>
|
||
<p>Il faudra faire quelque chose de similaire pour la suppression
|
||
multiple, lorsqu'un utilisateur souhaitera supprimer des
|
||
enregistrements sur lesquels il a les droits de lecture mais pas
|
||
d'écriture.</p>
|
||
</div>
|
||
<div class="section" id="le-modele-de-donnees">
|
||
<h2>Le modèle de données</h2>
|
||
<p>Pour avoir une idée des requêtes dans un backend SQL, voyons un peu ce
|
||
que donnerait le modèle de données.</p>
|
||
<div class="section" id="le-format-des-id">
|
||
<h3>Le format des ID</h3>
|
||
<p>Utiliser des URI comme identifiant des objets présente de nombreux
|
||
avantages (lisibilité, unicité, cohérence avec les URLs)</p>
|
||
<ul class="simple">
|
||
<li>bucket: <tt class="docutils literal">/buckets/blog</tt></li>
|
||
<li>groupe: <tt class="docutils literal">/buckets/blog/group/moderators</tt></li>
|
||
<li>collection: <tt class="docutils literal">/buckets/blog/collections/articles</tt></li>
|
||
<li>record: <tt class="docutils literal"><span class="pre">/buckets/blog/collections/articles/records/02f3f76f-7059-4ae4-888f-2ac9824e9200</span></tt></li>
|
||
</ul>
|
||
</div>
|
||
<div class="section" id="les-tables">
|
||
<h3>Les tables</h3>
|
||
<p>Pour le stockage des principals et des permissions:</p>
|
||
<div class="highlight"><pre><span></span><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="k">user</span><span class="p">(</span><span class="n">id</span> <span class="nb">TEXT</span><span class="p">,</span> <span class="n">principals</span> <span class="nb">TEXT</span><span class="p">[]);</span>
|
||
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">perms</span><span class="p">(</span><span class="n">ace</span> <span class="nb">TEXT</span><span class="p">,</span> <span class="n">principals</span> <span class="nb">TEXT</span><span class="p">[]);</span>
|
||
</pre></div>
|
||
<p>La table <em>perms</em> va associer des <em>principals</em> à chaque <em>ACE</em>
|
||
(e.g.``/buckets/blog:write``).</p>
|
||
<p>Pour le stockage des données:</p>
|
||
<div class="highlight"><pre><span></span><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="k">object</span><span class="p">(</span><span class="n">id</span> <span class="nb">TEXT</span><span class="p">,</span> <span class="k">type</span> <span class="nb">TEXT</span><span class="p">,</span> <span class="n">parent_id</span> <span class="nb">TEXT</span><span class="p">,</span> <span class="k">data</span> <span class="n">JSONB</span><span class="p">,</span>
|
||
<span class="n">write_principals</span> <span class="nb">TEXT</span><span class="p">[],</span> <span class="n">read_principals</span> <span class="nb">TEXT</span><span class="p">[]);</span>
|
||
</pre></div>
|
||
<p>La colonne <em>parent_id</em> permet de savoir à qui appartient l'objet
|
||
(e.g. groupe d'un <em>bucket</em>, collection d'un <em>bucket</em>, <em>record</em> d'une
|
||
collection, ...).</p>
|
||
</div>
|
||
<div class="section" id="exemple-d-utilisateur">
|
||
<h3>Exemple d'utilisateur</h3>
|
||
<div class="highlight"><pre><span></span><span class="k">INSERT</span> <span class="k">INTO</span> <span class="k">user</span> <span class="p">(</span><span class="n">id</span><span class="p">,</span> <span class="n">principals</span><span class="p">)</span>
|
||
<span class="k">VALUES</span> <span class="p">(</span><span class="s1">&#39;fxa:alexis&#39;</span><span class="p">,</span> <span class="s1">&#39;{}&#39;</span><span class="p">);</span>
|
||
|
||
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="k">user</span> <span class="p">(</span><span class="n">id</span><span class="p">,</span> <span class="n">principals</span><span class="p">)</span>
|
||
<span class="k">VALUES</span> <span class="p">(</span><span class="s1">&#39;fxa:natim&#39;</span><span class="p">,</span>
|
||
<span class="s1">&#39;{&quot;/buckets/blog/groups/moderators&quot;}&#39;</span><span class="p">);</span>
|
||
</pre></div>
|
||
</div>
|
||
<div class="section" id="exemple-d-objets">
|
||
<h3>Exemple d'objets</h3>
|
||
<div class="section" id="bucket">
|
||
<h4>Bucket</h4>
|
||
<div class="highlight"><pre><span></span><span class="k">INSERT</span> <span class="k">INTO</span> <span class="k">object</span> <span class="p">(</span><span class="n">id</span><span class="p">,</span> <span class="k">type</span><span class="p">,</span> <span class="n">parent_id</span><span class="p">,</span> <span class="k">data</span><span class="p">,</span>
|
||
<span class="n">read_principals</span><span class="p">,</span> <span class="n">write_principals</span><span class="p">)</span>
|
||
<span class="k">VALUES</span> <span class="p">(</span>
|
||
<span class="s1">&#39;/buckets/blog&#39;</span><span class="p">,</span>
|
||
<span class="s1">&#39;bucket&#39;</span><span class="p">,</span>
|
||
<span class="k">NULL</span><span class="p">,</span>
|
||
<span class="s1">&#39;{&quot;name&quot;: &quot;blog&quot;}&#39;</span><span class="p">::</span><span class="n">JSONB</span><span class="p">,</span>
|
||
<span class="s1">&#39;{}&#39;</span><span class="p">,</span> <span class="s1">&#39;{&quot;fxa:alexis&quot;}&#39;</span><span class="p">);</span>
|
||
</pre></div>
|
||
</div>
|
||
<div class="section" id="group">
|
||
<h4>Group</h4>
|
||
<div class="highlight"><pre><span></span><span class="k">INSERT</span> <span class="k">INTO</span> <span class="k">object</span> <span class="p">(</span><span class="n">id</span><span class="p">,</span> <span class="k">type</span><span class="p">,</span> <span class="n">parent_id</span><span class="p">,</span> <span class="k">data</span><span class="p">,</span>
|
||
<span class="n">read_principals</span><span class="p">,</span> <span class="n">write_principals</span><span class="p">)</span>
|
||
<span class="k">VALUES</span> <span class="p">(</span>
|
||
<span class="s1">&#39;/buckets/blog/groups/moderators&#39;</span><span class="p">,</span>
|
||
<span class="s1">&#39;group&#39;</span><span class="p">,</span>
|
||
<span class="s1">&#39;/buckets/blog&#39;</span><span class="p">,</span>
|
||
<span class="s1">&#39;{&quot;name&quot;: &quot;moderators&quot;, &quot;members&quot;: [&#39;</span><span class="n">fxa</span><span class="p">:</span><span class="n">natim</span><span class="s1">&#39;]}&#39;</span><span class="p">::</span><span class="n">JSONB</span><span class="p">,</span>
|
||
<span class="s1">&#39;{}&#39;</span><span class="p">,</span> <span class="s1">&#39;{}&#39;</span><span class="p">);</span>
|
||
</pre></div>
|
||
<p>Ce groupe peut être gére par <tt class="docutils literal">fxa:alexis</tt> puisqu'il a la permission
|
||
<tt class="docutils literal">write</tt> dans le <em>bucket</em> parent.</p>
|
||
</div>
|
||
<div class="section" id="collection">
|
||
<h4>Collection</h4>
|
||
<div class="highlight"><pre><span></span><span class="k">INSERT</span> <span class="k">INTO</span> <span class="k">object</span> <span class="p">(</span><span class="n">id</span><span class="p">,</span> <span class="k">type</span><span class="p">,</span> <span class="n">parent_id</span><span class="p">,</span> <span class="k">data</span><span class="p">,</span>
|
||
<span class="n">read_principals</span><span class="p">,</span> <span class="n">write_principals</span><span class="p">)</span>
|
||
<span class="k">VALUES</span> <span class="p">(</span>
|
||
<span class="s1">&#39;/buckets/blog/collections/articles&#39;</span><span class="p">,</span>
|
||
<span class="s1">&#39;collection&#39;</span><span class="p">,</span>
|
||
<span class="s1">&#39;/buckets/blog&#39;</span><span class="p">,</span>
|
||
<span class="s1">&#39;{&quot;name&quot;: &quot;article&quot;}&#39;</span><span class="p">::</span><span class="n">JSONB</span><span class="p">,</span>
|
||
<span class="s1">&#39;{&quot;system.Everyone&quot;}&#39;</span><span class="p">,</span>
|
||
<span class="s1">&#39;{&quot;/buckets/blog/groups/moderators&quot;}&#39;</span><span class="p">);</span>
|
||
</pre></div>
|
||
<p>Cette collection d'articles peut être lue par tout le monde,
|
||
et gérée par les membres du groupe <tt class="docutils literal">moderators</tt>, ainsi que
|
||
<tt class="docutils literal">fxa:alexis</tt>, via le <em>bucket</em>.</p>
|
||
</div>
|
||
<div class="section" id="records">
|
||
<h4>Records</h4>
|
||
<div class="highlight"><pre><span></span><span class="k">INSERT</span> <span class="k">INTO</span> <span class="k">object</span> <span class="p">(</span><span class="n">id</span><span class="p">,</span> <span class="k">type</span><span class="p">,</span> <span class="n">parent_id</span><span class="p">,</span> <span class="k">data</span><span class="p">,</span>
|
||
<span class="n">read_principals</span><span class="p">,</span> <span class="n">write_principals</span><span class="p">)</span>
|
||
<span class="k">VALUES</span> <span class="p">(</span>
|
||
<span class="s1">&#39;/buckets/blog/collections/articles/records/02f3f76f-7059-4ae4-888f-2ac9824e9200&#39;</span><span class="p">,</span>
|
||
<span class="s1">&#39;record&#39;</span><span class="p">,</span>
|
||
<span class="s1">&#39;/buckets/blog/collections/articles&#39;</span><span class="p">,</span>
|
||
<span class="s1">&#39;{&quot;name&quot;: &quot;02f3f76f-7059-4ae4-888f-2ac9824e9200&quot;,</span>
|
||
<span class="s1"> &quot;title&quot;: &quot;Stocker les permissions&quot;, ...}&#39;</span><span class="p">::</span><span class="n">JSONB</span><span class="p">,</span>
|
||
<span class="s1">&#39;{}&#39;</span><span class="p">,</span> <span class="s1">&#39;{}&#39;</span><span class="p">);</span>
|
||
</pre></div>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="interroger-les-permissions">
|
||
<h3>Interroger les permissions</h3>
|
||
<div class="section" id="id1">
|
||
<h4>Obtenir la liste des &quot;principals&quot; d'un ACE</h4>
|
||
<p>Comme vu plus haut, pour vérifier une permission, on fait l'union des
|
||
<em>principals</em> requis par les objets hérités, et on teste leur
|
||
intersection avec ceux de l'utilisateur:</p>
|
||
<div class="highlight"><pre><span></span><span class="k">WITH</span> <span class="n">required_principals</span> <span class="k">AS</span> <span class="p">(</span>
|
||
<span class="k">SELECT</span> <span class="k">unnest</span><span class="p">(</span><span class="n">principals</span><span class="p">)</span> <span class="k">AS</span> <span class="n">p</span>
|
||
<span class="k">FROM</span> <span class="n">perms</span>
|
||
<span class="k">WHERE</span> <span class="n">ace</span> <span class="k">IN</span> <span class="p">(</span>
|
||
<span class="s1">&#39;/buckets/blog:write&#39;</span><span class="p">,</span>
|
||
<span class="s1">&#39;/buckets/blog:read&#39;</span><span class="p">,</span>
|
||
<span class="s1">&#39;/buckets/blog/collections/article:write&#39;</span><span class="p">,</span>
|
||
<span class="s1">&#39;/buckets/blog/collections/article:read&#39;</span><span class="p">)</span>
|
||
<span class="p">),</span>
|
||
<span class="n">user_principals</span> <span class="k">AS</span> <span class="p">(</span>
|
||
<span class="k">SELECT</span> <span class="k">unnest</span><span class="p">(</span><span class="n">principals</span><span class="p">)</span>
|
||
<span class="k">FROM</span> <span class="k">user</span>
|
||
<span class="k">WHERE</span> <span class="n">id</span> <span class="o">=</span> <span class="s1">&#39;fxa:natim&#39;</span>
|
||
<span class="p">)</span>
|
||
<span class="k">SELECT</span> <span class="k">COUNT</span><span class="p">(</span><span class="o">*</span><span class="p">)</span>
|
||
<span class="k">FROM</span> <span class="n">user_principals</span> <span class="n">a</span>
|
||
<span class="k">INNER</span> <span class="k">JOIN</span> <span class="n">required_principals</span> <span class="n">b</span>
|
||
<span class="k">ON</span> <span class="n">a</span><span class="p">.</span><span class="n">p</span> <span class="o">=</span> <span class="n">b</span><span class="p">.</span><span class="n">p</span><span class="p">;</span>
|
||
</pre></div>
|
||
</div>
|
||
<div class="section" id="filtrer-les-objets-en-fonction-des-permissions">
|
||
<h4>Filtrer les objets en fonction des permissions</h4>
|
||
<p>Pour filtrer les objets, on fait une simple intersection de liste
|
||
(<em>merci PostgreSQL</em>):</p>
|
||
<div class="highlight"><pre><span></span><span class="k">SELECT</span> <span class="k">data</span>
|
||
<span class="k">FROM</span> <span class="k">object</span> <span class="n">o</span><span class="p">,</span> <span class="k">user</span> <span class="n">u</span>
|
||
<span class="k">WHERE</span> <span class="n">o</span><span class="p">.</span><span class="k">type</span> <span class="o">=</span> <span class="s1">&#39;record&#39;</span>
|
||
<span class="k">AND</span> <span class="n">o</span><span class="p">.</span><span class="n">parent_id</span> <span class="o">=</span> <span class="s1">&#39;/buckets/blog/collections/article&#39;</span>
|
||
<span class="k">AND</span> <span class="p">(</span><span class="n">o</span><span class="p">.</span><span class="n">read_principals</span> <span class="o">&amp;&amp;</span> <span class="n">u</span><span class="p">.</span><span class="n">principals</span> <span class="k">OR</span>
|
||
<span class="n">o</span><span class="p">.</span><span class="n">write_principals</span> <span class="o">&amp;&amp;</span> <span class="n">u</span><span class="p">.</span><span class="n">principals</span><span class="p">)</span>
|
||
<span class="k">AND</span> <span class="n">u</span><span class="p">.</span><span class="n">id</span> <span class="o">=</span> <span class="s1">&#39;fxa:natim&#39;</span><span class="p">;</span>
|
||
</pre></div>
|
||
<p>Les listes s'indexent bien, notamment grâce aux <a class="reference external" href="http://www.postgresql.org/docs/current/static/indexes-types.html">index GIN</a>.</p>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="avec-redis">
|
||
<h3>Avec Redis</h3>
|
||
<p><em>Redis</em> présente plusieurs avantages pour ce genre de
|
||
problématiques. Notamment, il gère les <em>set</em> nativement (listes de
|
||
valeurs uniques), ainsi que les opérations d'intersection et d'union.</p>
|
||
<p>Avec <em>Redis</em> on peut écrire l'obtention des <em>principals</em> pour un <em>ACE</em>
|
||
comme cela :</p>
|
||
<div class="highlight"><pre><span></span>SUNIONSTORE temp_perm:/buckets/blog/collections/articles:write permission:/buckets/blog:write permission:/buckets/blog/collections/articles:write
|
||
SINTER temp_perm:/buckets/blog/collections/articles:write principals:fxa:alexis
|
||
</pre></div>
|
||
<ul class="simple">
|
||
<li><tt class="docutils literal">SUNIONSTORE</tt> permet de créer un set contenant les éléments de
|
||
l'union de tous les set suivants. Dans notre cas on le nomme
|
||
<tt class="docutils literal"><span class="pre">temp_perm:/buckets/blog/collections/articles:write</span></tt> et il contient
|
||
l'union des sets d'ACLs suivants:
|
||
- <tt class="docutils literal"><span class="pre">permission:/buckets/blog:write</span></tt>
|
||
- <tt class="docutils literal"><span class="pre">permission:/buckets/blog/collections/articles:write</span></tt></li>
|
||
<li><tt class="docutils literal">SINTER</tt> retourne l'intersection de tous les sets passés en paramètres dans notre cas :
|
||
- <tt class="docutils literal"><span class="pre">temp_perm:/buckets/blog/collections/articles:write</span></tt>
|
||
- <tt class="docutils literal">principals:fxa:alexis</tt></li>
|
||
</ul>
|
||
<p>Plus d'informations sur :
|
||
- <a class="reference external" href="http://redis.io/commands/sinter">http://redis.io/commands/sinter</a>
|
||
- <a class="reference external" href="http://redis.io/commands/sunionstore">http://redis.io/commands/sunionstore</a></p>
|
||
<p>Si le set résultant de la commande <tt class="docutils literal">SINTER</tt> n'est pas vide, alors
|
||
l'utilisateur possède la permission.</p>
|
||
<p>On peut ensuite supprimer la clé temporaire <tt class="docutils literal">temp_perm</tt>.</p>
|
||
<p>En utilisant <tt class="docutils literal">MULTI</tt> on peut <a class="reference external" href="https://gist.github.com/Natim/77c8f61c1d42e476cef8#file-permission-py-L117-L124">même faire tout cela au sein d'une
|
||
transaction</a>
|
||
et garantir ainsi l'intégrité de la requête.</p>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="conclusion">
|
||
<h2>Conclusion</h2>
|
||
<p>La solution a l'air simple mais nous a demandé beaucoup de réflexion
|
||
en passant par plusieurs propositions.</p>
|
||
<p>L'idée finale est d'avoir :</p>
|
||
<ul class="simple">
|
||
<li>Un backend spécifique permettant de stocker les <em>principals</em> des
|
||
utilisateurs et des <em>ACE</em> (e.g. avec les sets Redis) ;</li>
|
||
<li>La liste des principals read et write sur la table des objets.</li>
|
||
</ul>
|
||
<p>C'est dommage d'avoir le concept de permissions à deux endroits, mais
|
||
cela permet de connaître rapidement la permission d'un utilisateur sur
|
||
un objet et également de pouvoir récupérer tous les objets d'une
|
||
collection pour un utilisateur si celui-ci n'a pas accès à tous les
|
||
records de la collection, ou toutes les collections du bucket.</p>
|
||
</div>
|
||
</content></entry><entry><title>Les problèmes de PGP</title><link href="https://blog.notmyidea.org/les-problemes-de-pgp.html" rel="alternate"></link><published>2015-05-25T00:00:00+02:00</published><updated>2015-05-25T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2015-05-25:/les-problemes-de-pgp.html</id><summary type="html"><blockquote>
|
||
<p>Flip a bit in the communication between sender and recipient and they
|
||
will experience decryption or verification errors. How high are the
|
||
chances they will start to exchange the data in the clear rather than
|
||
trying to hunt down the man in the middle?</p>
|
||
<p>-- <a href="http://secushare.org/PGP">http://secushare.org/PGP</a></p>
|
||
</blockquote>
|
||
<p>Une fois …</p></summary><content type="html"><blockquote>
|
||
<p>Flip a bit in the communication between sender and recipient and they
|
||
will experience decryption or verification errors. How high are the
|
||
chances they will start to exchange the data in the clear rather than
|
||
trying to hunt down the man in the middle?</p>
|
||
<p>-- <a href="http://secushare.org/PGP">http://secushare.org/PGP</a></p>
|
||
</blockquote>
|
||
<p>Une fois passé l'euphorie du "il faut utiliser PGP pour l'ensemble de
|
||
nos communications", j'ai réalisé lors de discussions que PGP avait
|
||
plusieurs problèmes, parmi ceux-ci:</p>
|
||
<ul>
|
||
<li>Les <em>meta données</em> (y compris le champ "sujet" de la conversation)
|
||
sont quand même échangées en clair (il est possible de savoir qu'un
|
||
message à été échangé entre telle et telle personne, a telle date);</li>
|
||
<li>PGP se base sur un protocole de communication qui est lui non
|
||
chiffré, et il est donc facile de soit se tromper, soit dégrader le
|
||
mode de conversation vers une méthode non chiffrée;</li>
|
||
<li>Il est facile de connaître votre réseau social avec PGP, puisque
|
||
tout le principe est de signer les clés des personnes dont vous
|
||
validez l'identité;</li>
|
||
<li>En cas de fuite de votre clé privée, tous les messages que vous avez
|
||
chiffrés avec elle sont compromis. On dit que PGP ne fournit pas de
|
||
<em>forward secrecy</em>;</li>
|
||
<li>La découverte de la clé de pairs se passe souvent <em>en clair</em>, sans
|
||
utiliser une connexion "sécurisée" (HTTPS). Tout le monde peut donc
|
||
voir ces échanges et savoir de qui vous cherchez la clé;</li>
|
||
<li>Les discussions de groupes sont très difficiles: il faut chiffrer
|
||
pour chacun des destinataires (ou que ceux-ci partagent une paire de
|
||
clés).</li>
|
||
</ul>
|
||
<p>Je suis en train de creuser à propos les alternatives à PGP, par exemple
|
||
<a href="https://pond.imperialviolet.org/">Pond</a>, qui lui ne construit pas par
|
||
dessus un standard déjà établi, et donc n'hérite pas de ses défauts
|
||
(mais pas non plus de son réseau déjà établi).</p>
|
||
<p>En attendant, quelques bonnes pratiques sur PGP ;)</p>
|
||
<h2 id="bonnes-pratiques">Bonnes pratiques</h2>
|
||
<p>Il est en fait assez facile d'utiliser PGP de travers. Riseup à fait <a href="https://help.riseup.net/en/security/message-security/openpgp/best-practices">un
|
||
excellent
|
||
guide</a>
|
||
qui explique comment configurer son installation correctement.</p>
|
||
<ul>
|
||
<li>J'en ai déjà parlé, mais il faut absolument choisir des phrases de
|
||
passes suffisamment longues. Pas facile de les retenir, mais
|
||
indispensable. Vous pouvez aussi avoir un document chiffré avec une
|
||
clé que vous ne mettez jamais en ligne, qui contiens ces phrases de
|
||
passe, au cas ou vous les oubliez.</li>
|
||
<li>Générez des clés RSA de 4096 bits, en utilisant sha512;</li>
|
||
<li>Il faut utiliser une date d'expiration de nos clés suffisamment
|
||
proche (2 ans). Il est possible de repousser cette date si
|
||
nécessaire, par la suite.</li>
|
||
</ul>
|
||
<p>Parmi les choses les plus frappantes que j'ai rencontrées:</p>
|
||
<ul>
|
||
<li>Utiliser le <em>flag</em> –hidden-recipient avec PGP pour ne pas dévoiler
|
||
qui est le destinataire du message;</li>
|
||
<li>Ne pas envoyer les messages de brouillons sur votre serveur, ils le
|
||
seraient en clair !;</li>
|
||
<li>Utilisez HPKS pour communiquer avec les serveurs de clés, sinon tout
|
||
le trafic est en clair.</li>
|
||
</ul>
|
||
<p>Le <a href="https://bitmask.net/">projet Bitmask</a> vise lui à rendre les outils
|
||
de chiffrement d'échanges de messages et de VPN simples à utiliser,
|
||
encore quelque chose à regarder.</p>
|
||
<p>Enfin bref, y'a du taf.</p></content></entry><entry><title>Simplifier les preuves d'identités</title><link href="https://blog.notmyidea.org/simplifier-les-preuves-didentites.html" rel="alternate"></link><published>2015-05-11T00:00:00+02:00</published><updated>2015-05-11T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2015-05-11:/simplifier-les-preuves-didentites.html</id><summary type="html">
|
||
<ul>
|
||
<li>headline<br>
|
||
Qu'est-ce que Keybase.io et comment essayent-ils de simplifier la
|
||
création de preuves d'identité.</li>
|
||
</ul>
|
||
<p>L'un des problèmes non réellement résolu actuellement quant au
|
||
chiffrement des échanges est lié à l'authenticité des clés. Si quelqu'un
|
||
décide de publier une clé en mon nom, et en utilisant mon adresse email,
|
||
cela …</p></summary><content type="html">
|
||
<ul>
|
||
<li>headline<br>
|
||
Qu'est-ce que Keybase.io et comment essayent-ils de simplifier la
|
||
création de preuves d'identité.</li>
|
||
</ul>
|
||
<p>L'un des problèmes non réellement résolu actuellement quant au
|
||
chiffrement des échanges est lié à l'authenticité des clés. Si quelqu'un
|
||
décide de publier une clé en mon nom, et en utilisant mon adresse email,
|
||
cela lui est assez facile.</p>
|
||
<p>Il est donc nécessaire d'avoir des moyens de prouver que la clé publique
|
||
que j'utilise est réellement la mienne.</p>
|
||
<p>Traditionnellement, il est nécessaire de faire signer ma clé publique
|
||
par d'autres personnes, via une rencontre en personne ou des échanges
|
||
hors du réseau. C'est par exemple ce qui est réalisé lors des <a href="https://fr.wikipedia.org/wiki/Key_signing_party">Key
|
||
Signing parties</a>.</p>
|
||
<p>Une manière simple d'effectuer ces vérifications serait, en plus de
|
||
donner son adresse email, sa signature de clé, ou a minima de donner un
|
||
mot clé pour valider que les échanges proviennent bien de la bonne
|
||
personne.</p>
|
||
<p>PGP propose un mécanisme de signature des clés d'autrui, une fois celles
|
||
ci validées, ce qui permet de placer sa confiance dans les signataires
|
||
de la clé.</p>
|
||
<p><a href="https://keybase.io">Keybase.io</a> est un service qui vise à rendre la
|
||
création de ces preuves plus facile, en partant du principe qu'il est
|
||
possible d'utiliser différents moyens afin de prouver l'identité des
|
||
personnes. Par exemple, leurs comptes Twitter, GitHub ou leurs noms de
|
||
domaines. De la même manière qu'il est possible de signer (valider) les
|
||
clés de nos amis, il est possible de les "tracker" selon le jargon de
|
||
keybase.</p>
|
||
<p>Donc, en somme, <em>Keybase.io</em> est un annuaire, qui tente de rendre plus
|
||
facile la création de preuves. Bien.</p>
|
||
<h2 id="quelques-points-dombre">Quelques points d'ombre</h2>
|
||
<p>Il s'agit d'une <em>startup</em> américaine, domiciliée dans le Delaware, qui
|
||
se trouve être un des paradis fiscaux qui <a href="https://fr.wikipedia.org/wiki/Delaware">est connu pour être un
|
||
paradis fiscal au coeur même des
|
||
États-Unis</a>. Je ne veux pas
|
||
faire de raccourcis trop rapides, bien évidemment, alors <a href="https://github.com/keybase/keybase-issues/issues/1569">j'ai ouvert un
|
||
ticket sur GitHub pour en savoir
|
||
plus</a> (après
|
||
tout, le fait d'être un paradis fiscal permet peut-être d'échapper à
|
||
certaines lois sur la requêtes de données). D'autant plus étonnant, la
|
||
startup n'a pour l'instant <a href="https://github.com/keybase/keybase-issues/issues/788">pas de <em>business
|
||
model</em></a> (ce qui en
|
||
un sens est assez rassurant, même si on peut se poser la question de
|
||
pourquoi faire une startup dans ces cas là).</p>
|
||
<p>Le service (bien qu'en Alpha), n'est pas mis à disposition sous licence
|
||
libre, ce qui pour l'instant empêche quiconque de créer son propre
|
||
serveur Keybase. <a href="https://github.com/keybase/">Une partie des composants, cependant, le sont (open
|
||
source)</a>.</p>
|
||
<p>J'ai du mal à croire en des initiatives qui veulent sauver le monde,
|
||
mais dans leur coin, je ne comprends pas pourquoi il n'y à pas de
|
||
documentation sur comment monter son propre serveur, ou comment les
|
||
aider à travailler sur la fédération. Mais bon, c'est pour l'instant une
|
||
initiative encore fraîche, et je lui laisse le bénéfice du doute.</p>
|
||
<p>Sur le long terme, une infrastructure comme <em>Keybase.io</em>, devra
|
||
évidemment être
|
||
<a href="https://github.com/keybase/keybase-issues/issues/162">distribuée</a>.</p>
|
||
<blockquote>
|
||
<p>We've been talking about a total decentralization, but we have to
|
||
solve a couple things, synchronization in particular. Right now
|
||
someone can mirror us and a client can trust a mirror just as easily
|
||
as the server at keybase.io, but there needs to be a way of announcing
|
||
proofs to any server and having them cooperate with each other. We'd
|
||
be so happy to get this right.</p>
|
||
<p>-- <a href="http://chris.beams.io/posts/keybase/">Chris Coyne, co-founder of
|
||
Keybase</a></p>
|
||
</blockquote>
|
||
<p>Afin de se "passer" de leur service centralisé, les preuves générées
|
||
(qui sont la force du système qu'ils mettent en place) pourraient être
|
||
exportées sur des serveurs de clés existants. C'est quelque chose
|
||
<a href="https://github.com/keybase/keybase-issues/issues/890">qu'ils souhaitent réaliser
|
||
.</a>.</p>
|
||
<p>Bref, une initiative quand même importante et utile, même si elle
|
||
soulève des questions qui méritent qu'on s'y attarde un brin.</p>
|
||
<p>Par ailleurs, <a href="https://leap.se/nicknym">d'autres projets qui visent des objectifs
|
||
similaires</a> existent, via le projet LEAP, mais
|
||
je n'ai pas encore creusé.</p></content></entry><entry><title>Phrases de passe et bonnes pratiques</title><link href="https://blog.notmyidea.org/phrases-de-passe-et-bonnes-pratiques.html" rel="alternate"></link><published>2015-05-09T00:00:00+02:00</published><updated>2015-05-09T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2015-05-09:/phrases-de-passe-et-bonnes-pratiques.html</id><summary type="html">
|
||
<ul>
|
||
<li>headline<br>
|
||
Communiquer de manière chiffrée n'est pas aisée, et nécessite de
|
||
mémoriser des phrases de passes complexes. Comment s'en sortir ?</li>
|
||
</ul>
|
||
<blockquote>
|
||
<p>Au contraire des autres mots de passe, les mots de passe
|
||
cryptographiques ont specifiquement besoin d'être longs et extremement
|
||
difficiles à deviner. La raison est qu'un ordinateur (ou un cluster …</p></blockquote></summary><content type="html">
|
||
<ul>
|
||
<li>headline<br>
|
||
Communiquer de manière chiffrée n'est pas aisée, et nécessite de
|
||
mémoriser des phrases de passes complexes. Comment s'en sortir ?</li>
|
||
</ul>
|
||
<blockquote>
|
||
<p>Au contraire des autres mots de passe, les mots de passe
|
||
cryptographiques ont specifiquement besoin d'être longs et extremement
|
||
difficiles à deviner. La raison est qu'un ordinateur (ou un cluster de
|
||
plusieurs ordinateurs) peut être programmé pour faire des trillions
|
||
d'essais de manière automatique. Si le mot de passe choisi est trop
|
||
faible ou construit d'une manière trop prédictible, cette attaque par
|
||
la force pourrait se revéler fructueuse en essayant toutes les
|
||
possibilités.</p>
|
||
<p>-- <a href="https://www.eff.org/wp/defending-privacy-us-border-guide-travelers-carrying-digital-devices">The Electronic Frontier
|
||
Foundation</a>
|
||
(traduction de mon fait)</p>
|
||
</blockquote>
|
||
<p>Comprendre les concepts et l'écosystème qui permettent d'avoir une vie
|
||
numérique chiffrée n'est pas quelque chose d'aisé.
|
||
<a href="https://emailselfdefense.fsf.org/fr/">Plusieurs</a>
|
||
<a href="http://www.controle-tes-donnees.net/outils/GnuPG.html">guides</a> ont été
|
||
écrits à ce propos, et pour autant je me rends compte que naïvement il
|
||
est possible de mal utiliser les outils existants.</p>
|
||
<blockquote>
|
||
<p>Utilisez un <em>bon</em> mot de passe pour votre session utilisateur et une
|
||
<em>bonne</em> phrase de passe pour proteger votre clé privée. Cette phrase
|
||
de passe est la partie la plus fragile de tout le système.</p>
|
||
<p>-- La page de manuel de GPG.</p>
|
||
</blockquote>
|
||
<p>Une phrase de passe devrait:</p>
|
||
<ul>
|
||
<li>Être suffisamment longue pour être difficile à deviner;</li>
|
||
<li>Ne pas être une citation connue (littérature, livres sacrés etc);</li>
|
||
<li>Difficile à deviner même pour vos proches;</li>
|
||
<li>Facile à se souvenir et à taper;</li>
|
||
<li>être unique et non partagée entre différents sites / applications
|
||
etc.</li>
|
||
</ul>
|
||
<p>Une des techniques consiste à utiliser des mots du dictionnaire,
|
||
sélectionnés de manière aléatoire, puis modifiés.</p>
|
||
<p><img alt="" src="https://imgs.xkcd.com/comics/password_strength.png"></p>
|
||
<p>Micah Lee <a href="https://github.com/micahflee/passphrases">travaille également sur un
|
||
outil</a> qui vise à rendre la
|
||
mémorisation des phrases de passe plus aisée, de par leur répétition
|
||
avec des pauses de plus en plus longues.</p>
|
||
<p><img alt="" src="%7Bfilename%7D/static/passphrases.png"></p>
|
||
<p>Oui, ce n'est pas aussi simple que ce qu'il y parait. Pour ma part, j'ai
|
||
une copie en local de mes clés, dans un fichier chiffré avec une autre
|
||
clé que j'ai généré pour l'occasion et que je ne partagerait pas. J'ai
|
||
par ailleurs <a href="https://github.com/jamessan/vim-gnupg">configuré</a> mon
|
||
éditeur de texte pour pouvoir chiffrer les documents textes par défaut.</p>
|
||
<p>J'ai donc regénéré une nouvelle fois mes clés de travail et
|
||
personnelles, en utilisant des phrases de passe plus complexes.</p>
|
||
<p>Reste encore la question de la sauvegarde de ces clés privées de manière
|
||
chiffrée, que je n'ai pas encore résolue. Bref, tout cela me semble bien
|
||
compliqué pour réussir à l'expliquer à des novices, qui pour certains ne
|
||
sont même pas sur de l'intérêt de la chose.</p></content></entry><entry><title>Eco-système et stockage générique</title><link href="https://blog.notmyidea.org/eco-systeme-et-stockage-generique.html" rel="alternate"></link><published>2015-04-30T00:00:00+02:00</published><updated>2015-04-30T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2015-04-30:/eco-systeme-et-stockage-generique.html</id><summary type="html">
|
||
<p><strong>tl;dr Nous devons construire un service de suivi de paiements, et nous
|
||
hésitons à continuer à nous entêter avec notre propre solution de
|
||
stockage/synchronisation.</strong></p>
|
||
<p>Comme nous l'écrivions <a href="%7Bfilename%7D/2015.04.service-de-nuages.rst">dans l'article
|
||
précédent</a>, nous
|
||
souhaitons construire une solution de stockage générique. On refait
|
||
<a href="http://daybed.readthedocs.org">Daybed</a> chez Mozilla !</p>
|
||
<p>Notre objectif est simple: permettre …</p></summary><content type="html">
|
||
<p><strong>tl;dr Nous devons construire un service de suivi de paiements, et nous
|
||
hésitons à continuer à nous entêter avec notre propre solution de
|
||
stockage/synchronisation.</strong></p>
|
||
<p>Comme nous l'écrivions <a href="%7Bfilename%7D/2015.04.service-de-nuages.rst">dans l'article
|
||
précédent</a>, nous
|
||
souhaitons construire une solution de stockage générique. On refait
|
||
<a href="http://daybed.readthedocs.org">Daybed</a> chez Mozilla !</p>
|
||
<p>Notre objectif est simple: permettre aux développeurs d'application,
|
||
internes à Mozilla ou du monde entier, de faire persister et
|
||
synchroniser facilement des données associées à un utilisateur.</p>
|
||
<div id="storage-specs">
|
||
|
||
Les aspects de l'architecture qui nous semblent incontournables:
|
||
|
||
</div>
|
||
|
||
<ul>
|
||
<li>La solution doit reposer sur un protocole, et non sur une
|
||
implémentation ;</li>
|
||
<li>L'auto-hébergement de l'ensemble doit être simplissime ;</li>
|
||
<li>L'authentification doit être <em>pluggable</em>, voire décentralisée
|
||
(OAuth2, FxA, Persona) ;</li>
|
||
<li>Les enregistrements doivent pouvoir être validés par le serveur ;</li>
|
||
<li>Les données doivent pouvoir être stockées dans n'importe quel
|
||
backend ;</li>
|
||
<li>Un système de permissions doit permettre de protéger des
|
||
collections, ou de partager des enregistrements de manière fine ;</li>
|
||
<li>La résolution de conflits doit pouvoir avoir lieu sur le serveur ;</li>
|
||
<li>Le client doit être pensé «*offline-first*» ;</li>
|
||
<li>Le client doit pouvoir réconcilier les données simplement ;</li>
|
||
<li>Le client doit pouvoir être utilisé aussi bien dans le navigateur
|
||
que côté serveur ;</li>
|
||
<li>Tous les composants se doivent d´être simples et substituables
|
||
facilement.</li>
|
||
</ul>
|
||
<p>La première question qui nous a été posée fût «*Pourquoi vous
|
||
n'utilisez pas PouchDB ou Remote Storage ?*»</p>
|
||
<h2 id="remote-storage">Remote Storage</h2>
|
||
<p>Remote Storage est un standard ouvert pour du stockage par utilisateur.
|
||
<a href="http://tools.ietf.org/html/draft-dejong-remotestorage-04">La
|
||
specification</a>
|
||
se base sur des standards déjà existants et éprouvés: Webfinger, OAuth
|
||
2, CORS et REST.</p>
|
||
<p>L'API est simple, des <a href="http://blog.cozycloud.cc/news/2014/08/12/when-unhosted-meets-cozy-cloud/">projets prestigieux
|
||
l'utilisent</a>.
|
||
Il y a plusieurs <a href="https://github.com/jcoglan/restore">implémentations</a>
|
||
du serveur, et il existe <a href="https://www.npmjs.com/package/remotestorage-server">un squelette
|
||
Node</a> pour
|
||
construire un serveur sur mesure.</p>
|
||
<p><img alt="Remote Storage widget" src="%7Bfilename%7D/images/remotestorage-widget.png"></p>
|
||
<p>Le client
|
||
<a href="https://github.com/remotestorage/remotestorage.js/">remoteStorage.js</a>
|
||
permet d'intégrer la solution dans les applications Web. Il se charge du
|
||
«store local», du cache, de la synchronization, et fournit un widget qui
|
||
permet aux utilisateurs des applications de choisir le serveur qui
|
||
recevra les données (via Webfinger).</p>
|
||
<p><a href="https://github.com/michielbdejong/ludbud">ludbud</a>, la version épurée de
|
||
<em>remoteStorage.js</em>, se limite à l'abstraction du stockage distant. Cela
|
||
permettrait à terme, d'avoir une seule bibliothèque pour stocker dans un
|
||
serveur <em>remoteStorage</em>, <em>ownCloud</em> ou chez les méchants comme <em>Google
|
||
Drive</em> ou <em>Dropbox</em>.</p>
|
||
<p>Au premier abord, la spécification correspond à ce que nous voulons
|
||
accomplir:</p>
|
||
<ul>
|
||
<li>La philosophie du protocole est saine;</li>
|
||
<li>L'éco-système est bien fichu;</li>
|
||
<li>La vision politique colle: redonner le contrôle des données aux
|
||
utilisateurs (voir <a href="http://unhosted.org/">unhosted</a>);</li>
|
||
<li>Les choix techniques compatibles avec ce qu'on a commencé (CORS,
|
||
REST, OAuth 2);</li>
|
||
</ul>
|
||
<p>En revanche, vis à vis de la manipulation des données, il y a plusieurs
|
||
différences avec ce que nous souhaitons faire:</p>
|
||
<ul>
|
||
<li>L'API suit globalement une métaphore «fichiers» (dossier/documents),
|
||
plutôt que «données» (collection/enregistrements) ;</li>
|
||
<li>Il n'y a pas de validation des enregistrements selon un schéma (même
|
||
si <a href="https://remotestorage.io/doc/code/files/baseclient/types-js.html">certaines
|
||
implémentations</a>
|
||
du protocole le font) ;</li>
|
||
<li>Il n'y a pas la possibilité de trier/filtrer les enregistrements
|
||
selon des attributs ;</li>
|
||
<li>Les permissions <a href="https://groups.google.com/forum/#!topic/unhosted/5_NOGq8BPTo">se limitent à
|
||
privé/public</a>
|
||
(et <a href="https://github.com/remotestorage/spec/issues/58#issue-27249452">l'auteur envisage plutôt un modèle à la
|
||
Git</a>)[1]
|
||
;</li>
|
||
</ul>
|
||
<p>En résumé, il semblerait que ce que nous souhaitons faire avec le
|
||
stockage d'enregistrements validés est complémentaire avec <em>Remote
|
||
Storage</em>.</p>
|
||
<p>Si des besoins de persistence orientés «fichiers» se présentent, a
|
||
priori nous aurions tort de réinventer les solutions apportées par cette
|
||
spécification. Il y a donc de grandes chances que nous l´intégrions à
|
||
terme, et que <em>Remote Storage</em> devienne une facette de notre solution.</p>
|
||
<h2 id="pouchdb">PouchDB</h2>
|
||
<p><a href="http://pouchdb.com/">PouchDB</a> est une bibliothèque JavaScript qui
|
||
permet de manipuler des enregistrements en local et de les synchroniser
|
||
vers une base distante.</p>
|
||
<p>``` sourceCode javascript
|
||
var db = new PouchDB('dbname');</p>
|
||
<p>db.put({
|
||
_id: 'dave@gmail.com',
|
||
name: 'David',
|
||
age: 68
|
||
});</p>
|
||
<p>db.replicate.to('http://example.com/mydb');
|
||
```</p>
|
||
<p>Le projet a le vent en poupe, bénéficie de nombreux contributeurs,
|
||
l'éco-système est très riche et l'adoption par des projets <a href="https://github.com/hoodiehq/wip-hoodie-store-on-pouchdb">comme
|
||
Hoodie</a> ne fait
|
||
que confirmer la pertinence de l'outil pour les développeurs frontend.</p>
|
||
<p><em>PouchDB</em> gère un « store » local, dont la persistence est abstraite et
|
||
<a href="http://pouchdb.com/2014/07/25/pouchdb-levels-up.html">repose sur</a> l'API
|
||
<a href="https://github.com/level/levelup#relationship-to-leveldown">LevelDown</a>
|
||
pour persister les données dans <a href="https://github.com/Level/levelup/wiki/Modules#storage-back-ends">n'importe quel
|
||
backend</a>.</p>
|
||
<p>Même si <em>PouchDB</em> adresse principalement les besoins des applications
|
||
«*offline-first*», il peut être utilisé aussi bien dans le navigateur
|
||
que côté serveur, via Node.</p>
|
||
<h3 id="synchronisation">Synchronisation</h3>
|
||
<p>La synchronisation (ou réplication) des données locales s'effectue sur
|
||
un <a href="http://couchdb.apache.org/">CouchDB</a> distant.</p>
|
||
<p>Le projet <a href="https://github.com/pouchdb/pouchdb-server">PouchDB Server</a>
|
||
implémente l'API de CouchDB en NodeJS. Comme <em>PouchDB</em> est utilisé, on
|
||
obtient un service qui se comporte comme un <em>CouchDB</em> mais qui stocke
|
||
ses données n'importe où, dans un <em>Redis</em> ou un <em>PostgreSQL</em> par
|
||
exemple.</p>
|
||
<p>La synchronisation est complète. Autrement dit, tous les enregistrements
|
||
qui sont sur le serveur se retrouvent synchronisés dans le client. Il
|
||
est possible de filtrer les collections synchronisées, mais cela <a href="http://pouchdb.com/2015/04/05/filtered-replication.html">n'a
|
||
pas pour objectif de sécuriser l'accès aux
|
||
données</a>.</p>
|
||
<p>L'approche recommandée pour cloisonner les données par utilisateur
|
||
consiste à créer <a href="https://github.com/nolanlawson/pouchdb-authentication#some-people-can-read-some-docs-some-people-can-write-those-same-docs">une base de données par
|
||
utilisateur</a>.</p>
|
||
<p>Ce n'est pas forcément un problème, CouchDB <a href="https://mail-archives.apache.org/mod_mbox/couchdb-user/201401.mbox/%3C52CEB873.7080404@ironicdesign.com%3E">supporte des centaines de
|
||
milliers de bases sans
|
||
sourciller</a>.
|
||
Mais selon les cas d'utilisation, le cloisement n'est pas toujours
|
||
facile à déterminer (par rôle, par application, par collection, ...).</p>
|
||
<h2 id="le-cas-dutilisation-payments">Le cas d'utilisation « Payments »</h2>
|
||
<p><img alt="Put Payments Here -- Before the Internet - CC-NC-SA Katy Silberger
|
||
https://www.flickr.com/photos/katysilbs/11163812186" src="%7Bfilename%7D/images/put-payments.jpg"></p>
|
||
<p>Dans les prochaines semaines, nous devrons mettre sur pied un prototype
|
||
pour tracer l'historique des paiements et abonnements d'un utilisateur.</p>
|
||
<p>Le besoin est simple:</p>
|
||
<ul>
|
||
<li>l'application « Payment » enregistre les paiements et abonnements
|
||
d'un utilisateur pour une application donnée;</li>
|
||
<li>l'application « Donnée » interroge le service pour vérifier qu'un
|
||
utilisateur a payé ou est abonné;</li>
|
||
<li>l'utilisateur interroge le service pour obtenir la liste de tous ses
|
||
abonnements.</li>
|
||
</ul>
|
||
<p>Seule l'application « Payment » a le droit de créer/modifier/supprimer
|
||
des enregistrements, les deux autres ne peuvent que consulter en lecture
|
||
seule.</p>
|
||
<p>Une application donnée ne peut pas accéder aux paiements des autres
|
||
applications, et un utilisateur ne peut pas accéder aux paiements des
|
||
autres utilisateurs.</p>
|
||
<h3 id="avec-remotestorage">Avec RemoteStorage</h3>
|
||
<p><img alt="Remote Love - CC-BY-NC Julie
|
||
https://www.flickr.com/photos/mamajulie2008/2609549461" src="%7Bfilename%7D/images/remote-love.jpg"></p>
|
||
<p>Clairement, l'idée de <em>RemoteStorage</em> est de dissocier l'application
|
||
executée, et les données créées par l'utilisateur avec celle-ci.</p>
|
||
<p>Dans notre cas, c'est l'application « Payment » qui manipule des données
|
||
concernant un utilisateur. Mais celles-ci ne lui appartiennent pas
|
||
directement: certes un utilisateur doit pouvoir les supprimer, surtout
|
||
pas en créer ou les modifier!</p>
|
||
<p>La notion de permissions limitée à privé/publique ne suffit pas dans ce
|
||
cas précis.</p>
|
||
<h3 id="avec-pouchdb">Avec PouchDB</h3>
|
||
<p>Il va falloir créer une <em>base de données</em> par utilisateur, afin d'isoler
|
||
les enregistrements de façon sécurisée. Seule l'application « Payment »
|
||
aura tous les droits sur les databases.</p>
|
||
<p>Mais cela ne suffit pas.</p>
|
||
<p>Il ne faut pas qu'une application puisse voir les paiements des autres
|
||
applications, donc il va aussi falloir recloisonner, et créer une <em>base
|
||
de données</em> par application.</p>
|
||
<p>Quand un utilisateur voudra accéder à l'ensemble de ses paiements, il
|
||
faudra agréger les <em>databases</em> de toutes les applications. Quand
|
||
l'équipe marketing voudra faire des statistiques sur l'ensemble des
|
||
applications, il faudra agrégér des centaines de milliers de
|
||
<em>databases</em>.</p>
|
||
<p>Ce qui est fort dommage, puisqu'il est probable que les paiements ou
|
||
abonnements d'un utilisateur pour une application se comptent sur les
|
||
doigts d'une main. Des centaines de milliers de bases contenant moins de
|
||
5 enregistrements ?</p>
|
||
<p>De plus, dans le cas de l'application « Payment », le serveur est
|
||
implémenté en Python. Utiliser un wrapper JavaScript comme le fait
|
||
<a href="https://pythonhosted.org/Python-PouchDB/">python-pouchdb</a> cela ne nous
|
||
fait pas trop rêver.</p>
|
||
<h2 id="un-nouvel-eco-systeme">Un nouvel éco-système ?</h2>
|
||
<p><img alt="Wagon wheel - CC-BY-NC-SA arbyreed
|
||
https://www.flickr.com/photos/19779889@N00/16161808220" src="%7Bfilename%7D/images/wagon-wheel.jpg"></p>
|
||
<p>Évidemment, quand on voit la richesse des projets <em>PouchDB</em> et <em>Remote
|
||
Storage</em> et la dynamique de ces communautés, il est légitime d'hésiter
|
||
avant de développer une solution alternative.</p>
|
||
<p>Quand nous avons créé le serveur <em>Reading List</em>, nous l'avons construit
|
||
avec <a href="http://cliquet.readthedocs.org/">Cliquet</a>, ce fût l'occasion de
|
||
mettre au point <a href="http://cliquet.readthedocs.org/en/latest/api/">un protocole très
|
||
simple</a>, fortement
|
||
inspiré de <a href="http://en.wikipedia.org/wiki/Firefox_Sync">Firefox Sync</a>,
|
||
pour faire de la synchronisation d'enregistrements.</p>
|
||
<p>Et si les clients <em>Reading List</em> ont pu être implémentés en quelques
|
||
semaines, que ce soit en JavaScript, Java (Android) et ASM (Add-on
|
||
Firefox), c'est que le principe «*offline first*» du service est
|
||
trivial.</p>
|
||
<h3 id="les-compromis">Les compromis</h3>
|
||
<p>Évidemment, nous n'avons pas la prétention de concurrencer <em>CouchDB</em>.
|
||
Nous faisons plusieurs concessions:</p>
|
||
<ul>
|
||
<li>De base, les collections d'enregistrements sont cloisonnées par
|
||
utilisateur;</li>
|
||
<li>Pas d'historique des révisions;</li>
|
||
<li>Pas de diff sur les enregistrements entre révisions;</li>
|
||
<li>De base, pas de résolution de conflit automatique;</li>
|
||
<li>Pas de synchronisation par flux (<em>streams</em>);</li>
|
||
</ul>
|
||
<p>Jusqu'à preuve du contraire, ces compromis excluent la possibilité
|
||
d'implémenter un <a href="https://github.com/pouchdb/pouchdb/blob/master/lib/adapters/http/http.js#L721-L946">adapter
|
||
PouchDB</a>
|
||
pour la synchronisation avec le protocole HTTP de <em>Cliquet</em>.</p>
|
||
<p>Dommage puisque capitaliser sur l'expérience client de <em>PouchDB</em> au
|
||
niveau synchro client semble être une très bonne idée.</p>
|
||
<p>En revanche, nous avons plusieurs fonctionnalités intéressantes:</p>
|
||
<ul>
|
||
<li>Pas de map-reduce;</li>
|
||
<li>Synchronisation partielle et/ou ordonnée et/ou paginée ;</li>
|
||
<li>Le client choisit, via des headers, d'écraser la donnée ou de
|
||
respecter la version du serveur ;</li>
|
||
<li>Un seul serveur à déployer pour N applications ;</li>
|
||
<li>Auto-hébergement simplissime ;</li>
|
||
<li>Le client peut choisir de ne pas utiliser de « store local » du tout
|
||
;</li>
|
||
<li>Dans le client JS, la gestion du « store local » sera externalisée
|
||
(on pense à <a href="https://github.com/mozilla/localForage">LocalForage</a> ou
|
||
<a href="https://github.com/dfahlander/Dexie.js">Dexie.js</a>) ;</li>
|
||
</ul>
|
||
<p>Et, on répond au reste des <a href="#storage-specs">specifications mentionnées au début de
|
||
l'article</a> !</p>
|
||
<h3 id="les-arguments-philosophiques">Les arguments philosophiques</h3>
|
||
<p>Il est <a href="http://en.wikipedia.org/wiki/Law_of_the_instrument">illusoire de penser qu'on peut tout faire avec un seul
|
||
outil</a>.</p>
|
||
<p>Nous avons d'autres cas d'utilisations dans les cartons qui semblent
|
||
correspondre au scope de <em>PouchDB</em> (<em>pas de notion de permissions ou de
|
||
partage, environnement JavaScript, ...</em>). Nous saurons en tirer profit
|
||
quand cela s'avèrera pertinent !</p>
|
||
<p>L'éco-système que nous voulons construire tentera de couvrir les cas
|
||
d'utilisation qui sont mal adressés par <em>PouchDB</em>. Il se voudra:</p>
|
||
<ul>
|
||
<li>Basé sur notre protocole très simple ;</li>
|
||
<li>Minimaliste et multi-usages (<em>comme la fameuse 2CV</em>) ;</li>
|
||
<li>Naïf (<em>pas de rocket science</em>) ;</li>
|
||
<li>Sans magie (<em>explicite et facile à réimplémenter from scratch</em>) ;</li>
|
||
</ul>
|
||
<p><a href="http://cliquet.readthedocs.org/en/latest/rationale.html">La philosophie et les fonctionnalités du toolkit python
|
||
Cliquet</a> seront
|
||
bien entendu à l'honneur :)</p>
|
||
<p>Quant à <em>Remote Storage</em>, dès que le besoin se présentera, nous serons
|
||
très fier de rejoindre l'initiative, mais pour l'instant cela nous
|
||
paraît risqué de démarrer en tordant la solution.</p>
|
||
<h3 id="les-arguments-pratiques">Les arguments pratiques</h3>
|
||
<p>Avant d'accepter de déployer une solution à base de <em>CouchDB</em>, les <em>ops</em>
|
||
de Mozilla vont nous demander de leur prouver par A+B que ce n'est pas
|
||
faisable avec les stacks qui sont déjà rodées en interne (i.e. MySQL,
|
||
Redis, PostgreSQL).</p>
|
||
<p>De plus, on doit s'engager sur une pérennité d'au moins 5 ans pour les
|
||
données. Avec <em>Cliquet</em>, en utilisant le backend PostgreSQL, les données
|
||
sont persistées à plat dans un <a href="https://github.com/mozilla-services/cliquet/blob/40aa33/cliquet/storage/postgresql/schema.sql#L14-L28">schéma PostgreSQL tout
|
||
bête</a>.
|
||
Ce qui ne sera pas le cas d'un adapteur LevelDown qui va manipuler des
|
||
notions de révisions éclatées dans un schéma clé-valeur.</p>
|
||
<p>Si nous basons le service sur <em>Cliquet</em>, comme c'est le cas avec
|
||
<a href="http://kinto.readthedocs.org">Kinto</a>, tout le travail d'automatisation
|
||
de la mise en production (<em>monitoring, builds RPM, Puppet...</em>) que nous
|
||
avons fait pour <em>Reading List</em> est complètement réutilisable.</p>
|
||
<p>De même, si on repart avec une stack complètement différente, nous
|
||
allons devoir recommencer tout le travail de rodage, de profiling et
|
||
d'optimisation effectué au premier trimestre.</p>
|
||
<h2 id="les-prochaines-etapes">Les prochaines étapes</h2>
|
||
<p>Et il est encore temps de changer de stratégie :) Nous aimerions avoir
|
||
un maximum de retours ! C'est toujours une décision difficile à
|
||
prendre... <code>&lt;/appel à troll&gt;</code></p>
|
||
<ul>
|
||
<li>Tordre un éco-système existant vs. constuire sur mesure ;</li>
|
||
<li>Maîtriser l'ensemble vs. s'intégrer ;</li>
|
||
<li>Contribuer vs. refaire ;</li>
|
||
<li>Guider vs. suivre.</li>
|
||
</ul>
|
||
<p>Nous avons vraiment l'intention de rejoindre l'initiative
|
||
<a href="https://nobackend.org/">no-backend</a>, et ce premier pas n'exclue pas que
|
||
nous convergions à terme ! Peut-être que nous allons finir par rendre
|
||
notre service compatible avec <em>Remote Storage</em>, et peut-être que
|
||
<em>PouchDB</em> deviendra plus agnostique quand au protocole de
|
||
synchronisation...</p>
|
||
<p><img alt="XKCD — Standards
|
||
https://xkcd.com/927/" src="%7Bfilename%7D/images/standards.png"></p>
|
||
<p>Utiliser ce nouvel écosystème pour le projet « Payments » va nous
|
||
permettre de mettre au point un système de permissions (<em>probablement
|
||
basé sur les scopes OAuth</em>) qui correspond au besoin exprimé. Et nous
|
||
avons bien l'intention de puiser dans <a href="http://blog.daybed.io/daybed-revival.html">notre expérience avec Daybed sur
|
||
le sujet</a>.</p>
|
||
<p>Nous extrairons aussi le code des clients implémentés pour <em>Reading
|
||
List</em> afin de faire un client JavaScript minimaliste.</p>
|
||
<p>En partant dans notre coin, nous prenons plusieurs risques:</p>
|
||
<ul>
|
||
<li>réinventer une roue dont nous n'avons pas connaissance ;</li>
|
||
<li>échouer à faire de l'éco-système <em>Cliquet</em> un projet communautaire ;</li>
|
||
<li>échouer à positionner <em>Cliquet</em> dans la niche des cas non couverts
|
||
par PouchDB :)</li>
|
||
</ul>
|
||
<p>Comme <a href="http://pouchdb.com/2015/04/05/filtered-replication.html">le dit Giovanni
|
||
Ornaghi</a>:</p>
|
||
<blockquote>
|
||
<p>Rolling out your set of webservices, push notifications, or background
|
||
services might give you more control, but at the same time it will
|
||
force you to engineer, write, test, and maintain a whole new
|
||
ecosystem.</p>
|
||
</blockquote>
|
||
<p>C'est justement l'éco-système dont est responsable l'équipe <em>Mozilla
|
||
Cloud Services</em>!</p>
|
||
<ol>
|
||
<li>Il existe le <a href="https://sharesome.5apps.com/">projet Sharesome</a> qui
|
||
permet de partager publiquement des ressources de son <em>remote
|
||
Storage</em>.</li>
|
||
</ol></content></entry><entry><title>What's Hawk and how to use it?</title><link href="https://blog.notmyidea.org/whats-hawk-and-how-to-use-it.html" rel="alternate"></link><published>2014-07-31T00:00:00+02:00</published><updated>2014-07-31T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2014-07-31:/whats-hawk-and-how-to-use-it.html</id><summary type="html">
|
||
<p>At Mozilla, we recently had to implement <a href="https://github.com/hueniverse/hawk">the Hawk authentication
|
||
scheme</a> for a number of projects,
|
||
and we came up creating two libraries to ease integration into pyramid
|
||
and node.js apps.</p>
|
||
<p>But maybe you don't know Hawk.</p>
|
||
<p>Hawk is a relatively new technology, crafted by one of the original …</p></summary><content type="html">
|
||
<p>At Mozilla, we recently had to implement <a href="https://github.com/hueniverse/hawk">the Hawk authentication
|
||
scheme</a> for a number of projects,
|
||
and we came up creating two libraries to ease integration into pyramid
|
||
and node.js apps.</p>
|
||
<p>But maybe you don't know Hawk.</p>
|
||
<p>Hawk is a relatively new technology, crafted by one of the original
|
||
<a href="https://en.wikipedia.org/wiki/OAuth">OAuth</a> specification authors, that
|
||
intends to replace the 2-legged OAuth authentication scheme using a
|
||
simpler approach.</p>
|
||
<p>It is an authentication scheme for HTTP, built around <a href="https://en.wikipedia.org/wiki/Hmac">HMAC
|
||
digests</a> of requests and responses.</p>
|
||
<p>Every authenticated client request has an Authorization header
|
||
containing a MAC (Message Authentication Code) and some additional
|
||
metadata, then each server response to authenticated requests contains a
|
||
Server-Authorization header that authenticates the response, so the
|
||
client is sure it comes from the right server.</p>
|
||
<h2 id="exchange-of-the-hawk-id-and-hawk-key">Exchange of the hawk id and hawk key</h2>
|
||
<p>To sign the requests, a client needs to retrieve a token id and a token
|
||
key from the server.</p>
|
||
<p>Hawk itself does not define how these credentials should be exchanged
|
||
between the server and the client. The excellent team behind <a href="http://accounts.firefox.com">Firefox
|
||
Accounts</a> put together a scheme to do that,
|
||
which acts like the following:</p>
|
||
<div class="note">
|
||
|
||
<div class="admonition-title">
|
||
|
||
Note
|
||
|
||
</div>
|
||
|
||
All this derivation crazyness might seem a bit complicated, but don't
|
||
worry, we put together some libraries that takes care of that for you
|
||
automatically.
|
||
|
||
If you are not interested into these details, you can directly jump to
|
||
the next section to see how to use the libraries.
|
||
|
||
</div>
|
||
|
||
<p>When your server application needs to send you the credentials, it will
|
||
return it inside a specific Hawk-Session-Token header. This token can be
|
||
derived to split this string in two values (hawk id and hawk key) that
|
||
you will use to sign your next requests.</p>
|
||
<p>In order to get the hawk credentials, you'll need to:</p>
|
||
<p>First, do an <a href="http://en.wikipedia.org/wiki/HKDF">HKDF derivation</a> on the
|
||
given session token. You'll need to use the following
|
||
parameters:</p>
|
||
<div class="highlight"><pre><span></span><span class="n">key_material</span> <span class="o">=</span> <span class="n">HKDF</span><span class="p">(</span><span class="n">hawk_session</span><span class="p">,</span> <span class="ss">&quot;&quot;</span><span class="p">,</span> <span class="s1">&#39;identity.mozilla.com/picl/v1/sessionToken&#39;</span><span class="p">,</span> <span class="mi">32</span><span class="o">*</span><span class="mi">2</span><span class="p">)</span>
|
||
</pre></div>
|
||
|
||
|
||
<div class="note">
|
||
|
||
<div class="admonition-title">
|
||
|
||
Note
|
||
|
||
</div>
|
||
|
||
The `identity.mozilla.com/picl/v1/sessionToken` is a reference to this
|
||
way of deriving the credentials, not an actual URL.
|
||
|
||
</div>
|
||
|
||
<p>Then, the key material you'll get out of the HKDF need to be separated
|
||
into two parts, the first 32 hex caracters are the hawk id, and the next
|
||
32 ones are the hawk key.</p>
|
||
<p>Credentials:</p>
|
||
<p>``` sourceCode javascript
|
||
credentials = {
|
||
'id': keyMaterial[0:32],
|
||
'key': keyMaterial[32:64],
|
||
'algorithm': 'sha256'
|
||
}</p>
|
||
<div class="highlight"><pre><span></span><span class="c1">## Httpie</span>
|
||
|
||
<span class="n">To</span> <span class="n">showcase</span> <span class="n">APIs</span> <span class="ow">in</span> <span class="n">the</span> <span class="n">documentation</span><span class="p">,</span> <span class="n">I</span> <span class="n">like</span> <span class="n">to</span> <span class="n">use</span>
|
||
<span class="p">[</span><span class="n">httpie</span><span class="p">](</span><span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="n">github</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">jakubroztocil</span><span class="o">/</span><span class="n">httpie</span><span class="p">),</span> <span class="n">a</span> <span class="n">curl</span><span class="o">-</span><span class="n">replacement</span>
|
||
<span class="k">with</span> <span class="n">a</span> <span class="n">nicer</span> <span class="n">API</span><span class="p">,</span> <span class="n">built</span> <span class="n">around</span> <span class="p">[</span><span class="n">the</span> <span class="n">python</span> <span class="n">requests</span>
|
||
<span class="n">library</span><span class="p">](</span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">python</span><span class="o">-</span><span class="n">requests</span><span class="o">.</span><span class="n">org</span><span class="p">)</span><span class="o">.</span>
|
||
|
||
<span class="n">Luckily</span><span class="p">,</span> <span class="n">HTTPie</span> <span class="n">allows</span> <span class="n">you</span> <span class="n">to</span> <span class="n">plug</span> <span class="n">different</span> <span class="n">authentication</span> <span class="n">schemes</span> <span class="k">for</span>
|
||
<span class="n">it</span><span class="p">,</span> <span class="n">so</span> <span class="p">[</span><span class="n">I</span> <span class="n">wrote</span> <span class="n">a</span>
|
||
<span class="n">wrapper</span><span class="p">](</span><span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="n">github</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">mozilla</span><span class="o">-</span><span class="n">services</span><span class="o">/</span><span class="n">requests</span><span class="o">-</span><span class="n">hawk</span><span class="p">)</span> <span class="n">around</span>
|
||
<span class="p">[</span><span class="n">mohawk</span><span class="p">](</span><span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="n">github</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">kumar303</span><span class="o">/</span><span class="n">mohawk</span><span class="p">)</span> <span class="n">to</span> <span class="n">add</span> <span class="n">hawk</span> <span class="n">support</span> <span class="n">to</span> <span class="n">the</span>
|
||
<span class="n">requests</span> <span class="n">lib</span><span class="o">.</span>
|
||
|
||
<span class="n">Doing</span> <span class="n">hawk</span> <span class="n">requests</span> <span class="ow">in</span> <span class="n">your</span> <span class="n">terminal</span> <span class="ow">is</span> <span class="n">now</span> <span class="k">as</span> <span class="n">simple</span> <span class="k">as</span><span class="p">:</span>
|
||
|
||
<span class="err">$</span> <span class="n">pip</span> <span class="n">install</span> <span class="n">requests</span><span class="o">-</span><span class="n">hawk</span> <span class="n">httpie</span>
|
||
<span class="err">$</span> <span class="n">http</span> <span class="n">GET</span> <span class="n">localhost</span><span class="p">:</span><span class="mi">5000</span><span class="o">/</span><span class="n">registration</span> <span class="o">--</span><span class="n">auth</span><span class="o">-</span><span class="nb">type</span><span class="o">=</span><span class="n">hawk</span> <span class="o">--</span><span class="n">auth</span><span class="o">=</span><span class="s1">&#39;id:key&#39;</span>
|
||
|
||
<span class="n">In</span> <span class="n">addition</span><span class="p">,</span> <span class="n">it</span> <span class="n">will</span> <span class="n">help</span> <span class="n">you</span> <span class="n">to</span> <span class="n">craft</span> <span class="n">requests</span> <span class="n">using</span> <span class="n">the</span> <span class="n">requests</span>
|
||
<span class="n">library</span><span class="p">:</span>
|
||
|
||
<span class="sb">``</span><span class="err">`</span> <span class="n">sourceCode</span> <span class="n">python</span>
|
||
<span class="kn">import</span> <span class="nn">requests</span>
|
||
<span class="kn">from</span> <span class="nn">requests_hawk</span> <span class="kn">import</span> <span class="n">HawkAuth</span>
|
||
|
||
<span class="n">hawk_auth</span> <span class="o">=</span> <span class="n">HawkAuth</span><span class="p">(</span>
|
||
<span class="n">credentials</span><span class="o">=</span><span class="p">{</span><span class="s1">&#39;id&#39;</span><span class="p">:</span> <span class="nb">id</span><span class="p">,</span> <span class="s1">&#39;key&#39;</span><span class="p">:</span> <span class="n">key</span><span class="p">,</span> <span class="s1">&#39;algorithm&#39;</span><span class="p">:</span> <span class="s1">&#39;sha256&#39;</span><span class="p">})</span>
|
||
|
||
<span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="s2">&quot;/url&quot;</span><span class="p">,</span> <span class="n">auth</span><span class="o">=</span><span class="n">hawk_auth</span><span class="p">)</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>Alternatively, if you don't have the token id and key, you can pass the
|
||
hawk session token I talked about earlier and the lib will take care of
|
||
the derivation for you:</p>
|
||
<p>``` sourceCode python
|
||
hawk_auth = HawkAuth(
|
||
hawk_session=resp.headers['hawk-session-token'],
|
||
server_url=self.server_url
|
||
)
|
||
requests.post("/url", auth=hawk_auth)</p>
|
||
<div class="highlight"><pre><span></span><span class="c1">## Integrate with python pyramid apps</span>
|
||
|
||
<span class="n">If</span> <span class="n">you</span><span class="s1">&#39;re writing pyramid applications, you&#39;</span><span class="n">ll</span> <span class="n">be</span> <span class="n">happy</span> <span class="n">to</span> <span class="n">learn</span> <span class="n">that</span>
|
||
<span class="p">[</span><span class="n">Ryan</span> <span class="n">Kelly</span><span class="p">](</span><span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="n">www</span><span class="o">.</span><span class="n">rfk</span><span class="o">.</span><span class="n">id</span><span class="o">.</span><span class="n">au</span><span class="o">/</span><span class="n">blog</span><span class="o">/</span><span class="p">)</span> <span class="n">put</span> <span class="n">together</span> <span class="n">a</span> <span class="n">library</span> <span class="n">that</span>
|
||
<span class="n">makes</span> <span class="n">Hawk</span> <span class="n">work</span> <span class="k">as</span> <span class="n">an</span> <span class="n">Authentication</span> <span class="n">provider</span> <span class="k">for</span> <span class="n">them</span><span class="o">.</span> <span class="n">I</span><span class="s1">&#39;m chocked how</span>
|
||
<span class="n">simple</span> <span class="n">it</span> <span class="ow">is</span> <span class="n">to</span> <span class="n">use</span> <span class="n">it</span><span class="o">.</span>
|
||
|
||
<span class="n">Here</span> <span class="ow">is</span> <span class="n">a</span> <span class="n">demo</span> <span class="n">of</span> <span class="n">how</span> <span class="n">we</span> <span class="n">implemented</span> <span class="n">it</span> <span class="k">for</span> <span class="n">Daybed</span><span class="p">:</span>
|
||
|
||
<span class="sb">``</span><span class="err">`</span> <span class="n">sourceCode</span> <span class="n">python</span>
|
||
<span class="kn">from</span> <span class="nn">pyramid_hawkauth</span> <span class="kn">import</span> <span class="n">HawkAuthenticationPolicy</span>
|
||
|
||
<span class="n">policy</span> <span class="o">=</span> <span class="n">HawkAuthenticationPolicy</span><span class="p">(</span><span class="n">decode_hawk_id</span><span class="o">=</span><span class="n">get_hawk_id</span><span class="p">)</span>
|
||
<span class="n">config</span><span class="o">.</span><span class="n">set_authentication_policy</span><span class="p">(</span><span class="n">authn_policy</span><span class="p">)</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>The get_hawk_id function is a function that takes a request and a
|
||
tokenid and returns a tuple of (token_id, token_key).</p>
|
||
<p>How you want to store the tokens and retrieve them is up to you. The
|
||
default implementation (e.g. if you don't pass a decode_hawk_id
|
||
function) decodes the key from the token itself, using a master secret
|
||
on the server (so you don't need to store anything).</p>
|
||
<h2 id="integrate-with-nodejs-express-apps">Integrate with node.js Express apps</h2>
|
||
<p>We had to implement Hawk authentication for two node.js projects and
|
||
finally came up factorizing everything in a library for express, named
|
||
<a href="https://github.com/mozilla-services/express-hawkauth">express-hawkauth</a>.</p>
|
||
<p>In order to plug it in your application, you'll need to use it as a
|
||
middleware:</p>
|
||
<p>``` sourceCode javascript
|
||
var express = require("express");
|
||
var hawk = require("express-hawkauth");
|
||
app = express();</p>
|
||
<p>var hawkMiddleware = hawk.getMiddleware({
|
||
hawkOptions: {},
|
||
getSession: function(tokenId, cb) {
|
||
// A function which pass to the cb the key and algorithm for the
|
||
// given token id. First argument of the callback is a potential
|
||
// error.
|
||
cb(null, {key: "key", algorithm: "sha256"});
|
||
},
|
||
createSession: function(id, key, cb) {
|
||
// A function which stores a session for the given id and key.
|
||
// Argument returned is a potential error.
|
||
cb(null);
|
||
},
|
||
setUser: function(req, res, tokenId, cb) {
|
||
// A function that uses req and res, the hawkId when they're known so
|
||
// that it can tweak it. For instance, you can store the tokenId as the
|
||
// user.
|
||
req.user = tokenId;
|
||
}
|
||
});</p>
|
||
<p>app.get("/hawk-enabled-endpoint", hawkMiddleware);
|
||
```</p>
|
||
<p>If you pass the createSession parameter, all non-authenticated requests
|
||
will create a new hawk session and return it with the response, in the
|
||
Hawk-Session-Token header.</p>
|
||
<p>If you want to only check a valid hawk session exists (without creating
|
||
a new one), just create a middleware which doesn't have any
|
||
createSession parameter defined.</p>
|
||
<h2 id="some-reference-implementations">Some reference implementations</h2>
|
||
<p>As a reference, here is how we're using the libraries I'm talking about,
|
||
in case that helps you to integrate with your projects.</p>
|
||
<ul>
|
||
<li>The Mozilla Loop server <a href="https://github.com/mozilla-services/loop-server/blob/master/loop/index.js#L70-L133">uses hawk as authentication once you're
|
||
logged in with a valid BrowserID
|
||
assertion</a>;
|
||
request, to keep a session between client and server;</li>
|
||
<li><a href="https://github.com/spiral-project/daybed/commit/f178b4e43015fa077430798dcd3d0886c7611caf">I recently added hawk support on the Daybed
|
||
project</a>
|
||
(that's a pyramid / cornice) app.</li>
|
||
<li>It's also interesting to note that Kumar put together <a href="http://hawkrest.readthedocs.org/en/latest/">hawkrest, for
|
||
the django rest
|
||
framework</a></li>
|
||
</ul></content></entry><entry><title>Implementing CORS in Cornice</title><link href="https://blog.notmyidea.org/implementing-cors-in-cornice.html" rel="alternate"></link><published>2013-02-04T00:00:00+01:00</published><updated>2013-02-04T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2013-02-04:/implementing-cors-in-cornice.html</id><summary type="html">
|
||
<div class="note">
|
||
|
||
<div class="admonition-title">
|
||
|
||
Note
|
||
|
||
</div>
|
||
|
||
I'm cross-posting [on the mozilla services
|
||
weblog](https://blog.mozilla.org/services/). Since this is the first
|
||
time we're doing that, I though it could be useful to point you there.
|
||
Check it out and expect more technical articles there in the future.
|
||
|
||
</div>
|
||
|
||
<p>For security reasons, it's not possible …</p></summary><content type="html">
|
||
<div class="note">
|
||
|
||
<div class="admonition-title">
|
||
|
||
Note
|
||
|
||
</div>
|
||
|
||
I'm cross-posting [on the mozilla services
|
||
weblog](https://blog.mozilla.org/services/). Since this is the first
|
||
time we're doing that, I though it could be useful to point you there.
|
||
Check it out and expect more technical articles there in the future.
|
||
|
||
</div>
|
||
|
||
<p>For security reasons, it's not possible to do cross-domain requests. In
|
||
other words, if you have a page served from the domain lolnet.org, it
|
||
will not be possible for it to get data from notmyidea.org.</p>
|
||
<p>Well, it's possible, using tricks and techniques like
|
||
<a href="http://en.wikipedia.org/wiki/JSONP">JSONP</a>, but that doesn't work all
|
||
the time (see <a href="#how-this-is-different-from-jsonp">the section below</a>). I
|
||
remember myself doing some simple proxies on my domain server to be able
|
||
to query other's API.</p>
|
||
<p>Thankfully, there is a nicer way to do this, namely, "Cross Origin
|
||
Resource-Sharing", or <a href="http://www.w3.org/TR/cors/">CORS</a>.</p>
|
||
<h2 id="you-want-an-icecream-go-ask-your-dad-first">You want an icecream? Go ask your dad first.</h2>
|
||
<p>If you want to use CORS, you need the API you're querying to support it;
|
||
on the server side.</p>
|
||
<p>The HTTP server need to answer to the OPTIONS verb, and with the
|
||
appropriate response headers.</p>
|
||
<p>OPTIONS is sent as what the authors of the spec call a "preflight
|
||
request"; just before doing a request to the API, the <em>User-Agent</em> (the
|
||
browser most of the time) asks the permission to the resource, with an
|
||
OPTIONS call.</p>
|
||
<p>The server answers, and tell what is available and what isn't:</p>
|
||
<p><img alt="The CORS flow (from the HTML5 CORS tutorial)" src="images/cors_flow.png"></p>
|
||
<ul>
|
||
<li>
|
||
<p>1a. The User-Agent, rather than doing the call directly, asks the
|
||
server, the API, the permission to do the request. It does so with
|
||
the following headers:</p>
|
||
<ul>
|
||
<li><strong>Access-Control-Request-Headers</strong>, contains the headers the
|
||
User-Agent want to access.</li>
|
||
<li><strong>Access-Control-Request-Method</strong> contains the method the
|
||
User-Agent want to access.</li>
|
||
</ul>
|
||
</li>
|
||
<li>
|
||
<p>1b. The API answers what is authorized:</p>
|
||
<ul>
|
||
<li><strong>Access-Control-Allow-Origin</strong> the origin that's accepted. Can
|
||
be * or the domain name.</li>
|
||
<li><strong>Access-Control-Allow-Methods</strong> a <em>list</em> of allowed methods.
|
||
This can be cached. Note than the request asks permission for
|
||
one method and the server should return a list of accepted
|
||
methods.</li>
|
||
<li><strong>Access-Allow-Headers</strong> a list of allowed headers, for all of
|
||
the methods, since this can be cached as well.</li>
|
||
</ul>
|
||
</li>
|
||
<li>
|
||
<ol>
|
||
<li>The User-Agent can do the "normal" request.</li>
|
||
</ol>
|
||
</li>
|
||
</ul>
|
||
<p>So, if you want to access the /icecream resource, and do a PUT there,
|
||
you'll have the following flow:</p>
|
||
<div class="highlight"><pre><span></span><span class="o">&gt;</span> <span class="k">OPTIONS</span> <span class="o">/</span><span class="n">icecream</span>
|
||
<span class="o">&gt;</span> <span class="k">Access</span><span class="o">-</span><span class="n">Control</span><span class="o">-</span><span class="n">Request</span><span class="o">-</span><span class="n">Methods</span> <span class="o">=</span> <span class="n">PUT</span>
|
||
<span class="o">&gt;</span> <span class="n">Origin</span><span class="p">:</span> <span class="n">notmyidea</span><span class="p">.</span><span class="n">org</span>
|
||
<span class="o">&lt;</span> <span class="k">Access</span><span class="o">-</span><span class="n">Control</span><span class="o">-</span><span class="n">Allow</span><span class="o">-</span><span class="n">Origin</span> <span class="o">=</span> <span class="n">notmyidea</span><span class="p">.</span><span class="n">org</span>
|
||
<span class="o">&lt;</span> <span class="k">Access</span><span class="o">-</span><span class="n">Control</span><span class="o">-</span><span class="n">Allow</span><span class="o">-</span><span class="n">Methods</span> <span class="o">=</span> <span class="n">PUT</span><span class="p">,</span><span class="k">GET</span><span class="p">,</span><span class="k">DELETE</span>
|
||
<span class="mi">200</span> <span class="n">OK</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>You can see that we have an Origin Header in the request, as well as a
|
||
Access-Control-Request-Methods. We're here asking if we have the right,
|
||
as notmyidea.org, to do a PUT request on /icecream.</p>
|
||
<p>And the server tells us that we can do that, as well as GET and DELETE.</p>
|
||
<p>I'll not cover all the details of the CORS specification here, but bear
|
||
in mind than with CORS, you can control what are the authorized methods,
|
||
headers, origins, and if the client is allowed to send authentication
|
||
information or not.</p>
|
||
<h2 id="a-word-about-security">A word about security</h2>
|
||
<p>CORS is not an answer for every cross-domain call you want to do,
|
||
because you need to control the service you want to call. For instance,
|
||
if you want to build a feed reader and access the feeds on different
|
||
domains, you can be pretty much sure that the servers will not implement
|
||
CORS, so you'll need to write a proxy yourself, to provide this.</p>
|
||
<p>Secondly, if misunderstood, CORS can be insecure, and cause problems.
|
||
Because the rules apply when a client wants to do a request to a server,
|
||
you need to be extra careful about who you're authorizing.</p>
|
||
<p>An incorrectly secured CORS server can be accessed by a malicious client
|
||
very easily, bypassing network security. For instance, if you host a
|
||
server on an intranet that is only available from behind a VPN but
|
||
accepts every cross-origin call. A bad guy can inject javascript into
|
||
the browser of a user who has access to your protected server and make
|
||
calls to your service, which is probably not what you want.</p>
|
||
<h2 id="how-this-is-different-from-jsonp">How this is different from JSONP?</h2>
|
||
<p>You may know the <a href="http://en.wikipedia.org/wiki/JSONP">JSONP</a> protocol.
|
||
JSONP allows cross origin, but for a particular use case, and does have
|
||
some drawbacks (for instance, it's not possible to do DELETEs or PUTs
|
||
with JSONP).</p>
|
||
<p>JSONP exploits the fact that it is possible to get information from
|
||
another domain when you are asking for javascript code, using the
|
||
\&lt;script> element.</p>
|
||
<blockquote>
|
||
<p>Exploiting the open policy for \&lt;script> elements, some pages use
|
||
them to retrieve JavaScript code that operates on dynamically
|
||
generated JSON-formatted data from other origins. This usage pattern
|
||
is known as JSONP. Requests for JSONP retrieve not JSON, but arbitrary
|
||
JavaScript code. They are evaluated by the JavaScript interpreter, not
|
||
parsed by a JSON parser.</p>
|
||
</blockquote>
|
||
<h2 id="using-cors-in-cornice">Using CORS in Cornice</h2>
|
||
<p>Okay, things are hopefully clearer about CORS, let's see how we
|
||
implemented it on the server-side.</p>
|
||
<p>Cornice is a toolkit that lets you define resources in python and takes
|
||
care of the heavy lifting for you, so I wanted it to take care of the
|
||
CORS support as well.</p>
|
||
<p>In Cornice, you define a service like this:</p>
|
||
<p>``` sourceCode python
|
||
from cornice import Service</p>
|
||
<p>foobar = Service(name="foobar", path="/foobar")</p>
|
||
<h1 id="and-then-you-do-something-with-it">and then you do something with it</h1>
|
||
<p>@foobar.get()
|
||
def get_foobar(request):
|
||
# do something with the request.</p>
|
||
<div class="highlight"><pre><span></span><span class="k">To</span><span class="w"> </span><span class="k">add</span><span class="w"> </span><span class="n">CORS</span><span class="w"> </span><span class="n">support</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">this</span><span class="w"> </span><span class="n">resource</span><span class="p">,</span><span class="w"> </span><span class="n">you</span><span class="w"> </span><span class="n">can</span><span class="w"> </span><span class="k">go</span><span class="w"> </span><span class="n">this</span><span class="w"> </span><span class="n">way</span><span class="p">,</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="n">the</span><span class="w"></span>
|
||
<span class="n">cors</span><span class="err">\</span><span class="n">_origins</span><span class="w"> </span><span class="k">parameter</span><span class="err">:</span><span class="w"></span>
|
||
|
||
<span class="err">```</span><span class="w"> </span><span class="n">sourceCode</span><span class="w"> </span><span class="n">python</span><span class="w"></span>
|
||
<span class="n">foobar</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Service</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">&#39;foobar&#39;</span><span class="p">,</span><span class="w"> </span><span class="k">path</span><span class="o">=</span><span class="s1">&#39;/foobar&#39;</span><span class="p">,</span><span class="w"> </span><span class="n">cors_origins</span><span class="o">=</span><span class="p">(</span><span class="s1">&#39;*&#39;</span><span class="p">,))</span><span class="w"></span>
|
||
</pre></div>
|
||
|
||
|
||
<p>Ta-da! You have enabled CORS for your service. <strong>Be aware that you're
|
||
authorizing anyone to query your server, that may not be what you
|
||
want.</strong></p>
|
||
<p>Of course, you can specify a list of origins you trust, and you don't
|
||
need to stick with *, which means "authorize everyone".</p>
|
||
<h3 id="headers">Headers</h3>
|
||
<p>You can define the headers you want to expose for the service:</p>
|
||
<p>``` sourceCode python
|
||
foobar = Service(name='foobar', path='/foobar', cors_origins=('*',))</p>
|
||
<p>@foobar.get(cors_headers=('X-My-Header', 'Content-Type'))
|
||
def get_foobars_please(request):
|
||
return "some foobar for you"</p>
|
||
<div class="highlight"><pre><span></span><span class="nv">I</span><span class="s1">&#39;</span><span class="s">ve done some testing and it wasn</span><span class="s1">&#39;</span><span class="nv">t</span> <span class="nv">working</span> <span class="nv">on</span> <span class="nv">Chrome</span> <span class="nv">because</span> <span class="nv">I</span> <span class="nv">wasn</span><span class="s1">&#39;</span><span class="s">t</span>
|
||
<span class="nv">handling</span> <span class="nv">the</span> <span class="nv">headers</span> <span class="nv">the</span> <span class="nv">right</span> <span class="nv">way</span> <span class="ss">(</span><span class="nv">The</span> <span class="nv">missing</span> <span class="nv">one</span> <span class="nv">was</span> <span class="nv">Content</span><span class="o">-</span><span class="nv">Type</span>,
|
||
<span class="nv">that</span> <span class="nv">Chrome</span> <span class="nv">was</span> <span class="nv">asking</span> <span class="k">for</span><span class="ss">)</span>. <span class="nv">With</span> <span class="nv">my</span> <span class="nv">first</span> <span class="nv">version</span> <span class="nv">of</span> <span class="nv">the</span>
|
||
<span class="nv">implementation</span>, <span class="nv">I</span> <span class="nv">needed</span> <span class="nv">the</span> <span class="nv">service</span> <span class="nv">implementers</span> <span class="nv">to</span> <span class="nv">explicitely</span> <span class="nv">list</span>
|
||
<span class="nv">all</span> <span class="nv">the</span> <span class="nv">headers</span> <span class="nv">that</span> <span class="nv">should</span> <span class="nv">be</span> <span class="nv">exposed</span>. <span class="k">While</span> <span class="nv">this</span> <span class="nv">improves</span> <span class="nv">security</span>, <span class="nv">it</span>
|
||
<span class="nv">can</span> <span class="nv">be</span> <span class="nv">frustrating</span> <span class="k">while</span> <span class="nv">developing</span>.
|
||
|
||
<span class="nv">So</span> <span class="nv">I</span> <span class="nv">introduced</span> <span class="nv">an</span> <span class="nv">expose</span>\<span class="nv">_all</span>\<span class="nv">_headers</span> <span class="nv">flag</span>, <span class="nv">which</span> <span class="nv">is</span> <span class="nv">set</span> <span class="nv">to</span> <span class="nv">True</span> <span class="nv">by</span>
|
||
<span class="nv">default</span>, <span class="k">if</span> <span class="nv">the</span> <span class="nv">service</span> <span class="nv">supports</span> <span class="nv">CORS</span>.
|
||
|
||
### <span class="nv">Cookies</span> <span class="o">/</span> <span class="nv">Credentials</span>
|
||
|
||
<span class="nv">By</span> <span class="nv">default</span>, <span class="nv">the</span> <span class="nv">requests</span> <span class="nv">you</span> <span class="k">do</span> <span class="nv">to</span> <span class="nv">your</span> <span class="nv">API</span> <span class="nv">endpoint</span> <span class="nv">don</span><span class="s1">&#39;</span><span class="s">t include the</span>
|
||
<span class="nv">credential</span> <span class="nv">information</span> <span class="k">for</span> <span class="nv">security</span> <span class="nv">reasons</span>. <span class="k">If</span> <span class="nv">you</span> <span class="nv">really</span> <span class="nv">want</span> <span class="nv">to</span> <span class="k">do</span>
|
||
<span class="nv">that</span>, <span class="nv">you</span> <span class="nv">need</span> <span class="nv">to</span> <span class="nv">enable</span> <span class="nv">it</span> <span class="nv">using</span> <span class="nv">the</span> <span class="nv">cors</span>\<span class="nv">_credentials</span> <span class="nv">parameter</span>. <span class="nv">You</span>
|
||
<span class="nv">can</span> <span class="nv">activate</span> <span class="nv">this</span> <span class="nv">one</span> <span class="nv">on</span> <span class="nv">a</span> <span class="nv">per</span><span class="o">-</span><span class="nv">service</span> <span class="nv">basis</span> <span class="nv">or</span> <span class="nv">on</span> <span class="nv">a</span> <span class="nv">per</span><span class="o">-</span><span class="nv">method</span> <span class="nv">basis</span>.
|
||
|
||
### <span class="nv">Caching</span>
|
||
|
||
<span class="nv">When</span> <span class="nv">you</span> <span class="k">do</span> <span class="nv">a</span> <span class="nv">preflight</span> <span class="nv">request</span>, <span class="nv">the</span> <span class="nv">information</span> <span class="nv">returned</span> <span class="nv">by</span> <span class="nv">the</span> <span class="nv">server</span>
|
||
<span class="nv">can</span> <span class="nv">be</span> <span class="nv">cached</span> <span class="nv">by</span> <span class="nv">the</span> <span class="nv">User</span><span class="o">-</span><span class="nv">Agent</span> <span class="nv">so</span> <span class="nv">that</span> <span class="nv">it</span><span class="s1">&#39;</span><span class="s">s not redone before each</span>
|
||
<span class="nv">actual</span> <span class="nv">call</span>.
|
||
|
||
<span class="nv">The</span> <span class="nv">caching</span> <span class="nv">period</span> <span class="nv">is</span> <span class="nv">defined</span> <span class="nv">by</span> <span class="nv">the</span> <span class="nv">server</span>, <span class="nv">using</span> <span class="nv">the</span>
|
||
<span class="nv">Access</span><span class="o">-</span><span class="nv">Control</span><span class="o">-</span><span class="nv">Max</span><span class="o">-</span><span class="nv">Age</span> <span class="nv">header</span>. <span class="nv">You</span> <span class="nv">can</span> <span class="nv">configure</span> <span class="nv">this</span> <span class="nv">timing</span> <span class="nv">using</span> <span class="nv">the</span>
|
||
<span class="nv">cors</span>\<span class="nv">_max</span>\<span class="nv">_age</span> <span class="nv">parameter</span>.
|
||
|
||
### <span class="nv">Simplifying</span> <span class="nv">the</span> <span class="nv">API</span>
|
||
|
||
<span class="nv">We</span> <span class="nv">have</span> <span class="nv">cors</span>\<span class="nv">_headers</span>, <span class="nv">cors</span>\<span class="nv">_enabled</span>, <span class="nv">cors</span>\<span class="nv">_origins</span>, <span class="nv">cors</span>\<span class="nv">_credentials</span>,
|
||
<span class="nv">cors</span>\<span class="nv">_max</span>\<span class="nv">_age</span>, <span class="nv">cors</span>\<span class="nv">_expose</span>\<span class="nv">_all</span>\<span class="nv">_headers</span> … <span class="nv">a</span> <span class="nv">fair</span> <span class="nv">number</span> <span class="nv">of</span>
|
||
<span class="nv">parameters</span>. <span class="k">If</span> <span class="nv">you</span> <span class="nv">want</span> <span class="nv">to</span> <span class="nv">have</span> <span class="nv">a</span> <span class="nv">specific</span> <span class="nv">CORS</span><span class="o">-</span><span class="nv">policy</span> <span class="k">for</span> <span class="nv">your</span>
|
||
<span class="nv">services</span>, <span class="nv">that</span> <span class="nv">can</span> <span class="nv">be</span> <span class="nv">a</span> <span class="nv">bit</span> <span class="nv">tedious</span> <span class="nv">to</span> <span class="nv">pass</span> <span class="nv">these</span> <span class="nv">to</span> <span class="nv">your</span> <span class="nv">services</span> <span class="nv">all</span>
|
||
<span class="nv">the</span> <span class="nv">time</span>.
|
||
|
||
<span class="nv">I</span> <span class="nv">introduced</span> <span class="nv">another</span> <span class="nv">way</span> <span class="nv">to</span> <span class="nv">pass</span> <span class="nv">the</span> <span class="nv">CORS</span> <span class="nv">policy</span>, <span class="nv">so</span> <span class="nv">you</span> <span class="nv">can</span> <span class="k">do</span>
|
||
<span class="nv">something</span> <span class="nv">like</span> <span class="nv">that</span>:
|
||
|
||
``` <span class="nv">sourceCode</span> <span class="nv">python</span>
|
||
<span class="nv">policy</span> <span class="o">=</span> <span class="nv">dict</span><span class="ss">(</span><span class="nv">enabled</span><span class="o">=</span><span class="nv">False</span>,
|
||
<span class="nv">headers</span><span class="o">=</span><span class="ss">(</span><span class="s1">&#39;</span><span class="s">X-My-Header</span><span class="s1">&#39;</span>, <span class="s1">&#39;</span><span class="s">Content-Type</span><span class="s1">&#39;</span><span class="ss">)</span>,
|
||
<span class="nv">origins</span><span class="o">=</span><span class="ss">(</span><span class="s1">&#39;</span><span class="s">*.notmyidea.org</span><span class="s1">&#39;</span><span class="ss">)</span>,
|
||
<span class="nv">credentials</span><span class="o">=</span><span class="nv">True</span>,
|
||
<span class="nv">max_age</span><span class="o">=</span><span class="mi">42</span><span class="ss">)</span>
|
||
|
||
<span class="nv">foobar</span> <span class="o">=</span> <span class="nv">Service</span><span class="ss">(</span><span class="nv">name</span><span class="o">=</span><span class="s1">&#39;</span><span class="s">foobar</span><span class="s1">&#39;</span>, <span class="nv">path</span><span class="o">=</span><span class="s1">&#39;</span><span class="s">/foobar</span><span class="s1">&#39;</span>, <span class="nv">cors_policy</span><span class="o">=</span><span class="nv">policy</span><span class="ss">)</span>
|
||
</pre></div>
|
||
|
||
|
||
<h2 id="comparison-with-other-implementations">Comparison with other implementations</h2>
|
||
<p>I was curious to have a look at other implementations of CORS, in django
|
||
for instance, and I found <a href="https://gist.github.com/426829.js">a gist about
|
||
it</a>.</p>
|
||
<p>Basically, this adds a middleware that adds the "rights" headers to the
|
||
answer, depending on the request.</p>
|
||
<p>While this approach works, it's not implementing the specification
|
||
completely. You need to add support for all the resources at once.</p>
|
||
<p>We can think about a nice way to implement this specifying a definition
|
||
of what's supposed to be exposed via CORS and what shouldn't directly in
|
||
your settings. In my opinion, CORS support should be handled at the
|
||
service definition level, except for the list of authorized hosts.
|
||
Otherwise, you don't know exactly what's going on when you look at the
|
||
definition of the service.</p>
|
||
<h2 id="resources">Resources</h2>
|
||
<p>There are a number of good resources that can be useful to you if you
|
||
want to either understand how CORS works, or if you want to implement it
|
||
yourself.</p>
|
||
<ul>
|
||
<li><a href="http://enable-cors.org/">http://enable-cors.org/</a> is useful to get started when you don't
|
||
know anything about CORS.</li>
|
||
<li>There is a W3C wiki page containing information that may be useful
|
||
about clients, common pitfalls etc:
|
||
<a href="http://www.w3.org/wiki/CORS_Enabled">http://www.w3.org/wiki/CORS_Enabled</a></li>
|
||
<li><em>HTML5 rocks</em> has a tutorial explaining how to implement CORS, with
|
||
<a href="http://www.html5rocks.com/en/tutorials/cors/#toc-adding-cors-support-to-the-server">a nice section about the
|
||
server-side</a>.</li>
|
||
<li>Be sure to have a look at the <a href="http://caniuse.com/#search=cors">clients support-matrix for this
|
||
feature</a>.</li>
|
||
<li>About security, <a href="https://code.google.com/p/html5security/wiki/CrossOriginRequestSecurity">check out this
|
||
page</a></li>
|
||
<li>If you want to have a look at the implementation code, check <a href="https://github.com/mozilla-services/cornice/pull/98/files">on
|
||
github</a></li>
|
||
</ul>
|
||
<p>Of course, the W3C specification is the best resource to rely on. This
|
||
specification isn't hard to read, so you may want to go through it.
|
||
Especially the <a href="http://www.w3.org/TR/cors/#resource-processing-model">"resource processing model"
|
||
section</a></p></content></entry><entry><title>Status board</title><link href="https://blog.notmyidea.org/status-board.html" rel="alternate"></link><published>2012-12-29T00:00:00+01:00</published><updated>2012-12-29T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2012-12-29:/status-board.html</id><summary type="html">
|
||
<p>À force de démarrer des services web pour un oui et pour un non, de
|
||
proposer à des copains d'héberger leurs sites, de faire pareil pour
|
||
quelques assos etc, je me suis retrouvé avec, comme dirait l'autre, <em>une
|
||
bonne platrée</em> de sites et de services à gérer sur lolnet.org …</p></summary><content type="html">
|
||
<p>À force de démarrer des services web pour un oui et pour un non, de
|
||
proposer à des copains d'héberger leurs sites, de faire pareil pour
|
||
quelques assos etc, je me suis retrouvé avec, comme dirait l'autre, <em>une
|
||
bonne platrée</em> de sites et de services à gérer sur lolnet.org, mon
|
||
serveur.</p>
|
||
<p>Jusqu'à très récemment, rien de tout ça n'était sauvegardé, et non plus
|
||
monitoré. Après quelques recherches, je suis tombé sur
|
||
<a href="http://www.stashboard.org/">stashboard</a>, un "status board" qu'il est
|
||
bien fait. Le seul problème, c'est écrit pour se lancer sur GAE, <em>Google
|
||
App Engine</em>. Heureusement, c'est open-source, et ça a été forké pour
|
||
donner naissance à
|
||
<a href="https://github.com/bfirsh/whiskerboard">whiskerboard</a> (la planche
|
||
moustachue, pour les non anglophones).</p>
|
||
<p><img alt="Capture d'écran du site." src="images/status_board.png"></p>
|
||
<h2 id="verifier-le-statut-des-services">Vérifier le statut des services</h2>
|
||
<p>Donc, c'est chouette, c'est facile à installer, tout ça, mais… mais ça ne fait en fait pas ce que je veux: ça ne fait que m'afficher le statut des services, mais ça ne vérifie pas que tout est bien "up".</p>
|
||
<p>Bon, un peu embêtant pour moi, parce que c'est vraiment ça que je voulais. Pas grave, je sais un peu coder, autant que ça serve. J'ai ajouté quelques fonctionnalités au soft, qui sont disponibles sur mon fork, sur github:: <a href="https://github.com/almet/whiskerboard">https://github.com/almet/whiskerboard</a> .</p>
|
||
<p>Entres autres, il est désormais possible de lancer
|
||
<a href="http://celeryproject.org/">celery</a> en tache de fond et de vérifier périodiquement que les services sont toujours bien vivants, en utilisant une tache spécifique.</p>
|
||
<p>C'était un bonheur de développer ça (on a fait ça à deux, avec guillaume, avec un mumble + tmux en pair prog, en une petite soirée, ça dépote).</p>
|
||
<p>Les modifications sont assez simples, vous pouvez aller jeter un œil aux changements ici:
|
||
<a href="https://github.com/almet/whiskerboard/compare/b539337416...master">https://github.com/almet/whiskerboard/compare/b539337416...master</a></p>
|
||
<p>En gros:</p>
|
||
<ul>
|
||
<li>ajout d'une connection_string aux services (de la forme
|
||
protocol://host:port)</li>
|
||
<li>ajout d'une commande check_status qui s'occupe d'itérer sur les
|
||
services et de lancer des taches celery qui vont bien, en fonction
|
||
du protocole</li>
|
||
<li>ajout des taches en question</li>
|
||
</ul>
|
||
<h2 id="deploiement">Déploiement</h2>
|
||
<p>Le plus long a été de le déployer en fin de compte, parce que je ne
|
||
voulais pas déployer mon service de supervision sur mon serveur,
|
||
forcément.</p>
|
||
<p>Après un essai (plutôt rapide en fait) sur <a href="http://heroku.com">heroku</a>,
|
||
je me suis rendu compte qu'il me fallait payer pas loin de 35$ par mois
|
||
pour avoir un process celeryd qui tourne, donc j'ai un peu cherché
|
||
ailleurs, pour finalement déployer la chose chez
|
||
<a href="https://www.alwaysdata.com/">alwaysdata</a></p>
|
||
<p>Après quelques péripéties, j'ai réussi à faire tourner le tout, ça à été
|
||
un peu la bataille au départ pour installer virtualenv (j'ai du faire
|
||
des changements dans mon PATH pour que ça puisse marcher), voici mon
|
||
`.bash_profile`:</p>
|
||
<div class="highlight"><pre><span></span><span class="n">export</span> <span class="n">PYTHONPATH</span><span class="o">=~/</span><span class="n">modules</span><span class="o">/</span>
|
||
<span class="n">export</span> <span class="n">PATH</span><span class="o">=</span><span class="err">$</span><span class="n">HOME</span><span class="o">/</span><span class="n">modules</span><span class="o">/</span><span class="n">bin</span><span class="p">:</span><span class="err">$</span><span class="n">HOME</span><span class="o">/</span><span class="n">modules</span><span class="o">/</span><span class="p">:</span><span class="err">$</span><span class="n">PATH</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>Et après y'a plus qu'à installer avec `easy_install`:</p>
|
||
<div class="highlight"><pre><span></span><span class="n">easy_install</span> <span class="c1">--install-dir ~/modules -U pip</span>
|
||
<span class="n">easy_install</span> <span class="c1">--install-dir ~/modules -U virtualenv</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>Et à créer le virtualenv:</p>
|
||
<div class="highlight"><pre><span></span><span class="n">virtualenv</span> <span class="n">venv</span>
|
||
<span class="n">venv</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">pip</span> <span class="n">install</span> <span class="o">-</span><span class="n">r</span> <span class="n">requirements</span><span class="p">.</span><span class="n">txt</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>Dernière étape, la création d'un fichier application.wsgi qui s'occupe
|
||
de rendre l'application disponible, avec le bon venv:</p>
|
||
<h2 id="ssl-et-requests">SSL et Requests</h2>
|
||
<p>Quelques tours de manivelle plus loin, j'ai un celeryd qui tourne et qui
|
||
consomme les taches qui lui sont envoyées (pour des questions de
|
||
simplicité, j'ai utilisé le backend django de celery, donc pas besoin
|
||
d'AMQP, par exemple).</p>
|
||
<p>Problème, les ressources que je vérifie en SSL (HTTPS) me jettent. Je
|
||
sais pas exactement pourquoi à l'heure qu'il est, mais il semble que
|
||
lorsque je fais une requête avec
|
||
<a href="http://docs.python-requests.org/en/latest/">Requests</a> je me récupère
|
||
des <em>Connection Refused</em>. Peut être une sombre histoire de proxy ? En
|
||
attendant, les appels avec CURL fonctionnent, donc j'ai fait <a href="https://github.com/ametaireau/whiskerboard/blob/master/board/tasks.py#L17">un
|
||
fallback vers CURL lorsque les autres méthodes
|
||
échouent</a>.
|
||
Pas super propre, mais ça fonctionne.</p>
|
||
<p><strong>EDIT</strong> Finalement, il se trouve que mon serveur était mal configuré.
|
||
J'utilisais haproxy + stunnel, et la négiciation SSL se passait mal. Une
|
||
fois SSL et TLS activés, et SSLv2 désactivé, tout fonctionne mieux.</p>
|
||
<h2 id="et-voila">Et voilà</h2>
|
||
<p>Finalement, j'ai mon joli status-board qui tourne à merveille sur
|
||
<a href="http://status.lolnet.org">http://status.lolnet.org</a> :-)</p></content></entry><entry><title>Astuces SSH</title><link href="https://blog.notmyidea.org/astuces-ssh.html" rel="alternate"></link><published>2012-12-27T00:00:00+01:00</published><updated>2012-12-27T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2012-12-27:/astuces-ssh.html</id><summary type="html">
|
||
<h2 id="tunelling">Tunelling</h2>
|
||
<p>Parce que je m'en rapelle jamais (tête de linote):</p>
|
||
<div class="highlight"><pre><span></span>$ ssh -f hote -L local:lolnet.org:destination -N
|
||
</pre></div>
|
||
|
||
|
||
<h2 id="sshconfig">.ssh/config</h2>
|
||
<p>(merci <a href="http://majerti.fr">gaston</a> !)</p>
|
||
<p>La directive suivante dans .ssh/config permet de sauter d'hôte en hôte
|
||
séparés par des "+" :</p>
|
||
<div class="highlight"><pre><span></span><span class="k">Host</span> <span class="o">*+*</span>
|
||
<span class="n">ProxyCommand</span> <span class="n">ssh</span> <span class="err">$</span><span class="p">(</span><span class="n">echo</span> <span class="o">%</span><span class="n">h</span> <span class="o">|</span> <span class="n">sed</span>
|
||
<span class="s1">&#39;s/+[^+]*$//;s/\([^+%%]*\)%%\([^+]*\)$/\2 -l \1/;s …</span></pre></div></summary><content type="html">
|
||
<h2 id="tunelling">Tunelling</h2>
|
||
<p>Parce que je m'en rapelle jamais (tête de linote):</p>
|
||
<div class="highlight"><pre><span></span>$ ssh -f hote -L local:lolnet.org:destination -N
|
||
</pre></div>
|
||
|
||
|
||
<h2 id="sshconfig">.ssh/config</h2>
|
||
<p>(merci <a href="http://majerti.fr">gaston</a> !)</p>
|
||
<p>La directive suivante dans .ssh/config permet de sauter d'hôte en hôte
|
||
séparés par des "+" :</p>
|
||
<div class="highlight"><pre><span></span><span class="k">Host</span> <span class="o">*+*</span>
|
||
<span class="n">ProxyCommand</span> <span class="n">ssh</span> <span class="err">$</span><span class="p">(</span><span class="n">echo</span> <span class="o">%</span><span class="n">h</span> <span class="o">|</span> <span class="n">sed</span>
|
||
<span class="s1">&#39;s/+[^+]*$//;s/\([^+%%]*\)%%\([^+]*\)$/\2 -l \1/;s/:/ -p /&#39;</span><span class="p">)</span>
|
||
<span class="n">PATH</span><span class="o">=</span><span class="p">.:</span><span class="err">\$</span><span class="n">PATH</span> <span class="n">nc</span> <span class="o">-</span><span class="n">w1</span> <span class="err">$</span><span class="p">(</span><span class="n">echo</span> <span class="o">%</span><span class="n">h</span> <span class="o">|</span> <span class="n">sed</span> <span class="s1">&#39;s/^.*+//;/:/!s/$/ %p/;s/:/ /&#39;</span><span class="p">)</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>On peut donc spécifier des "sauts" ssh du style:</p>
|
||
<div class="highlight"><pre><span></span><span class="n">ssh</span> <span class="n">root</span><span class="mf">@91.25.25.25</span><span class="o">+</span><span class="mf">192.168.1.1</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>Ensuite on peut essayer de rajouter:</p>
|
||
<div class="highlight"><pre><span></span><span class="k">Host</span> <span class="o">&lt;</span><span class="n">label_pour_mon_serveur_privé</span><span class="o">&gt;</span>
|
||
<span class="k">user</span> <span class="o">&lt;</span><span class="n">monuser</span><span class="p">(</span><span class="n">root</span><span class="p">)</span><span class="o">&gt;</span>
|
||
<span class="n">IdentityFile</span> <span class="o">&lt;</span><span class="n">chemin</span> <span class="n">vers</span> <span class="n">ma</span> <span class="n">clé</span> <span class="n">ssh</span> <span class="n">pour</span> <span class="n">le</span> <span class="n">serveur</span> <span class="n">publique</span><span class="o">&gt;</span>
|
||
<span class="n">hostname</span> <span class="n">ip_serveur_publique</span><span class="o">+</span><span class="n">ip_serveur_privé</span>
|
||
</pre></div></content></entry><entry><title>Gnome 3, extensions</title><link href="https://blog.notmyidea.org/gnome-3-extensions.html" rel="alternate"></link><published>2012-12-27T00:00:00+01:00</published><updated>2012-12-27T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2012-12-27:/gnome-3-extensions.html</id><summary type="html">
|
||
<p>Après avoir tenté pendant un bout de temps unity, le bureau par defaut
|
||
de ubuntu, j'ai eu envie de changements, et j'ai donc essayé un peu de
|
||
regarder du coté de gnome 3, à nouveau.</p>
|
||
<p>Et finalement, j'ai trouvé quelques extensions qui sont vraiment utiles,
|
||
que je liste ici.</p>
|
||
<ul>
|
||
<li><a href="https://extensions.gnome.org/extension/547/antisocial-menu/">Antisocial …</a></li></ul></summary><content type="html">
|
||
<p>Après avoir tenté pendant un bout de temps unity, le bureau par defaut
|
||
de ubuntu, j'ai eu envie de changements, et j'ai donc essayé un peu de
|
||
regarder du coté de gnome 3, à nouveau.</p>
|
||
<p>Et finalement, j'ai trouvé quelques extensions qui sont vraiment utiles,
|
||
que je liste ici.</p>
|
||
<ul>
|
||
<li><a href="https://extensions.gnome.org/extension/547/antisocial-menu/">Antisocial
|
||
Menu</a>
|
||
vire les boutons et textes en rapport avec le web social. J'en avais
|
||
pas besoin puisque je suis connecté à mon instant messenger dans un
|
||
terminal, en utilisant weechat.</li>
|
||
<li><a href="https://extensions.gnome.org/extension/97/coverflow-alt-tab/">Coverflow
|
||
Alt-Tab</a>
|
||
change le switcher d'applications par defaut. Je le trouve bien plus
|
||
pratique que celui par defaut puisqu'il me permet de voir "en grand"
|
||
quelle est la fenêtre que je vais afficher.</li>
|
||
<li><a href="https://extensions.gnome.org/extension/55/media-player-indicator/">Media player
|
||
indicator</a>
|
||
me permet de voir en temps réel ce qui se passe dans mon lecteur
|
||
audio. Ça semble ne pas être grand chose, mais ça me manquait. Ça
|
||
s'intègre niquel avec Spotify, et ça c'est chouette.</li>
|
||
<li><a href="https://extensions.gnome.org/extension/149/search-firefox-bookmarks-provider/">Rechercher dans les bookmarks
|
||
firefox</a>
|
||
permet de… à votre avis ?</li>
|
||
</ul>
|
||
<p>Un peu moins utile mais sait on jamais:</p>
|
||
<ul>
|
||
<li>“<a href="https://extensions.gnome.org/extension/130/advanced-settings-in-usermenu/">Advanced Settings in
|
||
UserMenu</a>”
|
||
permet d'avoir un raccourci vers les paramètres avancés dans le menu
|
||
utilisateur (en haut à droite)</li>
|
||
<li>Une <a href="https://extensions.gnome.org/extension/409/gtg-integration/">intégration à Getting things
|
||
Gnome</a>
|
||
(un truc de GTD). Je suis en train d'expérimenter avec cet outil,
|
||
donc je ne sais pas encore si ça va rester, mais pourquoi pas.</li>
|
||
</ul>
|
||
<p>Vous pouvez aller faire un tour sur <a href="https://extensions.gnome.org/">https://extensions.gnome.org/</a> pour
|
||
en trouver d'autres à votre gout.</p></content></entry><entry><title>Cheese & code - Wrap-up</title><link href="https://blog.notmyidea.org/cheese-code-wrap-up.html" rel="alternate"></link><published>2012-10-22T00:00:00+02:00</published><updated>2012-10-22T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2012-10-22:/cheese-code-wrap-up.html</id><summary type="html"><h1 id="cheese-code-wrap-up">Cheese &amp; code - Wrap-up</h1>
|
||
<p>This week-end I hosted a <em>cheese &amp; code</em> session in the country-side of
|
||
Angers, France.</p>
|
||
<p>We were a bunch of python hackers and it rained a lot, wich forced us to
|
||
stay inside and to code. Bad.</p>
|
||
<p>We were not enough to get rid of all the cheese …</p></summary><content type="html"><h1 id="cheese-code-wrap-up">Cheese &amp; code - Wrap-up</h1>
|
||
<p>This week-end I hosted a <em>cheese &amp; code</em> session in the country-side of
|
||
Angers, France.</p>
|
||
<p>We were a bunch of python hackers and it rained a lot, wich forced us to
|
||
stay inside and to code. Bad.</p>
|
||
<p>We were not enough to get rid of all the cheese and the awesome meals,
|
||
but well, we finally managed it pretty well.</p>
|
||
<p>Here is a summary of what we worked on:</p>
|
||
<h2 id="daybed">Daybed</h2>
|
||
<p>Daybed started some time ago, and intend to be a replacement to google
|
||
forms, in term of features, but backed as a REST web service, in python,
|
||
and open source.</p>
|
||
<p>In case you wonder, daybed is effectively the name of a couch. We chose
|
||
this name because of the similarities (in the sound) with <strong>db</strong>, and
|
||
because we're using <strong>CouchDB</strong> as a backend.</p>
|
||
<p><img alt="Daybed is a big couch!" src="images/daybed.jpg"></p>
|
||
<p>We mainly hacked on daybed and are pretty close to the release of the
|
||
first version, meaning that we have something working.</p>
|
||
<p><a href="http://github.com/spiral-project/daybed">The code</a> is available on
|
||
github, and we also wrote <a href="http://daybed.rtfd.org">a small
|
||
documentation</a> for it.</p>
|
||
<p>Mainly, we did a lot of cleanup, rewrote a bunch of tests so that it
|
||
would be easier to continue to work on the project, and implemented some
|
||
minor features. I'm pretty confidend that we now have really good basis
|
||
for this project.</p>
|
||
<p>Also, we will have a nice todolist application, with the backend <strong>and</strong>
|
||
the frontend, in javascript / html / css, you'll know more when it'll be
|
||
ready :-)</p>
|
||
<p>Once we have something good enough, we'll release the first version and
|
||
I'll host it somewhere so that people can play with it.</p>
|
||
<h2 id="cornice">Cornice</h2>
|
||
<p>Daybed is built on top of <a href="http://cornice.rtfd.org">Cornice</a>, a
|
||
framework to ease the creation of web-services.</p>
|
||
<p>At Pycon France, we had the opportunity to attend a good presentation
|
||
about <a href="https://github.com/SPORE/specifications">SPORE</a>. SPORE is a way
|
||
to describe your REST web services, as WSDL is for WS-* services. This
|
||
allows to ease the creation of generic SPORE clients, which are able to
|
||
consume any REST API with a SPORE endpoint.</p>
|
||
<p>Here is how you can let cornice describe your web service for you</p>
|
||
<p>``` sourceCode python
|
||
from cornice.ext.spore import generate_spore_description
|
||
from cornice.service import Service, get_services</p>
|
||
<p>spore = Service('spore', path='/spore', renderer='jsonp')
|
||
@spore.get
|
||
def get_spore(request):
|
||
services = get_services()
|
||
return generate_spore_description(services, 'Service name',
|
||
request.application_url, '1.0')</p>
|
||
<div class="highlight"><pre><span></span><span class="nv">And</span> <span class="nv">you</span><span class="s1">&#39;</span><span class="s">ll get a definition of your service, in SPORE, available at</span>
|
||
<span class="o">/</span><span class="nv">spore</span>.
|
||
|
||
<span class="nv">Of</span> <span class="nv">course</span>, <span class="nv">you</span> <span class="nv">can</span> <span class="nv">use</span> <span class="nv">it</span> <span class="nv">to</span> <span class="k">do</span> <span class="nv">other</span> <span class="nv">things</span>, <span class="nv">like</span> <span class="nv">generating</span> <span class="nv">the</span> <span class="nv">file</span>
|
||
<span class="nv">locally</span> <span class="nv">and</span> <span class="nv">exporting</span> <span class="nv">it</span> <span class="nv">wherever</span> <span class="nv">it</span> <span class="nv">makes</span> <span class="nv">sense</span> <span class="nv">to</span> <span class="nv">you</span>, <span class="nv">etc</span>.
|
||
|
||
<span class="nv">I</span> <span class="nv">released</span> <span class="nv">today</span> [<span class="nv">Cornice</span> <span class="mi">0</span>.<span class="mi">11</span>]<span class="ss">(</span><span class="nv">http</span>:<span class="o">//</span><span class="nv">crate</span>.<span class="nv">io</span><span class="o">/</span><span class="nv">packages</span><span class="o">/</span><span class="nv">cornice</span><span class="o">/</span><span class="ss">)</span>,
|
||
<span class="nv">which</span> <span class="nv">adds</span> <span class="nv">into</span> <span class="nv">other</span> <span class="nv">things</span> <span class="nv">the</span> <span class="nv">support</span> <span class="k">for</span> <span class="nv">SPORE</span>, <span class="nv">plus</span> <span class="nv">some</span> <span class="nv">other</span>
|
||
<span class="nv">fixes</span> <span class="nv">we</span> <span class="nv">found</span> <span class="nv">on</span> <span class="nv">our</span> <span class="nv">way</span>.
|
||
|
||
## <span class="nv">Respire</span>
|
||
|
||
<span class="nv">Once</span> <span class="nv">you</span> <span class="nv">have</span> <span class="nv">the</span> <span class="nv">description</span> <span class="nv">of</span> <span class="nv">the</span> <span class="nv">service</span>, <span class="nv">you</span> <span class="nv">can</span> <span class="k">do</span> <span class="nv">generic</span> <span class="nv">clients</span>
|
||
<span class="nv">consuming</span> <span class="nv">them</span>\<span class="o">!</span>
|
||
|
||
<span class="nv">We</span> <span class="nv">first</span> <span class="nv">wanted</span> <span class="nv">to</span> <span class="nv">contribute</span> <span class="nv">to</span> [<span class="nv">spyre</span>]<span class="ss">(</span><span class="nv">https</span>:<span class="o">//</span><span class="nv">github</span>.<span class="nv">com</span><span class="o">/</span><span class="nv">bl0b</span><span class="o">/</span><span class="nv">spyre</span><span class="ss">)</span>
|
||
<span class="nv">but</span> <span class="nv">it</span> <span class="nv">was</span> <span class="nv">written</span> <span class="nv">in</span> <span class="nv">a</span> <span class="nv">way</span> <span class="nv">that</span> <span class="nv">wasn</span><span class="s1">&#39;</span><span class="s">t supporting to POST data, and</span>
|
||
<span class="nv">they</span> <span class="nv">were</span> <span class="nv">using</span> <span class="nv">their</span> <span class="nv">own</span> <span class="nv">stack</span> <span class="nv">to</span> <span class="nv">handle</span> <span class="nv">HTTP</span>. <span class="nv">A</span> <span class="nv">lot</span> <span class="nv">of</span> <span class="nv">code</span> <span class="nv">that</span>
|
||
<span class="nv">already</span> <span class="nv">exists</span> <span class="nv">in</span> <span class="nv">other</span> <span class="nv">libraries</span>.
|
||
|
||
<span class="k">While</span> <span class="nv">waiting</span> <span class="nv">the</span> <span class="nv">train</span> <span class="nv">with</span> [<span class="nv">R</span>é<span class="nv">my</span>]<span class="ss">(</span><span class="nv">http</span>:<span class="o">//</span><span class="nv">natim</span>.<span class="nv">ionyse</span>.<span class="nv">com</span><span class="o">/</span><span class="ss">)</span>, <span class="nv">we</span> <span class="nv">hacked</span>
|
||
<span class="nv">something</span> <span class="nv">together</span>, <span class="nv">named</span> <span class="s2">&quot;</span><span class="s">Respire</span><span class="s2">&quot;</span>, <span class="nv">a</span> <span class="nv">thin</span> <span class="nv">layer</span> <span class="nv">on</span> <span class="nv">top</span> <span class="nv">of</span> <span class="nv">the</span> <span class="nv">awesome</span>
|
||
[<span class="nv">Requests</span>]<span class="ss">(</span><span class="nv">http</span>:<span class="o">//</span><span class="nv">python</span><span class="o">-</span><span class="nv">requests</span>.<span class="nv">org</span><span class="ss">)</span> <span class="nv">library</span>.
|
||
|
||
<span class="nv">We</span> <span class="nv">have</span> <span class="nv">a</span> <span class="nv">first</span> <span class="nv">version</span>, <span class="nv">feel</span> <span class="nv">free</span> <span class="nv">to</span> <span class="nv">have</span> <span class="nv">a</span> <span class="nv">look</span> <span class="nv">at</span> <span class="nv">it</span> <span class="nv">and</span> <span class="nv">provide</span>
|
||
<span class="nv">enhancements</span> <span class="k">if</span> <span class="nv">you</span> <span class="nv">feel</span> <span class="nv">like</span> <span class="nv">it</span>. <span class="nv">We</span><span class="s1">&#39;</span><span class="s">re still hacking on it so it may</span>
|
||
<span class="k">break</span> <span class="ss">(</span><span class="k">for</span> <span class="nv">the</span> <span class="nv">better</span><span class="ss">)</span>, <span class="nv">but</span> <span class="nv">that</span> <span class="nv">had</span> <span class="nv">been</span> <span class="nv">working</span> <span class="nv">pretty</span> <span class="nv">well</span> <span class="k">for</span> <span class="nv">us</span> <span class="nv">so</span>
|
||
<span class="nv">far</span>.
|
||
|
||
<span class="nv">You</span> <span class="nv">can</span> [<span class="nv">find</span> <span class="nv">the</span> <span class="nv">project</span> <span class="nv">on</span>
|
||
<span class="nv">github</span>]<span class="ss">(</span><span class="nv">http</span>:<span class="o">//</span><span class="nv">github</span>.<span class="nv">com</span><span class="o">/</span><span class="nv">spiral</span><span class="o">-</span><span class="nv">project</span><span class="o">/</span><span class="nv">respire</span><span class="ss">)</span>, <span class="nv">but</span> <span class="nv">here</span> <span class="nv">is</span> <span class="nv">how</span> <span class="nv">to</span>
|
||
<span class="nv">use</span> <span class="nv">it</span>, <span class="nv">really</span> <span class="nv">quickly</span> <span class="ss">(</span><span class="nv">these</span> <span class="nv">examples</span> <span class="nv">are</span> <span class="nv">how</span> <span class="nv">to</span> <span class="nv">interact</span> <span class="nv">with</span> <span class="nv">daybed</span><span class="ss">)</span>
|
||
|
||
``` <span class="nv">sourceCode</span> <span class="nv">python</span>
|
||
<span class="o">&gt;&gt;&gt;</span> <span class="nv">from</span> <span class="nv">respire</span> <span class="nv">import</span> <span class="nv">client_from_url</span>
|
||
|
||
<span class="o">&gt;&gt;&gt;</span> # <span class="nv">create</span> <span class="nv">the</span> <span class="nv">client</span> <span class="nv">from</span> <span class="nv">the</span> <span class="nv">SPORE</span> <span class="nv">definition</span>
|
||
<span class="o">&gt;&gt;&gt;</span> <span class="nv">cl</span> <span class="o">=</span> <span class="nv">client_from_url</span><span class="ss">(</span><span class="s1">&#39;</span><span class="s">http://localhost:8000/spore</span><span class="s1">&#39;</span><span class="ss">)</span>
|
||
|
||
<span class="o">&gt;&gt;&gt;</span> # <span class="nv">in</span> <span class="nv">daybed</span>, <span class="nv">create</span> <span class="nv">a</span> <span class="nv">new</span> <span class="nv">definition</span>
|
||
<span class="o">&gt;&gt;&gt;</span> <span class="nv">todo_def</span> <span class="o">=</span> {
|
||
... <span class="s2">&quot;</span><span class="s">title</span><span class="s2">&quot;</span>: <span class="s2">&quot;</span><span class="s">todo</span><span class="s2">&quot;</span>,
|
||
... <span class="s2">&quot;</span><span class="s">description</span><span class="s2">&quot;</span>: <span class="s2">&quot;</span><span class="s">A list of my stuff to do</span><span class="s2">&quot;</span>,
|
||
... <span class="s2">&quot;</span><span class="s">fields</span><span class="s2">&quot;</span>: [
|
||
... {
|
||
... <span class="s2">&quot;</span><span class="s">name</span><span class="s2">&quot;</span>: <span class="s2">&quot;</span><span class="s">item</span><span class="s2">&quot;</span>,
|
||
... <span class="s2">&quot;</span><span class="s">type</span><span class="s2">&quot;</span>: <span class="s2">&quot;</span><span class="s">string</span><span class="s2">&quot;</span>,
|
||
... <span class="s2">&quot;</span><span class="s">description</span><span class="s2">&quot;</span>: <span class="s2">&quot;</span><span class="s">The item</span><span class="s2">&quot;</span>
|
||
... },
|
||
... {
|
||
... <span class="s2">&quot;</span><span class="s">name</span><span class="s2">&quot;</span>: <span class="s2">&quot;</span><span class="s">status</span><span class="s2">&quot;</span>,
|
||
... <span class="s2">&quot;</span><span class="s">type</span><span class="s2">&quot;</span>: <span class="s2">&quot;</span><span class="s">enum</span><span class="s2">&quot;</span>,
|
||
... <span class="s2">&quot;</span><span class="s">choices</span><span class="s2">&quot;</span>: [
|
||
... <span class="s2">&quot;</span><span class="s">done</span><span class="s2">&quot;</span>,
|
||
... <span class="s2">&quot;</span><span class="s">todo</span><span class="s2">&quot;</span>
|
||
... ],
|
||
... <span class="s2">&quot;</span><span class="s">description</span><span class="s2">&quot;</span>: <span class="s2">&quot;</span><span class="s">is it done or not</span><span class="s2">&quot;</span>
|
||
... }
|
||
... ]}
|
||
<span class="o">&gt;&gt;&gt;</span> <span class="nv">cl</span>.<span class="nv">put_definition</span><span class="ss">(</span><span class="nv">model_name</span><span class="o">=</span><span class="s1">&#39;</span><span class="s">todo</span><span class="s1">&#39;</span>, <span class="nv">data</span><span class="o">=</span><span class="nv">todo_def</span><span class="ss">)</span>
|
||
<span class="o">&gt;&gt;&gt;</span> <span class="nv">cl</span>.<span class="nv">post_data</span><span class="ss">(</span><span class="nv">model_name</span><span class="o">=</span><span class="s1">&#39;</span><span class="s">todo</span><span class="s1">&#39;</span>, <span class="nv">data</span><span class="o">=</span><span class="nv">dict</span><span class="ss">(</span><span class="nv">item</span><span class="o">=</span><span class="s1">&#39;</span><span class="s">make it work</span><span class="s1">&#39;</span>, <span class="nv">status</span><span class="o">=</span><span class="s1">&#39;</span><span class="s">todo</span><span class="s1">&#39;</span><span class="ss">))</span>
|
||
{<span class="nv">u</span><span class="s1">&#39;</span><span class="s">id</span><span class="s1">&#39;</span>: <span class="nv">u</span><span class="s1">&#39;</span><span class="s">9f2c90c0529a442cfdc03c191b022cf7</span><span class="s1">&#39;</span>}
|
||
<span class="o">&gt;&gt;&gt;</span> <span class="nv">cl</span>.<span class="nv">get_data</span><span class="ss">(</span><span class="nv">model_name</span><span class="o">=</span><span class="s1">&#39;</span><span class="s">todo</span><span class="s1">&#39;</span><span class="ss">)</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>Finally, we were out of cheese so everyone headed back to their
|
||
respective houses and cities.</p>
|
||
<p>Until next time?</p></content></entry><entry><title>Circus sprint at PyconFR</title><link href="https://blog.notmyidea.org/circus-sprint-at-pyconfr.html" rel="alternate"></link><published>2012-09-17T00:00:00+02:00</published><updated>2012-09-17T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2012-09-17:/circus-sprint-at-pyconfr.html</id><summary type="html">
|
||
<p>Last Thursday to Sunday, <a href="http://pycon.fr">Pycon France</a> took place, in
|
||
Paris. It was the opportunity to meet a lot of people and to talk about
|
||
python awesomness in general.</p>
|
||
<p>We had three tracks this year, plus sprints the two first days. We
|
||
sprinted on <a href="http://circus.io">Circus</a>, the process and socket manager
|
||
we're …</p></summary><content type="html">
|
||
<p>Last Thursday to Sunday, <a href="http://pycon.fr">Pycon France</a> took place, in
|
||
Paris. It was the opportunity to meet a lot of people and to talk about
|
||
python awesomness in general.</p>
|
||
<p>We had three tracks this year, plus sprints the two first days. We
|
||
sprinted on <a href="http://circus.io">Circus</a>, the process and socket manager
|
||
we're using at Mozilla for some of our setups.</p>
|
||
<p>The project gathered some interest, and we ended up with 5 persons
|
||
working on it. Of course, we spent some time explaining what is Circus,
|
||
how it had been built, a lot of time talking about use-cases and
|
||
possible improvements, but we also managed to add new features.</p>
|
||
<p>Having people wanting to sprint on our projects is exciting because
|
||
that's when making things in the open unleashes its full potential. You
|
||
can't imagine how happy I was to have some friends come and work on this
|
||
with us :)</p>
|
||
<p>Here is a wrap-up of the sprint:</p>
|
||
<h2 id="autocompletion-on-the-command-line">Autocompletion on the command-line</h2>
|
||
<p><a href="http://natim.ionyse.com">Remy Hubscher</a> worked on the command-line
|
||
autocompletion. Now we have a fancy command-line interface which is able
|
||
to aucomplete if you're using bash. It seems that not that much work is
|
||
needed to make it happen on zsh as well :)</p>
|
||
<p><a href="https://github.com/mozilla-services/circus/blob/master/extras/circusctl_bash_completion">Have a look at the
|
||
feature</a></p>
|
||
<p>On the same topic, we now have a cool shell for Circus. If you start the
|
||
circusctl command without any option, you'll end-up with a cool shell.
|
||
Thanks <a href="https://github.com/jojax">Jonathan Dorival</a> for the work on
|
||
this! You can have a look at <a href="https://github.com/mozilla-services/circus/pull/268">the pull
|
||
request</a>.</p>
|
||
<h2 id="future-changes-to-the-web-ui">Future changes to the web ui</h2>
|
||
<p><a href="https://twitter.com/rachbelaid">Rachid Belaid</a> had a deep look at the
|
||
source code and is much more familiarized to it now than before. We
|
||
discussed the possibility to change the implementation of the web ui,
|
||
and I'm glad of this. Currently, it's done with bottle.py and we want to
|
||
switch to pyramid.</p>
|
||
<p>He fixed some issues that were in the tracker, so we now can have the
|
||
age of watchers in the webui, for instance.</p>
|
||
<h2 id="bug-and-doc-fixing">Bug and doc fixing</h2>
|
||
<p>While reading the source code, we found some inconsistencies and fixed
|
||
them, with <a href="http://mathieu.agopian.info/">Mathieu Agopian</a>. We also
|
||
tried to improve the documentation at different levels.</p>
|
||
<p>Documentation still needs a lot of love, and I'm planning to spend some
|
||
time on this shortly. I've gathered a bunch of feedback on this</p>
|
||
<h2 id="circus-clustering-capabilities">Circus clustering capabilities</h2>
|
||
<p>One feature I wanted to work on during this sprint was the clustering
|
||
abilities of Circus. Nick Pellegrino made an internship on this topic at
|
||
Mozilla so we spent some time to review his pull requests.</p>
|
||
<p>A lot of code was written for this so we discussed a bunch of things
|
||
regarding all of this. It took us more time than expected (and I still
|
||
need to spend more time on this to provide appropriate feedback), but it
|
||
allowed us to have a starting-point about what this clustering thing
|
||
could be.</p>
|
||
<p>Remy wrote <a href="http://tech.novapost.fr/circus-clustering-management-en.html">a good summary about our
|
||
brainstorming</a>
|
||
so I'll not do it again here, but feel free to contact us if you have
|
||
ideas on this, they're very welcome!</p>
|
||
<h2 id="project-management">Project management</h2>
|
||
<p>We've had some inquiries telling us that's not as easy as it should to
|
||
get started with the Circus project. Some of the reasons are that we
|
||
don't have any release schedule, and that the documentation is hairy
|
||
enough to lost people, at some point :)</p>
|
||
<p>That's something we'll try to fix soon :)</p>
|
||
<p>PyconFR was a very enjoyable event. I'm looking forward to meet the
|
||
community again and discuss how Circus can evolve in ways that are
|
||
interesting to everyone.</p>
|
||
<p>Tarek and me are going to <a href="http://python.ie/pycon/2012/">Pycon ireland</a>,
|
||
feel free to reach us if you're going there, we'll be happy to meet and
|
||
enjoy beers!</p></content></entry><entry><title>Refactoring Cornice</title><link href="https://blog.notmyidea.org/refactoring-cornice.html" rel="alternate"></link><published>2012-05-01T00:00:00+02:00</published><updated>2012-05-01T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2012-05-01:/refactoring-cornice.html</id><summary type="html">
|
||
<p>After working for a while with <a href="http://cornice.readthedocs.com">Cornice</a>
|
||
to define our APIs at <a href="http://docs.services.mozilla.com">Services</a>, it
|
||
turned out that the current implementation wasn't flexible enough to
|
||
allow us to do what we wanted to do.</p>
|
||
<p>Cornice started as a toolkit on top of the
|
||
<a href="http://docs.pylonsproject.org/en/latest/docs/pyramid.html">pyramid</a>
|
||
routing system, allowing to register services in …</p></summary><content type="html">
|
||
<p>After working for a while with <a href="http://cornice.readthedocs.com">Cornice</a>
|
||
to define our APIs at <a href="http://docs.services.mozilla.com">Services</a>, it
|
||
turned out that the current implementation wasn't flexible enough to
|
||
allow us to do what we wanted to do.</p>
|
||
<p>Cornice started as a toolkit on top of the
|
||
<a href="http://docs.pylonsproject.org/en/latest/docs/pyramid.html">pyramid</a>
|
||
routing system, allowing to register services in a simpler way. Then we
|
||
added some niceties such as the ability to automatically generate the
|
||
services documentation or returning the correct HTTP headers <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html">as defined
|
||
by the HTTP
|
||
specification</a>
|
||
without the need from the developer to deal with them nor to know them.</p>
|
||
<p>If you're not familiar with Cornice, here is how you define a simple
|
||
service with it:</p>
|
||
<p>``` sourceCode python
|
||
from cornice.service import Service
|
||
bar = Service(path="/bar")</p>
|
||
<p>@bar.get(validators=validators, accept='application/json')
|
||
def get_drink(request):
|
||
# do something with the request (with moderation).</p>
|
||
<div class="highlight"><pre><span></span><span class="nv">This</span> <span class="nv">external</span> <span class="nv">API</span> <span class="nv">is</span> <span class="nv">quite</span> <span class="nv">cool</span>, <span class="nv">as</span> <span class="nv">it</span> <span class="nv">allows</span> <span class="nv">to</span> <span class="k">do</span> <span class="nv">a</span> <span class="nv">bunch</span> <span class="nv">of</span> <span class="nv">things</span>
|
||
<span class="nv">quite</span> <span class="nv">easily</span>. <span class="k">For</span> <span class="nv">instance</span>, <span class="nv">we</span><span class="s1">&#39;</span><span class="s">ve written our</span>
|
||
[<span class="nv">token</span><span class="o">-</span><span class="nv">server</span>]<span class="ss">(</span><span class="nv">https</span>:<span class="o">//</span><span class="nv">github</span>.<span class="nv">com</span><span class="o">/</span><span class="nv">mozilla</span><span class="o">-</span><span class="nv">services</span><span class="o">/</span><span class="nv">tokenserver</span><span class="ss">)</span> <span class="nv">code</span> <span class="nv">on</span>
|
||
<span class="nv">top</span> <span class="nv">of</span> <span class="nv">this</span> <span class="nv">in</span> <span class="nv">a</span> <span class="nv">blast</span>.
|
||
|
||
## <span class="nv">The</span> <span class="nv">burden</span>
|
||
|
||
<span class="nv">The</span> <span class="nv">problem</span> <span class="nv">with</span> <span class="nv">this</span> <span class="nv">was</span> <span class="nv">that</span> <span class="nv">we</span> <span class="nv">were</span> <span class="nv">mixing</span> <span class="nv">internally</span> <span class="nv">the</span> <span class="nv">service</span>
|
||
<span class="nv">description</span> <span class="nv">logic</span> <span class="nv">with</span> <span class="nv">the</span> <span class="nv">route</span> <span class="nv">registration</span> <span class="nv">one</span>. <span class="nv">The</span> <span class="nv">way</span> <span class="nv">we</span> <span class="nv">were</span> <span class="nv">doing</span>
|
||
<span class="nv">this</span> <span class="nv">was</span> <span class="nv">via</span> <span class="nv">an</span> <span class="nv">extensive</span> <span class="nv">use</span> <span class="nv">of</span> <span class="nv">decorators</span> <span class="nv">internally</span>.
|
||
|
||
<span class="nv">The</span> <span class="nv">API</span> <span class="nv">of</span> <span class="nv">the</span> <span class="nv">cornice</span>.<span class="nv">service</span>.<span class="nv">Service</span> <span class="nv">class</span> <span class="nv">was</span> <span class="nv">as</span> <span class="nv">following</span>
|
||
<span class="ss">(</span><span class="nv">simplified</span> <span class="nv">so</span> <span class="nv">you</span> <span class="nv">can</span> <span class="nv">get</span> <span class="nv">the</span> <span class="nv">gist</span> <span class="nv">of</span> <span class="nv">it</span><span class="ss">)</span>.
|
||
|
||
``` <span class="nv">sourceCode</span> <span class="nv">python</span>
|
||
<span class="nv">class</span> <span class="nv">Service</span><span class="ss">(</span><span class="nv">object</span><span class="ss">)</span>:
|
||
|
||
<span class="nv">def</span> <span class="nv">__init__</span><span class="ss">(</span><span class="nv">self</span>, <span class="o">**</span><span class="nv">service_kwargs</span><span class="ss">)</span>:
|
||
# <span class="nv">some</span> <span class="nv">information</span>, <span class="nv">such</span> <span class="nv">as</span> <span class="nv">the</span> <span class="nv">colander</span> <span class="nv">schemas</span> <span class="ss">(</span><span class="k">for</span> <span class="nv">validation</span><span class="ss">)</span>,
|
||
# <span class="nv">the</span> <span class="nv">defined</span> <span class="nv">methods</span> <span class="nv">that</span> <span class="nv">had</span> <span class="nv">been</span> <span class="nv">registered</span> <span class="k">for</span> <span class="nv">this</span> <span class="nv">service</span> <span class="nv">and</span>
|
||
# <span class="nv">some</span> <span class="nv">other</span> <span class="nv">things</span> <span class="nv">were</span> <span class="nv">registered</span> <span class="nv">as</span> <span class="nv">instance</span> <span class="nv">variables</span>.
|
||
<span class="nv">self</span>.<span class="nv">schemas</span> <span class="o">=</span> <span class="nv">service_kwargs</span>.<span class="nv">get</span><span class="ss">(</span><span class="nv">schema</span><span class="s1">&#39;</span><span class="s">, None)</span>
|
||
<span class="nv">self</span>.<span class="nv">defined_methods</span> <span class="o">=</span> []
|
||
<span class="nv">self</span>.<span class="nv">definitions</span> <span class="o">=</span> []
|
||
|
||
<span class="nv">def</span> <span class="nv">api</span><span class="ss">(</span><span class="nv">self</span>, <span class="o">**</span><span class="nv">view_kwargs</span><span class="ss">)</span>:
|
||
<span class="s2">&quot;&quot;&quot;</span><span class="s">This method is a decorator that is being used by some alias</span>
|
||
<span class="nv">methods</span>.
|
||
<span class="s2">&quot;&quot;&quot;</span>
|
||
<span class="nv">def</span> <span class="nv">wrapper</span><span class="ss">(</span><span class="nv">view</span><span class="ss">)</span>:
|
||
# <span class="nv">all</span> <span class="nv">the</span> <span class="nv">logic</span> <span class="nv">goes</span> <span class="nv">here</span>. <span class="nv">And</span> <span class="nv">when</span> <span class="nv">I</span> <span class="nv">mean</span> <span class="nv">all</span> <span class="nv">the</span> <span class="nv">logic</span>, <span class="nv">I</span>
|
||
# <span class="nv">mean</span> <span class="nv">it</span>.
|
||
# <span class="mi">1</span>. <span class="nv">we</span> <span class="nv">are</span> <span class="nv">registering</span> <span class="nv">a</span> <span class="nv">callback</span> <span class="nv">to</span> <span class="nv">the</span> <span class="nv">pyramid</span> <span class="nv">routing</span>
|
||
# <span class="nv">system</span> <span class="nv">so</span> <span class="nv">it</span> <span class="nv">gets</span> <span class="nv">called</span> <span class="nv">whenever</span> <span class="nv">the</span> <span class="nv">module</span> <span class="nv">using</span> <span class="nv">the</span>
|
||
# <span class="nv">decorator</span> <span class="nv">is</span> <span class="nv">used</span>.
|
||
# <span class="mi">2</span>. <span class="nv">we</span> <span class="nv">are</span> <span class="nv">transforming</span> <span class="nv">the</span> <span class="nv">passed</span> <span class="nv">arguments</span> <span class="nv">so</span> <span class="nv">they</span> <span class="nv">conform</span>
|
||
# <span class="nv">to</span> <span class="nv">what</span> <span class="nv">is</span> <span class="nv">expected</span> <span class="nv">by</span> <span class="nv">the</span> <span class="nv">pyramid</span> <span class="nv">routing</span> <span class="nv">system</span>.
|
||
# <span class="mi">3</span>. <span class="nv">We</span> <span class="nv">are</span> <span class="nv">storing</span> <span class="nv">some</span> <span class="nv">of</span> <span class="nv">the</span> <span class="nv">passed</span> <span class="nv">arguments</span> <span class="nv">into</span> <span class="nv">the</span>
|
||
# <span class="nv">object</span> <span class="nv">so</span> <span class="nv">we</span> <span class="nv">can</span> <span class="nv">retrieve</span> <span class="nv">them</span> <span class="nv">later</span> <span class="nv">on</span>.
|
||
# <span class="mi">4</span>. <span class="nv">Also</span>, <span class="nv">we</span> <span class="nv">are</span> <span class="nv">transforming</span> <span class="nv">the</span> <span class="nv">passed</span> <span class="nv">view</span> <span class="nv">before</span>
|
||
# <span class="nv">registering</span> <span class="nv">it</span> <span class="nv">in</span> <span class="nv">the</span> <span class="nv">pyramid</span> <span class="nv">routing</span> <span class="nv">system</span> <span class="nv">so</span> <span class="nv">that</span> <span class="nv">it</span>
|
||
# <span class="nv">can</span> <span class="k">do</span> <span class="nv">what</span> <span class="nv">Cornice</span> <span class="nv">wants</span> <span class="nv">it</span> <span class="nv">to</span> <span class="k">do</span> <span class="ss">(</span><span class="nv">checking</span> <span class="nv">some</span> <span class="nv">rules</span>,
|
||
# <span class="nv">applying</span> <span class="nv">validators</span> <span class="nv">and</span> <span class="nv">filters</span> <span class="nv">etc</span>.
|
||
<span class="k">return</span> <span class="nv">wrapper</span>
|
||
|
||
<span class="nv">def</span> <span class="nv">get</span><span class="ss">(</span><span class="nv">self</span>, <span class="o">**</span><span class="nv">kwargs</span><span class="ss">)</span>:
|
||
<span class="s2">&quot;&quot;&quot;</span><span class="s">A shortcut of the api decorator</span><span class="s2">&quot;&quot;&quot;</span>
|
||
<span class="k">return</span> <span class="nv">self</span>.<span class="nv">api</span><span class="ss">(</span><span class="nv">request_method</span><span class="o">=</span><span class="s2">&quot;</span><span class="s">GET</span><span class="s2">&quot;</span>, <span class="o">**</span><span class="nv">kwargs</span><span class="ss">)</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>I encourage you to go read <a href="https://github.com/mozilla-services/cornice/blob/4e0392a2ae137b6a11690459bcafd7325e86fa9e/cornice/service.py#L44">the entire
|
||
file</a>.
|
||
on github so you can get a better opinion on how all of this was done.</p>
|
||
<p>A bunch of things are wrong:</p>
|
||
<ul>
|
||
<li>first, we are not separating the description logic from the
|
||
registration one. This causes problems when we need to access the
|
||
parameters passed to the service, because the parameters you get are
|
||
not exactly the ones you passed but the ones that the pyramid
|
||
routing system is expecting. For instance, if you want to get the
|
||
view get_drink, you will instead get a decorator which contains
|
||
this view.</li>
|
||
<li>second, we are using decorators as APIs we expose. Even if
|
||
decorators are good as shortcuts, they shouldn't be the default way
|
||
to deal with an API. A good example of this is <a href="https://github.com/mozilla-services/cornice/blob/4e0392a2ae137b6a11690459bcafd7325e86fa9e/cornice/resource.py#L56">how the resource
|
||
module consumes this
|
||
API</a>.
|
||
This is quite hard to follow.</li>
|
||
<li>Third, in the api method, a bunch of things are done regarding
|
||
inheritance of parameters that are passed to the service or to its
|
||
decorator methods. This leaves you with a really hard to follow path
|
||
when it comes to add new parameters to your API.</li>
|
||
</ul>
|
||
<h2 id="how-do-we-improve-this">How do we improve this?</h2>
|
||
<p>Python is great because it allows you to refactor things in an easy way.
|
||
What I did isn't breaking our APIs, but make things way simpler to
|
||
hack-on. One example is that it allowed me to add features that we
|
||
wanted to bring to Cornice really quickly (a matter of minutes), without
|
||
touching the API that much.</p>
|
||
<p>Here is the gist of the new architecture:</p>
|
||
<p>``` sourceCode python
|
||
class Service(object):
|
||
# we define class-level variables that will be the default values for
|
||
# this service. This makes things more extensible than it was before.
|
||
renderer = 'simplejson'
|
||
default_validators = DEFAULT_VALIDATORS
|
||
default_filters = DEFAULT_FILTERS</p>
|
||
<div class="highlight"><pre><span></span># <span class="nv">we</span> <span class="nv">also</span> <span class="nv">have</span> <span class="nv">some</span> <span class="nv">class</span><span class="o">-</span><span class="nv">level</span> <span class="nv">parameters</span> <span class="nv">that</span> <span class="nv">are</span> <span class="nv">useful</span> <span class="nv">to</span> <span class="nv">know</span>
|
||
# <span class="nv">which</span> <span class="nv">parameters</span> <span class="nv">are</span> <span class="nv">supposed</span> <span class="nv">to</span> <span class="nv">be</span> <span class="nv">lists</span> <span class="ss">(</span><span class="nv">and</span> <span class="nv">so</span> <span class="nv">converted</span> <span class="nv">as</span> <span class="nv">such</span><span class="ss">)</span>
|
||
# <span class="nv">or</span> <span class="nv">which</span> <span class="nv">are</span> <span class="nv">mandatory</span>.
|
||
<span class="nv">mandatory_arguments</span> <span class="o">=</span> <span class="ss">(</span><span class="s1">&#39;</span><span class="s">renderer</span><span class="s1">&#39;</span>,<span class="ss">)</span>
|
||
<span class="nv">list_arguments</span> <span class="o">=</span> <span class="ss">(</span><span class="s1">&#39;</span><span class="s">validators</span><span class="s1">&#39;</span>, <span class="s1">&#39;</span><span class="s">filters</span><span class="s1">&#39;</span><span class="ss">)</span>
|
||
|
||
<span class="nv">def</span> <span class="nv">__init__</span><span class="ss">(</span><span class="nv">self</span>, <span class="nv">name</span>, <span class="nv">path</span>, <span class="nv">description</span><span class="o">=</span><span class="nv">None</span>, <span class="o">**</span><span class="nv">kw</span><span class="ss">)</span>:
|
||
# <span class="nv">setup</span> <span class="nv">name</span>, <span class="nv">path</span> <span class="nv">and</span> <span class="nv">description</span> <span class="nv">as</span> <span class="nv">instance</span> <span class="nv">variables</span>
|
||
<span class="nv">self</span>.<span class="nv">name</span> <span class="o">=</span> <span class="nv">name</span>
|
||
<span class="nv">self</span>.<span class="nv">path</span> <span class="o">=</span> <span class="nv">path</span>
|
||
<span class="nv">self</span>.<span class="nv">description</span> <span class="o">=</span> <span class="nv">description</span>
|
||
|
||
# <span class="nv">convert</span> <span class="nv">the</span> <span class="nv">arguments</span> <span class="nv">passed</span> <span class="nv">to</span> <span class="nv">something</span> <span class="nv">we</span> <span class="nv">want</span> <span class="nv">to</span> <span class="nv">store</span>
|
||
# <span class="nv">and</span> <span class="k">then</span> <span class="nv">store</span> <span class="nv">them</span> <span class="nv">as</span> <span class="nv">attributes</span> <span class="nv">of</span> <span class="nv">the</span> <span class="nv">instance</span> <span class="ss">(</span><span class="nv">because</span> <span class="nv">they</span>
|
||
# <span class="nv">were</span> <span class="nv">passed</span> <span class="nv">to</span> <span class="nv">the</span> <span class="nv">constructor</span>
|
||
<span class="nv">self</span>.<span class="nv">arguments</span> <span class="o">=</span> <span class="nv">self</span>.<span class="nv">get_arguments</span><span class="ss">(</span><span class="nv">kw</span><span class="ss">)</span>
|
||
<span class="k">for</span> <span class="nv">key</span>, <span class="nv">value</span> <span class="nv">in</span> <span class="nv">self</span>.<span class="nv">arguments</span>.<span class="nv">items</span><span class="ss">()</span>:
|
||
<span class="nv">setattr</span><span class="ss">(</span><span class="nv">self</span>, <span class="nv">key</span>, <span class="nv">value</span><span class="ss">)</span>
|
||
|
||
# <span class="nv">we</span> <span class="nv">keep</span> <span class="nv">having</span> <span class="nv">the</span> <span class="nv">defined_methods</span> <span class="nv">tuple</span> <span class="nv">and</span> <span class="nv">the</span> <span class="nv">list</span> <span class="nv">of</span>
|
||
# <span class="nv">definitions</span> <span class="nv">that</span> <span class="nv">are</span> <span class="nv">done</span> <span class="k">for</span> <span class="nv">this</span> <span class="nv">service</span>
|
||
<span class="nv">self</span>.<span class="nv">defined_methods</span> <span class="o">=</span> []
|
||
<span class="nv">self</span>.<span class="nv">definitions</span> <span class="o">=</span> []
|
||
|
||
<span class="nv">def</span> <span class="nv">get_arguments</span><span class="ss">(</span><span class="nv">self</span>, <span class="nv">conf</span><span class="o">=</span><span class="nv">None</span><span class="ss">)</span>:
|
||
<span class="s2">&quot;&quot;&quot;</span><span class="s">Returns a dict of arguments. It does all the conversions for</span>
|
||
<span class="nv">you</span>, <span class="nv">and</span> <span class="nv">uses</span> <span class="nv">the</span> <span class="nv">information</span> <span class="nv">that</span> <span class="nv">were</span> <span class="nv">defined</span> <span class="nv">at</span> <span class="nv">the</span> <span class="nv">instance</span>
|
||
<span class="nv">level</span> <span class="nv">as</span> <span class="nv">fallbacks</span>.
|
||
<span class="s2">&quot;&quot;&quot;</span>
|
||
|
||
<span class="nv">def</span> <span class="nv">add_view</span><span class="ss">(</span><span class="nv">self</span>, <span class="nv">method</span>, <span class="nv">view</span>, <span class="o">**</span><span class="nv">kwargs</span><span class="ss">)</span>:
|
||
<span class="s2">&quot;&quot;&quot;</span><span class="s">Add a view to this service.</span><span class="s2">&quot;&quot;&quot;</span>
|
||
# <span class="nv">this</span> <span class="nv">is</span> <span class="nv">really</span> <span class="nv">simple</span> <span class="nv">and</span> <span class="nv">looks</span> <span class="nv">a</span> <span class="nv">lot</span> <span class="nv">like</span> <span class="nv">this</span>
|
||
<span class="nv">method</span> <span class="o">=</span> <span class="nv">method</span>.<span class="nv">upper</span><span class="ss">()</span>
|
||
<span class="nv">self</span>.<span class="nv">definitions</span>.<span class="nv">append</span><span class="ss">((</span><span class="nv">method</span>, <span class="nv">view</span>, <span class="nv">args</span><span class="ss">))</span>
|
||
<span class="k">if</span> <span class="nv">method</span> <span class="nv">not</span> <span class="nv">in</span> <span class="nv">self</span>.<span class="nv">defined_methods</span>:
|
||
<span class="nv">self</span>.<span class="nv">defined_methods</span>.<span class="nv">append</span><span class="ss">(</span><span class="nv">method</span><span class="ss">)</span>
|
||
|
||
<span class="nv">def</span> <span class="nv">decorator</span><span class="ss">(</span><span class="nv">self</span>, <span class="nv">method</span>, <span class="o">**</span><span class="nv">kwargs</span><span class="ss">)</span>:
|
||
<span class="s2">&quot;&quot;&quot;</span><span class="s">This is only another interface to the add_view method, exposing a</span>
|
||
<span class="nv">decorator</span> <span class="nv">interface</span><span class="s2">&quot;&quot;&quot;</span>
|
||
<span class="nv">def</span> <span class="nv">wrapper</span><span class="ss">(</span><span class="nv">view</span><span class="ss">)</span>:
|
||
<span class="nv">self</span>.<span class="nv">add_view</span><span class="ss">(</span><span class="nv">method</span>, <span class="nv">view</span>, <span class="o">**</span><span class="nv">kwargs</span><span class="ss">)</span>
|
||
<span class="k">return</span> <span class="nv">view</span>
|
||
<span class="k">return</span> <span class="nv">wrapper</span>
|
||
</pre></div>
|
||
|
||
|
||
<div class="highlight"><pre><span></span><span class="n">So</span><span class="p">,</span> <span class="n">the</span> <span class="n">service</span> <span class="k">is</span> <span class="n">now</span> <span class="k">only</span> <span class="n">storing</span> <span class="n">the</span> <span class="n">information</span> <span class="n">that</span><span class="err">&#39;</span><span class="n">s</span> <span class="n">passed</span> <span class="k">to</span> <span class="n">it</span>
|
||
<span class="k">and</span> <span class="k">nothing</span> <span class="k">more</span><span class="p">.</span> <span class="k">No</span> <span class="k">more</span> <span class="n">route</span> <span class="n">registration</span> <span class="n">logic</span> <span class="n">goes</span> <span class="n">here</span><span class="p">.</span> <span class="k">Instead</span><span class="p">,</span> <span class="n">I</span>
|
||
<span class="n">added</span> <span class="n">this</span> <span class="k">as</span> <span class="n">another</span> <span class="n">feature</span><span class="p">,</span> <span class="n">even</span> <span class="k">in</span> <span class="n">a</span> <span class="n">different</span> <span class="n">module</span><span class="p">.</span> <span class="n">The</span> <span class="k">function</span>
|
||
<span class="k">is</span> <span class="n">named</span> <span class="n">register</span><span class="err">\</span><span class="n">_service</span><span class="err">\</span><span class="n">_views</span> <span class="k">and</span> <span class="n">has</span> <span class="n">the</span> <span class="n">following</span> <span class="n">signature</span><span class="p">:</span>
|
||
|
||
<span class="o">```</span> <span class="n">sourceCode</span> <span class="n">python</span>
|
||
<span class="n">register_service_views</span><span class="p">(</span><span class="n">config</span><span class="p">,</span> <span class="n">service</span><span class="p">)</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>To sum up, here are the changes I made:</p>
|
||
<ol>
|
||
<li>Service description is now separated from the route registration.</li>
|
||
<li>cornice.service.Service now provides a hook_view method, which is
|
||
not a decorator. decorators are still present but they are optional
|
||
(you don't need to use them if you don't want to).</li>
|
||
<li>Everything has been decoupled as much as possible, meaning that you
|
||
really can use the Service class as a container of information about
|
||
the services you are describing. This is especially useful when
|
||
generating documentation.</li>
|
||
</ol>
|
||
<p>As a result, it is now possible to use Cornice with other frameworks. It
|
||
means that you can stick with the service description but plug any other
|
||
framework on top of it. cornice.services.Service is now only a
|
||
description tool. To register routes, one would need to read the
|
||
information contained into this service and inject the right parameters
|
||
into their preferred routing system.</p>
|
||
<p>However, no integration with other frameworks is done at the moment even
|
||
if the design allows it.</p>
|
||
<p>The same way, the sphinx description layer is now only a consumer of
|
||
this service description tool: it looks at what's described and build-up
|
||
the documentation from it.</p>
|
||
<p>The resulting branch is not merged yet. Still, you can <a href="https://github.com/mozilla-services/cornice/tree/refactor-the-world">have a look at
|
||
it</a>.</p>
|
||
<p>Any suggestions are of course welcome :-)</p></content></entry><entry><title>Djangocong 2012</title><link href="https://blog.notmyidea.org/djangocong-2012.html" rel="alternate"></link><published>2012-04-16T00:00:00+02:00</published><updated>2012-04-16T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2012-04-16:/djangocong-2012.html</id><summary type="html">
|
||
<p>Ce week-end, c'était <a href="http://rencontres.django-fr.org">djangocong</a>, une
|
||
conférence autour de <a href="http://djangoproject.org">django</a>, de
|
||
<a href="http://python.org">python</a> et du web, qui avait lieu dans le sud, à
|
||
Carnon-plage, à quelques kilomètres de Montpellier la belle.</p>
|
||
<p>J'ai vraiment apprécié les trois jours passés avec cette bande de geeks.
|
||
Je m'attendais à des <em>nerds</em>, j'y ai trouvé une …</p></summary><content type="html">
|
||
<p>Ce week-end, c'était <a href="http://rencontres.django-fr.org">djangocong</a>, une
|
||
conférence autour de <a href="http://djangoproject.org">django</a>, de
|
||
<a href="http://python.org">python</a> et du web, qui avait lieu dans le sud, à
|
||
Carnon-plage, à quelques kilomètres de Montpellier la belle.</p>
|
||
<p>J'ai vraiment apprécié les trois jours passés avec cette bande de geeks.
|
||
Je m'attendais à des <em>nerds</em>, j'y ai trouvé une qualité d'écoute, des
|
||
personnes qui partagent des valeurs qui leur sont chères, mais qui ne
|
||
limitent pas leurs discussions à du technique. Eeeh ouais, encore un
|
||
préjugé qui tombe, tiens :)</p>
|
||
<p>En tant que <em>hackers</em>, on a le moyen de créer des outils qui sont utiles
|
||
à tous, et qui peuvent être utiles pour favoriser la collaboration et la
|
||
mise en commun des données. J'ai eu l'occasion de discuter de projets
|
||
tournant autour de l'entraide, que ça soit pour mettre en lien des
|
||
associations d'économie sociale et solidaire (ESS) ou simplement pour
|
||
que les populations <em>non tech</em> <a href="http://blog.notmyidea.org/quels-usages-pour-linformatique-fr.html">puissent utiliser toute la puissance de
|
||
l'outil qu'est le
|
||
web</a>.</p>
|
||
<p>Au niveau du format des conférences, je ne savais pas trop à quoi
|
||
m'attendre, au vu des échos de l'an dernier, mais c'était adapté: des
|
||
mini-confs de 12mn le samedi matin + début d'aprem, en mode no-wifi pour
|
||
récupérer une qualité d'écoute. Et contrairement à mes attentes, ce
|
||
n'est pas trop court. Pas mal de retours d'expérience pour le coup, et
|
||
une matinée pas vraiment techniques, mais ça pose le décor et permet de
|
||
savoir qui fait quoi.</p>
|
||
<p>Parmi l'ensemble des conférences du matin, je retiens principalement
|
||
celle de Mathieu Leplatre, "des cartes d'un autre monde", qui m'a
|
||
réellement bluffée quand à la facilité de créer des cartes avec
|
||
<a href="http://mapbox.com/tilemill/">TileMill</a>, et qui me pousse à reconsidérer
|
||
le fait que "la carto, c'est compliqué". <a href="https://www.youtube.com/watch?v=7NPQo54NbJ8">La vidéo est (déja !)
|
||
disponible en ligne</a>, je
|
||
vous invite à la regarder (c'est une 15aine de minutes) pour vous faire
|
||
un avis ;)</p>
|
||
<p>Une fois les conf passées, ça reste très intéressant, voire plus: il
|
||
reste un jour et demi pour discuter avec les autres présents. On a pu se
|
||
retrouver avec Mathieu pour discuter de "notre" projet <a href="http://blog.notmyidea.org/carto-forms-fr.html">"carto
|
||
forms"</a>, qui à finalement
|
||
pu se redéfinir un peu plus et donner naissance à un
|
||
<a href="https://github.com/spiral-project/daybed/blob/master/README.rst">README</a>.
|
||
On en à profité pour lui choisir un nouveau nom: "daybed", en référence
|
||
à couchdb.</p>
|
||
<p>Ça devrait se transformer en code d'ici peu. La curiosité aidant, on a
|
||
pu discuter du projet avec d'autres personnes et affiner les attentes de
|
||
chacun pour finalement arriver à quelque chose d'assez sympathique.</p>
|
||
<p>J'ai aussi pu me rendre compte que pas mal de monde utilise
|
||
<a href="http://pelican.notmyidea.org">pelican</a>, le bout de code que j'ai codé
|
||
pour générer ce blog, et avoir des retours utiles ! Probablement des
|
||
réflexions à venir sur comment éviter qu'un projet open-source ne
|
||
devienne chronophage, et sur comment réussir à garder une qualité dans
|
||
le code source tout en ne froissant pas les contributeurs.</p>
|
||
<p>Bien évidemment, c'était aussi l'occaz de rencontrer des gens qu'on ne
|
||
voir que sur les inter-nets, et de discuter un brin de tout ce qui fait
|
||
que notre monde est chouette et moins chouette.</p>
|
||
<p>Entres autres faits notoires, JMad a perdu au baby-foot face à Exirel,
|
||
même en m'ayant à ses cotés pour le déconcentrer (et je suis un joueur
|
||
d'un autre monde - en d'autres termes, je suis nul), David`bgk ne s'est
|
||
pas levé pour aller courir le dimanche matin (il avait dit 5 heures!),
|
||
Les suisses ont essayé de me convertir à coup d'abricotine, j'ai perdu
|
||
au skulls-n-roses en quelques tours et on a allumé un feu chez Stéphane
|
||
le dimanche soir (oui oui, à montpellier, mi avril, je vous le dis
|
||
qu'ils mentent avec leur soit disant soleil).</p>
|
||
<p>Et c'est sans parler de <a href="http://jehaisleprintemps.net/blog/fr/2012/04/15/j-ecris-ton-nom/">la
|
||
brasucade</a>
|
||
…</p>
|
||
<p>Bref, vivement la prochaine (et allez, cette fois ci je ferais une
|
||
présentation !)</p></content></entry><entry><title>Génération de formulaires, geolocalisés ?</title><link href="https://blog.notmyidea.org/generation-de-formulaires-geolocalises.html" rel="alternate"></link><published>2012-04-02T00:00:00+02:00</published><updated>2012-04-02T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2012-04-02:/generation-de-formulaires-geolocalises.html</id><summary type="html">
|
||
<p>On a un plan. Un "truc de ouf".</p>
|
||
<p>À plusieurs reprises, des amis m'ont demandé de leur coder la même
|
||
chose, à quelques détails près: une page web avec un formulaire qui
|
||
permettrait de soumettre des informations géographiques, lié à une carte
|
||
et des manières de filtrer l'information.</p>
|
||
<p>L'idée fait …</p></summary><content type="html">
|
||
<p>On a un plan. Un "truc de ouf".</p>
|
||
<p>À plusieurs reprises, des amis m'ont demandé de leur coder la même
|
||
chose, à quelques détails près: une page web avec un formulaire qui
|
||
permettrait de soumettre des informations géographiques, lié à une carte
|
||
et des manières de filtrer l'information.</p>
|
||
<p>L'idée fait son bout de chemin, et je commence à penser qu'on peut même
|
||
avoir quelque chose de vraiment flexible et utile. J'ai nommé le projet
|
||
<em>carto-forms</em> pour l'instant (mais c'est uniquement un nom de code).</p>
|
||
<p>Pour résumer: et si on avait un moyen de construire des formulaires, un
|
||
peu comme Google forms, mais avec des informations géographiques en
|
||
plus?</p>
|
||
<p>Si vous ne connaissez pas Google forms, il s'agit d'une interface simple
|
||
d'utilisation pour générer des formulaires et récupérer des informations
|
||
depuis ces derniers.</p>
|
||
<p>Google forms est un super outil mais à mon avis manque deux choses
|
||
importantes: premièrement, il s'agit d'un outil propriétaire (oui, on
|
||
peut aussi dire privateur) et il n'est donc pas possible de le hacker un
|
||
peu pour le faire devenir ce qu'on souhaite, ni l'installer sur notre
|
||
propre serveur. Deuxièmement, il ne sait pas vraiment fonctionner avec
|
||
des informations géographiques, et il n'y à pas d'autre moyen de filtrer
|
||
les informations que l'utilisation de leur système de feuilles de
|
||
calcul.</p>
|
||
<p>Après avoir réfléchi un petit peu à ça, j'ai contacté
|
||
<a href="http://blog.mathieu-leplatre.info/">Mathieu</a> et les anciens collègues
|
||
de chez <a href="http://makina-corpus.com">Makina Corpus</a>, puisque les projets
|
||
libres à base de carto sont à même de les intéresser.</p>
|
||
<p>Imaginez le cas suivant:</p>
|
||
<ol>
|
||
<li>Dans une "mapping party", on choisit un sujet particulier à
|
||
cartographier et on design un formulaire (liste des champs (tags) a
|
||
remplir + description + le type d'information) ;</li>
|
||
<li>Sur place, les utilisateurs remplissent les champs du formulaire
|
||
avec ce qu'ils voient. Les champs géolocalisés peuvent être remplis
|
||
automatiquement avec la géolocalisation du téléphone ;</li>
|
||
<li>À la fin de la journée, il est possible de voir une carte des
|
||
contributions, avec le formulaire choisi ;</li>
|
||
<li>Un script peut importer les résultats et les publier vers
|
||
OpenStreetMap.</li>
|
||
</ol>
|
||
<h2 id="quelques-cas-dutilisation">Quelques cas d'utilisation</h2>
|
||
<p>J'arrive à imaginer différents cas d'utilisation pour cet outil. Le
|
||
premier est celui que j'ai approximativement décrit plus haut: la
|
||
génération de cartes de manière collaborative, avec des filtres à
|
||
facettes. Voici un flux d'utilisation général:</p>
|
||
<ul>
|
||
<li>
|
||
<p>Un "administrateur" se rend sur le site web et crée un nouveau
|
||
formulaire pour l'ensemble des évènements alternatifs. Il crée les
|
||
champs suivants:</p>
|
||
<ul>
|
||
<li>Nom: le champ qui contient le nom de l'évènement.</li>
|
||
<li>Catégorie: la catégorie de l'évènement (marche, concert,
|
||
manifestation…). Il peut s'agir d'un champ à multiples
|
||
occurrences.</li>
|
||
<li>Le lieu de l'évènement. Celui-ci peut être donné soit par une
|
||
adresse soit en sélectionnant un point sur une carte.</li>
|
||
<li>Date: la date de l'évènement (un "date picker" peut permettre
|
||
cela facilement)</li>
|
||
</ul>
|
||
<p>Chaque champ dans le formulaire a des informations sémantiques
|
||
associées (oui/non, multiple sélection, date, heure, champ géocodé,
|
||
sélection carto, etc.)</p>
|
||
</li>
|
||
<li>
|
||
<p>Une fois terminé, le formulaire est généré et une URL permet d'y
|
||
accéder. (par exemple <a href="http://forms.notmyidea.org/alternatives">http://forms.notmyidea.org/alternatives</a>).</p>
|
||
</li>
|
||
<li>
|
||
<p>Une API REST permet à d'autres applications d'accéder aux
|
||
informations et d'en ajouter / modifier de nouvelles.</p>
|
||
</li>
|
||
<li>
|
||
<p>Il est maintenant possible de donner l'URL à qui voudra en faire bon
|
||
usage. N'importe qui peut ajouter des informations. On peut
|
||
également imaginer une manière de modérer les modifications si
|
||
besoin est.</p>
|
||
</li>
|
||
<li>
|
||
<p>Bien sur, la dernière phase est la plus intéressante: il est
|
||
possible de filtrer les informations par lieu, catégorie ou date, le
|
||
tout soit via une API REST, soit via une jolie carte et quelques
|
||
contrôles bien placés, dans le navigateur.</p>
|
||
</li>
|
||
</ul>
|
||
<p>Vous avez dû remarquer que le processus de création d'un formulaire est
|
||
volontairement très simple. L'idée est que n'importe qui puisse créer
|
||
des cartes facilement, en quelques clics. Si une API bien pensée suit,
|
||
on peut imaginer faire de la validation coté serveur et même faire des
|
||
applications pour téléphone assez simplement.</p>
|
||
<p>Pour aller un peu plus loin, si on arrive à penser un format de
|
||
description pour le formulaire, il sera possible de construire les
|
||
formulaires de manière automatisée sur différentes plateformes et
|
||
également sur des clients génériques.</p>
|
||
<p>On imagine pas mal d'exemples pour ce projet: des points de recyclage,
|
||
les endroits accessibles (pour fauteuils roulants etc.), identification
|
||
des arbres, bons coins à champignons, recensement des espèces en voie de
|
||
disparition (l'aigle de Bonelli est actuellement suivi en utilisant une
|
||
feuille de calcul partagée !), suivi des espèces dangereuses (le frelon
|
||
asiatique par exemple), cartographier les points d'affichage
|
||
publicitaires, participation citoyenne (graffitis, nids de poule, voir
|
||
<a href="http://fixmystreet.ca">http://fixmystreet.ca</a>), geocaching, trajectoires (randonnées,
|
||
coureurs, cyclistes)…</p>
|
||
<p>Voici quelques exemples où ce projet pourrait être utile (la liste n'est
|
||
pas exhaustive):</p>
|
||
<h3 id="un-backend-sig-simple-a-utiliser">Un backend SIG simple à utiliser</h3>
|
||
<p>Disons que vous êtes développeur mobile. Vous ne voulez pas vous
|
||
encombrer avec PostGIS ou écrire du code spécifique pour récupérer et
|
||
insérer des données SIG! Vous avez besoin de <em>Carto-Forms</em>! Une API
|
||
simple vous aide à penser vos modèles et vos formulaires, et cette même
|
||
API vous permet d'insérer et de récupérer des données. Vous pouvez vous
|
||
concentrer sur votre application et non pas sur la manière dont les
|
||
données géographiques sont stockées et gérées.</p>
|
||
<p>En d'autres termes, vous faites une distinction entre le stockage des
|
||
informations et leur affichage.</p>
|
||
<p>Si vous êtes un développeur django, plomino, drupal etc. vous pouvez
|
||
développer un module pour "plugger" vos modèles et votre interface
|
||
utilisateur avec celle de <em>Carto-Forms</em>. De cette manière, il est
|
||
possible d'exposer les formulaires aux utilisateurs de vos backoffices.
|
||
De la même manière, il est possible d'écrire des widgets qui consomment
|
||
des données et les affichent (en utilisant par exemple une bibliothèque
|
||
javascript de webmapping).</p>
|
||
<h3 id="un-outil-de-visualisation">Un outil de visualisation</h3>
|
||
<p>Puisque les données peuvent être proposées de manière automatisée en
|
||
utilisant l'API, vous pouvez utiliser la page de résultat de Carto-forms
|
||
comme un outil de visualisation.</p>
|
||
<p>Il est possible d'explorer mon jeu de données en utilisant des filtres
|
||
sur chacun des champs. La recherche à facettes peut être une idée pour
|
||
faciliter ce filtrage. Une carte affiche le résultat. Vous avez
|
||
l'impressoin d'être en face d'un système d'aide à la décision !</p>
|
||
<p>Évidemment, il est possible de télécharger les données brutes (geojson,
|
||
xml). Idéalement, le mieux serait d'obtenir ces données filtrées
|
||
directement depuis une API Web, et un lien permet de partager la page
|
||
avec l'état des filtres et le niveau de zoom / la localisation de la
|
||
carte.</p>
|
||
<h3 id="un-service-generique-pour-gerer-les-formulaires">Un service générique pour gérer les formulaires</h3>
|
||
<p>Si vous souhaitez générer un fichier de configuration (ou ce que vous
|
||
voulez, messages emails, …) vous aurez besoin d'un formulaire et d'un
|
||
template pour injecter les données proposées par les utilisateurs et
|
||
récupérer un résultat.</p>
|
||
<p>Un service de gestion des formulaires pourrait être utile pour créer des
|
||
formulaires de manière automatique et récupérer les données "nettoyées"
|
||
et "validées".</p>
|
||
<p>On peut imaginer par exemple l'utilisation d'un système de templates
|
||
externe reposant sur <em>carto-forms</em>. Celui-ci "parserait" le contenu des
|
||
templates et pourrait le lier aux informations ajoutées par les
|
||
utilisateurs via un formulaire.</p>
|
||
<p>Pour ce cas particulier, il n'y a pas besoin d'informations
|
||
géographiques (SIG). Il s'agit quasiment du service proposé
|
||
actuellement par Google forms.</p>
|
||
<h2 id="ca-nexiste-pas-deja-tout-ca">Ça n'existe pas déjà tout ça ?</h2>
|
||
<p>Bien sur, il y a Google forms, qui vous permet de faire ce genre de
|
||
choses, mais comme je l'ai précisé plus haut, il ne s'agit pas
|
||
exactement de la même chose.</p>
|
||
<p>Nous avons découvert <a href="https://webform.com">https://webform.com</a> qui permet de créer des
|
||
formulaires avec un système de drag'n'drop. J'adorerais reproduire
|
||
quelque chose de similaire pour l'interface utilisateur. Par contre ce
|
||
projet ne gère pas les appels via API et les informations de
|
||
géolocalisation …</p>
|
||
<p>L'idée de <a href="http://thoth.io">http://thoth.io</a> est également assez sympathique: une api
|
||
très simple pour stocker et récupérer des données. En plus de ça,
|
||
<em>carto-forms</em> proposerait de la validation de données et proposerait un
|
||
support des points SIG (point, ligne, polygone).</p>
|
||
<p><a href="http://mapbox.com">http://mapbox.com</a> fait également un superbe travail autour de la
|
||
cartographie, mais ne prends pas en compte le coté auto-génération de
|
||
formulaires…</p>
|
||
<h2 id="on-est-parti-33">On est parti ?!</h2>
|
||
<p>Comme vous avez pu vous en rendre compte, il ne s'agit pas d'un problème
|
||
outrageusement complexe. On a pas mal discuté avec Mathieu, à propos de
|
||
ce qu'on souhaite faire et du comment. Il se trouve qu'on peut sûrement
|
||
s'en sortir avec une solution élégante sans trop de problèmes. Mathieu
|
||
est habitué à travailler autour des projets de SIG (ce qui est parfait
|
||
parce que ce n'est pas mon cas) et connaît son sujet. Une bonne
|
||
opportunité d'apprendre!</p>
|
||
<p>On sera tous les deux à <a href="http://rencontres.django-fr.org">Djangocong</a> le
|
||
14 et 15 Avril, et on prévoit une session de <em>tempête de cerveau</em> et un
|
||
sprint sur ce projet. Si vous êtes dans le coin et que vous souhaitez
|
||
discuter ou nous filer un coup de patte, n'hésitez pas!</p>
|
||
<p>On ne sait pas encore si on utilisera django ou quelque chose d'autre.
|
||
On a pensé un peu à CouchDB, son système de couchapps et geocouch, mais
|
||
rien n'est encore gravé dans le marbre ! N'hésitez pas à proposer vos
|
||
solutions ou suggestions.</p>
|
||
<p>Voici le document etherpad sur lequel on a travaillé jusqu'à maintenant:
|
||
<a href="http://framapad.org/carto-forms">http://framapad.org/carto-forms</a>. N'hésitez pas à l'éditer et à ajouter
|
||
vos commentaires, c'est son objectif!</p>
|
||
<p>Merci à <a href="http://sneakernet.fr/">Arnaud</a> pour la relecture et la
|
||
correction de quelques typos dans le texte :)</p></content></entry><entry><title>Thoughts about a form generation service, GIS enabled</title><link href="https://blog.notmyidea.org/thoughts-about-a-form-generation-service-gis-enabled.html" rel="alternate"></link><published>2012-04-02T00:00:00+02:00</published><updated>2012-04-02T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2012-04-02:/thoughts-about-a-form-generation-service-gis-enabled.html</id><summary type="html">
|
||
<ul>
|
||
<li>
|
||
<p>slug<br>
|
||
carto-forms</p>
|
||
</li>
|
||
<li>
|
||
<p>date<br>
|
||
02-04-2012</p>
|
||
</li>
|
||
<li>
|
||
<p>author<br>
|
||
Alexis Métaireau, Mathieu Leplatre</p>
|
||
</li>
|
||
<li>
|
||
<p>tags<br>
|
||
GIS, forms</p>
|
||
</li>
|
||
<li>
|
||
<p>lang<br>
|
||
en</p>
|
||
</li>
|
||
<li>
|
||
<p>category<br>
|
||
tech</p>
|
||
</li>
|
||
</ul>
|
||
<p>We have a plan. A "fucking good" one.</p>
|
||
<p>A bunch of friends asked me twice for quite the same thing: a webpage
|
||
with a form, tied to a map generation with some information filtering …</p></summary><content type="html">
|
||
<ul>
|
||
<li>
|
||
<p>slug<br>
|
||
carto-forms</p>
|
||
</li>
|
||
<li>
|
||
<p>date<br>
|
||
02-04-2012</p>
|
||
</li>
|
||
<li>
|
||
<p>author<br>
|
||
Alexis Métaireau, Mathieu Leplatre</p>
|
||
</li>
|
||
<li>
|
||
<p>tags<br>
|
||
GIS, forms</p>
|
||
</li>
|
||
<li>
|
||
<p>lang<br>
|
||
en</p>
|
||
</li>
|
||
<li>
|
||
<p>category<br>
|
||
tech</p>
|
||
</li>
|
||
</ul>
|
||
<p>We have a plan. A "fucking good" one.</p>
|
||
<p>A bunch of friends asked me twice for quite the same thing: a webpage
|
||
with a form, tied to a map generation with some information filtering.
|
||
They didn't explicitly ask that but that's the gist of it.</p>
|
||
<p>This idea has been stuck in my head since then and I even think that we
|
||
can come out with something a little bit more flexible and useful. I've
|
||
named it <em>carto-forms</em> for now, but that's only the "codename".</p>
|
||
<p>To put it shortly: what if we had a way to build forms, ala Google
|
||
forms, but with geographic information in them?</p>
|
||
<p>If you don't know Google forms, it means having an user-friendly way to
|
||
build forms and to use them to gather information from different users.</p>
|
||
<p>In my opinion, Google forms is missing two important things: first, it's
|
||
not open-source, so it's not possible to hack it or even to run it on
|
||
your own server. Second, it doesn't really know how to deal with
|
||
geographic data, and there is no way to filter the information more than
|
||
in a spreadsheet.</p>
|
||
<p>I knew that <a href="http://blog.mathieu-leplatre.info/">Mathieu</a> and some folks
|
||
at <a href="http://makina-corpus.com">Makina Corpus</a> would be interested in
|
||
this, so I started a discussion with him on IRC and we refined the
|
||
details of the project and its objectives.</p>
|
||
<p>Imagine the following:</p>
|
||
<ol>
|
||
<li>For a mapping party, we choose a specific topic to map and design
|
||
the form (list of fields (i.e. tags) to be filled + description +
|
||
type of the information) ;</li>
|
||
<li>In situ, users fill the form fields with what they see. Geo fields
|
||
can be pre-populated using device geolocation ;</li>
|
||
<li>At the end of the day, we can see a map with all user contributions
|
||
seized through this particular form ;</li>
|
||
<li>If relevant, a script could eventually import the resulting dataset
|
||
and publish/merge with OpenStreetMap.</li>
|
||
</ol>
|
||
<h2 id="some-use-cases">Some use cases</h2>
|
||
<p>I can see some use cases for this. The first one is a collaborative map,
|
||
with facet filtering. Let's draw a potential user flow:</p>
|
||
<ul>
|
||
<li>
|
||
<p>An "administrator" goes to the website and creates a form to list
|
||
all the alternative-related events. He creates the following fields:</p>
|
||
<ul>
|
||
<li>Name: a plain text field containing the name of the event.</li>
|
||
<li>Category: the category of the event. Can be a finite list.</li>
|
||
<li>Location: The location of the event. It could be provided by
|
||
selecting a point on a map or by typing an address.</li>
|
||
<li>Date: the date of the event (a datepicker could do the trick)</li>
|
||
</ul>
|
||
<p>Each field in the form has semantic information associated with it
|
||
(yes/no, multiple selection, date-time, geocoding carto, carto
|
||
selection etc)</p>
|
||
</li>
|
||
<li>
|
||
<p>Once finished, the form is generated and the user gets an url (say
|
||
<a href="http://forms.notmyidea.org/alternatives">http://forms.notmyidea.org/alternatives</a>) for it.</p>
|
||
</li>
|
||
<li>
|
||
<p>REST APIs allow third parties to get the form description and to
|
||
push/edit/get information from there.</p>
|
||
</li>
|
||
<li>
|
||
<p>He can communicate the address in any way he wants to his community
|
||
so they can go to the page and add information to it.</p>
|
||
</li>
|
||
<li>
|
||
<p>Then, it is possible to filter the results per location / date or
|
||
category. This can be done via API calls (useful for third parties)
|
||
or via a nice interface in the browser.</p>
|
||
</li>
|
||
</ul>
|
||
<p>So, as you may have noticed, this would allow us to create interactive
|
||
maps really easily. It's almost just a matter of some clicks to the
|
||
users. If we also come up with a nice Web API for this, we could do
|
||
server-side validation and build even phone applications easily.</p>
|
||
<p>To push the cursor a bit further, if we can come with a cool description
|
||
format for the forms, we could even build the forms dynamically on
|
||
different platforms, with generic clients.</p>
|
||
<p>As mentioned before, the idea of a simple tool to support collaborative
|
||
mapping fullfils a recurring necessity !</p>
|
||
<p>We envision a lot of example uses for this : recycling spots, accessible
|
||
spots (wheelchairs, etc.), trees identification, mushrooms picking
|
||
areas, tracking of endangered species (e.g. Bonelli's Eagle is currently
|
||
tracked by sharing a spreadsheet), spotting of dangerous species (e.g.
|
||
asian predatory wasps), map advertisement boards (most cities do not
|
||
track them!), citizen reporting (e.g. graffiti, potholes, garbage,
|
||
lightning like <a href="http://fixmystreet.ca">http://fixmystreet.ca</a>), geocaching, trajectories (e.g
|
||
hiking, runners, cyclists)...</p>
|
||
<p>Here are some other examples of where <em>carto-forms</em> could be useful:</p>
|
||
<h3 id="simple-gis-storage-backend">Simple GIS storage backend</h3>
|
||
<p>Let's say you are a mobile developer, you don't want to bother with
|
||
PostGIS nor write a custom and insecure code to insert and retrieve your
|
||
GIS data! You need carto-forms! A simple API helps you design your
|
||
models/forms and the same API allows you to CRUD and query your data.
|
||
Thus, you only need to focus on your application, not on how GIS data
|
||
will be handled.</p>
|
||
<p>We make a distinction between storage and widgets.</p>
|
||
<p>Besides, if you are a django / drupal / plomino... maintainer : you can
|
||
develop a module to "plug" your models (content types) and UI to
|
||
carto-forms! Carto forms are then exposed to your backoffice users (ex:
|
||
drupal admin UI, django adminsite), and likewise you can write your own
|
||
HTML widgets that consume datasets in frontend views (facets in
|
||
JSON/XML, and map data in GeoJSON).</p>
|
||
<h3 id="visualization-tool">Visualization tool</h3>
|
||
<p>Since data submission can be done programmatically using the API, you
|
||
could use Carto-forms results page as a visualization tool.</p>
|
||
<p>You can explore your dataset content using filters related to each form
|
||
field. Facets filtering is a great advantage, and a map shows the
|
||
resulting features set. You feel like you're in front of a decision
|
||
support system!</p>
|
||
<p>Of course, filtered raw data can be downloaded (GeoJSON, XML) and a
|
||
permalink allows to share the page with the state of the filters and the
|
||
zoom/location of the map.</p>
|
||
<h3 id="generic-forms-service">Generic forms service</h3>
|
||
<p>If you want to generate a configuration file (or whatever, email
|
||
messages, ...), you will need a form and a template to inlay user
|
||
submitted values and get the result.</p>
|
||
<p>A form service would be really useful to create forms programmatically
|
||
and retrieve cleaned and validated input values.</p>
|
||
<p>You could run a dedicated template service based on <em>carto-forms</em>!
|
||
Parsing a template content, this external service could create a form
|
||
dynamically and bind them together. The output of the form service
|
||
(fields => values) would be bound to the input of a template engine
|
||
(variables => final result).</p>
|
||
<p>Note that for this use-case, there is no specific need of GIS data nor
|
||
storage of records for further retrieval.</p>
|
||
<h2 id="whats-out-in-the-wild-already">What's out in the wild already?</h2>
|
||
<p>Of course, there is Google forms, which allows you to do these kind of
|
||
things, but it's closed and not exactly what we are describing here.</p>
|
||
<p>We've discovered the interesting <a href="https://webform.com/">https://webform.com/</a> which allows one
|
||
to create forms with a nice drag-n-drop flow. I would love to reproduce
|
||
something similar for the user experience. However, the project doesn't
|
||
handle APIs and geolocation information.</p>
|
||
<p>The idea of <a href="http://thoth.io">http://thoth.io</a> is very attractive : an extremely simple
|
||
web API to store and retrieve data. In addition, <em>carto-forms</em> would do
|
||
datatype validation and have basic GIS fields (point, line, polygon).</p>
|
||
<p><a href="http://mapbox.com">http://mapbox.com</a> also did an awesome work on cartography, but didn't
|
||
take into account the form aspect we're leveraging here.</p>
|
||
<h2 id="so-lets-get-it-real33">So… Let's get it real!</h2>
|
||
<p>As you may have understood, this isn't a really complicated problem. We
|
||
have been sometimes chatting about that with Mathieu about what we would
|
||
need and how we could achieve this.</p>
|
||
<p>We can probably come with an elegant solution without too much pain.
|
||
Mathieu is used to work with GIS systems (which is really cool because
|
||
I'm not at all) and knows his subject, so that's an opportunity to learn
|
||
;-)</p>
|
||
<p>We will be at <a href="http://rencontres.django-fr.org">Djangocong</a> on April 14
|
||
and 15 and will probably have a brainstorming session and a sprint on
|
||
this, so if you are around and want to help us, or just to discuss, feel
|
||
free to join!</p>
|
||
<p>We don't know yet if we will be using django for this or something else.
|
||
We have been thinking about couchdb, couchapps and geocouch but nothing
|
||
is written in stone yet. Comments and proposals are welcome!</p>
|
||
<p>Here is the etherpad document we worked on so far:
|
||
<a href="http://framapad.org/carto-forms">http://framapad.org/carto-forms</a>. Don't hesitate to add your thoughts
|
||
and edit it, that's what it's made for!</p>
|
||
<p>Thanks to <a href="http://sneakernet.fr/">Arnaud</a> and
|
||
<a href="http://qwerty.fuzz.me.uk/">Fuzzmz</a> for proof-reading and typo fixing.</p></content></entry><entry><title>Introducing Cornice</title><link href="https://blog.notmyidea.org/introducing-cornice.html" rel="alternate"></link><published>2011-12-07T00:00:00+01:00</published><updated>2011-12-07T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2011-12-07:/introducing-cornice.html</id><summary type="html">
|
||
<p>Wow, already my third working day at Mozilla. Since Monday, I've been
|
||
working with <a href="http://ziade.org">Tarek Ziadé</a>, on a pyramid REST-ish
|
||
toolkit named <a href="https://github.com/mozilla-services/cornice">Cornice</a>.</p>
|
||
<p>Its goal is to take care for you of what you're usually missing so you
|
||
can focus on what's important. Cornice provides you facilities for
|
||
validation of …</p></summary><content type="html">
|
||
<p>Wow, already my third working day at Mozilla. Since Monday, I've been
|
||
working with <a href="http://ziade.org">Tarek Ziadé</a>, on a pyramid REST-ish
|
||
toolkit named <a href="https://github.com/mozilla-services/cornice">Cornice</a>.</p>
|
||
<p>Its goal is to take care for you of what you're usually missing so you
|
||
can focus on what's important. Cornice provides you facilities for
|
||
validation of any kind.</p>
|
||
<p>The goal is to simplify your work, but we don't want to reinvent the
|
||
wheel, so it is easily pluggable with validations frameworks, such as
|
||
<a href="http://docs.pylonsproject.org/projects/colander/en/latest/">Colander</a>.</p>
|
||
<h2 id="handling-errors-and-validation">Handling errors and validation</h2>
|
||
<p>Here is how it works:</p>
|
||
<p>``` sourceCode python
|
||
service = Service(name="service", path="/service")</p>
|
||
<p>def is_awesome(request):
|
||
if not 'awesome' in request.GET:
|
||
request.errors.add('query', 'awesome',
|
||
'the awesome parameter is required')</p>
|
||
<p>@service.get(validator=is_awesome)
|
||
def get1(request):
|
||
return {"test": "yay!"}</p>
|
||
<div class="highlight"><pre><span></span><span class="nv">All</span> <span class="nv">the</span> <span class="nv">errors</span> <span class="nv">collected</span> <span class="nv">during</span> <span class="nv">the</span> <span class="nv">validation</span> <span class="nv">process</span>, <span class="nv">or</span> <span class="nv">after</span>, <span class="nv">are</span>
|
||
<span class="nv">collected</span> <span class="nv">before</span> <span class="nv">returning</span> <span class="nv">the</span> <span class="nv">request</span>. <span class="k">If</span> <span class="nv">any</span>, <span class="nv">a</span> <span class="nv">error</span> <span class="mi">400</span> <span class="nv">is</span> <span class="nv">fired</span> <span class="nv">up</span>,
|
||
<span class="nv">with</span> <span class="nv">the</span> <span class="nv">list</span> <span class="nv">of</span> <span class="nv">problems</span> <span class="nv">encountered</span> <span class="nv">returned</span> <span class="nv">as</span> <span class="nv">a</span> <span class="nv">nice</span> <span class="nv">json</span> <span class="nv">list</span>
|
||
<span class="nv">response</span> <span class="ss">(</span><span class="nv">we</span> <span class="nv">plan</span> <span class="nv">to</span> <span class="nv">support</span> <span class="nv">multiple</span> <span class="nv">formats</span> <span class="nv">in</span> <span class="nv">the</span> <span class="nv">future</span><span class="ss">)</span>
|
||
|
||
<span class="nv">As</span> <span class="nv">you</span> <span class="nv">might</span> <span class="nv">have</span> <span class="nv">seen</span>, <span class="nv">request</span>.<span class="nv">errors</span>.<span class="nv">add</span> <span class="nv">takes</span> <span class="nv">three</span> <span class="nv">parameters</span>:
|
||
<span class="o">**</span><span class="nv">location</span><span class="o">**</span>, <span class="o">**</span><span class="nv">name</span><span class="o">**</span> <span class="nv">and</span> <span class="o">**</span><span class="nv">description</span><span class="o">**</span>.
|
||
|
||
<span class="o">**</span><span class="nv">location</span><span class="o">**</span> <span class="nv">is</span> <span class="nv">where</span> <span class="nv">the</span> <span class="nv">error</span> <span class="nv">is</span> <span class="nv">located</span> <span class="nv">in</span> <span class="nv">the</span> <span class="nv">request</span>. <span class="nv">It</span> <span class="nv">can</span> <span class="nv">either</span>
|
||
<span class="nv">be</span> <span class="s2">&quot;</span><span class="s">body</span><span class="s2">&quot;</span>, <span class="s2">&quot;</span><span class="s">query</span><span class="s2">&quot;</span>, <span class="s2">&quot;</span><span class="s">headers</span><span class="s2">&quot;</span> <span class="nv">or</span> <span class="s2">&quot;</span><span class="s">path</span><span class="s2">&quot;</span>. <span class="o">**</span><span class="nv">name</span><span class="o">**</span> <span class="nv">is</span> <span class="nv">the</span> <span class="nv">name</span> <span class="nv">of</span> <span class="nv">the</span>
|
||
<span class="nv">variable</span> <span class="nv">causing</span> <span class="nv">problem</span>, <span class="k">if</span> <span class="nv">any</span>, <span class="nv">and</span> <span class="o">**</span><span class="nv">description</span><span class="o">**</span> <span class="nv">contains</span> <span class="nv">a</span> <span class="nv">more</span>
|
||
<span class="nv">detailed</span> <span class="nv">message</span>.
|
||
|
||
<span class="nv">Let</span><span class="s1">&#39;</span><span class="s">s run this simple service and send some queries to it:</span>
|
||
|
||
$ <span class="nv">curl</span> <span class="o">-</span><span class="nv">v</span> <span class="nv">http</span>:<span class="o">//</span><span class="mi">127</span>.<span class="mi">0</span>.<span class="mi">0</span>.<span class="mi">1</span>:<span class="mi">5000</span><span class="o">/</span><span class="nv">service</span>
|
||
<span class="o">&gt;</span> <span class="nv">GET</span> <span class="o">/</span><span class="nv">service</span> <span class="nv">HTTP</span><span class="o">/</span><span class="mi">1</span>.<span class="mi">1</span>
|
||
<span class="o">&gt;</span> <span class="nv">Host</span>: <span class="mi">127</span>.<span class="mi">0</span>.<span class="mi">0</span>.<span class="mi">1</span>:<span class="mi">5000</span>
|
||
<span class="o">&gt;</span> <span class="nv">Accept</span>: <span class="o">*/*</span>
|
||
<span class="o">&gt;</span>
|
||
<span class="o">*</span> <span class="nv">HTTP</span> <span class="mi">1</span>.<span class="mi">0</span>, <span class="nv">assume</span> <span class="nv">close</span> <span class="nv">after</span> <span class="nv">body</span>
|
||
<span class="o">&lt;</span> <span class="nv">HTTP</span><span class="o">/</span><span class="mi">1</span>.<span class="mi">0</span> <span class="mi">400</span> <span class="nv">Bad</span> <span class="nv">Request</span>
|
||
<span class="o">&lt;</span> <span class="nv">Content</span><span class="o">-</span><span class="nv">Type</span>: <span class="nv">application</span><span class="o">/</span><span class="nv">json</span><span class="c1">; charset=UTF-8</span>
|
||
[{<span class="s2">&quot;</span><span class="s">location</span><span class="s2">&quot;</span>: <span class="s2">&quot;</span><span class="s">query</span><span class="s2">&quot;</span>, <span class="s2">&quot;</span><span class="s">name</span><span class="s2">&quot;</span>: <span class="s2">&quot;</span><span class="s">awesome</span><span class="s2">&quot;</span>, <span class="s2">&quot;</span><span class="s">description</span><span class="s2">&quot;</span>: <span class="s2">&quot;</span><span class="s">You lack awesomeness!</span><span class="s2">&quot;</span>}
|
||
|
||
<span class="nv">I</span><span class="s1">&#39;</span><span class="s">ve removed the extra clutter from the curl</span><span class="s1">&#39;</span><span class="nv">s</span> <span class="nv">output</span>, <span class="nv">but</span> <span class="nv">you</span> <span class="nv">got</span> <span class="nv">the</span>
|
||
<span class="nv">general</span> <span class="nv">idea</span>.
|
||
|
||
<span class="nv">The</span> <span class="nv">content</span> <span class="nv">returned</span> <span class="nv">is</span> <span class="nv">in</span> <span class="nv">JSON</span>, <span class="nv">and</span> <span class="nv">I</span> <span class="nv">know</span> <span class="nv">exactly</span> <span class="nv">what</span> <span class="nv">I</span> <span class="nv">have</span> <span class="nv">to</span> <span class="k">do</span>:
|
||
<span class="nv">add</span> <span class="nv">an</span> <span class="s2">&quot;</span><span class="s">awesome</span><span class="s2">&quot;</span> <span class="nv">parameter</span> <span class="nv">in</span> <span class="nv">my</span> <span class="nv">query</span>. <span class="nv">Let</span><span class="s1">&#39;</span><span class="s">s do it again:</span>
|
||
|
||
$ <span class="nv">curl</span> <span class="nv">http</span>:<span class="o">//</span><span class="mi">127</span>.<span class="mi">0</span>.<span class="mi">0</span>.<span class="mi">1</span>:<span class="mi">5000</span><span class="o">/</span><span class="nv">service</span>?<span class="nv">awesome</span><span class="o">=</span><span class="nv">yeah</span>
|
||
{<span class="s2">&quot;</span><span class="s">test</span><span class="s2">&quot;</span>: <span class="s2">&quot;</span><span class="s">yay!</span><span class="s2">&quot;</span>}
|
||
|
||
<span class="nv">Validators</span> <span class="nv">can</span> <span class="nv">also</span> <span class="nv">convert</span> <span class="nv">parts</span> <span class="nv">of</span> <span class="nv">the</span> <span class="nv">request</span> <span class="nv">and</span> <span class="nv">store</span> <span class="nv">the</span> <span class="nv">converted</span>
|
||
<span class="nv">value</span> <span class="nv">in</span> <span class="nv">request</span>.<span class="nv">validated</span>. <span class="nv">It</span> <span class="nv">is</span> <span class="nv">a</span> <span class="nv">standard</span> <span class="nv">dict</span> <span class="nv">automatically</span> <span class="nv">attached</span>
|
||
<span class="nv">to</span> <span class="nv">the</span> <span class="nv">requests</span>.
|
||
|
||
<span class="k">For</span> <span class="nv">instance</span>, <span class="nv">in</span> <span class="nv">our</span> <span class="nv">validator</span>, <span class="nv">we</span> <span class="nv">can</span> <span class="nv">chose</span> <span class="nv">to</span> <span class="nv">validate</span> <span class="nv">the</span> <span class="nv">parameter</span>
|
||
<span class="nv">passed</span> <span class="nv">and</span> <span class="nv">use</span> <span class="nv">it</span> <span class="nv">in</span> <span class="nv">the</span> <span class="nv">body</span> <span class="nv">of</span> <span class="nv">the</span> <span class="nv">webservice</span>:
|
||
|
||
``` <span class="nv">sourceCode</span> <span class="nv">python</span>
|
||
<span class="nv">service</span> <span class="o">=</span> <span class="nv">Service</span><span class="ss">(</span><span class="nv">name</span><span class="o">=</span><span class="s2">&quot;</span><span class="s">service</span><span class="s2">&quot;</span>, <span class="nv">path</span><span class="o">=</span><span class="s2">&quot;</span><span class="s">/service</span><span class="s2">&quot;</span><span class="ss">)</span>
|
||
|
||
|
||
<span class="nv">def</span> <span class="nv">is_awesome</span><span class="ss">(</span><span class="nv">request</span><span class="ss">)</span>:
|
||
<span class="k">if</span> <span class="nv">not</span> <span class="s1">&#39;</span><span class="s">awesome</span><span class="s1">&#39;</span> <span class="nv">in</span> <span class="nv">request</span>.<span class="nv">GET</span>:
|
||
<span class="nv">request</span>.<span class="nv">errors</span>.<span class="nv">add</span><span class="ss">(</span><span class="s1">&#39;</span><span class="s">query</span><span class="s1">&#39;</span>, <span class="s1">&#39;</span><span class="s">awesome</span><span class="s1">&#39;</span>,
|
||
<span class="s1">&#39;</span><span class="s">the awesome parameter is required</span><span class="s1">&#39;</span><span class="ss">)</span>
|
||
<span class="k">else</span>:
|
||
<span class="nv">request</span>.<span class="nv">validated</span>[<span class="s1">&#39;</span><span class="s">awesome</span><span class="s1">&#39;</span>] <span class="o">=</span> <span class="s1">&#39;</span><span class="s">awesome </span><span class="s1">&#39;</span> <span class="o">+</span> <span class="nv">request</span>.<span class="nv">GET</span>[<span class="s1">&#39;</span><span class="s">awesome</span><span class="s1">&#39;</span>]
|
||
|
||
|
||
@<span class="nv">service</span>.<span class="nv">get</span><span class="ss">(</span><span class="nv">validator</span><span class="o">=</span><span class="nv">is_awesome</span><span class="ss">)</span>
|
||
<span class="nv">def</span> <span class="nv">get1</span><span class="ss">(</span><span class="nv">request</span><span class="ss">)</span>:
|
||
<span class="k">return</span> {<span class="s2">&quot;</span><span class="s">test</span><span class="s2">&quot;</span>: <span class="nv">request</span>.<span class="nv">validated</span>[<span class="s1">&#39;</span><span class="s">awesome</span><span class="s1">&#39;</span>]}
|
||
</pre></div>
|
||
|
||
|
||
<p>The output would look like this:</p>
|
||
<div class="highlight"><pre><span></span><span class="n">curl</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="mi">127</span><span class="p">.</span><span class="mi">0</span><span class="p">.</span><span class="mi">0</span><span class="p">.</span><span class="mi">1</span><span class="p">:</span><span class="mi">5000</span><span class="o">/</span><span class="n">service</span><span class="o">?</span><span class="n">awesome</span><span class="o">=</span><span class="n">yeah</span>
|
||
<span class="err">{</span><span class="ss">&quot;test&quot;</span><span class="p">:</span> <span class="ss">&quot;awesome yeah&quot;</span><span class="err">}</span>
|
||
</pre></div>
|
||
|
||
|
||
<h2 id="dealing-with-accept-headers">Dealing with "Accept" headers</h2>
|
||
<p>The HTTP spec defines a <strong>Accept</strong> header the client can send so the
|
||
response is encoded the right way. A resource, available at an URL, can
|
||
be available in different formats. This is especially true for web
|
||
services.</p>
|
||
<p>Cornice can help you dealing with this. The services you define can tell
|
||
which Content-Type values they can deal with and this will be checked
|
||
against the <strong>Accept</strong> headers sent by the client.</p>
|
||
<p>Let's refine a bit our previous example, by specifying which
|
||
content-types are supported, using the accept
|
||
parameter:</p>
|
||
<p>``` sourceCode python
|
||
@service.get(validator=is_awesome, accept=("application/json", "text/json"))
|
||
def get1(request):
|
||
return {"test": "yay!"}</p>
|
||
<div class="highlight"><pre><span></span><span class="nv">Now</span>, <span class="k">if</span> <span class="nv">you</span> <span class="nv">specifically</span> <span class="nv">ask</span> <span class="k">for</span> <span class="nv">XML</span>, <span class="nv">Cornice</span> <span class="nv">will</span> <span class="nv">throw</span> <span class="nv">a</span> <span class="mi">406</span> <span class="nv">with</span> <span class="nv">the</span>
|
||
<span class="nv">list</span> <span class="nv">of</span> <span class="nv">accepted</span> <span class="nv">Content</span><span class="o">-</span><span class="nv">Type</span> <span class="nv">values</span>:
|
||
|
||
$ <span class="nv">curl</span> <span class="o">-</span><span class="nv">vH</span> <span class="s2">&quot;</span><span class="s">Accept: application/xml</span><span class="s2">&quot;</span> <span class="nv">http</span>:<span class="o">//</span><span class="mi">127</span>.<span class="mi">0</span>.<span class="mi">0</span>.<span class="mi">1</span>:<span class="mi">5000</span><span class="o">/</span><span class="nv">service</span>
|
||
<span class="o">&gt;</span> <span class="nv">GET</span> <span class="o">/</span><span class="nv">service</span> <span class="nv">HTTP</span><span class="o">/</span><span class="mi">1</span>.<span class="mi">1</span>
|
||
<span class="o">&gt;</span> <span class="nv">Host</span>: <span class="mi">127</span>.<span class="mi">0</span>.<span class="mi">0</span>.<span class="mi">1</span>:<span class="mi">5000</span>
|
||
<span class="o">&gt;</span> <span class="nv">Accept</span>: <span class="nv">application</span><span class="o">/</span><span class="nv">xml</span>
|
||
<span class="o">&gt;</span>
|
||
<span class="o">&lt;</span> <span class="nv">HTTP</span><span class="o">/</span><span class="mi">1</span>.<span class="mi">0</span> <span class="mi">406</span> <span class="nv">Not</span> <span class="nv">Acceptable</span>
|
||
<span class="o">&lt;</span> <span class="nv">Content</span><span class="o">-</span><span class="nv">Type</span>: <span class="nv">application</span><span class="o">/</span><span class="nv">json</span><span class="c1">; charset=UTF-8</span>
|
||
<span class="o">&lt;</span> <span class="nv">Content</span><span class="o">-</span><span class="nv">Length</span>: <span class="mi">33</span>
|
||
<span class="o">&lt;</span>
|
||
[<span class="s2">&quot;</span><span class="s">application/json</span><span class="s2">&quot;</span>, <span class="s2">&quot;</span><span class="s">text/json</span><span class="s2">&quot;</span>]
|
||
|
||
## <span class="nv">Building</span> <span class="nv">your</span> <span class="nv">documentation</span> <span class="nv">automatically</span>
|
||
|
||
<span class="nv">writing</span> <span class="nv">documentation</span> <span class="k">for</span> <span class="nv">web</span> <span class="nv">services</span> <span class="nv">can</span> <span class="nv">be</span> <span class="nv">painful</span>, <span class="nv">especially</span> <span class="nv">when</span>
|
||
<span class="nv">your</span> <span class="nv">services</span> <span class="nv">evolve</span>. <span class="nv">Cornice</span> <span class="nv">provides</span> <span class="nv">a</span> <span class="nv">sphinx</span> <span class="nv">directive</span> <span class="nv">to</span>
|
||
<span class="nv">automatically</span> <span class="nv">document</span> <span class="nv">your</span> <span class="nv">API</span> <span class="nv">in</span> <span class="nv">your</span> <span class="nv">docs</span>.
|
||
|
||
``` <span class="nv">sourceCode</span> <span class="nv">rst</span>
|
||
.. <span class="nv">services</span>::
|
||
:<span class="nv">package</span>: <span class="nv">coolapp</span>
|
||
:<span class="nv">service</span>: <span class="nv">quote</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>Here is an example of what a generated page looks like:
|
||
<a href="http://packages.python.org/cornice/exampledoc.html">http://packages.python.org/cornice/exampledoc.html</a></p>
|
||
<h2 id="yay33-how-can-i-get-it">Yay! How can I get it?</h2>
|
||
<p>We just cut a 0.4 release, so it's available at
|
||
<a href="http://pypi.python.org/pypi/cornice">http://pypi.python.org/pypi/cornice</a> You can install it easily using
|
||
pip, for instance:</p>
|
||
<div class="highlight"><pre><span></span>$ pip install cornice
|
||
</pre></div>
|
||
|
||
|
||
<p>You can also have a look at the documentation at
|
||
<a href="http://packages.python.org/cornice/">http://packages.python.org/cornice/</a></p>
|
||
<h2 id="whats-next">What's next?</h2>
|
||
<p>We try to make our best to find how Cornice can help you build better
|
||
web services. Cool features we want for the future include the automatic
|
||
publication of a static definition of the services, so it can be used by
|
||
clients to discover services in a nice way.</p>
|
||
<p>Of course, we are open to all your ideas and patches! If you feel
|
||
haskish and want to see the sources, <a href="https://github.com/mozilla-services/cornice">go grab them on
|
||
github</a> , commit and send
|
||
us a pull request!</p></content></entry><entry><title>How are you handling your shared expenses?</title><link href="https://blog.notmyidea.org/how-are-you-handling-your-shared-expenses.html" rel="alternate"></link><published>2011-10-15T00:00:00+02:00</published><updated>2011-10-15T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2011-10-15:/how-are-you-handling-your-shared-expenses.html</id><summary type="html">
|
||
<p><strong>TL;DR:</strong> We're kick-starting a new application to manage your shared
|
||
expenses. Have a look at <a href="http://ihatemoney.notmyidea.org">http://ihatemoney.notmyidea.org</a></p>
|
||
<p>As a student, I lived in a lot of different locations, and the majority
|
||
of them had something in common: I lived with others. It usually was a
|
||
great experience …</p></summary><content type="html">
|
||
<p><strong>TL;DR:</strong> We're kick-starting a new application to manage your shared
|
||
expenses. Have a look at <a href="http://ihatemoney.notmyidea.org">http://ihatemoney.notmyidea.org</a></p>
|
||
<p>As a student, I lived in a lot of different locations, and the majority
|
||
of them had something in common: I lived with others. It usually was a
|
||
great experience (and I think I will continue to live with others). Most
|
||
of the time, we had to spend some time each month to compute who had to
|
||
pay what to the others.</p>
|
||
<p>I wanted to create a pet project using flask, so I wrote a little (\~150
|
||
lines) flask application to handle this. It worked out pretty well for
|
||
my housemates and me, and as we had to move into different locations,
|
||
one of them asked me if he could continue to use it for the year to
|
||
come.</p>
|
||
<p>I said yes and gave it some more thoughts: We probably aren't the only
|
||
ones interested by such kind of software. I decided to extend a bit more
|
||
the software to have a concept of projects and persons (the list of
|
||
persons was hard-coded in the first time, boooh!).</p>
|
||
<p>I then discussed with a friend of mine, who was excited about it and
|
||
wanted to learn python. Great! That's a really nice way to get started.
|
||
Some more friends were also interested in it and contributed some
|
||
features and provided feedback (thanks
|
||
<a href="http://www.sneakernet.fr/">Arnaud</a> and Quentin!)</p>
|
||
<p>Since that, the project now support multiple languages and provides a
|
||
REST API (android and iphone apps in the tubes!), into other things.
|
||
There is no need to register for an account or whatnot, just enter a
|
||
project name, a secret code and a contact email, invite friends and
|
||
that's it (this was inspired by doodle)!</p>
|
||
<p><img alt="Capture d'écran du site." src="images/ihatemoney.png"></p>
|
||
<p>You can try the project at <a href="http://ihatemoney.notmyidea.org">http://ihatemoney.notmyidea.org</a> for now,
|
||
and the code lives at <a href="https://github.com/spiral-project/ihatemoney/">https://github.com/spiral-project/ihatemoney/</a>.</p>
|
||
<h2 id="features">Features</h2>
|
||
<p>In the wild, currently, there already are some implementations of this
|
||
shared budget manager thing. The fact is that most of them are either
|
||
hard to use, with a too much fancy design or simply trying to do too
|
||
much things at once.</p>
|
||
<p>No, I don't want my budget manager to make my shopping list, or to run a
|
||
blog for me, thanks. I want it to let me focus on something else. Keep
|
||
out of my way.</p>
|
||
<h3 id="no-user-registration">No user registration</h3>
|
||
<p>You don't need to register an account on the website to start using it.
|
||
You just have to create a project, set a secret code for it, and give
|
||
both the url and the code to the people you want to share it with (or
|
||
the website can poke them for you).</p>
|
||
<h3 id="keeping-things-simple">Keeping things simple</h3>
|
||
<p>"Keep It Simple, Stupid" really matches our philosophy here: you want to
|
||
add a bill? Okay. Just do it. You just have to enter who paid, for who,
|
||
how much, and a description, like you would have done when you're back
|
||
from the farmer's market on raw paper.</p>
|
||
<h3 id="no-categories">No categories</h3>
|
||
<p>Some people like to organise their stuff into different "categories":
|
||
leisure, work, eating, etc. That's not something I want (at least to
|
||
begin with).</p>
|
||
<p>I want things to be simple. Got that? Great. Just add your bills!</p>
|
||
<h3 id="balance">Balance</h3>
|
||
<p>One of the most useful thing is to know what's your "balance" compared
|
||
to others. In other words, if you're negative, you owe money, if you're
|
||
positive, you have to receive money. This allows you to dispatch who has
|
||
to pay for the next thing, in order to re-equilibrate the balance.</p>
|
||
<p>Additionally, the system is able to compute for you who has to give how
|
||
much to who, in order to reduce the number of transactions needed to
|
||
restore the balance.</p>
|
||
<h3 id="api">API</h3>
|
||
<p>All of what's possible to do with the standard web interface is also
|
||
available through a REST API. I developed a simple REST toolkit for
|
||
flask for this (and I should release it!).</p>
|
||
<h2 id="interested">Interested?</h2>
|
||
<p>This project is open source. All of us like to share what we are doing
|
||
and would be happy to work with new people and implement new ideas. If
|
||
you have a nice idea about this, if you want to tweak it or to fill
|
||
bugs. Don't hesitate a second! The project lives at
|
||
<a href="http://github.com/spiral-project/ihatemoney/">http://github.com/spiral-project/ihatemoney/</a></p></content></entry><entry><title>Using dbpedia to get languages influences</title><link href="https://blog.notmyidea.org/using-dbpedia-to-get-languages-influences.html" rel="alternate"></link><published>2011-08-16T00:00:00+02:00</published><updated>2011-08-16T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2011-08-16:/using-dbpedia-to-get-languages-influences.html</id><summary type="html">
|
||
<p>While browsing the Python's wikipedia page, I found information about
|
||
the languages influenced by python, and the languages that influenced
|
||
python itself.</p>
|
||
<p>Well, that's kind of interesting to know which languages influenced
|
||
others, it could even be more interesting to have an overview of the
|
||
connexion between them, keeping python …</p></summary><content type="html">
|
||
<p>While browsing the Python's wikipedia page, I found information about
|
||
the languages influenced by python, and the languages that influenced
|
||
python itself.</p>
|
||
<p>Well, that's kind of interesting to know which languages influenced
|
||
others, it could even be more interesting to have an overview of the
|
||
connexion between them, keeping python as the main focus.</p>
|
||
<p>This information is available on the wikipedia page, but not in a really
|
||
exploitable format. Hopefully, this information is provided into the
|
||
information box present on the majority of wikipedia pages. And… guess
|
||
what? there is project with the goal to scrap and index all this
|
||
information in a more queriable way, using the semantic web
|
||
technologies.</p>
|
||
<p>Well, you may have guessed it, the project in question in dbpedia, and
|
||
exposes information in the form of RDF triples, which are way more easy
|
||
to work with than simple HTML.</p>
|
||
<p>For instance, let's take the page about python:
|
||
<a href="http://dbpedia.org/page/Python_%28programming_language%29">http://dbpedia.org/page/Python_%28programming_language%29</a></p>
|
||
<p>The interesting properties here are "Influenced" and "InfluencedBy",
|
||
which allows us to get a list of languages. Unfortunately, they are not
|
||
really using all the power of the Semantic Web here, and the list is
|
||
actually a string with coma separated values in it.</p>
|
||
<p>Anyway, we can use a simple rule: All wikipedia pages of programming
|
||
languages are either named after the name of the language itself, or
|
||
suffixed with "( programming language)", which is the case for python.</p>
|
||
<p>So I've built <a href="https://github.com/ametaireau/experiments/blob/master/influences/get_influences.py">a tiny script to extract the information from
|
||
dbpedia</a>
|
||
and transform them into a shiny graph using graphviz.</p>
|
||
<p>After a nice:</p>
|
||
<div class="highlight"><pre><span></span>$ python get_influences.py python dot <span class="p">|</span> dot -Tpng &gt; influences.png
|
||
</pre></div>
|
||
|
||
|
||
<p>The result is the following graph (<a href="http://files.lolnet.org/alexis/influences.png">see it directly
|
||
here</a>)</p>
|
||
<p><img alt="Graph des influances des langages les uns sur les
|
||
autres." src="http://files.lolnet.org/alexis/influences.png"></p>
|
||
<p>While reading this diagram, keep in mind that it is a) not listing all
|
||
the languages and b) keeping a python perspective.</p>
|
||
<p>This means that you can trust the scheme by following the arrows from
|
||
python to something and from something to python, it is not trying to
|
||
get the matching between all the languages at the same time to keep
|
||
stuff readable.</p>
|
||
<p>It would certainly be possible to have all the connections between all
|
||
languages (and the resulting script would be easier) to do so, but the
|
||
resulting graph would probably be way less readable.</p>
|
||
<p>You can find the script <a href="https://github.com/ametaireau/experiments">on my github
|
||
account</a>. Feel free to adapt
|
||
it for whatever you want if you feel hackish.</p></content></entry><entry><title>Pelican, 9 months later</title><link href="https://blog.notmyidea.org/pelican-9-months-later.html" rel="alternate"></link><published>2011-07-25T00:00:00+02:00</published><updated>2011-07-25T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2011-07-25:/pelican-9-months-later.html</id><summary type="html">
|
||
<p>Back in October, I released
|
||
<a href="http://docs.notmyidea.org/alexis/pelican">pelican</a>, a little piece of
|
||
code I wrote to power this weblog. I had simple needs: I wanted to be
|
||
able to use my text editor of choice (vim), a vcs (mercurial) and
|
||
restructured text. I started to write a really simple blog engine in …</p></summary><content type="html">
|
||
<p>Back in October, I released
|
||
<a href="http://docs.notmyidea.org/alexis/pelican">pelican</a>, a little piece of
|
||
code I wrote to power this weblog. I had simple needs: I wanted to be
|
||
able to use my text editor of choice (vim), a vcs (mercurial) and
|
||
restructured text. I started to write a really simple blog engine in
|
||
something like a hundred python lines and released it on github.</p>
|
||
<p>And people started contributing. I wasn't at all expecting to see people
|
||
interested in such a little piece of code, but it turned out that they
|
||
were. I refactored the code to make it evolve a bit more by two times
|
||
and eventually, in 9 months, got 49 forks, 139 issues and 73 pull
|
||
requests.</p>
|
||
<p><strong>Which is clearly awesome.</strong></p>
|
||
<p>I pulled features such as translations, tag clouds, integration with
|
||
different services such as twitter or piwik, import from dotclear and
|
||
rss, fixed a number of mistakes and improved a lot the codebase. This
|
||
was a proof that there is a bunch of people that are willing to make
|
||
better softwares just for the sake of fun.</p>
|
||
<p>Thank you, guys, you're why I like open source so much.</p></content></entry><entry><title>Using JPype to bridge python and Java</title><link href="https://blog.notmyidea.org/using-jpype-to-bridge-python-and-java.html" rel="alternate"></link><published>2011-06-11T00:00:00+02:00</published><updated>2011-06-11T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2011-06-11:/using-jpype-to-bridge-python-and-java.html</id><summary type="html">
|
||
<p>Java provides some interesting libraries that have no exact equivalent
|
||
in python. In my case, the awesome boilerpipe library allows me to
|
||
remove uninteresting parts of HTML pages, like menus, footers and other
|
||
"boilerplate" contents.</p>
|
||
<p>Boilerpipe is written in Java. Two solutions then: using java from
|
||
python or reimplement boilerpipe …</p></summary><content type="html">
|
||
<p>Java provides some interesting libraries that have no exact equivalent
|
||
in python. In my case, the awesome boilerpipe library allows me to
|
||
remove uninteresting parts of HTML pages, like menus, footers and other
|
||
"boilerplate" contents.</p>
|
||
<p>Boilerpipe is written in Java. Two solutions then: using java from
|
||
python or reimplement boilerpipe in python. I will let you guess which
|
||
one I chosen, meh.</p>
|
||
<p>JPype allows to bridge python project with java libraries. It takes
|
||
another point of view than Jython: rather than reimplementing python in
|
||
Java, both languages are interfacing at the VM level. This means you
|
||
need to start a VM from your python script, but it does the job and stay
|
||
fully compatible with Cpython and its C extensions.</p>
|
||
<h2 id="first-steps-with-jpype">First steps with JPype</h2>
|
||
<p>Once JPype installed (you'll have to hack a bit some files to integrate
|
||
seamlessly with your system) you can access java classes by doing
|
||
something like that:</p>
|
||
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">jpype</span>
|
||
<span class="n">jpype</span><span class="o">.</span><span class="n">startJVM</span><span class="p">(</span><span class="n">jpype</span><span class="o">.</span><span class="n">getDefaultJVMPath</span><span class="p">())</span>
|
||
|
||
<span class="c1"># you can then access to the basic java functions</span>
|
||
<span class="n">jpype</span><span class="o">.</span><span class="n">java</span><span class="o">.</span><span class="n">lang</span><span class="o">.</span><span class="n">System</span><span class="o">.</span><span class="n">out</span><span class="o">.</span><span class="n">println</span><span class="p">(</span><span class="s2">&quot;hello world&quot;</span><span class="p">)</span>
|
||
|
||
<span class="c1"># and you have to shutdown the VM at the end</span>
|
||
<span class="n">jpype</span><span class="o">.</span><span class="n">shutdownJVM</span><span class="p">()</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>Okay, now we have a hello world, but what we want seems somehow more
|
||
complex. We want to interact with java classes, so we will have to load
|
||
them.</p>
|
||
<h2 id="interfacing-with-boilerpipe">Interfacing with Boilerpipe</h2>
|
||
<p>To install boilerpipe, you just have to run an ant script:</p>
|
||
<div class="highlight"><pre><span></span>$ <span class="nb">cd</span> boilerpipe
|
||
$ ant
|
||
</pre></div>
|
||
|
||
|
||
<p>Here is a simple example of how to use boilerpipe in Java, from their
|
||
sources</p>
|
||
<div class="highlight"><pre><span></span><span class="kn">package</span> <span class="nn">de.l3s.boilerpipe.demo</span><span class="o">;</span>
|
||
<span class="kn">import</span> <span class="nn">java.net.URL</span><span class="o">;</span>
|
||
<span class="kn">import</span> <span class="nn">de.l3s.boilerpipe.extractors.ArticleExtractor</span><span class="o">;</span>
|
||
|
||
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">Oneliner</span> <span class="o">{</span>
|
||
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="kd">final</span> <span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
|
||
<span class="kd">final</span> <span class="n">URL</span> <span class="n">url</span> <span class="o">=</span> <span class="k">new</span> <span class="n">URL</span><span class="o">(</span><span class="s">&quot;http://notmyidea.org&quot;</span><span class="o">);</span>
|
||
<span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">ArticleExtractor</span><span class="o">.</span><span class="na">INSTANCE</span><span class="o">.</span><span class="na">getText</span><span class="o">(</span><span class="n">url</span><span class="o">));</span>
|
||
<span class="o">}</span>
|
||
<span class="o">}</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>To run it:</p>
|
||
<div class="highlight"><pre><span></span>$ javac -cp dist/boilerpipe-1.1-dev.jar:lib/nekohtml-1.9.13.jar:lib/xerces-2.9.1.jar src/demo/de/l3s/boilerpipe/demo/Oneliner.java
|
||
$ java -cp src/demo:dist/boilerpipe-1.1-dev.jar:lib/nekohtml-1.9.13.jar:lib/xerces-2.9.1.jar de.l3s.boilerpipe.demo.Oneliner
|
||
</pre></div>
|
||
|
||
|
||
<p>Yes, this is kind of ugly, sorry for your eyes. Let's try something
|
||
similar, but from python</p>
|
||
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">jpype</span>
|
||
|
||
<span class="c1"># start the JVM with the good classpaths</span>
|
||
<span class="n">classpath</span> <span class="o">=</span> <span class="s2">&quot;dist/boilerpipe-1.1-dev.jar:lib/nekohtml-1.9.13.jar:lib/xerces-2.9.1.jar&quot;</span>
|
||
<span class="n">jpype</span><span class="o">.</span><span class="n">startJVM</span><span class="p">(</span><span class="n">jpype</span><span class="o">.</span><span class="n">getDefaultJVMPath</span><span class="p">(),</span> <span class="s2">&quot;-Djava.class.path=</span><span class="si">%s</span><span class="s2">&quot;</span> <span class="o">%</span> <span class="n">classpath</span><span class="p">)</span>
|
||
|
||
<span class="c1"># get the Java classes we want to use</span>
|
||
<span class="n">DefaultExtractor</span> <span class="o">=</span> <span class="n">jpype</span><span class="o">.</span><span class="n">JPackage</span><span class="p">(</span><span class="s2">&quot;de&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">l3s</span><span class="o">.</span><span class="n">boilerpipe</span><span class="o">.</span><span class="n">extractors</span><span class="o">.</span><span class="n">DefaultExtractor</span>
|
||
|
||
<span class="c1"># call them !</span>
|
||
<span class="k">print</span> <span class="n">DefaultExtractor</span><span class="o">.</span><span class="n">INSTANCE</span><span class="o">.</span><span class="n">getText</span><span class="p">(</span><span class="n">jpype</span><span class="o">.</span><span class="n">java</span><span class="o">.</span><span class="n">net</span><span class="o">.</span><span class="n">URL</span><span class="p">(</span><span class="s2">&quot;http://blog.notmyidea.org&quot;</span><span class="p">))</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>And you get what you want.</p>
|
||
<p>I must say I didn't thought it could work so easily. This will allow me
|
||
to extract text content from URLs and remove the <em>boilerplate</em> text
|
||
easily for infuse (my master thesis project), without having to write
|
||
java code, nice!</p></content></entry><entry><title>Un coup de main pour mon mémoire33</title><link href="https://blog.notmyidea.org/un-coup-de-main-pour-mon-memoire33.html" rel="alternate"></link><published>2011-05-25T00:00:00+02:00</published><updated>2011-05-25T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2011-05-25:/un-coup-de-main-pour-mon-memoire33.html</id><summary type="html"><h1 id="un-coup-de-main-pour-mon-memoire33">Un coup de main pour mon mémoire!</h1>
|
||
<p>Ça y est, bientôt la fin. LA FIN. La fin des études, et le début du
|
||
reste. En attendant je bosse sur mon mémoire de fin d'études et j'aurais
|
||
besoin d'un petit coup de main.</p>
|
||
<p>Mon mémoire porte sur les systèmes de recommandation …</p></summary><content type="html"><h1 id="un-coup-de-main-pour-mon-memoire33">Un coup de main pour mon mémoire!</h1>
|
||
<p>Ça y est, bientôt la fin. LA FIN. La fin des études, et le début du
|
||
reste. En attendant je bosse sur mon mémoire de fin d'études et j'aurais
|
||
besoin d'un petit coup de main.</p>
|
||
<p>Mon mémoire porte sur les systèmes de recommandation. Pour ceux qui
|
||
connaissent last.fm, je fais quelque chose de similaire mais pour les
|
||
sites internet: en me basant sur ce que vous visitez quotidiennement et
|
||
comment vous le visitez (quelles horaires, quelle emplacement
|
||
géographique, etc.) je souhaites proposer des liens qui vous
|
||
intéresseront potentiellement, en me basant sur l'avis des personnes
|
||
qui ont des profils similaires au votre.</p>
|
||
<p>Le projet est loin d'être terminé, mais la première étape est de
|
||
récupérer des données de navigation, idéalement beaucoup de données de
|
||
navigation. Donc si vous pouvez me filer un coup de main je vous en
|
||
serais éternellement reconnaissant (pour ceux qui font semblant de pas
|
||
comprendre, entendez "tournée générale").</p>
|
||
<p>J'ai créé un petit site web (en anglais) qui résume un peu le concept,
|
||
qui vous propose de vous inscrire et de télécharger un plugin firefox
|
||
qui m'enverra des information sur les sites que vous visitez (si vous
|
||
avez l'habitude d'utiliser chrome vous pouvez considérer de switcher à
|
||
firefox4 pour les deux prochains mois pour me filer un coup de main). Il
|
||
est possible de désactiver le plugin d'un simple clic si vous souhaitez
|
||
garder votre vie privée privée ;-)</p>
|
||
<p>Le site est par là: <a href="http://infuse.notmyidea.org">http://infuse.notmyidea.org</a>. Une fois le plugin
|
||
téléchargé et le compte créé il faut renseigner vos identifiants dans
|
||
le plugin en question, et c'est tout!</p>
|
||
<p>A votre bon cœur ! Je récupérerais probablement des données durant les
|
||
2 prochains mois pour ensuite les analyser correctement.</p>
|
||
<p>Merci pour votre aide !</p></content></entry><entry><title>Analyse users' browsing context to build up a web recommender</title><link href="https://blog.notmyidea.org/analyse-users-browsing-context-to-build-up-a-web-recommender.html" rel="alternate"></link><published>2011-04-01T00:00:00+02:00</published><updated>2011-04-01T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2011-04-01:/analyse-users-browsing-context-to-build-up-a-web-recommender.html</id><summary type="html">
|
||
<p>No, this is not an april's fool ;)</p>
|
||
<p>Wow, it's been a long time. My year in Oxford is going really well. I
|
||
realized few days ago that the end of the year is approaching really
|
||
quickly. Exams are coming in one month or such and then I'll be working
|
||
full …</p></summary><content type="html">
|
||
<p>No, this is not an april's fool ;)</p>
|
||
<p>Wow, it's been a long time. My year in Oxford is going really well. I
|
||
realized few days ago that the end of the year is approaching really
|
||
quickly. Exams are coming in one month or such and then I'll be working
|
||
full time on my dissertation topic.</p>
|
||
<p>When I learned we'll have about 6 month to work on something, I first
|
||
thought about doing a packaging related stuff, but finally decided to
|
||
start something new. After all, that's the good time to learn.</p>
|
||
<p>Since a long time, I'm being impressed by the <a href="http://last.fm">last.fm</a>
|
||
recommender system. They're <em>scrobbling</em> the music I listen to since
|
||
something like 5 years now and the recommendations they're doing are
|
||
really nice and accurate (I discovered <strong>a lot</strong> of great artists
|
||
listening to the "neighbour radio".) (by the way, <a href="http://lastfm.com/user/akounet/">here
|
||
is</a> my lastfm account)</p>
|
||
<p>So I decided to work on recommender systems, to better understand what
|
||
is it about.</p>
|
||
<p>Recommender systems are usually used to increase the sales of products
|
||
(like Amazon.com does) which is not really what I'm looking for (The one
|
||
who know me a bit know I'm kind of sick about all this consumerism going
|
||
on).</p>
|
||
<p>Actually, the most simple thing I thought of was the web: I'm browsing
|
||
it quite every day and each time new content appears. I've stopped to
|
||
follow <a href="https://bitbucket.org/bruno/aspirator/">my feed reader</a> because
|
||
of the information overload, and reduced drastically the number of
|
||
people I follow <a href="http://twitter.com/ametaireau/">on twitter</a>.</p>
|
||
<p>Too much information kills the information.</p>
|
||
<p>You shall got what will be my dissertation topic: a recommender system
|
||
for the web. Well, such recommender systems already exists, so I will
|
||
try to add contextual information to them: you're probably not
|
||
interested by the same topics at different times of the day, or
|
||
depending on the computer you're using. We can also probably make good
|
||
use of the way you browse to create groups into the content you're
|
||
browsing (or even use the great firefox4 tab group feature).</p>
|
||
<p>There is a large part of concerns to have about user's privacy as well.</p>
|
||
<p>Here is my proposal (copy/pasted from the one I had to do for my master)</p>
|
||
<h2 id="introduction-and-rationale">Introduction and rationale</h2>
|
||
<p>Nowadays, people surf the web more and more often. New web pages are
|
||
created each day so the amount of information to retrieve is more
|
||
important as the time passes. These users uses the web in different
|
||
contexts, from finding cooking recipes to technical articles.</p>
|
||
<p>A lot of people share the same interest to various topics, and the
|
||
quantity of information is such than it's really hard to triage them
|
||
efficiently without spending hours doing it. Firstly because of the huge
|
||
quantity of information but also because the triage is something
|
||
relative to each person. Although, this triage can be facilitated by
|
||
fetching the browsing information of all particular individuals and put
|
||
the in perspective.</p>
|
||
<p>Machine learning is a branch of Artificial Intelligence (AI) which deals
|
||
with how a program can learn from data. Recommendation systems are a
|
||
particular application area of machine learning which is able to
|
||
recommend things (links in our case) to the users, given a particular
|
||
database containing the previous choices users have made.</p>
|
||
<p>This browsing information is currently available in browsers. Even if it
|
||
is not in a very usable format, it is possible to transform it to
|
||
something useful. This information gold mine just wait to be used.
|
||
Although, it is not as simple as it can seems at the first approach: It
|
||
is important to take care of the context the user is in while browsing
|
||
links. For instance, It's more likely that during the day, a computer
|
||
scientist will browse computing related links, and that during the
|
||
evening, he browse cooking recipes or something else.</p>
|
||
<p>Page contents are also interesting to analyse, because that's what
|
||
people browse and what actually contain the most interesting part of the
|
||
information. The raw data extracted from the browsing can then be
|
||
translated into something more useful (namely tags, type of resource,
|
||
visit frequency, navigation context etc.)</p>
|
||
<p>The goal of this dissertation is to create a recommender system for web
|
||
links, including this context information.</p>
|
||
<p>At the end of the dissertation, different pieces of software will be
|
||
provided, from raw data collection from the browser to a recommendation
|
||
system.</p>
|
||
<h2 id="background-review">Background Review</h2>
|
||
<p>This dissertation is mainly about data extraction, analysis and
|
||
recommendation systems. Two different research area can be isolated:
|
||
Data preprocessing and Information filtering.</p>
|
||
<p>The first step in order to make recommendations is to gather some data.
|
||
The more data we have available, the better it is (T. Segaran, 2007).
|
||
This data can be retrieved in various ways, one of them is to get it
|
||
directly from user's browsers.</p>
|
||
<h3 id="data-preparation-and-extraction">Data preparation and extraction</h3>
|
||
<p>The data gathered from browsers is basically URLs and additional
|
||
information about the context of the navigation. There is clearly a need
|
||
to extract more information about the meaning of the data the user is
|
||
browsing, starting by the content of the web pages.</p>
|
||
<p>Because the information provided on the current Web is not meant to be
|
||
read by machines (T. Berners Lee, 2001) there is a need of tools to
|
||
extract meaning from web pages. The information needs to be preprocessed
|
||
before stored in a machine readable format, allowing to make
|
||
recommendations (Choochart et Al, 2004).</p>
|
||
<p>Data preparation is composed of two steps: cleaning and structuring (
|
||
Castellano et Al, 2007). Because raw data can contain a lot of un-needed
|
||
text (such as menus, headers etc.) and need to be cleaned prior to be
|
||
stored. Multiple techniques can be used here and belongs to boilerplate
|
||
removal and full text extraction (Kohlschütter et Al, 2010).</p>
|
||
<p>Then, structuring the information: category, type of content (news,
|
||
blog, wiki) can be extracted from raw data. This kind of information is
|
||
not clearly defined by HTML pages so there is a need of tools to
|
||
recognise them.</p>
|
||
<p>Some context-related information can also be inferred from each
|
||
resource. It can go from the visit frequency to the navigation group the
|
||
user was in while browsing. It is also possible to determine if the user
|
||
"liked" a resource, and determine a mark for it, which can be used by
|
||
information filtering a later step (T. Segaran, 2007).</p>
|
||
<p>At this stage, structuring the data is required. Storing this kind of
|
||
information in RDBMS can be a bit tedious and require complex queries to
|
||
get back the data in an usable format. Graph databases can play a major
|
||
role in the simplification of information storage and querying.</p>
|
||
<h3 id="information-filtering">Information filtering</h3>
|
||
<p>To filter the information, three techniques can be used (Balabanovic et
|
||
Al, 1997):</p>
|
||
<ul>
|
||
<li>The content-based approach states that if an user have liked
|
||
something in the past, he is more likely to like similar things in
|
||
the future. So it's about establishing a profile for the user and
|
||
compare new items against it.</li>
|
||
<li>The collaborative approach will rather recommend items that other
|
||
similar users have liked. This approach consider only the
|
||
relationship between users, and not the profile of the user we are
|
||
making recommendations to.</li>
|
||
<li>the hybrid approach, which appeared recently combine both of the
|
||
previous approaches, giving recommendations when items score high
|
||
regarding user's profile, or if a similar user already liked it.</li>
|
||
</ul>
|
||
<p>Grouping is also something to consider at this stage (G. Myatt, 2007).
|
||
Because we are dealing with huge amount of data, it can be useful to
|
||
detect group of data that can fit together. Data clustering is able to
|
||
find such groups (T. Segaran, 2007).</p>
|
||
<p>References:</p>
|
||
<ul>
|
||
<li>Balabanović, M., &amp; Shoham, Y. (1997). Fab: content-based,
|
||
collaborative recommendation. Communications of the ACM, 40(3),
|
||
66–72. ACM. Retrieved March 1, 2011, from
|
||
<a href="http://portal.acm.org/citation.cfm?id=245108.245124&amp;">http://portal.acm.org/citation.cfm?id=245108.245124&amp;</a>;.</li>
|
||
<li>Berners-Lee, T., Hendler, J., &amp; Lassila, O. (2001). The semantic
|
||
web: Scientific american. Scientific American, 284(5), 34–43.
|
||
Retrieved November 21, 2010, from
|
||
<a href="http://www.citeulike.org/group/222/article/1176986">http://www.citeulike.org/group/222/article/1176986</a>.</li>
|
||
<li>Castellano, G., Fanelli, A., &amp; Torsello, M. (2007). LODAP: a LOg
|
||
DAta Preprocessor for mining Web browsing patterns. Proceedings of
|
||
the 6th Conference on 6th WSEAS Int. Conf. on Artificial
|
||
Intelligence, Knowledge Engineering and Data Bases-Volume 6 (p.
|
||
12–17). World Scientific and Engineering Academy and Society
|
||
(WSEAS). Retrieved March 8, 2011, from
|
||
<a href="http://portal.acm.org/citation.cfm?id=1348485.1348488">http://portal.acm.org/citation.cfm?id=1348485.1348488</a>.</li>
|
||
<li>Kohlschutter, C., Fankhauser, P., &amp; Nejdl, W. (2010). Boilerplate
|
||
detection using shallow text features. Proceedings of the third ACM
|
||
international conference on Web search and data mining (p. 441–450).
|
||
ACM. Retrieved March 8, 2011, from
|
||
<a href="http://portal.acm.org/citation.cfm?id=1718542">http://portal.acm.org/citation.cfm?id=1718542</a>.</li>
|
||
<li>Myatt, G. J. (2007). Making Sense of Data: A Practical Guide to
|
||
Exploratory Data Analysis and Data Mining.</li>
|
||
<li>Segaran, T. (2007). Collective Intelligence.</li>
|
||
</ul>
|
||
<h2 id="privacy">Privacy</h2>
|
||
<p>The first thing that's come to people minds when it comes to process
|
||
their browsing data is privacy. People don't want to be stalked. That's
|
||
perfectly right, and I don't either.</p>
|
||
<p>But such a system don't have to deal with people identities. It's
|
||
completely possible to process completely anonymous data, and that's
|
||
probably what I'm gonna do.</p>
|
||
<p>By the way, if you have interesting thoughts about that, if you do know
|
||
projects that do seems related, fire the comments !</p>
|
||
<h2 id="whats-the-plan">What's the plan ?</h2>
|
||
<p>There is a lot of different things to explore, especially because I'm a
|
||
complete novice in that field.</p>
|
||
<ul>
|
||
<li>I want to develop a firefox plugin, to extract the browsing
|
||
informations ( still, I need to know exactly which kind of
|
||
informations to retrieve). The idea is to provide some <em>raw</em>
|
||
browsing data, and then to transform it and to store it in the
|
||
better possible way.</li>
|
||
<li>Analyse how to store the informations in a graph database. What can
|
||
be the different methods to store this data and to visualize the
|
||
relationship between different pieces of data? How can I define the
|
||
different contexts, and add those informations in the db?</li>
|
||
<li>Process the data using well known recommendation algorithms. Compare
|
||
the results and criticize their value.</li>
|
||
</ul>
|
||
<p>There is plenty of stuff I want to try during this experimentation:</p>
|
||
<ul>
|
||
<li>I want to try using Geshi to visualize the connexion between the
|
||
links, and the contexts</li>
|
||
<li>Try using graph databases such as Neo4j</li>
|
||
<li>Having a deeper look at tools such as scikit.learn (a machine
|
||
learning toolkit in python)</li>
|
||
<li>Analyse web pages in order to categorize them. Processing their
|
||
contents as well, to do some keyword based classification will be
|
||
done.</li>
|
||
</ul>
|
||
<p>Lot of work on its way, yay !</p></content></entry><entry><title>Working directly on your server? How to backup and sync your dev environment with unison</title><link href="https://blog.notmyidea.org/working-directly-on-your-server-how-to-backup-and-sync-your-dev-environment-with-unison.html" rel="alternate"></link><published>2011-03-16T00:00:00+01:00</published><updated>2011-03-16T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2011-03-16:/working-directly-on-your-server-how-to-backup-and-sync-your-dev-environment-with-unison.html</id><summary type="html">
|
||
<p>I have a server running freebsd since some time now, and was wondering
|
||
about the possibility to directly have a development environment ready
|
||
to use when I get a internet connexion, even if I'm not on my computer.</p>
|
||
<p>Since I use vim to code, and spend most of my time …</p></summary><content type="html">
|
||
<p>I have a server running freebsd since some time now, and was wondering
|
||
about the possibility to directly have a development environment ready
|
||
to use when I get a internet connexion, even if I'm not on my computer.</p>
|
||
<p>Since I use vim to code, and spend most of my time in a console while
|
||
developing, it's possible to work via ssh, from everywhere.</p>
|
||
<p>The only problem is the synchronisation of the source code, config files
|
||
etc. from my machine to the server.</p>
|
||
<p>Unison provides an interesting way to synchronise two folders, even over
|
||
a network. So let's do it !</p>
|
||
<h2 id="creating-the-jail">Creating the jail</h2>
|
||
<p>In case you don't use FreeBSD, you can skip this section.</p>
|
||
<div class="highlight"><pre><span></span><span class="o">#</span> <span class="n">I</span> <span class="n">have</span> <span class="n">a</span> <span class="n">flavour</span> <span class="n">jail</span> <span class="n">named</span> <span class="k">default</span>
|
||
<span class="err">$</span> <span class="n">ezjail</span><span class="o">-</span><span class="k">admin</span> <span class="o">-</span><span class="n">f</span> <span class="k">default</span> <span class="n">workspace</span><span class="p">.</span><span class="n">notmyidea</span><span class="p">.</span><span class="n">org</span> <span class="mi">172</span><span class="p">.</span><span class="mi">19</span><span class="p">.</span><span class="mi">1</span><span class="p">.</span><span class="mi">6</span>
|
||
<span class="err">$</span> <span class="n">ezjail</span><span class="o">-</span><span class="k">admin</span> <span class="k">start</span> <span class="n">workspace</span><span class="p">.</span><span class="n">notmyidea</span><span class="p">.</span><span class="n">org</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>In my case, because the "default" flavour contains already a lot of
|
||
interesting things, my jail come already setup with ssh, bash and vim
|
||
for instance, but maybe you'll need it in your case.</p>
|
||
<p>I want to be redirected to the ssh of the jail when I connect to the
|
||
host with the 20006 port. Add lines in <code>/etc/pf.conf</code>:</p>
|
||
<div class="highlight"><pre><span></span> <span class="nv">workspace_jail</span><span class="o">=</span><span class="s2">&quot;172.19.1.6&quot;</span>
|
||
rdr on <span class="nv">$ext_if</span> proto tcp from any to <span class="nv">$ext_ip</span> port <span class="m">20006</span> -&gt; <span class="nv">$workspace_jail</span> port <span class="m">22</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>Reload packet filter rules</p>
|
||
<div class="highlight"><pre><span></span>$ /etc/rc.d/pf reload
|
||
</pre></div>
|
||
|
||
|
||
<h2 id="working-with-unison">Working with unison</h2>
|
||
<p>Now that we've set up the jail. Set up unison on the server and on your
|
||
client. Unison is available on the freebsd ports so just install it</p>
|
||
<div class="highlight"><pre><span></span>$ ssh notmyidea.org -p <span class="m">20006</span>
|
||
$ make -C /usr/ports/net/unison-nox11 config-recursive
|
||
$ make -C /usr/ports/net/unison-nox11 package-recursive
|
||
</pre></div>
|
||
|
||
|
||
<p>Install as well unison on your local machine. Double check to install
|
||
the same version on the client and on the server. Ubuntu contains the
|
||
2.27.57 as well as the 2.32.52.</p>
|
||
<p>Check that unison is installed and reachable via ssh from your machine</p>
|
||
<div class="highlight"><pre><span></span>$ ssh notmyidea.org -p <span class="m">20006</span> unison -version
|
||
unison version <span class="m">2</span>.27.157
|
||
$ unison -version
|
||
unison version <span class="m">2</span>.27.57
|
||
</pre></div>
|
||
|
||
|
||
<h2 id="let-sync-our-folders">Let sync our folders</h2>
|
||
<p>The first thing I want to sync is my vim configuration. Well, it's
|
||
already <a href="http://github.com/ametaireau/dotfiles/">in a git repository</a>
|
||
but let's try to use unison for it right now.</p>
|
||
<p>I have two machines then: workspace, the jail, and ecureuil my laptop.</p>
|
||
<div class="highlight"><pre><span></span>unison .vim ssh://notmyidea.org:20006/.vim
|
||
unison .vimrc ssh://notmyidea.org:20006/.vimrc
|
||
</pre></div>
|
||
|
||
|
||
<p>It is also possible to put all the informations in a config file, and
|
||
then to only run unison. (fire up vim \~/.unison/default.prf.</p>
|
||
<p>Here is my config:</p>
|
||
<div class="highlight"><pre><span></span> <span class="na">root</span> <span class="o">=</span> <span class="s">/home/alexis</span>
|
||
<span class="s"> root = ssh://notmyidea.org:20006</span>
|
||
|
||
<span class="na">path</span> <span class="o">=</span> <span class="s">.vimrc</span>
|
||
<span class="s"> path = dotfiles</span>
|
||
<span class="s"> path = dev</span>
|
||
|
||
<span class="na">follow</span> <span class="o">=</span> <span class="s">Name *</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>My vimrc is in fact a symbolic link on my laptop, but I don't want to
|
||
specify each of the links to unison. That's why the follow = Name * is
|
||
for.</p>
|
||
<p>The folders you want to synchronize are maybe a bit large. If so,
|
||
considering others options such as rsync for the first import may be a
|
||
good idea (I enjoyed my university huge upload bandwith to upload 2GB in
|
||
20mn ;)</p>
|
||
<h2 id="run-the-script-frequently">Run the script frequently</h2>
|
||
<p>Once that done, you just need to run the unison command line some times
|
||
when you want to sync your two machines. I've wrote a tiny script to get
|
||
some feedback from the sync:</p>
|
||
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||
<span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
|
||
|
||
<span class="n">DEFAULT_LOGFILE</span> <span class="o">=</span> <span class="s2">&quot;~/unison.log&quot;</span>
|
||
<span class="n">PROGRAM_NAME</span> <span class="o">=</span> <span class="s2">&quot;Unison syncer&quot;</span>
|
||
|
||
<span class="k">def</span> <span class="nf">sync</span><span class="p">(</span><span class="n">logfile</span><span class="o">=</span><span class="n">DEFAULT_LOGFILE</span><span class="p">,</span> <span class="n">program_name</span><span class="o">=</span><span class="n">PROGRAM_NAME</span><span class="p">):</span>
|
||
<span class="c1"># init</span>
|
||
<span class="n">display_message</span> <span class="o">=</span> <span class="bp">True</span>
|
||
<span class="n">error</span> <span class="o">=</span> <span class="bp">False</span>
|
||
|
||
<span class="n">before</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
|
||
<span class="c1"># call unison to make the sync</span>
|
||
<span class="n">os</span><span class="o">.</span><span class="n">system</span><span class="p">(</span><span class="s1">&#39;unison -batch &gt; {0}&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">logfile</span><span class="p">))</span>
|
||
|
||
<span class="c1"># get the duration of the operation</span>
|
||
<span class="n">td</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">-</span> <span class="n">before</span>
|
||
<span class="n">delta</span> <span class="o">=</span> <span class="p">(</span><span class="n">td</span><span class="o">.</span><span class="n">microseconds</span> <span class="o">+</span> <span class="p">(</span><span class="n">td</span><span class="o">.</span><span class="n">seconds</span> <span class="o">+</span> <span class="n">td</span><span class="o">.</span><span class="n">days</span> <span class="o">*</span> <span class="mi">24</span> <span class="o">*</span> <span class="mi">3600</span><span class="p">)</span> <span class="o">*</span> <span class="mi">10</span><span class="o">**</span><span class="mi">6</span><span class="p">)</span> <span class="o">/</span> <span class="mi">10</span><span class="o">**</span><span class="mi">6</span>
|
||
|
||
<span class="c1"># check what was the last entry in the log</span>
|
||
<span class="n">log</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">expanduser</span><span class="p">(</span><span class="n">logfile</span><span class="p">))</span>
|
||
<span class="n">lines</span> <span class="o">=</span> <span class="n">log</span><span class="o">.</span><span class="n">readlines</span><span class="p">()</span>
|
||
<span class="k">if</span> <span class="s1">&#39;No updates to propagate&#39;</span> <span class="ow">in</span> <span class="n">lines</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]:</span>
|
||
<span class="n">display_message</span> <span class="o">=</span> <span class="bp">False</span>
|
||
<span class="k">else</span><span class="p">:</span>
|
||
<span class="n">output</span> <span class="o">=</span> <span class="p">[</span><span class="n">l</span> <span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="n">lines</span> <span class="k">if</span> <span class="s2">&quot;Synchronization&quot;</span> <span class="ow">in</span> <span class="n">l</span><span class="p">]</span>
|
||
|
||
<span class="n">message</span> <span class="o">=</span> <span class="n">output</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
|
||
<span class="n">message</span> <span class="o">+=</span> <span class="s2">&quot; It took {0}s.&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">delta</span><span class="p">)</span>
|
||
|
||
<span class="k">if</span> <span class="n">display_message</span><span class="p">:</span>
|
||
<span class="n">os</span><span class="o">.</span><span class="n">system</span><span class="p">(</span><span class="s1">&#39;notify-send -i {2} &quot;{0}&quot; &quot;{1}&quot;&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">program_name</span><span class="p">,</span> <span class="n">message</span><span class="p">,</span>
|
||
<span class="s1">&#39;error&#39;</span> <span class="k">if</span> <span class="n">error</span> <span class="k">else</span> <span class="s1">&#39;info&#39;</span><span class="p">))</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">&quot;__main__&quot;</span><span class="p">:</span>
|
||
<span class="n">sync</span><span class="p">()</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>This is probably perfectible, but that does the job.</p>
|
||
<p>Last step is to tell you machine to run that frequently. That's what
|
||
crontab is made for, so let's <code>crontab -e</code>:</p>
|
||
<div class="highlight"><pre><span></span> <span class="err">$</span> <span class="o">*</span> <span class="o">*/</span><span class="mi">3</span> <span class="o">*</span> <span class="o">*</span> <span class="o">*</span> <span class="p">.</span> <span class="o">~/</span><span class="p">.</span><span class="n">Xdbus</span><span class="p">;</span> <span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">python</span> <span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">alexis</span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">python</span><span class="o">/</span><span class="n">unison</span><span class="o">-</span><span class="n">syncer</span><span class="o">/</span><span class="n">sync</span><span class="p">.</span><span class="n">py</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>The \~/.Xdbus allows cron to communicate with your X11 session. Here is
|
||
its content.</p>
|
||
<div class="highlight"><pre><span></span><span class="ch">#!/bin/bash</span>
|
||
|
||
<span class="c1"># Get the pid of nautilus</span>
|
||
<span class="nv">nautilus_pid</span><span class="o">=</span><span class="k">$(</span>pgrep -u <span class="nv">$LOGNAME</span> -n nautilus<span class="k">)</span>
|
||
|
||
<span class="c1"># If nautilus isn&#39;t running, just exit silently</span>
|
||
<span class="k">if</span> <span class="o">[</span> -z <span class="s2">&quot;</span><span class="nv">$nautilus_pid</span><span class="s2">&quot;</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
|
||
<span class="nb">exit</span> <span class="m">0</span>
|
||
<span class="k">fi</span>
|
||
|
||
<span class="c1"># Grab the DBUS_SESSION_BUS_ADDRESS variable from nautilus&#39;s environment</span>
|
||
<span class="nb">eval</span> <span class="k">$(</span>tr <span class="s1">&#39;\0&#39;</span> <span class="s1">&#39;\n&#39;</span> &lt; /proc/<span class="nv">$nautilus_pid</span>/environ <span class="p">|</span> grep <span class="s1">&#39;^DBUS_SESSION_BUS_ADDRESS=&#39;</span><span class="k">)</span>
|
||
|
||
<span class="c1"># Check that we actually found it</span>
|
||
<span class="k">if</span> <span class="o">[</span> -z <span class="s2">&quot;</span><span class="nv">$DBUS_SESSION_BUS_ADDRESS</span><span class="s2">&quot;</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
|
||
<span class="nb">echo</span> <span class="s2">&quot;Failed to find bus address&quot;</span> &gt;<span class="p">&amp;</span><span class="m">2</span>
|
||
<span class="nb">exit</span> <span class="m">1</span>
|
||
<span class="k">fi</span>
|
||
|
||
<span class="c1"># export it so that child processes will inherit it</span>
|
||
<span class="nb">export</span> DBUS_SESSION_BUS_ADDRESS
|
||
</pre></div>
|
||
|
||
|
||
<p>And it comes from
|
||
<a href="http://ubuntuforums.org/showthread.php?p=10148738#post10148738">here</a>.</p>
|
||
<p>A sync takes about 20s + the upload time on my machine, which stay
|
||
acceptable for all of my developments.</p></content></entry><entry><title>Wrap up of the distutils2 paris' sprint</title><link href="https://blog.notmyidea.org/wrap-up-of-the-distutils2-paris-sprint.html" rel="alternate"></link><published>2011-02-08T00:00:00+01:00</published><updated>2011-02-08T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2011-02-08:/wrap-up-of-the-distutils2-paris-sprint.html</id><summary type="html">
|
||
<p>Finally, thanks to a bunch of people that helped me to pay my train and
|
||
bus tickets, I've made it to paris for the distutils2 sprint.</p>
|
||
<p>They have been a bit more than 10 people to come during the sprint, and
|
||
it was very productive. Here's a taste of what …</p></summary><content type="html">
|
||
<p>Finally, thanks to a bunch of people that helped me to pay my train and
|
||
bus tickets, I've made it to paris for the distutils2 sprint.</p>
|
||
<p>They have been a bit more than 10 people to come during the sprint, and
|
||
it was very productive. Here's a taste of what we've been working on:</p>
|
||
<ul>
|
||
<li>the datafiles, a way to specify and to handle the installation of
|
||
files which are not python-related (pictures, manpages and so on).</li>
|
||
<li>mkgcfg, a tool to help you to create a setup.cfg in minutes (and
|
||
with funny examples)</li>
|
||
<li>converters from setup.py scripts. We do now have a piece of code
|
||
which reads your current setup.py file and fill in some fields in
|
||
the setup.cfg for you.</li>
|
||
<li>a compatibility layer for distutils1, so it can read the setup.cfg
|
||
you will wrote for distutils2 :-)</li>
|
||
<li>the uninstaller, so it's now possible to uninstall what have been
|
||
installed by distutils2 (see PEP 376)</li>
|
||
<li>the installer, and the setuptools compatibility layer, which will
|
||
allow you to rely on setuptools' based distributions (and there are
|
||
plenty of them!)</li>
|
||
<li>The compilers, so they are more flexible than they were. Since
|
||
that's an obscure part of the code for distutils2 commiters (it
|
||
comes directly from the distutils1 ages), having some guys who
|
||
understood the problematics here was a must.</li>
|
||
</ul>
|
||
<p>Some people have also tried to port their packaging from distutils1 to
|
||
distutils2. They have spotted a number of bugs and made some
|
||
improvements to the code, to make it more friendly to use.</p>
|
||
<p>I'm really pleased to see how newcomers went trough the code, and
|
||
started hacking so fast. I must say it wasn't the case when we started
|
||
to work on distutils1 so that's a very good point: people now can hack
|
||
the code quicker than they could before.</p>
|
||
<p>Some of the features here are not <em>completely</em> finished yet, but are on
|
||
the tubes, and will be ready for a release (hopefully) at the end of the
|
||
week.</p>
|
||
<p>Big thanks to logilab for hosting (and sponsoring my train ticket) and
|
||
providing us food, and to bearstech for providing some money for
|
||
breakfast and bears^Wbeers.</p>
|
||
<p>Again, a big thanks to all the people who gave me money to pay the
|
||
transport, I really wasn't expecting such thing to happen :-)</p></content></entry><entry><title>PyPI on CouchDB</title><link href="https://blog.notmyidea.org/pypi-on-couchdb.html" rel="alternate"></link><published>2011-01-20T00:00:00+01:00</published><updated>2011-01-20T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2011-01-20:/pypi-on-couchdb.html</id><summary type="html">
|
||
<p>By now, there are two ways to retrieve data from PyPI (the Python
|
||
Package Index). You can both rely on xml/rpc or on the "simple" API. The
|
||
simple API is not so simple to use as the name suggest, and have several
|
||
existing drawbacks.</p>
|
||
<p>Basically, if you want to …</p></summary><content type="html">
|
||
<p>By now, there are two ways to retrieve data from PyPI (the Python
|
||
Package Index). You can both rely on xml/rpc or on the "simple" API. The
|
||
simple API is not so simple to use as the name suggest, and have several
|
||
existing drawbacks.</p>
|
||
<p>Basically, if you want to use informations coming from the simple API,
|
||
you will have to parse web pages manually, to extract informations using
|
||
some black vodoo magic. Badly, magic have a price, and it's sometimes
|
||
impossible to get exactly the informations you want to get from this
|
||
index. That's the technique currently being used by distutils2,
|
||
setuptools and pip.</p>
|
||
<p>On the other side, while XML/RPC is working fine, it's requiring extra
|
||
work to the python servers each time you request something, which can
|
||
lead to some outages from time to time. Also, it's important to point
|
||
out that, even if PyPI have a mirroring infrastructure, it's only for
|
||
the so-called <em>simple</em> API, and not for the XML/RPC.</p>
|
||
<h2 id="couchdb">CouchDB</h2>
|
||
<p>Here comes CouchDB. CouchDB is a document oriented database, that knows
|
||
how to speak REST and JSON. It's easy to use, and provides out of the
|
||
box a replication mechanism.</p>
|
||
<h2 id="so-what">So, what ?</h2>
|
||
<p>Hmm, I'm sure you got it. I've wrote a piece of software to link
|
||
informations from PyPI to a CouchDB instance. Then you can replicate all
|
||
the PyPI index with only one HTTP request on the CouchDB server. You can
|
||
also access the informations from the index directly using a REST API,
|
||
speaking json. Handy.</p>
|
||
<p>So PyPIonCouch is using the PyPI XML/RPC API to get data from PyPI, and
|
||
generate records in the CouchDB instance.</p>
|
||
<p>The final goal is to avoid to rely on this "simple" API, and rely on a
|
||
REST insterface instead. I have set up a couchdb server on my server,
|
||
which is available at
|
||
<a href="http://couchdb.notmyidea.org/_utils/database.html?pypi">http://couchdb.notmyidea.org/_utils/database.html?pypi</a>.</p>
|
||
<p>There is not a lot to see there for now, but I've done the first import
|
||
from PyPI yesterday and all went fine: it's possible to access the
|
||
metadata of all PyPI projects via a REST interface. Next step is to
|
||
write a client for this REST interface in distutils2.</p>
|
||
<h2 id="example">Example</h2>
|
||
<p>For now, you can use pypioncouch via the command line, or via the python
|
||
API.</p>
|
||
<h3 id="using-the-command-line">Using the command line</h3>
|
||
<p>You can do something like that for a full import. This <strong>will</strong> take
|
||
long, because it's fetching all the projects at pypi and importing their
|
||
metadata:</p>
|
||
<div class="highlight"><pre><span></span><span class="err">$</span> <span class="n">pypioncouch</span> <span class="o">--</span><span class="n">fullimport</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">your</span><span class="o">.</span><span class="n">couchdb</span><span class="o">.</span><span class="n">instance</span><span class="o">/</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>If you already have the data on your couchdb instance, you can just
|
||
update it with the last informations from pypi. <strong>However, I recommend
|
||
to just replicate the principal node, hosted at
|
||
<a href="http://couchdb.notmyidea.org/pypi/">http://couchdb.notmyidea.org/pypi/</a></strong>, to avoid the duplication of
|
||
nodes:</p>
|
||
<div class="highlight"><pre><span></span>$ pypioncouch --update http://your.couchdb.instance/
|
||
</pre></div>
|
||
|
||
|
||
<p>The principal node is updated once a day by now, I'll try to see if it's
|
||
enough, and ajust with the time.</p>
|
||
<h3 id="using-the-python-api">Using the python API</h3>
|
||
<p>You can also use the python API to interact with pypioncouch:</p>
|
||
<div class="highlight"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">pypioncouch</span> <span class="kn">import</span> <span class="n">XmlRpcImporter</span><span class="p">,</span> <span class="n">import_all</span><span class="p">,</span> <span class="n">update</span>
|
||
<span class="o">&gt;&gt;&gt;</span> <span class="n">full_import</span><span class="p">()</span>
|
||
<span class="o">&gt;&gt;&gt;</span> <span class="n">update</span><span class="p">()</span>
|
||
</pre></div>
|
||
|
||
|
||
<h2 id="whats-next">What's next ?</h2>
|
||
<p>I want to make a couchapp, in order to navigate PyPI easily. Here are
|
||
some of the features I want to propose:</p>
|
||
<ul>
|
||
<li>List all the available projects</li>
|
||
<li>List all the projects, filtered by specifiers</li>
|
||
<li>List all the projects by author/maintainer</li>
|
||
<li>List all the projects by keywords</li>
|
||
<li>Page for each project.</li>
|
||
<li>Provide a PyPI "Simple" API equivalent, even if I want to replace
|
||
it, I do think it will be really easy to setup mirrors that way,
|
||
with the out of the box couchdb replication</li>
|
||
</ul>
|
||
<p>I also still need to polish the import mechanism, so I can directly
|
||
store in couchdb:</p>
|
||
<ul>
|
||
<li>The OPML files for each project</li>
|
||
<li>The upload_time as couchdb friendly format (list of int)</li>
|
||
<li>The tags as lists (currently it's only a string separated by spaces</li>
|
||
</ul>
|
||
<p>The work I've done by now is available on
|
||
<a href="https://bitbucket.org/ametaireau/pypioncouch/">https://bitbucket.org/ametaireau/pypioncouch/</a>. Keep in mind that it's
|
||
still a work in progress, and everything can break at any time. However,
|
||
any feedback will be appreciated !</p></content></entry><entry><title>Help me to go to the distutils2 paris' sprint</title><link href="https://blog.notmyidea.org/help-me-to-go-to-the-distutils2-paris-sprint.html" rel="alternate"></link><published>2011-01-15T00:00:00+01:00</published><updated>2011-01-15T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2011-01-15:/help-me-to-go-to-the-distutils2-paris-sprint.html</id><summary type="html">
|
||
<p><strong>Edit: Thanks to logilab and some amazing people, I can make it to
|
||
paris for the sprint. Many thanks to them for the support!</strong></p>
|
||
<p>There will be a distutils2 sprint from the 27th to the 30th of january,
|
||
thanks to logilab which will host the event.</p>
|
||
<p>You can find more …</p></summary><content type="html">
|
||
<p><strong>Edit: Thanks to logilab and some amazing people, I can make it to
|
||
paris for the sprint. Many thanks to them for the support!</strong></p>
|
||
<p>There will be a distutils2 sprint from the 27th to the 30th of january,
|
||
thanks to logilab which will host the event.</p>
|
||
<p>You can find more informations about the sprint on the wiki page of the
|
||
event (<a href="http://wiki.python.org/moin/Distutils/SprintParis">http://wiki.python.org/moin/Distutils/SprintParis</a>).</p>
|
||
<p>I really want to go there but I'm unfortunately blocked in UK for money
|
||
reasons. The cheapest two ways I've found is about £80, which I can't
|
||
afford. Following some advices on #distutils, I've set up a ChipIn
|
||
account for that, so if some people want to help me making it to go
|
||
there, they can give me some money that way.</p>
|
||
<p>I'll probably work on the installer (to support old distutils and
|
||
setuptools distributions) and on the uninstaller (depending on the first
|
||
task). If I can't make it to paris, I'll hang around on IRC to give some
|
||
help while needed.</p>
|
||
<p>If you want to contribute some money to help me go there, feel free to
|
||
use this chipin page:
|
||
<a href="http://ametaireau.chipin.com/distutils2-sprint-in-paris">http://ametaireau.chipin.com/distutils2-sprint-in-paris</a></p>
|
||
<p>Thanks for your support !</p></content></entry><entry><title>How to reboot your bebox using the CLI</title><link href="https://blog.notmyidea.org/how-to-reboot-your-bebox-using-the-cli.html" rel="alternate"></link><published>2010-10-21T00:00:00+02:00</published><updated>2010-10-21T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2010-10-21:/how-to-reboot-your-bebox-using-the-cli.html</id><summary type="html">
|
||
<p>I've an internet connection which, for some obscure reasons, tend to be
|
||
very slow from time to time. After rebooting the box (yes, that's a hard
|
||
solution), all the things seems to go fine again.</p>
|
||
<h2 id="edit-using-grep">EDIT : Using grep</h2>
|
||
<p>After a bit of reflexion, that's also really easy to do using …</p></summary><content type="html">
|
||
<p>I've an internet connection which, for some obscure reasons, tend to be
|
||
very slow from time to time. After rebooting the box (yes, that's a hard
|
||
solution), all the things seems to go fine again.</p>
|
||
<h2 id="edit-using-grep">EDIT : Using grep</h2>
|
||
<p>After a bit of reflexion, that's also really easy to do using directly
|
||
the command line tools curl, grep and tail (but really harder to
|
||
read).</p>
|
||
<p><code>`` sourceCode bash
|
||
curl -X POST -u joel:joel http://bebox.config/cgi/b/info/restart/\?be\=0\&amp;l0\=1\&amp;l1\=0\&amp;tid\=RESTART -d "0=17&amp;2=</code>curl -u joel:joel http://bebox.config/cgi/b/info/restart/\?be\=0\&amp;l0\=1\&amp;l1\=0\&amp;tid\=RESTART | grep -o "name='2' value='[0-9]+" | grep -o "[0-9]+" | tail -n 1`&amp;1"</p>
|
||
<div class="highlight"><pre><span></span><span class="c1">## The Python version</span>
|
||
|
||
<span class="n">Well</span><span class="p">,</span> <span class="n">that</span><span class="s1">&#39;s not the optimal solution, that&#39;</span><span class="n">s</span> <span class="n">a</span> <span class="n">bit</span> <span class="s2">&quot;gruik&quot;</span><span class="p">,</span> <span class="n">but</span> <span class="n">it</span>
|
||
<span class="n">works</span><span class="o">.</span>
|
||
|
||
<span class="sb">``</span><span class="err">`</span> <span class="n">sourceCode</span> <span class="n">python</span>
|
||
<span class="kn">import</span> <span class="nn">urllib2</span>
|
||
<span class="kn">import</span> <span class="nn">urlparse</span>
|
||
<span class="kn">import</span> <span class="nn">re</span>
|
||
<span class="kn">import</span> <span class="nn">argparse</span>
|
||
|
||
<span class="n">REBOOT_URL</span> <span class="o">=</span> <span class="s1">&#39;/b/info/restart/?be=0&amp;l0=1&amp;l1=0&amp;tid=RESTART&#39;</span>
|
||
<span class="n">BOX_URL</span> <span class="o">=</span> <span class="s1">&#39;http://bebox.config/cgi&#39;</span>
|
||
|
||
<span class="k">def</span> <span class="nf">open_url</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="p">):</span>
|
||
<span class="n">passman</span> <span class="o">=</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">HTTPPasswordMgrWithDefaultRealm</span><span class="p">()</span>
|
||
<span class="n">passman</span><span class="o">.</span><span class="n">add_password</span><span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="n">url</span><span class="p">,</span> <span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span>
|
||
<span class="n">authhandler</span> <span class="o">=</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">HTTPBasicAuthHandler</span><span class="p">(</span><span class="n">passman</span><span class="p">)</span>
|
||
|
||
<span class="n">opener</span> <span class="o">=</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">build_opener</span><span class="p">(</span><span class="n">authhandler</span><span class="p">)</span>
|
||
|
||
<span class="n">urllib2</span><span class="o">.</span><span class="n">install_opener</span><span class="p">(</span><span class="n">opener</span><span class="p">)</span>
|
||
|
||
<span class="k">return</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">url</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
|
||
|
||
<span class="k">def</span> <span class="nf">reboot</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="p">):</span>
|
||
<span class="n">data</span> <span class="o">=</span> <span class="n">open_url</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span>
|
||
<span class="n">token</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s2">&quot;name\=</span><span class="se">\\</span><span class="s2">&#39;2</span><span class="se">\\</span><span class="s2">&#39; value=</span><span class="se">\\</span><span class="s2">&#39;([0-9]+)</span><span class="se">\\</span><span class="s2">&#39;&quot;</span><span class="p">,</span> <span class="n">data</span><span class="p">)[</span><span class="mi">1</span><span class="p">]</span>
|
||
<span class="n">urllib2</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">urllib2</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="n">url</span><span class="o">=</span><span class="n">url</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="s1">&#39;0=17&amp;2=</span><span class="si">%s</span><span class="s1">&amp;1&#39;</span> <span class="o">%</span> <span class="n">token</span><span class="p">))</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__file__</span> <span class="o">==</span> <span class="s1">&#39;__main__&#39;</span><span class="p">:</span>
|
||
<span class="n">parser</span> <span class="o">=</span> <span class="n">argparse</span><span class="o">.</span><span class="n">ArgumentParser</span><span class="p">(</span><span class="n">description</span><span class="o">=</span><span class="s2">&quot;&quot;&quot;Reboot your bebox !&quot;&quot;&quot;</span><span class="p">)</span>
|
||
|
||
<span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="n">dest</span><span class="o">=</span><span class="s1">&#39;user&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;username&#39;</span><span class="p">)</span>
|
||
<span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="n">dest</span><span class="o">=</span><span class="s1">&#39;password&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;password&#39;</span><span class="p">)</span>
|
||
<span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="n">boxurl</span><span class="o">=</span><span class="s1">&#39;boxurl&#39;</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="n">BOX_URL</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;Base box url. Default is </span><span class="si">%s</span><span class="s1">&#39;</span> <span class="o">%</span> <span class="n">BOX_URL</span><span class="p">)</span>
|
||
|
||
<span class="n">args</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">parse_args</span><span class="p">()</span>
|
||
<span class="n">url</span> <span class="o">=</span> <span class="n">urlparse</span><span class="o">.</span><span class="n">urljoin</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">boxurl</span><span class="p">,</span> <span class="n">REBOOT_URL</span><span class="p">)</span>
|
||
<span class="n">reboot</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">args</span><span class="o">.</span><span class="n">username</span><span class="p">,</span> <span class="n">args</span><span class="o">.</span><span class="n">password</span><span class="p">)</span>
|
||
</pre></div></content></entry><entry><title>Dynamically change your gnome desktop wallpaper</title><link href="https://blog.notmyidea.org/dynamically-change-your-gnome-desktop-wallpaper.html" rel="alternate"></link><published>2010-10-11T00:00:00+02:00</published><updated>2010-10-11T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2010-10-11:/dynamically-change-your-gnome-desktop-wallpaper.html</id><summary type="html">
|
||
<p>In gnome, you can can use a XML file to have a dynamic wallpaper. It's
|
||
not so easy, and you can't just tell: use the pictures in this folder to
|
||
do so.</p>
|
||
<p>You can have a look to the git repository if you want:
|
||
<a href="http://github.com/ametaireau/gnome-background-generator">http://github.com/ametaireau/gnome-background-generator</a></p>
|
||
<p>Some …</p></summary><content type="html">
|
||
<p>In gnome, you can can use a XML file to have a dynamic wallpaper. It's
|
||
not so easy, and you can't just tell: use the pictures in this folder to
|
||
do so.</p>
|
||
<p>You can have a look to the git repository if you want:
|
||
<a href="http://github.com/ametaireau/gnome-background-generator">http://github.com/ametaireau/gnome-background-generator</a></p>
|
||
<p>Some time ago, I've made a little python script to ease that, and you
|
||
can now use it too. It's named "gnome-background-generator", and you can
|
||
install it via pip for instance.</p>
|
||
<p>``` sourceCode shell
|
||
$ pip install gnome-background-generator</p>
|
||
<div class="highlight"><pre><span></span><span class="k">Then</span>, <span class="nv">you</span> <span class="nv">have</span> <span class="nv">just</span> <span class="nv">to</span> <span class="nv">use</span> <span class="nv">it</span> <span class="nv">this</span> <span class="nv">way</span>:
|
||
|
||
``` <span class="nv">sourceCode</span> <span class="nv">shell</span>
|
||
$ <span class="nv">gnome</span><span class="o">-</span><span class="nv">background</span><span class="o">-</span><span class="nv">generator</span> <span class="o">-</span><span class="nv">p</span> <span class="o">~/</span><span class="nv">Images</span><span class="o">/</span><span class="nv">walls</span> <span class="o">-</span><span class="nv">s</span>
|
||
<span class="o">/</span><span class="nv">home</span><span class="o">/</span><span class="nv">alexis</span><span class="o">/</span><span class="nv">Images</span><span class="o">/</span><span class="nv">walls</span><span class="o">/</span><span class="nv">dynamic</span><span class="o">-</span><span class="nv">wallpaper</span>.<span class="nv">xml</span> <span class="nv">generated</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>Here is a extract of the `--help`:</p>
|
||
<p>``` sourceCode shell
|
||
$ gnome-background-generator --help
|
||
usage: gnome-background-generator [-h] [-p PATH] [-o OUTPUT]
|
||
[-t TRANSITION_TIME] [-d DISPLAY_TIME] [-s]
|
||
[-b]</p>
|
||
<p>A simple command line tool to generate an XML file to use for gnome
|
||
wallpapers, to have dynamic walls</p>
|
||
<p>optional arguments:
|
||
-h, --help show this help message and exit
|
||
-p PATH, --path PATH Path to look for the pictures. If no output is
|
||
specified, will be used too for outputing the dynamic-
|
||
wallpaper.xml file. Default value is the current
|
||
directory (.)
|
||
-o OUTPUT, --output OUTPUT
|
||
Output filename. If no filename is specified, a
|
||
dynamic-wallpaper.xml file will be generated in the
|
||
path containing the pictures. You can also use "-" to
|
||
display the xml in the stdout.
|
||
-t TRANSITION_TIME, --transition-time TRANSITION_TIME
|
||
Time (in seconds) transitions must last (default value
|
||
is 2 seconds)
|
||
-d DISPLAY_TIME, --display-time DISPLAY_TIME
|
||
Time (in seconds) a picture must be displayed. Default
|
||
value is 900 (15mn)
|
||
-s, --set-background '''try to set the background using gnome-appearance-
|
||
properties
|
||
-b, --debug
|
||
```</p></content></entry><entry><title>How to install NGINX + PHP 5.3 on FreeBSD.</title><link href="https://blog.notmyidea.org/how-to-install-nginx-php-53-on-freebsd.html" rel="alternate"></link><published>2010-10-10T00:00:00+02:00</published><updated>2010-10-10T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2010-10-10:/how-to-install-nginx-php-53-on-freebsd.html</id><summary type="html">
|
||
<ul>
|
||
<li>
|
||
<p>date<br>
|
||
2010-10-10</p>
|
||
</li>
|
||
<li>
|
||
<p>category<br>
|
||
tech</p>
|
||
</li>
|
||
</ul>
|
||
<p>I've not managed so far to get completely rid of php, so here's a simple
|
||
reminder about how to install php on NGINX, for FreeBSD. Nothing hard,
|
||
but that's worse to have the piece of configuration somewhere !</p>
|
||
<div class="highlight"><pre><span></span><span class="o">#</span> <span class="k">update</span> <span class="n">the</span> <span class="n">ports</span>
|
||
<span class="err">$</span> <span class="n">portsnap</span> <span class="k">fetch</span> <span class="k">update</span>
|
||
|
||
<span class="o">#</span> <span class="n">install</span> <span class="n">php5</span> <span class="n">port …</span></pre></div></summary><content type="html">
|
||
<ul>
|
||
<li>
|
||
<p>date<br>
|
||
2010-10-10</p>
|
||
</li>
|
||
<li>
|
||
<p>category<br>
|
||
tech</p>
|
||
</li>
|
||
</ul>
|
||
<p>I've not managed so far to get completely rid of php, so here's a simple
|
||
reminder about how to install php on NGINX, for FreeBSD. Nothing hard,
|
||
but that's worse to have the piece of configuration somewhere !</p>
|
||
<div class="highlight"><pre><span></span><span class="o">#</span> <span class="k">update</span> <span class="n">the</span> <span class="n">ports</span>
|
||
<span class="err">$</span> <span class="n">portsnap</span> <span class="k">fetch</span> <span class="k">update</span>
|
||
|
||
<span class="o">#</span> <span class="n">install</span> <span class="n">php5</span> <span class="n">port</span>
|
||
<span class="err">$</span> <span class="n">make</span> <span class="n">config</span><span class="o">-</span><span class="k">recursive</span> <span class="o">-</span><span class="k">C</span> <span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">ports</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="n">php5</span><span class="o">-</span><span class="n">extensions</span>
|
||
<span class="err">$</span> <span class="n">make</span> <span class="n">package</span><span class="o">-</span><span class="k">recursive</span> <span class="o">-</span><span class="k">C</span> <span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">ports</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="n">php5</span><span class="o">-</span><span class="n">extensions</span>
|
||
|
||
<span class="o">#</span> <span class="n">install</span> <span class="n">nginx</span>
|
||
<span class="err">$</span> <span class="n">make</span> <span class="n">config</span><span class="o">-</span><span class="k">recursive</span> <span class="o">-</span><span class="k">C</span> <span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">ports</span><span class="o">/</span><span class="n">www</span><span class="o">/</span><span class="n">nginx</span><span class="o">-</span><span class="n">devel</span>
|
||
<span class="err">$</span> <span class="n">make</span> <span class="n">package</span><span class="o">-</span><span class="k">recursive</span> <span class="o">-</span><span class="k">C</span> <span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">ports</span><span class="o">/</span><span class="n">www</span><span class="o">/</span><span class="n">nginx</span><span class="o">-</span><span class="n">devel</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>Now we have all the dependencies installed, we need to configure a bit
|
||
the server.</p>
|
||
<p>That's a simple thing in fact, but it could be good to have something
|
||
that will work without effort over time.</p>
|
||
<p>Here's a sample of my configuration:</p>
|
||
<div class="highlight"><pre><span></span><span class="nv">server</span> {
|
||
<span class="nv">server_name</span> <span class="nv">ndd</span><span class="c1">;</span>
|
||
<span class="nv">set</span> $<span class="nv">path</span> <span class="o">/</span><span class="nv">path</span><span class="o">/</span><span class="nv">to</span><span class="o">/</span><span class="nv">your</span><span class="o">/</span><span class="nv">files</span><span class="c1">;</span>
|
||
<span class="nv">root</span> $<span class="nv">path</span><span class="c1">;</span>
|
||
|
||
<span class="nv">location</span> <span class="o">/</span> {
|
||
<span class="nv">index</span> <span class="nv">index</span>.<span class="nv">php</span><span class="c1">;</span>
|
||
}
|
||
|
||
<span class="nv">location</span> <span class="o">~*</span> <span class="o">^</span>.<span class="o">+</span>.<span class="ss">(</span><span class="nv">jpg</span><span class="o">|</span><span class="nv">jpeg</span><span class="o">|</span><span class="nv">gif</span><span class="o">|</span><span class="nv">css</span><span class="o">|</span><span class="nv">png</span><span class="o">|</span><span class="nv">js</span><span class="o">|</span><span class="nv">ico</span><span class="o">|</span><span class="nv">xml</span><span class="ss">)</span>$ {
|
||
<span class="nv">access_log</span> <span class="nv">off</span><span class="c1">;</span>
|
||
<span class="nv">expires</span> <span class="mi">30</span><span class="nv">d</span><span class="c1">;</span>
|
||
}
|
||
|
||
<span class="nv">location</span> <span class="o">~</span> .<span class="nv">php</span>$ {
|
||
<span class="nv">fastcgi_param</span> <span class="nv">SCRIPT_FILENAME</span> $<span class="nv">path</span><span class="mh">$fa</span><span class="nv">stcgi_script_name</span><span class="c1">;</span>
|
||
<span class="nv">fastcgi_pass</span> <span class="nv">backend</span><span class="c1">;</span>
|
||
<span class="k">include</span> <span class="nv">fastcgi_params</span><span class="c1">;</span>
|
||
}
|
||
}
|
||
|
||
<span class="nv">upstream</span> <span class="nv">backend</span> {
|
||
<span class="nv">server</span> <span class="mi">127</span>.<span class="mi">0</span>.<span class="mi">0</span>.<span class="mi">1</span>:<span class="mi">9000</span><span class="c1">;</span>
|
||
}
|
||
</pre></div>
|
||
|
||
|
||
<p>And that's it !</p></content></entry><entry><title>Pelican, a simple static blog generator in python</title><link href="https://blog.notmyidea.org/pelican-a-simple-static-blog-generator-in-python.html" rel="alternate"></link><published>2010-10-06T00:00:00+02:00</published><updated>2010-10-06T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2010-10-06:/pelican-a-simple-static-blog-generator-in-python.html</id><summary type="html">
|
||
<p>Those days, I've wrote a little python application to fit my blogging
|
||
needs. I'm an occasional blogger, a vim lover, I like restructured text
|
||
and DVCSes, so I've made a little tool that makes good use of all that.</p>
|
||
<p><a href="http://docs.getpelican.com">Pelican</a> (for calepin) is just a simple
|
||
tool to generate your …</p></summary><content type="html">
|
||
<p>Those days, I've wrote a little python application to fit my blogging
|
||
needs. I'm an occasional blogger, a vim lover, I like restructured text
|
||
and DVCSes, so I've made a little tool that makes good use of all that.</p>
|
||
<p><a href="http://docs.getpelican.com">Pelican</a> (for calepin) is just a simple
|
||
tool to generate your blog as static files, letting you using your
|
||
editor of choice (vim!). It's easy to extend, and has a template
|
||
support (via jinja2).</p>
|
||
<p>I've made it to fit <em>my</em> needs. I hope it will fit yours, but maybe it
|
||
wont, and it have not be designed to feet everyone's needs.</p>
|
||
<p>Need an example ? You're looking at it ! This weblog is using pelican
|
||
to be generated, also for the atom feeds.</p>
|
||
<p>I've released it under AGPL, since I want all the modifications to be
|
||
profitable to all the users.</p>
|
||
<p>You can find a repository to fork at
|
||
<a href="https://github.com/getpelican/pelican/">https://github.com/getpelican/pelican/</a>. feel free to hack it !</p>
|
||
<p>If you just want to get started, use your installer of choice (pip,
|
||
easy_install, …) And then have a look to the help (pelican --help)</p>
|
||
<p>``` sourceCode bash
|
||
$ pip install pelican</p>
|
||
<div class="highlight"><pre><span></span><span class="o">##</span> <span class="k">Usage</span>
|
||
|
||
<span class="n">Here</span><span class="err">&#39;</span><span class="n">s</span> <span class="n">a</span> <span class="n">sample</span> <span class="k">usage</span> <span class="k">of</span> <span class="n">pelican</span>
|
||
|
||
<span class="o">```</span> <span class="n">sourceCode</span> <span class="n">bash</span>
|
||
<span class="err">$</span> <span class="n">pelican</span> <span class="p">.</span>
|
||
<span class="n">writing</span> <span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">alexis</span><span class="o">/</span><span class="n">projets</span><span class="o">/</span><span class="n">notmyidea</span><span class="p">.</span><span class="n">org</span><span class="o">/</span><span class="k">output</span><span class="o">/</span><span class="k">index</span><span class="p">.</span><span class="n">html</span>
|
||
<span class="n">writing</span> <span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">alexis</span><span class="o">/</span><span class="n">projets</span><span class="o">/</span><span class="n">notmyidea</span><span class="p">.</span><span class="n">org</span><span class="o">/</span><span class="k">output</span><span class="o">/</span><span class="n">tags</span><span class="p">.</span><span class="n">html</span>
|
||
<span class="n">writing</span> <span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">alexis</span><span class="o">/</span><span class="n">projets</span><span class="o">/</span><span class="n">notmyidea</span><span class="p">.</span><span class="n">org</span><span class="o">/</span><span class="k">output</span><span class="o">/</span><span class="n">categories</span><span class="p">.</span><span class="n">html</span>
|
||
<span class="n">writing</span> <span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">alexis</span><span class="o">/</span><span class="n">projets</span><span class="o">/</span><span class="n">notmyidea</span><span class="p">.</span><span class="n">org</span><span class="o">/</span><span class="k">output</span><span class="o">/</span><span class="n">archives</span><span class="p">.</span><span class="n">html</span>
|
||
<span class="n">writing</span> <span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">alexis</span><span class="o">/</span><span class="n">projets</span><span class="o">/</span><span class="n">notmyidea</span><span class="p">.</span><span class="n">org</span><span class="o">/</span><span class="k">output</span><span class="o">/</span><span class="n">category</span><span class="o">/</span><span class="n">python</span><span class="p">.</span><span class="n">html</span>
|
||
<span class="n">writing</span>
|
||
<span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">alexis</span><span class="o">/</span><span class="n">projets</span><span class="o">/</span><span class="n">notmyidea</span><span class="p">.</span><span class="n">org</span><span class="o">/</span><span class="k">output</span><span class="o">/</span><span class="n">pelican</span><span class="o">-</span><span class="n">a</span><span class="o">-</span><span class="k">simple</span><span class="o">-</span><span class="k">static</span><span class="o">-</span><span class="n">blog</span><span class="o">-</span><span class="n">generator</span><span class="o">-</span><span class="k">in</span><span class="o">-</span><span class="n">python</span><span class="p">.</span><span class="n">html</span>
|
||
<span class="n">Done</span> <span class="o">!</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>You also can use the --help option for the command line to get more
|
||
informations</p>
|
||
<p>``` sourceCode bash
|
||
$pelican --help
|
||
usage: pelican [-h] [-t TEMPLATES] [-o OUTPUT] [-m MARKUP] [-s SETTINGS] [-b]
|
||
path</p>
|
||
<p>A tool to generate a static blog, with restructured text input files.</p>
|
||
<p>positional arguments:
|
||
path Path where to find the content files (default is
|
||
"content").</p>
|
||
<p>optional arguments:
|
||
-h, --help show this help message and exit
|
||
-t TEMPLATES, --templates-path TEMPLATES
|
||
Path where to find the templates. If not specified,
|
||
will uses the ones included with pelican.
|
||
-o OUTPUT, --output OUTPUT
|
||
Where to output the generated files. If not specified,
|
||
a directory will be created, named "output" in the
|
||
current path.
|
||
-m MARKUP, --markup MARKUP
|
||
the markup language to use. Currently only
|
||
ReSTreucturedtext is available.
|
||
-s SETTINGS, --settings SETTINGS
|
||
the settings of the application. Default to None.
|
||
-b, --debug
|
||
```</p>
|
||
<p>Enjoy :)</p></content></entry><entry><title>An amazing summer of code working on distutils2</title><link href="https://blog.notmyidea.org/an-amazing-summer-of-code-working-on-distutils2.html" rel="alternate"></link><published>2010-08-16T00:00:00+02:00</published><updated>2010-08-16T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2010-08-16:/an-amazing-summer-of-code-working-on-distutils2.html</id><summary type="html">
|
||
<p>The <a href="http://code.google.com/soc/">Google Summer of Code</a> I've spent
|
||
working on <a href="http://hg.python.org/distutils2/">distutils2</a> is over. It
|
||
was a really amazing experience, for many reasons.</p>
|
||
<p>First of all, we had a very good team, we were 5 students working on
|
||
distutils2: <a href="http://zubin71.wordpress.com">Zubin</a>,
|
||
<a href="http://wokslog.wordpress.com/">Éric</a>,
|
||
<a href="http://gsoc.djolonga.com/">Josip</a>,
|
||
<a href="http://konryd.blogspot.com/">Konrad</a> and me. In addition,
|
||
<a href="http://mouadino.blogspot.com/">Mouad</a> have worked on the …</p></summary><content type="html">
|
||
<p>The <a href="http://code.google.com/soc/">Google Summer of Code</a> I've spent
|
||
working on <a href="http://hg.python.org/distutils2/">distutils2</a> is over. It
|
||
was a really amazing experience, for many reasons.</p>
|
||
<p>First of all, we had a very good team, we were 5 students working on
|
||
distutils2: <a href="http://zubin71.wordpress.com">Zubin</a>,
|
||
<a href="http://wokslog.wordpress.com/">Éric</a>,
|
||
<a href="http://gsoc.djolonga.com/">Josip</a>,
|
||
<a href="http://konryd.blogspot.com/">Konrad</a> and me. In addition,
|
||
<a href="http://mouadino.blogspot.com/">Mouad</a> have worked on the PyPI testing
|
||
infrastructure. You could find what each person have done on <a href="http://bitbucket.org/tarek/distutils2/wiki/GSoC_2010_teams">the wiki
|
||
page of
|
||
distutils2</a>.</p>
|
||
<p>We were in contact with each others really often, helping us when
|
||
possible (in #distutils), and were continuously aware of the state of
|
||
the work of each participant. This, in my opinion, have bring us in a
|
||
good shape.</p>
|
||
<p>Then, I've learned a lot. Python packaging was completely new to me at
|
||
the time of the GSoC start, and I was pretty unfamiliar with python good
|
||
practices too, as I've been introducing myself to python in the late
|
||
2009.</p>
|
||
<p>I've recently looked at some python code I wrote just three months ago,
|
||
and I was amazed to think about many improvements to made on it. I guess
|
||
this is a good indicator of the path I've traveled since I wrote it.</p>
|
||
<p>This summer was awesome because I've learned about python good
|
||
practices, now having some strong
|
||
<a href="http://mercurial.selenic.com/">mercurial</a> knowledge, and I've seen a
|
||
little how the python community works.</p>
|
||
<p>Then, I would like to say a big thanks to all the mentors that have
|
||
hanged around while needed, on IRC or via mail, and especially my mentor
|
||
for this summer, <a href="http://tarek.ziade.org">Tarek Ziadé</a>.</p>
|
||
<p>Thanks a lot for your motivation, your leadership and your cheerfulness,
|
||
even with a new-born and a new work!</p>
|
||
<h2 id="why">Why ?</h2>
|
||
<p>I wanted to work on python packaging because, as the time pass, we were
|
||
having a sort of complex tools in this field. Each one wanted to add
|
||
features to distutils, but not in a standard way.</p>
|
||
<p>Now, we have PEPs that describes some format we agreed on (see PEP 345),
|
||
and we wanted to have a tool on which users can base their code on,
|
||
that's <a href="http://hg.python.org/distutils2/">distutils2</a>.</p>
|
||
<h2 id="my-job">My job</h2>
|
||
<p>I had to provide a way to crawl the PyPI indexes in a simple way, and do
|
||
some installation / uninstallation scripts.</p>
|
||
<p>All the work done is available in <a href="http://bitbucket.org/ametaireau/distutils2/">my bitbucket
|
||
repository</a>.</p>
|
||
<h3 id="crawling-the-pypi-indexes">Crawling the PyPI indexes</h3>
|
||
<p>There are two ways of requesting informations from the indexes: using
|
||
the "simple" index, that is a kind of REST index, and using XML-RPC.</p>
|
||
<p>I've done the two implementations, and a high level API to query those
|
||
twos. Basically, this supports the mirroring infrastructure defined in
|
||
PEP 381. So far, the work I've done is gonna be used in pip (they've
|
||
basically copy/paste the code, but this will change as soon as we get
|
||
something completely stable for distutils2), and that's a good news, as
|
||
it was the main reason for what I've done that.</p>
|
||
<p>I've tried to have an unified API for the clients, to switch from one to
|
||
another implementation easily. I'm already thinking of adding others
|
||
crawlers to this stuff, and it was made to be extensible.</p>
|
||
<p>If you want to get more informations about the crawlers/PyPI clients,
|
||
please refer to the distutils2 documentation, especially <a href="http://distutils2.notmyidea.org/library/distutils2.index.html">the pages
|
||
about
|
||
indexes</a>.</p>
|
||
<p>You can find the changes I made about this in the
|
||
<a href="http://hg.python.org/distutils2/">distutils2</a> source code .</p>
|
||
<h3 id="installation-uninstallation-scripts">Installation / Uninstallation scripts</h3>
|
||
<p>Next step was to think about an installation script, and an uninstaller.
|
||
I've not done the uninstaller part, and it's a smart part, as it's
|
||
basically removing some files from the system, so I'll probably do it in
|
||
a near future.</p>
|
||
<p><a href="http://hg.python.org/distutils2/">distutils2</a> provides a way to install
|
||
distributions, and to handle dependencies between releases. For now,
|
||
this support is only about the last version of the METADATA (1.2) (See,
|
||
the PEP 345), but I'm working on a compatibility layer for the old
|
||
metadata, and for the informations provided via PIP requires.txt, for
|
||
instance.</p>
|
||
<h3 id="extra-work">Extra work</h3>
|
||
<p>Also, I've done some extra work. this includes:</p>
|
||
<ul>
|
||
<li>working on the PEP 345, and having some discussion about it (about
|
||
the names of some fields).</li>
|
||
<li>writing a PyPI server mock, useful for tests. you can find more
|
||
information about it on the
|
||
<a href="http://distutils.notmyidea.org">documentation</a>.</li>
|
||
</ul>
|
||
<h2 id="futures-plans">Futures plans</h2>
|
||
<p>As I said, I've enjoyed working on distutils2, and the people I've met
|
||
here are really pleasant to work with. So I <em>want</em> to continue
|
||
contributing on python, and especially on python packaging, because
|
||
there is still a lot of things to do in this scope, to get something
|
||
really usable.</p>
|
||
<p>I'm not plainly satisfied by the work I've done, so I'll probably tweak
|
||
it a bit: the installer part is not yet completely finished, and I want
|
||
to add support for a real
|
||
<a href="http://en.wikipedia.org/wiki/Representational_State_Transfer">REST</a>
|
||
index in the future.</p>
|
||
<p>We'll talk again of this in the next months, probably, but we definitely
|
||
need a real
|
||
<a href="http://en.wikipedia.org/wiki/Representational_State_Transfer">REST</a> API
|
||
for <a href="http://pypi.python.org">PyPI</a>, as the "simple" index <em>is</em> an ugly
|
||
hack, in my opinion. I'll work on a serious proposition about this,
|
||
maybe involving <a href="http://couchdb.org">CouchDB</a>, as it seems to be a good
|
||
option for what we want here.</p>
|
||
<h2 id="issues">Issues</h2>
|
||
<p>I've encountered some issues during this summer. The main one is that's
|
||
hard to work remotely, especially being in the same room that we live,
|
||
with others. I like to just think about a project with other people, a
|
||
paper and a pencil, no computers. This have been not so possible at the
|
||
start of the project, as I needed to read a lot of code to understand
|
||
the codebase, and then to read/write emails.</p>
|
||
<p>I've finally managed to work in an office, so good point for home/office
|
||
separation.</p>
|
||
<p>I'd not planned there will be so a high number of emails to read, in
|
||
order to follow what's up in the python world, and be a part of the
|
||
community seems to takes some times to read/write emails, especially for
|
||
those (like me) that arent so confortable with english (but this had
|
||
brought me some english fu !).</p>
|
||
<h2 id="thanks-33">Thanks !</h2>
|
||
<p>A big thanks to <a href="http://www.graine-libre.fr/">Graine Libre</a> and <a href="http://www.makina-corpus.com/">Makina
|
||
Corpus</a>, which has offered me to come
|
||
into their offices from time to time, to share they cheerfulness ! Many
|
||
thanks too to the Google Summer of Code program for setting up such an
|
||
initiative. If you're a student, if you're interested about FOSS, dont
|
||
hesitate any second, it's a really good opportunity to work on
|
||
interesting projects!</p></content></entry><entry><title>Sprinting on distutils2 in Tours</title><link href="https://blog.notmyidea.org/sprinting-on-distutils2-in-tours.html" rel="alternate"></link><published>2010-07-10T00:00:00+02:00</published><updated>2010-07-10T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2010-07-10:/sprinting-on-distutils2-in-tours.html</id><summary type="html">
|
||
<ul>
|
||
<li>
|
||
<p>date<br>
|
||
2010-07-06</p>
|
||
</li>
|
||
<li>
|
||
<p>category<br>
|
||
tech</p>
|
||
</li>
|
||
</ul>
|
||
<p>Yesterday, as I was traveling to Tours, I've took some time to visit
|
||
Éric, another student who's working on distutils2 this summer, as a
|
||
part of the GSoC. Basically, it was to take a drink, discuss a bit about
|
||
distutils2, our respective tasks and general feelings …</p></summary><content type="html">
|
||
<ul>
|
||
<li>
|
||
<p>date<br>
|
||
2010-07-06</p>
|
||
</li>
|
||
<li>
|
||
<p>category<br>
|
||
tech</p>
|
||
</li>
|
||
</ul>
|
||
<p>Yesterday, as I was traveling to Tours, I've took some time to visit
|
||
Éric, another student who's working on distutils2 this summer, as a
|
||
part of the GSoC. Basically, it was to take a drink, discuss a bit about
|
||
distutils2, our respective tasks and general feelings, and to put a face
|
||
on a pseudonym. I'd really enjoyed this time, because Éric knows a lot
|
||
of things about mercurial and python good practices, and I'm eager to
|
||
learn about those. So, we have discussed about things, have not wrote so
|
||
much code, but have some things to propose so far, about documentation,
|
||
and I also provides here some bribes of conversations we had.</p>
|
||
<h2 id="documentation">Documentation</h2>
|
||
<p>While writing the PyPI simple index crawler documentation, I realized
|
||
that we miss some structure, or how-to about the documentation. Yep, you
|
||
read well. We lack documentation on how to make documentation. Heh.
|
||
We're missing some rules to follow, and this lead to a not-so-structured
|
||
final documentation. We probably target three type of publics, and we
|
||
can split the documentation regarding those:</p>
|
||
<ul>
|
||
<li><strong>Packagers</strong> who want to distribute their softwares.</li>
|
||
<li><strong>End users</strong> who need to understand how to use end user commands,
|
||
like the installer/uninstaller</li>
|
||
<li><strong>packaging coders</strong> who <em>use</em> distutils2, as a base for building a
|
||
package manager.</li>
|
||
</ul>
|
||
<p>We also need to discuss about a pattern to follow while writing
|
||
documentation. How many parts do we need ? Where to put the API
|
||
description ? etc. That's maybe seems to be not so important, but I
|
||
guess the readers would appreciate to have the same structure all along
|
||
distutils2 documentation.</p>
|
||
<h2 id="mercurial">Mercurial</h2>
|
||
<p>I'm really <em>not</em> a mercurial power user. I use it on daily basis, but I
|
||
lack of basic knowledge about it. Big thanks Éric for sharing yours with
|
||
me, you're of a great help. We have talked about some mercurial
|
||
extensions that seems to make the life simpler, while used the right
|
||
way. I've not used them so far, so consider this as a personal note.</p>
|
||
<ul>
|
||
<li>hg histedit, to edit the history</li>
|
||
<li>hg crecord, to select the changes to commit</li>
|
||
</ul>
|
||
<p>We have spent some time to review a merge I made sunday, to re-merge it,
|
||
and commit the changes as a new changeset. Awesome. These things make me
|
||
say I <strong>need</strong> to read <a href="http://hgbook.red-bean.com/read/">the hg book</a>,
|
||
and will do as soon as I got some spare time: mercurial seems to be
|
||
simply great. So ... Great. I'm a powerful merger now !</p>
|
||
<h2 id="on-using-tools">On using tools</h2>
|
||
<p>Because we <em>also</em> are <em>hackers</em>, we have shared a bit our ways to code,
|
||
the tools we use, etc. Both of us were using vim, and I've discovered
|
||
vimdiff and hgtk, which will completely change the way I navigate into
|
||
the mercurial history. We aren't "power users", so we have learned from
|
||
each other about vim tips. You can find <a href="http://github.com/ametaireau/dotfiles">my dotfiles on
|
||
github</a>, if it could help.
|
||
They're not perfect, and not intended to be, because changing all the
|
||
time, as I learn. Don't hesitate to have a look, and to propose
|
||
enhancements if you have !</p>
|
||
<h2 id="on-being-pythonic">On being pythonic</h2>
|
||
<p>My background as an old Java user disserves me so far, as the paradigms
|
||
are not the same while coding in python. Hard to find the more pythonic
|
||
way to do, and sometimes hard to unlearn my way to think about software
|
||
engineering. Well, it seems that the only solution is to read code, and
|
||
to re-read import this from times to times ! <a href="http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html">Coding like a
|
||
pythonista</a>
|
||
seems to be a must-read, so, I know what to do.</p>
|
||
<h2 id="conclusion">Conclusion</h2>
|
||
<p>It was really great. Next time, we'll need to focus a bit more on
|
||
distutils2, and to have a bullet list of things to do, but days like
|
||
this one are opportunities to catch ! We'll probably do another sprint
|
||
in a few weeks, stay tuned !</p></content></entry><entry><title>Introducing the distutils2 index crawlers</title><link href="https://blog.notmyidea.org/introducing-the-distutils2-index-crawlers.html" rel="alternate"></link><published>2010-07-06T00:00:00+02:00</published><updated>2010-07-06T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2010-07-06:/introducing-the-distutils2-index-crawlers.html</id><summary type="html">
|
||
<p>I'm working for about a month for distutils2, even if I was being a bit
|
||
busy (as I had some class courses and exams to work on)</p>
|
||
<p>I'll try do sum-up my general feelings here, and the work I've made so
|
||
far. You can also find, if you're interested, my …</p></summary><content type="html">
|
||
<p>I'm working for about a month for distutils2, even if I was being a bit
|
||
busy (as I had some class courses and exams to work on)</p>
|
||
<p>I'll try do sum-up my general feelings here, and the work I've made so
|
||
far. You can also find, if you're interested, my weekly summaries in <a href="http://wiki.notmyidea.org/distutils2_schedule">a
|
||
dedicated wiki page</a>.</p>
|
||
<h2 id="general-feelings">General feelings</h2>
|
||
<p>First, and it's a really important point, the GSoC is going very well,
|
||
for me as for other students, at least from my perspective. It's a
|
||
pleasure to work with such enthusiast people, as this make the global
|
||
atmosphere very pleasant to live.</p>
|
||
<p>First of all, I've spent time to read the existing codebase, and to
|
||
understand what we're going to do, and what's the rationale to do so.</p>
|
||
<p>It's really clear for me now: what we're building is the foundations of
|
||
a packaging infrastructure in python. The fact is that many projects
|
||
co-exists, and comes all with their good concepts. Distutils2 tries to
|
||
take the interesting parts of all, and to provide it in the python
|
||
standard libs, respecting the recently written PEP about packaging.</p>
|
||
<p>With distutils2, it will be simpler to make "things" compatible. So if
|
||
you think about a new way to deal with distributions and packaging in
|
||
python, you can use the Distutils2 APIs to do so.</p>
|
||
<h2 id="tasks">Tasks</h2>
|
||
<p>My main task while working on distutils2 is to provide an installation
|
||
and an un-installation command, as described in PEP 376. For this, I
|
||
first need to get informations about the existing distributions (what's
|
||
their version, name, metadata, dependencies, etc.)</p>
|
||
<p>The main index, you probably know and use, is PyPI. You can access it at
|
||
<a href="http://pypi.python.org">http://pypi.python.org</a>.</p>
|
||
<h2 id="pypi-index-crawling">PyPI index crawling</h2>
|
||
<p>There is two ways to get these informations from PyPI: using the simple
|
||
API, or via xml-rpc calls.</p>
|
||
<p>A goal was to use the version specifiers defined
|
||
in<a href="http://www.python.org/dev/peps/pep-0345/">PEP 345</a> and to provides a
|
||
way to sort the grabbed distributions depending our needs, to pick the
|
||
version we want/need.</p>
|
||
<h3 id="using-the-simple-api">Using the simple API</h3>
|
||
<p>The simple API is composed of HTML pages you can access at
|
||
<a href="http://pypi.python.org/simple/">http://pypi.python.org/simple/</a>.</p>
|
||
<p>Distribute and Setuptools already provides a crawler for that, but it
|
||
deals with their internal mechanisms, and I found that the code was not
|
||
so clear as I want, that's why I've preferred to pick up the good ideas,
|
||
and some implementation details, plus re-thinking the global
|
||
architecture.</p>
|
||
<p>The rules are simple: each project have a dedicated page, which allows
|
||
us to get informations about:</p>
|
||
<ul>
|
||
<li>the distribution download locations (for some versions)</li>
|
||
<li>homepage links</li>
|
||
<li>some other useful informations, as the bugtracker address, for
|
||
instance.</li>
|
||
</ul>
|
||
<p>If you want to find all the distributions of the "EggsAndSpam" project,
|
||
you could do the following (do not take so attention to the names here,
|
||
as the API will probably change a bit):</p>
|
||
<p>``` sourceCode python</p>
|
||
<blockquote>
|
||
<blockquote>
|
||
<blockquote>
|
||
<p>index = SimpleIndex()
|
||
index.find("EggsAndSpam")
|
||
[EggsAndSpam 1.1, EggsAndSpam 1.2, EggsAndSpam 1.3]</p>
|
||
</blockquote>
|
||
</blockquote>
|
||
</blockquote>
|
||
<div class="highlight"><pre><span></span><span class="n">We</span> <span class="n">also</span> <span class="n">could</span> <span class="n">use</span> <span class="k">version</span> <span class="n">specifiers</span><span class="p">:</span>
|
||
|
||
<span class="o">```</span> <span class="n">sourceCode</span> <span class="n">python</span>
|
||
<span class="o">&gt;&gt;&gt;</span> <span class="k">index</span><span class="p">.</span><span class="n">find</span><span class="p">(</span><span class="ss">&quot;EggsAndSpam (&lt; =1.2)&quot;</span><span class="p">)</span>
|
||
<span class="p">[</span><span class="n">EggsAndSpam</span> <span class="mi">1</span><span class="p">.</span><span class="mi">1</span><span class="p">,</span> <span class="n">EggsAndSpam</span> <span class="mi">1</span><span class="p">.</span><span class="mi">2</span><span class="p">]</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>Internally, what's done here is the following:</p>
|
||
<ul>
|
||
<li>it process the <a href="http://pypi.python.org/simple/FooBar/">http://pypi.python.org/simple/FooBar/</a> page,
|
||
searching for download URLs.</li>
|
||
<li>for each found distribution download URL, it creates an object,
|
||
containing informations about the project name, the version and the
|
||
URL where the archive remains.</li>
|
||
<li>it sort the found distributions, using version numbers. The default
|
||
behavior here is to prefer source distributions (over binary ones),
|
||
and to rely on the last "final" distribution (rather than beta,
|
||
alpha etc. ones)</li>
|
||
</ul>
|
||
<p>So, nothing hard or difficult here.</p>
|
||
<p>We provides a bunch of other features, like relying on the new PyPI
|
||
mirroring infrastructure or filter the found distributions by some
|
||
criterias. If you're curious, please browse the <a href="http://distutils2.notmyidea.org/">distutils2
|
||
documentation</a>.</p>
|
||
<h3 id="using-xml-rpc">Using xml-rpc</h3>
|
||
<p>We also can make some xmlrpc calls to retreive informations from PyPI.
|
||
It's a really more reliable way to get informations from from the index
|
||
(as it's just the index that provides the informations), but cost
|
||
processes on the PyPI distant server.</p>
|
||
<p>For now, this way of querying the xmlrpc client is not available on
|
||
Distutils2, as I'm working on it. The main pieces are already present
|
||
(I'll reuse some work I've made from the SimpleIndex querying, and <a href="http://github.com/ametaireau/pypiclient">some
|
||
code already set up</a>), what I
|
||
need to do is to provide a xml-rpc PyPI mock server, and that's on what
|
||
I'm actually working on.</p>
|
||
<h2 id="processes">Processes</h2>
|
||
<p>For now, I'm trying to follow the "documentation, then test, then code"
|
||
path, and that seems to be really needed while working with a community.
|
||
Code is hard to read/understand, compared to documentation, and it's
|
||
easier to change.</p>
|
||
<p>While writing the simple index crawling work, I must have done this to
|
||
avoid some changes on the API, and some loss of time.</p>
|
||
<p>Also, I've set up <a href="http://wiki.notmyidea.org/distutils2_schedule">a
|
||
schedule</a>, and the goal
|
||
is to be sure everything will be ready in time, for the end of the
|
||
summer. (And now, I need to learn to follow schedules ...)</p></content></entry><entry><title>Use Restructured Text (ReST) to power your presentations</title><link href="https://blog.notmyidea.org/use-restructured-text-rest-to-power-your-presentations.html" rel="alternate"></link><published>2010-06-25T00:00:00+02:00</published><updated>2010-06-25T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2010-06-25:/use-restructured-text-rest-to-power-your-presentations.html</id><summary type="html">
|
||
<ul>
|
||
<li>
|
||
<p>date<br>
|
||
2010-06-25</p>
|
||
</li>
|
||
<li>
|
||
<p>category<br>
|
||
tech</p>
|
||
</li>
|
||
</ul>
|
||
<p>Wednesday, we give a presentation, with some friends, about the CouchDB
|
||
Database, to <a href="http://www.toulibre.org">the Toulouse local LUG</a>. Thanks a
|
||
lot to all the presents for being there, it was a pleasure to talk about
|
||
this topic with you. Too bad the season is over now an …</p></summary><content type="html">
|
||
<ul>
|
||
<li>
|
||
<p>date<br>
|
||
2010-06-25</p>
|
||
</li>
|
||
<li>
|
||
<p>category<br>
|
||
tech</p>
|
||
</li>
|
||
</ul>
|
||
<p>Wednesday, we give a presentation, with some friends, about the CouchDB
|
||
Database, to <a href="http://www.toulibre.org">the Toulouse local LUG</a>. Thanks a
|
||
lot to all the presents for being there, it was a pleasure to talk about
|
||
this topic with you. Too bad the season is over now an I quit Toulouse
|
||
next year.</p>
|
||
<p>During our brainstorming about the topic, we used some paper, and we
|
||
wanted to make a presentation the simpler way. First thing that come to
|
||
my mind was using <a href="http://docutils.sourceforge.net/rst.html">restructured
|
||
text</a>, so I've wrote a simple
|
||
file containing our different bullet points. In fact, there is quite
|
||
nothing to do then, to have a working presentation.</p>
|
||
<p>So far, I've used <a href="http://code.google.com/p/rst2pdf/">the rst2pdf
|
||
program</a>, and a simple template, to
|
||
generate output. It's probably simple to have similar results using
|
||
latex + beamer, I'll try this next time, but as I'm not familiar with
|
||
latex syntax, restructured text was a great option.</p>
|
||
<p>Here are <a href="http://files.lolnet.org/alexis/rst-presentations/couchdb/couchdb.pdf">the final PDF
|
||
output</a>,
|
||
<a href="http://files.lolnet.org/alexis/rst-presentations/couchdb/couchdb.rst">Rhe ReST
|
||
source</a>,
|
||
<a href="http://files.lolnet.org/alexis/rst-presentations/slides.style">the theme
|
||
used</a>,
|
||
and the command line to generate the PDF:</p>
|
||
<div class="highlight"><pre><span></span><span class="n">rst2pdf</span> <span class="n">couchdb</span><span class="p">.</span><span class="n">rst</span> <span class="o">-</span><span class="n">b1</span> <span class="o">-</span><span class="n">s</span> <span class="p">..</span><span class="o">/</span><span class="n">slides</span><span class="p">.</span><span class="n">style</span>
|
||
</pre></div></content></entry><entry><title>first week working on distutils2</title><link href="https://blog.notmyidea.org/first-week-working-on-distutils2.html" rel="alternate"></link><published>2010-06-04T00:00:00+02:00</published><updated>2010-06-04T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2010-06-04:/first-week-working-on-distutils2.html</id><summary type="html">
|
||
<p>As I've been working on <a href="http://hg.python.org/distutils2/">Distutils2</a>
|
||
during the past week, taking part of the
|
||
<a href="http://code.google.com/intl/fr/soc/">GSOC</a> program, here is a short
|
||
summary of what I've done so far.</p>
|
||
<p>As my courses are not over yet, I've not worked as much as I wanted, and
|
||
this will continues until the end of …</p></summary><content type="html">
|
||
<p>As I've been working on <a href="http://hg.python.org/distutils2/">Distutils2</a>
|
||
during the past week, taking part of the
|
||
<a href="http://code.google.com/intl/fr/soc/">GSOC</a> program, here is a short
|
||
summary of what I've done so far.</p>
|
||
<p>As my courses are not over yet, I've not worked as much as I wanted, and
|
||
this will continues until the end of June. My main tasks are about
|
||
making installation and uninstallation commands, to have a simple way to
|
||
install distributions via
|
||
<a href="http://hg.python.org/distutils2/">Distutils2</a>.</p>
|
||
<p>To do this, we need to rely on informations provided by the Python
|
||
Package Index (<a href="http://pypi.python.org/">PyPI</a>), and there is at least
|
||
two ways to retreive informations from here: XML-RPC and the "simple"
|
||
API.</p>
|
||
<p>So, I've been working on porting some
|
||
<a href="http://bitbucket.org/tarek/distribute/">Distribute</a> related stuff to
|
||
<a href="http://hg.python.org/distutils2/">Distutils2</a>, cutting off all non
|
||
distutils' things, as we do not want to depend from Distribute's
|
||
internals. My main work has been about reading the whole code, writing
|
||
tests about this and making those tests possible.</p>
|
||
<p>In fact, there was a need of a pypi mocked server, and, after reading
|
||
and introducing myself to the distutils behaviors and code, I've taken
|
||
some time to improve the work <a href="http://bitbucket.org/konrad">Konrad</a>
|
||
makes about this mock.</p>
|
||
<h2 id="a-pypi-server-mock">A PyPI Server mock</h2>
|
||
<p>The mock is embeded in a thread, to make it available during the tests,
|
||
in a non blocking way. We first used <a href="http://wsgi.org">WSGI</a> and
|
||
<a href="http://docs.python.org/library/wsgiref.html">wsgiref</a> in order control
|
||
what to serve, and to log the requests made to the server, but finally
|
||
realised that <a href="http://docs.python.org/library/wsgiref.html">wsgiref</a> is
|
||
not python 2.4 compatible (and we <em>need</em> to be python 2.4 compatible in
|
||
Distutils2).</p>
|
||
<p>So, we switched to
|
||
<a href="http://docs.python.org/library/basehttpserver.html">BaseHTTPServer</a> and
|
||
<a href="http://docs.python.org/library/simplehttpserver.html">SimpleHTTPServer</a>,
|
||
and updated our tests accordingly. It's been an opportunity to realize
|
||
that <a href="http://wsgi.org">WSGI</a> has been a great step forward for making
|
||
HTTP servers, and expose a really simplest way to discuss with HTTP !</p>
|
||
<p>You can find <a href="http://bitbucket.org/ametaireau/distutils2/changesets">the modifications I
|
||
made</a>, and the
|
||
<a href="http://bitbucket.org/ametaireau/distutils2/src/tip/docs/source/test_framework.rst">related
|
||
docs</a>
|
||
about this on <a href="http://bitbucket.org/ametaireau/distutils2/">my bitbucket distutils2
|
||
clone</a>.</p>
|
||
<h2 id="the-pypi-simple-api">The PyPI Simple API</h2>
|
||
<p>So, back to the main problematic: make a python library to access and
|
||
request information stored on PyPI, via the simple API. As I said, I've
|
||
just grabbed the work made from
|
||
<a href="http://bitbucket.org/tarek/distribute/">Distribute</a>, and played a bit
|
||
with, in order to view what are the different use cases, and started to
|
||
write the related tests.</p>
|
||
<h2 id="the-work-to-come">The work to come</h2>
|
||
<p>So, once all use cases covered with tests, I'll rewrite a bit the
|
||
grabbed code, and do some software design work (to not expose all things
|
||
as privates methods, have a clear API, and other things like this), then
|
||
update the tests accordingly and write a documentation to make this
|
||
clear.</p>
|
||
<p>Next step is to a little client, as I've <a href="http://github.com/ametaireau/pypiclient">already started
|
||
here</a> I'll take you updated !</p></content></entry><entry><title>A Distutils2 GSoC</title><link href="https://blog.notmyidea.org/a-distutils2-gsoc.html" rel="alternate"></link><published>2010-05-01T00:00:00+02:00</published><updated>2010-05-01T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2010-05-01:/a-distutils2-gsoc.html</id><summary type="html">
|
||
<p>WOW. I've been accepted to be a part of the <a href="http://code.google.com/intl/fr/soc/">Google Summer Of
|
||
Code</a> program, and will work on
|
||
<a href="http://python.org/">python</a> <a href="http://hg.python.org/distutils2/">distutils2</a>, with <a href="http://pygsoc.wordpress.com/">a</a> <a href="http://konryd.blogspot.com/">lot</a> <a href="http://ziade.org/">of</a> (intersting !) <a href="http://zubin71.wordpress.com/">people</a>.</p>
|
||
<blockquote>
|
||
<p>So, it's about building the successor of Distutils2, ie. "the python
|
||
package manager". Today, there is too many ways to package a python …</p></blockquote></summary><content type="html">
|
||
<p>WOW. I've been accepted to be a part of the <a href="http://code.google.com/intl/fr/soc/">Google Summer Of
|
||
Code</a> program, and will work on
|
||
<a href="http://python.org/">python</a> <a href="http://hg.python.org/distutils2/">distutils2</a>, with <a href="http://pygsoc.wordpress.com/">a</a> <a href="http://konryd.blogspot.com/">lot</a> <a href="http://ziade.org/">of</a> (intersting !) <a href="http://zubin71.wordpress.com/">people</a>.</p>
|
||
<blockquote>
|
||
<p>So, it's about building the successor of Distutils2, ie. "the python
|
||
package manager". Today, there is too many ways to package a python
|
||
application (pip, setuptools, distribute, distutils, etc.) so there is
|
||
a huge effort to make in order to make all this packaging stuff
|
||
interoperable, as pointed out by
|
||
the <a href="http://www.python.org/dev/peps/pep-0376/">PEP 376</a>.</p>
|
||
</blockquote>
|
||
<p>In more details, I'm going to work on the Installer / Uninstaller features of Distutils2, and on a PyPI XML-RPC client for distutils2. Here are the already defined tasks:</p>
|
||
<ul>
|
||
<li>Implement Distutils2 APIs described in PEP 376.</li>
|
||
<li>Add the uninstall command.</li>
|
||
<li>think about a basic installer / uninstaller script. (with deps) --
|
||
similar to pip/easy_install</li>
|
||
<li>in a pypi subpackage;</li>
|
||
<li>Integrate a module similar to setuptools' package_index'</li>
|
||
<li>PyPI XML-RPC client for distutils 2:
|
||
<a href="http://bugs.python.org/issue8190">http://bugs.python.org/issue8190</a></li>
|
||
</ul>
|
||
<p>As I'm relatively new to python, I'll need some extra work in order to apply all good practice, among other things that can make a developper-life joyful. I'll post here, each week, my advancement, and my tought about python and especialy python packaging world.</p></content></entry><entry><title>Python ? go !</title><link href="https://blog.notmyidea.org/python-go.html" rel="alternate"></link><published>2009-12-17T00:00:00+01:00</published><updated>2009-12-17T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2009-12-17:/python-go.html</id><summary type="html">
|
||
<p>Cela fait maintenant un peu plus d'un mois que je travaille sur un
|
||
projet en <a href="http://www.djangoproject.org">django</a>, et que,
|
||
nécessairement, je me forme à <a href="http://python.org/">Python</a>. Je prends
|
||
un plaisir non dissimulé à découvrir ce langage (et à l'utiliser), qui
|
||
ne cesse de me surprendre. Les premiers mots qui me viennent à …</p></summary><content type="html">
|
||
<p>Cela fait maintenant un peu plus d'un mois que je travaille sur un
|
||
projet en <a href="http://www.djangoproject.org">django</a>, et que,
|
||
nécessairement, je me forme à <a href="http://python.org/">Python</a>. Je prends
|
||
un plaisir non dissimulé à découvrir ce langage (et à l'utiliser), qui
|
||
ne cesse de me surprendre. Les premiers mots qui me viennent à l'esprit
|
||
à propos de Python, sont "logique" et "simple". Et pourtant puissant
|
||
pour autant. Je ne manque d'ailleurs pas une occasion pour faire un peu
|
||
d'<em>évangélisation</em> auprès des quelques personnes qui veulent bien
|
||
m'écouter.</p>
|
||
<h2 id="the-zen-of-python">The Zen of Python</h2>
|
||
<p>Avant toute autre chose, je pense utile de citer Tim Peters, et <a href="http://www.python.org/dev/peps/pep-0020/">le
|
||
PEP20</a>, qui constituent une
|
||
très bonne introduction au langage, qui prends la forme d'un <em>easter
|
||
egg</em> présent dans python</p>
|
||
<p>``` sourceCode bash</p>
|
||
<blockquote>
|
||
<blockquote>
|
||
<blockquote>
|
||
<p>import this
|
||
The Zen of Python, by Tim Peters</p>
|
||
</blockquote>
|
||
</blockquote>
|
||
</blockquote>
|
||
<p>Beautiful is better than ugly.
|
||
Explicit is better than implicit.
|
||
Simple is better than complex.
|
||
Complex is better than complicated.
|
||
Flat is better than nested.
|
||
Sparse is better than dense.
|
||
Readability counts.
|
||
Special cases aren't special enough to break the rules.
|
||
Although practicality beats purity.
|
||
Errors should never pass silently.
|
||
Unless explicitly silenced.
|
||
In the face of ambiguity, refuse the temptation to guess.
|
||
There should be one-- and preferably only one --obvious way to do it.
|
||
Although that way may not be obvious at first unless you're Dutch.
|
||
Now is better than never.
|
||
Although never is often better than <em>right</em> now.
|
||
If the implementation is hard to explain, it's a bad idea.
|
||
If the implementation is easy to explain, it may be a good idea.
|
||
Namespaces are one honking great idea -- let's do more of those!
|
||
```</p>
|
||
<p>J'ai la vague impression que c'est ce que j'ai toujours cherché à faire
|
||
en PHP, et particulièrement dans <a href="http://www.spiral-project.org">le framework
|
||
Spiral</a>, mais en ajoutant ces concepts
|
||
dans une sur-couche au langage. Ici, c'est directement de <em>l'esprit</em> de
|
||
python qu'il s'agit, ce qui signifie que la plupart des bibliothèques
|
||
python suivent ces concepts. Elle est pas belle la vie ?</p>
|
||
<h2 id="comment-commencer-et-par-ou">Comment commencer, et par ou ?</h2>
|
||
<p>Pour ma part, j'ai commencé par la lecture de quelques livres et
|
||
articles intéressants, qui constituent une bonne entrée en matière sur
|
||
le sujet (La liste n'est bien évidemment pas exhaustive et vos
|
||
commentaires sont les bienvenus) :</p>
|
||
<ul>
|
||
<li><a href="http://diveintopython.adrahon.org/">Dive into python</a></li>
|
||
<li><a href="http://www.swaroopch.com/notes/Python_fr:Table_des_Matières">A byte of
|
||
python</a></li>
|
||
<li><a href="http://www.amazon.fr/Python-Petit-guide-lusage-développeur/dp/2100508830">Python: petit guide à l'usage du développeur
|
||
agile</a>
|
||
de <a href="http://tarekziade.wordpress.com/">Tarek Ziadé</a></li>
|
||
<li><a href="http://docs.python.org/index.html">La documentation officielle
|
||
python</a>, bien sûr !</li>
|
||
<li><a href="http://video.pycon.fr/videos/pycon-fr-2009/">Les vidéos du
|
||
pyconfr 2009</a>!</li>
|
||
<li>Un peu de temps, et une console python ouverte :)</li>
|
||
</ul>
|
||
<p>J'essaye par ailleurs de partager au maximum les ressources que je
|
||
trouve de temps à autres, que ce soit <a href="http://www.twitter.com/ametaireau">via
|
||
twitter</a> ou <a href="http://delicious.com/ametaireau">via mon compte
|
||
delicious</a>. Allez jeter un œil <a href="http://delicious.com/ametaireau/python">au tag
|
||
python</a> sur mon profil, peut
|
||
être que vous trouverez des choses intéressantes, qui sait!</p>
|
||
<h2 id="un-python-sexy">Un python sexy</h2>
|
||
<p>Quelques fonctionnalités qui devraient vous mettre l'eau à la bouche:</p>
|
||
<ul>
|
||
<li><a href="http://docs.python.org/library/stdtypes.html#comparisons">Le chaînage des opérateurs de
|
||
comparaison</a>
|
||
est possible (a\&lt;b \&lt;c dans une condition)</li>
|
||
<li>Assignation de valeurs multiples (il est possible de faire a,b,c =
|
||
1,2,3 par exemple)</li>
|
||
<li><a href="http://docs.python.org/tutorial/datastructures.html">Les listes</a>
|
||
sont simples à manipuler !</li>
|
||
<li>Les <a href="http://docs.python.org/tutorial/datastructures.html#list-comprehensions">list
|
||
comprehension</a>,
|
||
ou comment faire des opérations complexes sur les listes, de manière
|
||
simple.</li>
|
||
<li>Les
|
||
<a href="http://docs.python.org/library/doctest.html?highlight=doctest">doctests</a>:
|
||
ou comment faire des tests directement dans la documentation de vos
|
||
classes, tout en la documentant avec de vrais exemples.</li>
|
||
<li>Les
|
||
<a href="http://www.python.org/doc/essays/metaclasses/meta-vladimir.txt">métaclasses</a>,
|
||
ou comment contrôler la manière dont les classes se construisent</li>
|
||
<li>Python est <a href="http://wiki.python.org/moin/Why%20is%20Python%20a%20dynamic%20language%20and%20also%20a%20strongly%20typed%20language">un langage à typage fort
|
||
dynamique</a>:
|
||
c'est ce qui m'agaçait avec PHP qui est un langage à typage faible
|
||
dynamique.</li>
|
||
</ul>
|
||
<p>Cous pouvez également aller regarder <a href="http://video.pycon.fr/videos/free/53/">l'atelier donné par Victor Stinner
|
||
durant le Pyconfr 09</a>. Have fun
|
||
!</p></content></entry></feed> |