mirror of
https://github.com/almet/notmyidea.git
synced 2025-04-28 19:42:37 +02:00
2937 lines
No EOL
271 KiB
XML
2937 lines
No EOL
271 KiB
XML
<?xml version="1.0" encoding="utf-8"?>
|
||
<feed xmlns="http://www.w3.org/2005/Atom"><title>Carnets Web - tech</title><link href="https://blog.notmyidea.org/" rel="alternate"></link><link href="https://blog.notmyidea.org/feeds/tech.atom.xml" rel="self"></link><id>https://blog.notmyidea.org/</id><updated>2018-02-25T00:00:00+01:00</updated><entry><title>Webnotes</title><link href="https://blog.notmyidea.org/webnotes.html" rel="alternate"></link><published>2018-02-25T00:00:00+01:00</published><updated>2018-02-25T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2018-02-25:/webnotes.html</id><summary type="html"><p>Quand je navigue en ligne, j'aime bien prendre des notes sur ce que je lis. C'est utile pour les retrouver plus tard. Il existe quelques outils pour ce genre de cas, mais j'ai vraiment eu du mal à trouver un outil qui faisais ce que je voulais, de la manière …</p></summary><content type="html"><p>Quand je navigue en ligne, j'aime bien prendre des notes sur ce que je lis. C'est utile pour les retrouver plus tard. Il existe quelques outils pour ce genre de cas, mais j'ai vraiment eu du mal à trouver un outil qui faisais ce que je voulais, de la manière que je voulais, c'est à dire:</p>
|
||
<ul>
|
||
<li>enregistrer une sélection de texte ainsi que son contexte: heure, site web.</li>
|
||
<li>fonctionner sur Firefox;</li>
|
||
<li>stocker mes notes à un endroit que je contrôle (ce sont mes données, après tout !)</li>
|
||
<li>rester en dehors de mon chemin: je suis en train de lire, pas en train d'organiser mes notes.</li>
|
||
<li>automatiquement partager les notes sur une page web.</li>
|
||
</ul>
|
||
<p>J'ai donc pris un peu de temps pour fabriquer mon outil de prises de notes, que j'ai baptisé « Webnotes ». C'est <a href="https://addons.mozilla.org/en-US/firefox/addon/wwebnotes/">une extension Firefox</a>, qui se configure assez simplement, et qui stocke les données dans une instance de <a href="http://kinto-storage.org/">Kinto</a>.</p>
|
||
<p><img src="https://github.com/almet/webnotes/blob/master/webnotes.gif?raw=true" /></p>
|
||
<p>C'est aussi simple que sélectionner du texte, faire « clic droit » puis « save as webnote », entrer un tag et le tour est joué !</p>
|
||
<p>Mes notes sont disponibles <a href="https://notes.notmyidea.org">sur notes.notmyidea.org</a>, et voici <a href="https://github.com/almet/webnotes">le lien vers les sources</a>, si ça vous intéresse de regarder comment ça fonctionne !</p></content></entry><entry><title>Comment est-ce que vous générez vos formulaires ?</title><link href="https://blog.notmyidea.org/comment-est-ce-que-vous-generez-vos-formulaires.html" rel="alternate"></link><published>2016-05-31T00:00:00+02:00</published><updated>2016-05-31T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2016-05-31:/comment-est-ce-que-vous-generez-vos-formulaires.html</id><summary type="html"><p>TL; DR: Je viens à peine de <em>releaser</em> la première version d'un service de génération de formulaires.
|
||
Allez jeter un coup d'œil sur <a href="https://www.fourmilieres.net">https://www.fourmilieres.net</a></p>
|
||
<p><em>En février 2012, je parlais ici <a href="https://blog.notmyidea.org/carto-forms.html">d'un service de génération de formulaires</a>.
|
||
Depuis, pas mal d'eau à coulé sous les ponts, on est …</em></p></summary><content type="html"><p>TL; DR: Je viens à peine de <em>releaser</em> la première version d'un service de génération de formulaires.
|
||
Allez jeter un coup d'œil sur <a href="https://www.fourmilieres.net">https://www.fourmilieres.net</a></p>
|
||
<p><em>En février 2012, je parlais ici <a href="https://blog.notmyidea.org/carto-forms.html">d'un service de génération de formulaires</a>.
|
||
Depuis, pas mal d'eau à coulé sous les ponts, on est passé par pas mal d'étapes pour
|
||
finalement arriver à une première version de ce service de génération de
|
||
formulaires (à la </em>google forms<em>).</em></p>
|
||
<p>En tant qu'organisateurs d'évènements (petits et gros), je me retrouve souvent
|
||
dans une situation ou je dois créer des formulaires pour recueillir des
|
||
informations. Actuellement, la meilleure solution disponible est <em>Google Forms</em>,
|
||
mais celle ci à plusieurs problèmes, à commencer par le fait que le code n'est
|
||
pas libre et que les données sont stockées chez Google.</p>
|
||
<p>La plupart du temps, le besoin est assez simple: je veux spécifier quelques
|
||
questions, et donner un lien à mes amis pour qu'ils puissent y répondre.
|
||
Je reviens ensuite plus tard pour voir la liste des réponses apportées.</p>
|
||
<p><img alt="Capture de l'interface de création du formulaire" src="{filename}/static/formbuilder-build.png"></p>
|
||
<h2>Fonctionnalités</h2>
|
||
<p>Il existe pas mal de solutions techniques qui essayent de répondre à la même
|
||
problématique, mais la plupart d'entre elles sont assez souvent compliquées,
|
||
nécessitent de se créer un compte, et/ou ne vous laisse pas la main libre sur
|
||
les données générées, voire le code est assez difficile à faire évoluer ou à
|
||
déployer.</p>
|
||
<p>Je voulais donc quelque chose de simple à utiliser <em>et</em> pour les créateurs de
|
||
formulaires <em>et</em> pour les utilisateurs finaux. Pas de chichis, juste quelques
|
||
vues, et des URLs à sauvegarder une fois l'opération terminée.</p>
|
||
<p><img alt="Capture de l'écran avec les URLs générées" src="{filename}/static/formbuilder-created.png">
|
||
<img alt="Capture d'écran d'un exemple de formulaire" src="{filename}/static/formbuilder-form.png"></p>
|
||
<h3>Pas de compte</h3>
|
||
<p>Vous n'avez pas besoin d'avoir un compte sur le site pour commencer à l'utiliser.
|
||
Vous créez simplement un nouveau formulaire puis envoyez le lien à vos amis pour
|
||
qu'eux puissent à leur tour le remplir.</p>
|
||
<p><img alt="Capture de la page d'accueil, ou aucun compte n'est requis" src="{filename}/static/formbuilder-welcome.png"></p>
|
||
<h3>Gardez la main sur vos données</h3>
|
||
<p>Une fois que vous avez récupéré les réponses à vos questions, vous pouvez
|
||
récupérer les données sur votre machines dans un fichier <code>.csv</code>.</p>
|
||
<p><img alt="Capture de la page de resultats, il est possible de télécharger en CSV." src="{filename}/static/formbuilder-results.png"></p>
|
||
<h3>API</h3>
|
||
<p>L'ensemble des données sont en fait stockées dans <a href="https://kinto.readthedocs.org">Kinto</a>
|
||
qui est interrogeable très facilement en HTTP. Ce qui fait qu'il est très facile de
|
||
réutiliser les formulaires que vous avez construits (ou leurs réponses) depuis
|
||
d'autres outils.</p>
|
||
<h3>Auto-hébergeable</h3>
|
||
<p>Un des objectifs de ce projet est de vous redonner la main sur vos données.
|
||
Bien sur, vous pouvez utiliser l'instance qui est mise à votre disposition sur
|
||
<a href="https://www.fourmilieres.net">wwww.fourmilieres.net</a>, mais vous pouvez
|
||
également l'héberger vous même très
|
||
simplement, et vous êtes d'ailleurs fortement encouragés à le faire ! Notre
|
||
objectif n'est pas de stocker l'ensemble des formulaires du monde, mais de
|
||
(re)donner le contrôle aux utilisateurs !</p>
|
||
<h2>On commence petit…</h2>
|
||
<p>Cette <em>release</em> n'est (bien sur) pas parfaite, et il reste encore pas mal de
|
||
travail sur cet outil, mais je pense qu'il s'agit d'une base de travail
|
||
intéressante pour un futur où Google n'a pas la main sur toutes nos données.</p>
|
||
<p>La liste des champs supportés est pour l'instant assez faible (Texte court,
|
||
Texte long, Oui/Non, choix dans une liste) mais elle à vocation à s'étendre, en
|
||
fonction des besoins de chacun.</p>
|
||
<p>J'ai d'ailleurs créé <a href="https://www.fourmilieres.net/#/form/cfd878264cec4ed2">un formulaire pour que vous puissiez me faire part de vos
|
||
retours</a>, n'hésitez pas !</p>
|
||
<h2>Et, euh, comment ça marche ?</h2>
|
||
<p>Le <em>formbuilder</em>, comme j'aime l'appeler se compose en fin de compte de deux
|
||
parties distinctes:</p>
|
||
<ul>
|
||
<li><a href="https://kinto.readthedocs.org">Kinto</a>, un service qui stocke
|
||
des données coté serveur et qui les expose via des <strong>APIs HTTP</strong></li>
|
||
<li><a href="https://github.com/kinto/formbuilder">Le formbuilder</a>, une application
|
||
JavaScript qui ne tourne que coté client (dans votre navigateur) qui permet
|
||
de construire les formulaires et d'envoyer les données sur les <em>APIs</em> coté
|
||
serveur.</li>
|
||
</ul>
|
||
<p>Au niveau de la <em>stack</em> technique, le <strong>formbuilder</strong> est codé en ReactJS. Un
|
||
des points techniques intéressants du projet est qu'il génère en fin de compte du
|
||
<a href="http://jsonschema.net/">JSON Schema</a>, un format de validation de données <em>JSON</em>.</p>
|
||
<p>Donc, reprenons! Vous arrivez sur la page d'accueil puis cliquez sur
|
||
"Create a new form", puis vous vous retrouvez face à une interface ou vous pouvez
|
||
ajouter des champs de formulaire. Une fois ce travail effectué, vous appuyez sur
|
||
"Create the form".</p>
|
||
<ul>
|
||
<li>Le JSON Schema est alors envoyé au serveur Kinto, qui l'utilisera pour valider
|
||
les données qu'il recevra par la suite.</li>
|
||
<li>Ce JSON Schema sera aussi utilisé lors de l'affichage du formulaire aux
|
||
personnes qui le remplissent.</li>
|
||
<li>Un jeton d'accès est généré et ajouté à l'URL, il s'agit de l'identifiant du
|
||
formulaire.</li>
|
||
<li>Un second jeton d'accès administrateur et généré, il vous faut le garder de
|
||
coté pour avoir accès aux réponses.</li>
|
||
</ul>
|
||
<p>Bref, en espérant que ça vous serve ! Un petit pas dans la direction des données
|
||
rendues à leurs utilisateurs !</p></content></entry><entry><title>What's Hawk and how to use it?</title><link href="https://blog.notmyidea.org/whats-hawk-and-how-to-use-it.html" rel="alternate"></link><published>2014-07-31T00:00:00+02:00</published><updated>2014-07-31T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2014-07-31:/whats-hawk-and-how-to-use-it.html</id><summary type="html"><p>At Mozilla, we recently had to implement <a class="reference external" href="https://github.com/hueniverse/hawk">the Hawk authentication scheme</a> for a number of projects, and we came up
|
||
creating two libraries to ease integration into pyramid and node.js apps.</p>
|
||
<p>But maybe you don't know Hawk.</p>
|
||
<p>Hawk is a relatively new technology, crafted by one of the original …</p></summary><content type="html"><p>At Mozilla, we recently had to implement <a class="reference external" href="https://github.com/hueniverse/hawk">the Hawk authentication scheme</a> for a number of projects, and we came up
|
||
creating two libraries to ease integration into pyramid and node.js apps.</p>
|
||
<p>But maybe you don't know Hawk.</p>
|
||
<p>Hawk is a relatively new technology, crafted by one of the original <a class="reference external" href="https://en.wikipedia.org/wiki/OAuth">OAuth</a> specification authors, that intends to
|
||
replace the 2-legged OAuth authentication scheme using a simpler approach.</p>
|
||
<p>It is an authentication scheme for HTTP, built around <a class="reference external" href="https://en.wikipedia.org/wiki/Hmac">HMAC digests</a> of requests and responses.</p>
|
||
<p>Every authenticated client request has an Authorization header containing a MAC
|
||
(Message Authentication Code) and some additional metadata, then each server
|
||
response to authenticated requests contains a Server-Authorization header that
|
||
authenticates the response, so the client is sure it comes from the right
|
||
server.</p>
|
||
<div class="section" id="exchange-of-the-hawk-id-and-hawk-key">
|
||
<h2>Exchange of the hawk id and hawk key</h2>
|
||
<p>To sign the requests, a client needs to retrieve a token id and a token key
|
||
from the server.</p>
|
||
<p>Hawk itself does not define how these credentials should be exchanged
|
||
between the server and the client. The excellent team behind <a class="reference external" href="http://accounts.firefox.com">Firefox Accounts</a> put together a scheme to do that, which acts
|
||
like the following:</p>
|
||
<div class="admonition note">
|
||
<p class="first admonition-title">Note</p>
|
||
<p>All this derivation crazyness might seem a bit complicated, but don't worry,
|
||
we put together some libraries that takes care of that for you automatically.</p>
|
||
<p class="last">If you are not interested into these details, you can directly jump to the
|
||
next section to see how to use the libraries.</p>
|
||
</div>
|
||
<p>When your server application needs to send you the credentials, it will return
|
||
it inside a specific <cite>Hawk-Session-Token</cite> header. This token can be derived to
|
||
split this string in two values (hawk id and hawk key) that you will use to
|
||
sign your next requests.</p>
|
||
<p>In order to get the hawk credentials, you'll need to:</p>
|
||
<p>First, do an <a class="reference external" href="http://en.wikipedia.org/wiki/HKDF">HKDF derivation</a> on the
|
||
given session token. You'll need to use the following parameters:</p>
|
||
<pre class="literal-block">
|
||
key_material = HKDF(hawk_session, &quot;&quot;, 'identity.mozilla.com/picl/v1/sessionToken', 32*2)
|
||
</pre>
|
||
<div class="admonition note">
|
||
<p class="first admonition-title">Note</p>
|
||
<p class="last">The <tt class="docutils literal">identity.mozilla.com/picl/v1/sessionToken</tt> is a reference to this way of
|
||
deriving the credentials, not an actual URL.</p>
|
||
</div>
|
||
<p>Then, the key material you'll get out of the HKDF need to be separated into two
|
||
parts, the first 32 hex caracters are the hawk id, and the next 32 ones are the
|
||
hawk key.</p>
|
||
<p>Credentials:</p>
|
||
<div class="highlight"><pre><span></span><span class="nx">credentials</span> <span class="o">=</span> <span class="p">{</span>
|
||
<span class="s1">&#39;id&#39;</span><span class="o">:</span> <span class="nx">keyMaterial</span><span class="p">[</span><span class="mi">0</span><span class="o">:</span><span class="mi">32</span><span class="p">],</span>
|
||
<span class="s1">&#39;key&#39;</span><span class="o">:</span> <span class="nx">keyMaterial</span><span class="p">[</span><span class="mi">32</span><span class="o">:</span><span class="mi">64</span><span class="p">],</span>
|
||
<span class="s1">&#39;algorithm&#39;</span><span class="o">:</span> <span class="s1">&#39;sha256&#39;</span>
|
||
<span class="p">}</span>
|
||
</pre></div>
|
||
</div>
|
||
<div class="section" id="httpie">
|
||
<h2>Httpie</h2>
|
||
<p>To showcase APIs in the documentation, I like to use <a class="reference external" href="https://github.com/jakubroztocil/httpie">httpie</a>, a curl-replacement with a nicer
|
||
API, built around <a class="reference external" href="http://python-requests.org">the python requests library</a>.</p>
|
||
<p>Luckily, HTTPie allows you to plug different authentication schemes for it, so <a class="reference external" href="https://github.com/mozilla-services/requests-hawk">I wrote
|
||
a wrapper</a> around <a class="reference external" href="https://github.com/kumar303/mohawk">mohawk</a> to add hawk support to the requests lib.</p>
|
||
<p>Doing hawk requests in your terminal is now as simple as:</p>
|
||
<pre class="literal-block">
|
||
$ pip install requests-hawk httpie
|
||
$ http GET localhost:5000/registration --auth-type=hawk --auth='id:key'
|
||
</pre>
|
||
<p>In addition, it will help you to craft requests using the requests library:</p>
|
||
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">requests</span>
|
||
<span class="kn">from</span> <span class="nn">requests_hawk</span> <span class="kn">import</span> <span class="n">HawkAuth</span>
|
||
|
||
<span class="n">hawk_auth</span> <span class="o">=</span> <span class="n">HawkAuth</span><span class="p">(</span>
|
||
<span class="n">credentials</span><span class="o">=</span><span class="p">{</span><span class="s1">&#39;id&#39;</span><span class="p">:</span> <span class="nb">id</span><span class="p">,</span> <span class="s1">&#39;key&#39;</span><span class="p">:</span> <span class="n">key</span><span class="p">,</span> <span class="s1">&#39;algorithm&#39;</span><span class="p">:</span> <span class="s1">&#39;sha256&#39;</span><span class="p">})</span>
|
||
|
||
<span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="s2">&quot;/url&quot;</span><span class="p">,</span> <span class="n">auth</span><span class="o">=</span><span class="n">hawk_auth</span><span class="p">)</span>
|
||
</pre></div>
|
||
<p>Alternatively, if you don't have the token id and key, you can pass the hawk
|
||
session token I talked about earlier and the lib will take care of the
|
||
derivation for you:</p>
|
||
<div class="highlight"><pre><span></span><span class="n">hawk_auth</span> <span class="o">=</span> <span class="n">HawkAuth</span><span class="p">(</span>
|
||
<span class="n">hawk_session</span><span class="o">=</span><span class="n">resp</span><span class="o">.</span><span class="n">headers</span><span class="p">[</span><span class="s1">&#39;hawk-session-token&#39;</span><span class="p">],</span>
|
||
<span class="n">server_url</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">server_url</span>
|
||
<span class="p">)</span>
|
||
<span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="s2">&quot;/url&quot;</span><span class="p">,</span> <span class="n">auth</span><span class="o">=</span><span class="n">hawk_auth</span><span class="p">)</span>
|
||
</pre></div>
|
||
</div>
|
||
<div class="section" id="integrate-with-python-pyramid-apps">
|
||
<h2>Integrate with python pyramid apps</h2>
|
||
<p>If you're writing pyramid applications, you'll be happy to learn that <a class="reference external" href="https://www.rfk.id.au/blog/">Ryan
|
||
Kelly</a> put together a library that makes Hawk
|
||
work as an Authentication provider for them. I'm chocked how simple it
|
||
is to use it.</p>
|
||
<p>Here is a demo of how we implemented it for Daybed:</p>
|
||
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">pyramid_hawkauth</span> <span class="kn">import</span> <span class="n">HawkAuthenticationPolicy</span>
|
||
|
||
<span class="n">policy</span> <span class="o">=</span> <span class="n">HawkAuthenticationPolicy</span><span class="p">(</span><span class="n">decode_hawk_id</span><span class="o">=</span><span class="n">get_hawk_id</span><span class="p">)</span>
|
||
<span class="n">config</span><span class="o">.</span><span class="n">set_authentication_policy</span><span class="p">(</span><span class="n">authn_policy</span><span class="p">)</span>
|
||
</pre></div>
|
||
<p>The <cite>get_hawk_id</cite> function is a function that takes a request and
|
||
a tokenid and returns a tuple of <cite>(token_id, token_key)</cite>.</p>
|
||
<p>How you want to store the tokens and retrieve them is up to you. The default
|
||
implementation (e.g. if you don't pass a <cite>decode_hawk_id</cite> function) decodes the
|
||
key from the token itself, using a master secret on the server (so you don't
|
||
need to store anything).</p>
|
||
</div>
|
||
<div class="section" id="integrate-with-node-js-express-apps">
|
||
<h2>Integrate with node.js Express apps</h2>
|
||
<p>We had to implement Hawk authentication for two node.js projects and finally
|
||
came up factorizing everything in a library for express, named <a class="reference external" href="https://github.com/mozilla-services/express-hawkauth">express-hawkauth</a>.</p>
|
||
<p>In order to plug it in your application, you'll need to use it as
|
||
a middleware:</p>
|
||
<div class="highlight"><pre><span></span><span class="kd">var</span> <span class="nx">express</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s2">&quot;express&quot;</span><span class="p">);</span>
|
||
<span class="kd">var</span> <span class="nx">hawk</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s2">&quot;express-hawkauth&quot;</span><span class="p">);</span>
|
||
<span class="nx">app</span> <span class="o">=</span> <span class="nx">express</span><span class="p">();</span>
|
||
|
||
<span class="kd">var</span> <span class="nx">hawkMiddleware</span> <span class="o">=</span> <span class="nx">hawk</span><span class="p">.</span><span class="nx">getMiddleware</span><span class="p">({</span>
|
||
<span class="nx">hawkOptions</span><span class="o">:</span> <span class="p">{},</span>
|
||
<span class="nx">getSession</span><span class="o">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">tokenId</span><span class="p">,</span> <span class="nx">cb</span><span class="p">)</span> <span class="p">{</span>
|
||
<span class="c1">// A function which pass to the cb the key and algorithm for the</span>
|
||
<span class="c1">// given token id. First argument of the callback is a potential</span>
|
||
<span class="c1">// error.</span>
|
||
<span class="nx">cb</span><span class="p">(</span><span class="kc">null</span><span class="p">,</span> <span class="p">{</span><span class="nx">key</span><span class="o">:</span> <span class="s2">&quot;key&quot;</span><span class="p">,</span> <span class="nx">algorithm</span><span class="o">:</span> <span class="s2">&quot;sha256&quot;</span><span class="p">});</span>
|
||
<span class="p">},</span>
|
||
<span class="nx">createSession</span><span class="o">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">id</span><span class="p">,</span> <span class="nx">key</span><span class="p">,</span> <span class="nx">cb</span><span class="p">)</span> <span class="p">{</span>
|
||
<span class="c1">// A function which stores a session for the given id and key.</span>
|
||
<span class="c1">// Argument returned is a potential error.</span>
|
||
<span class="nx">cb</span><span class="p">(</span><span class="kc">null</span><span class="p">);</span>
|
||
<span class="p">},</span>
|
||
<span class="nx">setUser</span><span class="o">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">req</span><span class="p">,</span> <span class="nx">res</span><span class="p">,</span> <span class="nx">tokenId</span><span class="p">,</span> <span class="nx">cb</span><span class="p">)</span> <span class="p">{</span>
|
||
<span class="c1">// A function that uses req and res, the hawkId when they&#39;re known so</span>
|
||
<span class="c1">// that it can tweak it. For instance, you can store the tokenId as the</span>
|
||
<span class="c1">// user.</span>
|
||
<span class="nx">req</span><span class="p">.</span><span class="nx">user</span> <span class="o">=</span> <span class="nx">tokenId</span><span class="p">;</span>
|
||
<span class="p">}</span>
|
||
<span class="p">});</span>
|
||
|
||
<span class="nx">app</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="s2">&quot;/hawk-enabled-endpoint&quot;</span><span class="p">,</span> <span class="nx">hawkMiddleware</span><span class="p">);</span>
|
||
</pre></div>
|
||
<p>If you pass the <cite>createSession</cite> parameter, all non-authenticated requests will
|
||
create a new hawk session and return it with the response, in the
|
||
<cite>Hawk-Session-Token</cite> header.</p>
|
||
<p>If you want to only check a valid hawk session exists (without creating a new
|
||
one), just create a middleware which doesn't have any <cite>createSession</cite> parameter
|
||
defined.</p>
|
||
</div>
|
||
<div class="section" id="some-reference-implementations">
|
||
<h2>Some reference implementations</h2>
|
||
<p>As a reference, here is how we're using the libraries I'm talking about, in
|
||
case that helps you to integrate with your projects.</p>
|
||
<ul class="simple">
|
||
<li>The Mozilla Loop server <a class="reference external" href="https://github.com/mozilla-services/loop-server/blob/master/loop/index.js#L70-L133">uses hawk as authentication once you're logged in with
|
||
a valid BrowserID assertion</a>;
|
||
request, to keep a session between client and server;</li>
|
||
<li><a class="reference external" href="https://github.com/spiral-project/daybed/commit/f178b4e43015fa077430798dcd3d0886c7611caf">I recently added hawk support on the Daybed project</a>
|
||
(that's a pyramid / cornice) app.</li>
|
||
<li>It's also interesting to note that Kumar put together <a class="reference external" href="http://hawkrest.readthedocs.org/en/latest/">hawkrest, for the
|
||
django rest framework</a></li>
|
||
</ul>
|
||
</div>
|
||
</content></entry><entry><title>New year python meme, 2012</title><link href="https://blog.notmyidea.org/new-year-python-meme-2012.html" rel="alternate"></link><published>2013-07-01T00:00:00+02:00</published><updated>2013-07-01T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2013-07-01:/new-year-python-meme-2012.html</id><summary type="html"><p>This series, also known as &quot;<a class="reference external" href="http://ziade.org">Tarek Ziadé</a> strikes again&quot;,
|
||
is a good occasion to take a look back at 2012 and see what I've done related
|
||
to python. So, let's try.</p>
|
||
<p><strong>1. What’s the coolest Python application, framework or library you have
|
||
discovered in 2012 ?</strong></p>
|
||
<p>I discovered <a class="reference external" href="http://docs.pylonsproject.org/en/latest/docs/pyramid.html">Pyramid</a> this …</p></summary><content type="html"><p>This series, also known as &quot;<a class="reference external" href="http://ziade.org">Tarek Ziadé</a> strikes again&quot;,
|
||
is a good occasion to take a look back at 2012 and see what I've done related
|
||
to python. So, let's try.</p>
|
||
<p><strong>1. What’s the coolest Python application, framework or library you have
|
||
discovered in 2012 ?</strong></p>
|
||
<p>I discovered <a class="reference external" href="http://docs.pylonsproject.org/en/latest/docs/pyramid.html">Pyramid</a> this year and
|
||
quite enjoyed it. I like the way they bring back some nice concepts on the
|
||
table, and how they compete with other frameworks. That's nicely decomposed and
|
||
everything fits together really well. I learned a bit about its internal
|
||
routing mechanism when working on <a class="reference external" href="http://cornice.rtfd.org">Cornice</a>, and
|
||
it's really well done.</p>
|
||
<p><strong>2. What new programming technique did you learn in 2012 ?</strong></p>
|
||
<p>I learned about asynchronous programming, something I wasn't really aware of.
|
||
(Yes, that's a shame). This was really helpful to understand some bits of
|
||
<a class="reference external" href="http://circus.io">Circus</a>. I also started to learn about some related
|
||
concepts present in <a class="reference external" href="http://golang.org">go</a> or in <a class="reference external" href="http://erlang.org">erlang</a> with <a class="reference external" href="http://en.wikipedia.org/wiki/Actor_model">the actor model</a></p>
|
||
<p><strong>3. Which open source project did you contribute to the most in 2012 ? What did
|
||
you do ?</strong></p>
|
||
<p>I worked a bunch on <a class="reference external" href="http://cornice.rtfd.org">Cornice</a> and <a class="reference external" href="http://circus.io">Circus</a>, not as much as I would like, but that was two really
|
||
interesting projects. For Cornice, I completely <a class="reference external" href="http://blog.notmyidea.org/refactoring-cornice.html">refactored the internals</a> back in May, and since
|
||
that, I added support for <a class="reference external" href="https://github.com/SPORE/specifications">SPORE</a>
|
||
and am currently working on porting it to Python 3 and adding support for <a class="reference external" href="http://www.w3.org/TR/cors/">CORS</a>. For Circus, I worked on the web interface and
|
||
on other bits of the projects related to stats.</p>
|
||
<p>I didn't contributed that much to <a class="reference external" href="http://getpelican.com">Pelican</a>, mainly
|
||
because I'm less excited about it than I was previously: the project is
|
||
working and needs to focus more on code quality than features. We started in
|
||
this direction, hopefully it will pay-off; but I committed a lot less code than
|
||
previously.</p>
|
||
<p><strong>4. Which Python blog or website did you read the most in 2012 ?</strong></p>
|
||
<p>I really don't know. I found some interesting python-related news on <a class="reference external" href="http://news.ycombinator.com/">hacker
|
||
news</a> and on the printed version as well,
|
||
<a class="reference external" href="http://hackermonthly.com/">hacker monthly</a>. Twitter and IRC got me some
|
||
interesting articles as well.</p>
|
||
<p><strong>5. What are the three top things you want to learn in 2013 ?</strong></p>
|
||
<p>On the computer science side, I want to learn more about other paradigms, maybe
|
||
in other languages. I like python, but I want to learn about other concepts,
|
||
maybe some of them don't really fit with python.</p>
|
||
<p>For instance, I don't know that much about pattern matching or about the actor
|
||
model. Well I know what it is, but I would like to make good use of them in
|
||
order to have something nice and useful.</p>
|
||
<p>Also, I want to learn how to make a product. From scratch. Which means
|
||
designing, implementing and maintaining something. Maybe this will be the case
|
||
with <a class="reference external" href="https://github.com/spiral-project/daybed">daybed</a> ? Who knows.</p>
|
||
<p>I want to get better at building communities. Working with others is something
|
||
that can be complex and quite hard sometimes. I want to learn how to handle
|
||
this better.</p>
|
||
<p>On the other side, I want to learn about tons of other non-tech things: taking
|
||
pictures, finding a good balance between life and work, working with children,
|
||
teaching and animating workshops, writing articles, be better at reading (being
|
||
critic when I'm thinking!).</p>
|
||
<p><strong>6. What is the top software, application or library you wish someone would
|
||
write in 2013 ?</strong></p>
|
||
<p>The one missing thing, IMO, is a <strong>good</strong> webmail reader, in python. One we
|
||
could contribute to, one we could write features for, and one that could come
|
||
and compete with gmail, in term of features of UI.</p>
|
||
<p>However, most of the time, I'm just impressed by the new ideas that come from
|
||
others. I would like to have a library to handle the actor model in a nice way
|
||
in Python, I would like to see packaging fixed in python :-)</p>
|
||
<p>Want to do your own list? Here's how:</p>
|
||
<ul class="simple">
|
||
<li>copy-paste the questions and answer to them in your blog</li>
|
||
<li>tweet it with the #2012pythonmeme hashtag</li>
|
||
</ul>
|
||
</content></entry><entry><title>Implementing CORS in Cornice</title><link href="https://blog.notmyidea.org/cross-origin-in-cornice.html" rel="alternate"></link><published>2013-04-02T00:00:00+02:00</published><updated>2013-04-02T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2013-04-02:/cross-origin-in-cornice.html</id><summary type="html"><div class="admonition note">
|
||
<p class="first admonition-title">Note</p>
|
||
<p class="last">I'm cross-posting <a class="reference external" href="https://blog.mozilla.org/services/">on the mozilla services weblog</a>. Since this is the
|
||
first time we're doing that, I though it could be useful to point you
|
||
there. Check it out and expect more technical articles there in the future.</p>
|
||
</div>
|
||
<p>For security reasons, it's not possible to do cross-domain requests. In …</p></summary><content type="html"><div class="admonition note">
|
||
<p class="first admonition-title">Note</p>
|
||
<p class="last">I'm cross-posting <a class="reference external" href="https://blog.mozilla.org/services/">on the mozilla services weblog</a>. Since this is the
|
||
first time we're doing that, I though it could be useful to point you
|
||
there. Check it out and expect more technical articles there in the future.</p>
|
||
</div>
|
||
<p>For security reasons, it's not possible to do cross-domain requests. In other
|
||
words, if you have a page served from the domain <cite>lolnet.org</cite>, it will not be
|
||
possible for it to get data from <cite>notmyidea.org</cite>.</p>
|
||
<p>Well, it's possible, using tricks and techniques like <a class="reference external" href="http://en.wikipedia.org/wiki/JSONP">JSONP</a>, but that doesn't work all the time (see
|
||
<a class="reference external" href="#how-this-is-different-from-jsonp">the section below</a>). I remember myself
|
||
doing some simple proxies on my domain server to be able to query other's API.</p>
|
||
<p>Thankfully, there is a nicer way to do this, namely, &quot;Cross Origin
|
||
Resource-Sharing&quot;, or <a class="reference external" href="http://www.w3.org/TR/cors/">CORS</a>.</p>
|
||
<div class="section" id="you-want-an-icecream-go-ask-your-dad-first">
|
||
<h2>You want an icecream? Go ask your dad first.</h2>
|
||
<p>If you want to use CORS, you need the API you're querying to support it; on the
|
||
server side.</p>
|
||
<p>The HTTP server need to answer to the <cite>OPTIONS</cite> verb, and with the appropriate
|
||
response headers.</p>
|
||
<p><cite>OPTIONS</cite> is sent as what the authors of the spec call a &quot;preflight request&quot;;
|
||
just before doing a request to the API, the <em>User-Agent</em> (the browser most of
|
||
the time) asks the permission to the resource, with an <cite>OPTIONS</cite> call.</p>
|
||
<p>The server answers, and tell what is available and what isn't:</p>
|
||
<img alt="The CORS flow (from the HTML5 CORS tutorial)" src="images/cors_flow.png" />
|
||
<ul class="simple">
|
||
<li>1a. The User-Agent, rather than doing the call directly, asks the server, the
|
||
API, the permission to do the request. It does so with the following headers:<ul>
|
||
<li><strong>Access-Control-Request-Headers</strong>, contains the headers the User-Agent
|
||
want to access.</li>
|
||
<li><strong>Access-Control-Request-Method</strong> contains the method the User-Agent want
|
||
to access.</li>
|
||
</ul>
|
||
</li>
|
||
<li>1b. The API answers what is authorized:<ul>
|
||
<li><strong>Access-Control-Allow-Origin</strong> the origin that's accepted. Can be <cite>*</cite> or
|
||
the domain name.</li>
|
||
<li><strong>Access-Control-Allow-Methods</strong> a <em>list</em> of allowed methods. This can be
|
||
cached. Note than the request asks permission for one method and the
|
||
server should return a list of accepted methods.</li>
|
||
<li><strong>Access-Allow-Headers</strong> a list of allowed headers, for all of the
|
||
methods, since this can be cached as well.</li>
|
||
</ul>
|
||
</li>
|
||
<li><ol class="first arabic" start="2">
|
||
<li>The User-Agent can do the &quot;normal&quot; request.</li>
|
||
</ol>
|
||
</li>
|
||
</ul>
|
||
<p>So, if you want to access the <cite>/icecream</cite> resource, and do a PUT there, you'll
|
||
have the following flow:</p>
|
||
<pre class="literal-block">
|
||
&gt; OPTIONS /icecream
|
||
&gt; Access-Control-Request-Methods = PUT
|
||
&gt; Origin: notmyidea.org
|
||
&lt; Access-Control-Allow-Origin = notmyidea.org
|
||
&lt; Access-Control-Allow-Methods = PUT,GET,DELETE
|
||
200 OK
|
||
</pre>
|
||
<p>You can see that we have an <cite>Origin</cite> Header in the request, as well as
|
||
a <cite>Access-Control-Request-Methods</cite>. We're here asking if we have the right, as
|
||
<cite>notmyidea.org</cite>, to do a <cite>PUT</cite> request on <cite>/icecream</cite>.</p>
|
||
<p>And the server tells us that we can do that, as well as <cite>GET</cite> and <cite>DELETE</cite>.</p>
|
||
<p>I'll not cover all the details of the CORS specification here, but bear in mind
|
||
than with CORS, you can control what are the authorized methods, headers,
|
||
origins, and if the client is allowed to send authentication information or
|
||
not.</p>
|
||
</div>
|
||
<div class="section" id="a-word-about-security">
|
||
<h2>A word about security</h2>
|
||
<p>CORS is not an answer for every cross-domain call you want to do, because you
|
||
need to control the service you want to call. For instance, if you want to
|
||
build a feed reader and access the feeds on different domains, you can be
|
||
pretty much sure that the servers will not implement CORS, so you'll need to
|
||
write a proxy yourself, to provide this.</p>
|
||
<p>Secondly, if misunderstood, CORS can be insecure, and cause
|
||
problems. Because the rules apply when a client wants to do a request to
|
||
a server, you need to be extra careful about who you're authorizing.</p>
|
||
<p>An incorrectly secured CORS server can be accessed by a malicious client very easily,
|
||
bypassing network security. For instance, if you host a server on an intranet
|
||
that is only available from behind a VPN but accepts every cross-origin call. A bad guy
|
||
can inject javascript into the browser of a user who has access to your
|
||
protected server and make calls to your service, which is probably not what you want.</p>
|
||
</div>
|
||
<div class="section" id="how-this-is-different-from-jsonp">
|
||
<h2>How this is different from JSONP?</h2>
|
||
<p>You may know the <a class="reference external" href="http://en.wikipedia.org/wiki/JSONP">JSONP</a> protocol. JSONP
|
||
allows cross origin, but for a particular use case, and does have some
|
||
drawbacks (for instance, it's not possible to do DELETEs or PUTs with JSONP).</p>
|
||
<p>JSONP exploits the fact that it is possible to get information from another domain
|
||
when you are asking for javascript code, using the <cite>&lt;script&gt;</cite> element.</p>
|
||
<blockquote>
|
||
Exploiting the open policy for &lt;script&gt; elements, some pages use them to
|
||
retrieve JavaScript code that operates on dynamically generated
|
||
JSON-formatted data from other origins. This usage pattern is known as
|
||
JSONP. Requests for JSONP retrieve not JSON, but arbitrary JavaScript code.
|
||
They are evaluated by the JavaScript interpreter, not parsed by a JSON
|
||
parser.</blockquote>
|
||
</div>
|
||
<div class="section" id="using-cors-in-cornice">
|
||
<h2>Using CORS in Cornice</h2>
|
||
<p>Okay, things are hopefully clearer about CORS, let's see how we
|
||
implemented it on the server-side.</p>
|
||
<p>Cornice is a toolkit that lets you define resources in python and takes care of
|
||
the heavy lifting for you, so I wanted it to take care of the CORS support as
|
||
well.</p>
|
||
<p>In Cornice, you define a service like this:</p>
|
||
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">cornice</span> <span class="kn">import</span> <span class="n">Service</span>
|
||
|
||
<span class="n">foobar</span> <span class="o">=</span> <span class="n">Service</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s2">&quot;foobar&quot;</span><span class="p">,</span> <span class="n">path</span><span class="o">=</span><span class="s2">&quot;/foobar&quot;</span><span class="p">)</span>
|
||
|
||
<span class="c1"># and then you do something with it</span>
|
||
<span class="nd">@foobar.get</span><span class="p">()</span>
|
||
<span class="k">def</span> <span class="nf">get_foobar</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
|
||
<span class="c1"># do something with the request.</span>
|
||
</pre></div>
|
||
<p>To add CORS support to this resource, you can go this way, with the
|
||
<cite>cors_origins</cite> parameter:</p>
|
||
<div class="highlight"><pre><span></span><span class="n">foobar</span> <span class="o">=</span> <span class="n">Service</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">&#39;foobar&#39;</span><span class="p">,</span> <span class="n">path</span><span class="o">=</span><span class="s1">&#39;/foobar&#39;</span><span class="p">,</span> <span class="n">cors_origins</span><span class="o">=</span><span class="p">(</span><span class="s1">&#39;*&#39;</span><span class="p">,))</span>
|
||
</pre></div>
|
||
<p>Ta-da! You have enabled CORS for your service. <strong>Be aware that you're
|
||
authorizing anyone to query your server, that may not be what you want.</strong></p>
|
||
<p>Of course, you can specify a list of origins you trust, and you don't need
|
||
to stick with <cite>*</cite>, which means &quot;authorize everyone&quot;.</p>
|
||
<div class="section" id="headers">
|
||
<h3>Headers</h3>
|
||
<p>You can define the headers you want to expose for the service:</p>
|
||
<div class="highlight"><pre><span></span><span class="n">foobar</span> <span class="o">=</span> <span class="n">Service</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">&#39;foobar&#39;</span><span class="p">,</span> <span class="n">path</span><span class="o">=</span><span class="s1">&#39;/foobar&#39;</span><span class="p">,</span> <span class="n">cors_origins</span><span class="o">=</span><span class="p">(</span><span class="s1">&#39;*&#39;</span><span class="p">,))</span>
|
||
|
||
<span class="nd">@foobar.get</span><span class="p">(</span><span class="n">cors_headers</span><span class="o">=</span><span class="p">(</span><span class="s1">&#39;X-My-Header&#39;</span><span class="p">,</span> <span class="s1">&#39;Content-Type&#39;</span><span class="p">))</span>
|
||
<span class="k">def</span> <span class="nf">get_foobars_please</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="s2">&quot;some foobar for you&quot;</span>
|
||
</pre></div>
|
||
<p>I've done some testing and it wasn't working on Chrome because I wasn't
|
||
handling the headers the right way (The missing one was <cite>Content-Type</cite>, that
|
||
Chrome was asking for). With my first version of the implementation, I needed
|
||
the service implementers to explicitely list all the headers that should be
|
||
exposed. While this improves security, it can be frustrating while developing.</p>
|
||
<p>So I introduced an <cite>expose_all_headers</cite> flag, which is set to <cite>True</cite> by
|
||
default, if the service supports CORS.</p>
|
||
</div>
|
||
<div class="section" id="cookies-credentials">
|
||
<h3>Cookies / Credentials</h3>
|
||
<p>By default, the requests you do to your API endpoint don't include the
|
||
credential information for security reasons. If you really want to do that,
|
||
you need to enable it using the <cite>cors_credentials</cite> parameter. You can activate
|
||
this one on a per-service basis or on a per-method basis.</p>
|
||
</div>
|
||
<div class="section" id="caching">
|
||
<h3>Caching</h3>
|
||
<p>When you do a preflight request, the information returned by the server can be
|
||
cached by the User-Agent so that it's not redone before each actual call.</p>
|
||
<p>The caching period is defined by the server, using the <cite>Access-Control-Max-Age</cite>
|
||
header. You can configure this timing using the <cite>cors_max_age</cite> parameter.</p>
|
||
</div>
|
||
<div class="section" id="simplifying-the-api">
|
||
<h3>Simplifying the API</h3>
|
||
<p>We have cors_headers, cors_enabled, cors_origins, cors_credentials,
|
||
cors_max_age, cors_expose_all_headers … a fair number of
|
||
parameters. If you want to have a specific CORS-policy for your services, that
|
||
can be a bit tedious to pass these to your services all the time.</p>
|
||
<p>I introduced another way to pass the CORS policy, so you can do something like
|
||
that:</p>
|
||
<div class="highlight"><pre><span></span><span class="n">policy</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="n">enabled</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
|
||
<span class="n">headers</span><span class="o">=</span><span class="p">(</span><span class="s1">&#39;X-My-Header&#39;</span><span class="p">,</span> <span class="s1">&#39;Content-Type&#39;</span><span class="p">),</span>
|
||
<span class="n">origins</span><span class="o">=</span><span class="p">(</span><span class="s1">&#39;*.notmyidea.org&#39;</span><span class="p">),</span>
|
||
<span class="n">credentials</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
|
||
<span class="n">max_age</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
|
||
|
||
<span class="n">foobar</span> <span class="o">=</span> <span class="n">Service</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">&#39;foobar&#39;</span><span class="p">,</span> <span class="n">path</span><span class="o">=</span><span class="s1">&#39;/foobar&#39;</span><span class="p">,</span> <span class="n">cors_policy</span><span class="o">=</span><span class="n">policy</span><span class="p">)</span>
|
||
</pre></div>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="comparison-with-other-implementations">
|
||
<h2>Comparison with other implementations</h2>
|
||
<p>I was curious to have a look at other implementations of CORS, in
|
||
django for instance, and I found <a class="reference external" href="https://gist.github.com/426829.js">a gist about it</a>.</p>
|
||
<p>Basically, this adds a middleware that adds the &quot;rights&quot; headers to the answer,
|
||
depending on the request.</p>
|
||
<p>While this approach works, it's not implementing the specification completely.
|
||
You need to add support for all the resources at once.</p>
|
||
<p>We can think about a nice way to implement this specifying a definition of what's
|
||
supposed to be exposed via CORS and what shouldn't directly in your settings.
|
||
In my opinion, CORS support should be handled at the service definition level,
|
||
except for the list of authorized hosts.
|
||
Otherwise, you don't know exactly what's going on when you look at the definition of the service.</p>
|
||
</div>
|
||
<div class="section" id="resources">
|
||
<h2>Resources</h2>
|
||
<p>There are a number of good resources that can be useful to you if you want to
|
||
either understand how CORS works, or if you want to implement it yourself.</p>
|
||
<ul class="simple">
|
||
<li><a class="reference external" href="http://enable-cors.org/">http://enable-cors.org/</a> is useful to get started when you don't know anything
|
||
about CORS.</li>
|
||
<li>There is a W3C wiki page containing information that may be useful about
|
||
clients, common pitfalls etc: <a class="reference external" href="http://www.w3.org/wiki/CORS_Enabled">http://www.w3.org/wiki/CORS_Enabled</a></li>
|
||
<li><em>HTML5 rocks</em> has a tutorial explaining how to implement CORS, with <a class="reference external" href="http://www.html5rocks.com/en/tutorials/cors/#toc-adding-cors-support-to-the-server">a nice
|
||
section about the server-side</a>.</li>
|
||
<li>Be sure to have a look at the <a class="reference external" href="http://caniuse.com/#search=cors">clients support-matrix for this feature</a>.</li>
|
||
<li>About security, <a class="reference external" href="https://code.google.com/p/html5security/wiki/CrossOriginRequestSecurity">check out this page</a></li>
|
||
<li>If you want to have a look at the implementation code, check <a class="reference external" href="https://github.com/mozilla-services/cornice/pull/98/files">on github</a></li>
|
||
</ul>
|
||
<p>Of course, the W3C specification is the best resource to rely on. This
|
||
specification isn't hard to read, so you may want to go through it. Especially
|
||
the <a class="reference external" href="http://www.w3.org/TR/cors/#resource-processing-model">&quot;resource processing model&quot; section</a></p>
|
||
</div>
|
||
</content></entry><entry><title>Status board</title><link href="https://blog.notmyidea.org/status-board-fr.html" rel="alternate"></link><published>2012-12-29T00:00:00+01:00</published><updated>2012-12-29T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2012-12-29:/status-board-fr.html</id><summary type="html"><p>À force de démarrer des services web pour un oui et pour un non, de proposer
|
||
à des copains d'héberger leurs sites, de faire pareil pour quelques assos etc,
|
||
je me suis retrouvé avec, comme dirait l'autre, <em>une bonne platrée</em> de sites et
|
||
de services à gérer sur <cite>lolnet.org …</cite></p></summary><content type="html"><p>À force de démarrer des services web pour un oui et pour un non, de proposer
|
||
à des copains d'héberger leurs sites, de faire pareil pour quelques assos etc,
|
||
je me suis retrouvé avec, comme dirait l'autre, <em>une bonne platrée</em> de sites et
|
||
de services à gérer sur <cite>lolnet.org</cite>, mon serveur.</p>
|
||
<p>Jusqu'à très récemment, rien de tout ça n'était sauvegardé, et non plus monitoré.
|
||
Après quelques recherches, je suis tombé sur <a class="reference external" href="http://www.stashboard.org/">stashboard</a>, un &quot;status board&quot; qu'il est bien fait. Le seul
|
||
problème, c'est écrit pour se lancer sur GAE, <em>Google App Engine</em>.
|
||
Heureusement, c'est open-source, et ça a été forké pour donner naissance
|
||
à <a class="reference external" href="https://github.com/bfirsh/whiskerboard">whiskerboard</a> (la planche
|
||
moustachue, pour les non anglophones).</p>
|
||
<img alt="Capture d'écran du site." src="images/status_board.png" />
|
||
<div class="section" id="verifier-le-statut-des-services">
|
||
<h2>Vérifier le statut des services</h2>
|
||
<p>Donc, c'est chouette, c'est facile à installer, tout ça, mais… mais ça ne fait
|
||
en fait pas ce que je veux: ça ne fait que m'afficher le statut des services,
|
||
mais ça ne vérifie pas que tout est bien &quot;up&quot;.</p>
|
||
<p>Bon, un peu embêtant pour moi, parce que c'est vraiment ça que je voulais. Pas
|
||
grave, je sais un peu coder, autant que ça serve. J'ai ajouté quelques
|
||
fonctionnalités au soft, qui sont disponibles sur mon fork, sur
|
||
github:: <a class="reference external" href="https://github.com/ametaireau/whiskerboard">https://github.com/ametaireau/whiskerboard</a> .</p>
|
||
<p>Entres autres, il est désormais possible de lancer <a class="reference external" href="http://celeryproject.org/">celery</a> en tache de fond et de vérifier périodiquement
|
||
que les services sont toujours bien vivants, en utilisant une tache spécifique.</p>
|
||
<p>C'était un bonheur de développer ça (on a fait ça à deux, avec guillaume, avec
|
||
un mumble + tmux en pair prog, en une petite soirée, ça dépote).</p>
|
||
<p>Les modifications sont assez simples, vous pouvez aller jeter un œil aux
|
||
changements ici:
|
||
<a class="reference external" href="https://github.com/ametaireau/whiskerboard/compare/b539337416...master">https://github.com/ametaireau/whiskerboard/compare/b539337416...master</a></p>
|
||
<p>En gros:</p>
|
||
<ul class="simple">
|
||
<li>ajout d'une <cite>connection_string</cite> aux services (de la forme
|
||
protocol://host:port)</li>
|
||
<li>ajout d'une commande <cite>check_status</cite> qui s'occupe d'itérer sur les services et
|
||
de lancer des taches celery qui vont bien, en fonction du protocole</li>
|
||
<li>ajout des taches en question</li>
|
||
</ul>
|
||
</div>
|
||
<div class="section" id="deploiement">
|
||
<h2>Déploiement</h2>
|
||
<p>Le plus long a été de le déployer en fin de compte, parce que je ne voulais pas
|
||
déployer mon service de supervision sur mon serveur, forcément.</p>
|
||
<p>Après un essai (plutôt rapide en fait) sur <a class="reference external" href="http://heroku.com">heroku</a>, je me
|
||
suis rendu compte qu'il me fallait payer pas loin de 35$ par mois pour avoir un
|
||
process <cite>celeryd</cite> qui tourne, donc j'ai un peu cherché ailleurs, pour
|
||
finalement déployer la chose chez <a class="reference external" href="https://www.alwaysdata.com/">alwaysdata</a></p>
|
||
<p>Après quelques péripéties, j'ai réussi à faire tourner le tout, ça à été un peu
|
||
la bataille au départ pour installer virtualenv (j'ai du faire des changements
|
||
dans mon <cite>PATH</cite> pour que ça puisse marcher), voici mon <cite>.bash_profile</cite>:</p>
|
||
<pre class="literal-block">
|
||
export PYTHONPATH=~/modules/
|
||
export PATH=$HOME/modules/bin:$HOME/modules/:$PATH
|
||
</pre>
|
||
<p>Et après y'a plus qu'à installer avec <cite>easy_install</cite>:</p>
|
||
<pre class="literal-block">
|
||
easy_install --install-dir ~/modules -U pip
|
||
easy_install --install-dir ~/modules -U virtualenv
|
||
</pre>
|
||
<p>Et à créer le virtualenv:</p>
|
||
<pre class="literal-block">
|
||
virtualenv venv
|
||
venv/bin/pip install -r requirements.txt
|
||
</pre>
|
||
<p>Dernière étape, la création d'un fichier <cite>application.wsgi</cite> qui s'occupe de
|
||
rendre l'application disponible, avec le bon venv:</p>
|
||
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||
<span class="kn">import</span> <span class="nn">sys</span>
|
||
|
||
<span class="n">activate_this</span> <span class="o">=</span> <span class="s1">&#39;/home/lolnet/venv/bin/activate_this.</span>
|
||
<span class="nb">execfile</span><span class="p">(</span><span class="n">activate_this</span><span class="p">,</span> <span class="nb">dict</span><span class="p">(</span><span class="vm">__file__</span><span class="o">=</span><span class="n">activate_this</span><span class="p">))</span>
|
||
|
||
<span class="n">paths</span> <span class="o">=</span> <span class="p">[</span><span class="s1">&#39;/home/lolnet/public&#39;</span><span class="p">,]</span>
|
||
|
||
<span class="k">for</span> <span class="n">path</span> <span class="ow">in</span> <span class="n">paths</span><span class="p">:</span>
|
||
<span class="k">if</span> <span class="n">path</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">sys</span><span class="o">.</span><span class="n">path</span><span class="p">:</span>
|
||
<span class="n">sys</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
|
||
|
||
<span class="kn">import</span> <span class="nn">django.core.handlers.wsgi</span>
|
||
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s1">&#39;DJANGO_SETTINGS_MODULE&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="s1">&#39;settings&#39;</span>
|
||
<span class="n">application</span> <span class="o">=</span> <span class="n">django</span><span class="o">.</span><span class="n">core</span><span class="o">.</span><span class="n">handlers</span><span class="o">.</span><span class="n">wsgi</span><span class="o">.</span><span class="n">WSGIHandler</span><span class="p">()</span>
|
||
</pre></div>
|
||
</div>
|
||
<div class="section" id="ssl-et-requests">
|
||
<h2>SSL et Requests</h2>
|
||
<p>Quelques tours de manivelle plus loin, j'ai un celeryd qui tourne et qui
|
||
consomme les taches qui lui sont envoyées (pour des questions de simplicité,
|
||
j'ai utilisé le backend <cite>django</cite> de celery, donc pas besoin d'AMQP, par
|
||
exemple).</p>
|
||
<p>Problème, les ressources que je vérifie en SSL (HTTPS) me jettent. Je sais pas
|
||
exactement pourquoi à l'heure qu'il est, mais il semble que lorsque je fais une
|
||
requête avec <a class="reference external" href="http://docs.python-requests.org/en/latest/">Requests</a> je me
|
||
récupère des <em>Connection Refused</em>. Peut être une sombre histoire de proxy ? En
|
||
attendant, les appels avec <cite>CURL</cite> fonctionnent, donc j'ai fait <a class="reference external" href="https://github.com/ametaireau/whiskerboard/blob/master/board/tasks.py#L17">un fallback
|
||
vers CURL lorsque les autres méthodes échouent</a>.
|
||
Pas super propre, mais ça fonctionne.</p>
|
||
<p><strong>EDIT</strong> Finalement, il se trouve que mon serveur était mal configuré.
|
||
J'utilisais haproxy + stunnel, et la négiciation SSL se passait mal. Une fois
|
||
SSL et TLS activés, et SSLv2 désactivé, tout fonctionne mieux.</p>
|
||
</div>
|
||
<div class="section" id="et-voila">
|
||
<h2>Et voilà</h2>
|
||
<p>Finalement, j'ai mon joli status-board qui tourne à merveille sur
|
||
<a class="reference external" href="http://status.lolnet.org">http://status.lolnet.org</a> :-)</p>
|
||
</div>
|
||
</content></entry><entry><title>Gnome 3, extensions</title><link href="https://blog.notmyidea.org/gnome-3-extensions.html" rel="alternate"></link><published>2012-12-27T00:00:00+01:00</published><updated>2012-12-27T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2012-12-27:/gnome-3-extensions.html</id><summary type="html"><p>Après avoir tenté pendant un bout de temps unity, le bureau par defaut de
|
||
ubuntu, j'ai eu envie de changements, et j'ai donc essayé un peu de regarder du
|
||
coté de gnome 3, à nouveau.</p>
|
||
<p>Et finalement, j'ai trouvé quelques extensions qui sont vraiment utiles, que je
|
||
liste ici.</p>
|
||
<ul class="simple">
|
||
<li><a class="reference external" href="https://extensions.gnome.org/extension/547/antisocial-menu/">Antisocial …</a></li></ul></summary><content type="html"><p>Après avoir tenté pendant un bout de temps unity, le bureau par defaut de
|
||
ubuntu, j'ai eu envie de changements, et j'ai donc essayé un peu de regarder du
|
||
coté de gnome 3, à nouveau.</p>
|
||
<p>Et finalement, j'ai trouvé quelques extensions qui sont vraiment utiles, que je
|
||
liste ici.</p>
|
||
<ul class="simple">
|
||
<li><a class="reference external" href="https://extensions.gnome.org/extension/547/antisocial-menu/">Antisocial Menu</a> vire les
|
||
boutons et textes en rapport avec le web social. J'en avais pas besoin
|
||
puisque je suis connecté à mon instant messenger dans un terminal, en
|
||
utilisant weechat.</li>
|
||
<li><a class="reference external" href="https://extensions.gnome.org/extension/97/coverflow-alt-tab/">Coverflow Alt-Tab</a> change le
|
||
switcher d'applications par defaut. Je le trouve bien plus pratique que celui
|
||
par defaut puisqu'il me permet de voir &quot;en grand&quot; quelle est la fenêtre que
|
||
je vais afficher.</li>
|
||
<li><a class="reference external" href="https://extensions.gnome.org/extension/55/media-player-indicator/">Media player indicator</a> me
|
||
permet de voir en temps réel ce qui se passe dans mon lecteur audio. Ça
|
||
semble ne pas être grand chose, mais ça me manquait. Ça s'intègre niquel avec
|
||
Spotify, et ça c'est chouette.</li>
|
||
<li><a class="reference external" href="https://extensions.gnome.org/extension/149/search-firefox-bookmarks-provider/">Rechercher dans les bookmarks firefox</a>
|
||
permet de… à votre avis ?</li>
|
||
</ul>
|
||
<p>Un peu moins utile mais sait on jamais:</p>
|
||
<ul class="simple">
|
||
<li>“<a class="reference external" href="https://extensions.gnome.org/extension/130/advanced-settings-in-usermenu/">Advanced Settings in UserMenu</a>”
|
||
permet d'avoir un raccourci vers les paramètres avancés dans le menu
|
||
utilisateur (en haut à droite)</li>
|
||
<li>Une <a class="reference external" href="https://extensions.gnome.org/extension/409/gtg-integration/">intégration à Getting things Gnome</a> (un truc de
|
||
GTD). Je suis en train d'expérimenter avec cet outil, donc je ne sais pas
|
||
encore si ça va rester, mais pourquoi pas.</li>
|
||
</ul>
|
||
<p>Vous pouvez aller faire un tour sur <a class="reference external" href="https://extensions.gnome.org/">https://extensions.gnome.org/</a>
|
||
pour en trouver d'autres à votre gout.</p>
|
||
</content><category term="note"></category></entry><entry><title>Cheese & code - Wrap-up</title><link href="https://blog.notmyidea.org/cheese-code-wrap-up.html" rel="alternate"></link><published>2012-10-22T00:00:00+02:00</published><updated>2012-10-22T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2012-10-22:/cheese-code-wrap-up.html</id><summary type="html"><p>This week-end I hosted a <em>cheese &amp; code</em> session in the country-side of Angers,
|
||
France.</p>
|
||
<p>We were a bunch of python hackers and it rained a lot, wich forced us to stay
|
||
inside and to code. Bad.</p>
|
||
<p>We were not enough to get rid of all the cheese and the awesome …</p></summary><content type="html"><p>This week-end I hosted a <em>cheese &amp; code</em> session in the country-side of Angers,
|
||
France.</p>
|
||
<p>We were a bunch of python hackers and it rained a lot, wich forced us to stay
|
||
inside and to code. Bad.</p>
|
||
<p>We were not enough to get rid of all the cheese and the awesome meals, but
|
||
well, we finally managed it pretty well.</p>
|
||
<p>Here is a summary of what we worked on:</p>
|
||
<div class="section" id="daybed">
|
||
<h2>Daybed</h2>
|
||
<p>Daybed started some time ago, and intend to be a replacement to google forms,
|
||
in term of features, but backed as a REST web service, in python, and open
|
||
source.</p>
|
||
<p>In case you wonder, daybed is effectively the name of a couch. We chose this
|
||
name because of the similarities (in the sound) with <strong>db</strong>, and because
|
||
we're using <strong>CouchDB</strong> as a backend.</p>
|
||
<img alt="Daybed is a big couch!" src="images/daybed.jpg" style="width: 400px;" />
|
||
<p>We mainly hacked on daybed and are pretty close to the release of the first
|
||
version, meaning that we have something working.</p>
|
||
<p><a class="reference external" href="http://github.com/spiral-project/daybed">The code</a> is available on github,
|
||
and we also wrote <a class="reference external" href="http://daybed.rtfd.org">a small documentation</a> for it.</p>
|
||
<p>Mainly, we did a lot of cleanup, rewrote a bunch of tests so that it would be
|
||
easier to continue to work on the project, and implemented some minor features.
|
||
I'm pretty confidend that we now have really good basis for this project.</p>
|
||
<p>Also, we will have a nice todolist application, with the backend <strong>and</strong> the
|
||
frontend, in javascript / html / css, you'll know more when it'll be ready :-)</p>
|
||
<p>Once we have something good enough, we'll release the first version and I'll
|
||
host it somewhere so that people can play with it.</p>
|
||
</div>
|
||
<div class="section" id="cornice">
|
||
<h2>Cornice</h2>
|
||
<p>Daybed is built on top of <a class="reference external" href="http://cornice.rtfd.org">Cornice</a>, a framework to
|
||
ease the creation of web-services.</p>
|
||
<p>At Pycon France, we had the opportunity to attend a good presentation about <a class="reference external" href="https://github.com/SPORE/specifications">SPORE</a>. SPORE is a way to describe your
|
||
REST web services, as WSDL is for WS-* services. This allows to ease the
|
||
creation of generic SPORE clients, which are able to consume any REST API with
|
||
a SPORE endpoint.</p>
|
||
<p>Here is how you can let cornice describe your web service for you</p>
|
||
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">cornice.ext.spore</span> <span class="kn">import</span> <span class="n">generate_spore_description</span>
|
||
<span class="kn">from</span> <span class="nn">cornice.service</span> <span class="kn">import</span> <span class="n">Service</span><span class="p">,</span> <span class="n">get_services</span>
|
||
|
||
<span class="n">spore</span> <span class="o">=</span> <span class="n">Service</span><span class="p">(</span><span class="s1">&#39;spore&#39;</span><span class="p">,</span> <span class="n">path</span><span class="o">=</span><span class="s1">&#39;/spore&#39;</span><span class="p">,</span> <span class="n">renderer</span><span class="o">=</span><span class="s1">&#39;jsonp&#39;</span><span class="p">)</span>
|
||
<span class="nd">@spore.get</span>
|
||
<span class="k">def</span> <span class="nf">get_spore</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
|
||
<span class="n">services</span> <span class="o">=</span> <span class="n">get_services</span><span class="p">()</span>
|
||
<span class="k">return</span> <span class="n">generate_spore_description</span><span class="p">(</span><span class="n">services</span><span class="p">,</span> <span class="s1">&#39;Service name&#39;</span><span class="p">,</span>
|
||
<span class="n">request</span><span class="o">.</span><span class="n">application_url</span><span class="p">,</span> <span class="s1">&#39;1.0&#39;</span><span class="p">)</span>
|
||
</pre></div>
|
||
<p>And you'll get a definition of your service, in SPORE, available at <cite>/spore</cite>.</p>
|
||
<p>Of course, you can use it to do other things, like generating the file locally
|
||
and exporting it wherever it makes sense to you, etc.</p>
|
||
<p>I released today <a class="reference external" href="http://crate.io/packages/cornice/">Cornice 0.11</a>, which adds
|
||
into other things the support for SPORE, plus some other fixes we found on our
|
||
way.</p>
|
||
</div>
|
||
<div class="section" id="respire">
|
||
<h2>Respire</h2>
|
||
<p>Once you have the description of the service, you can do generic clients
|
||
consuming them!</p>
|
||
<p>We first wanted to contribute to <a class="reference external" href="https://github.com/bl0b/spyre">spyre</a> but
|
||
it was written in a way that wasn't supporting to <cite>POST</cite> data, and they
|
||
were using their own stack to handle HTTP. A lot of code that already exists in
|
||
other libraries.</p>
|
||
<p>While waiting the train with <a class="reference external" href="http://natim.ionyse.com/">Rémy</a>, we hacked
|
||
something together, named &quot;Respire&quot;, a thin layer on top of the awesome
|
||
<a class="reference external" href="http://python-requests.org">Requests</a> library.</p>
|
||
<p>We have a first version, feel free to have a look at it and provide
|
||
enhancements if you feel like it. We're still hacking on it so it may break
|
||
(for the better), but that had been working pretty well for us so far.</p>
|
||
<p>You can <a class="reference external" href="http://github.com/spiral-project/respire">find the project on github</a>, but here is how to use it, really
|
||
quickly (these examples are how to interact with daybed)</p>
|
||
<div class="highlight"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">respire</span> <span class="kn">import</span> <span class="n">client_from_url</span>
|
||
|
||
<span class="o">&gt;&gt;&gt;</span> <span class="c1"># create the client from the SPORE definition</span>
|
||
<span class="o">&gt;&gt;&gt;</span> <span class="n">cl</span> <span class="o">=</span> <span class="n">client_from_url</span><span class="p">(</span><span class="s1">&#39;http://localhost:8000/spore&#39;</span><span class="p">)</span>
|
||
|
||
<span class="o">&gt;&gt;&gt;</span> <span class="c1"># in daybed, create a new definition</span>
|
||
<span class="o">&gt;&gt;&gt;</span> <span class="n">todo_def</span> <span class="o">=</span> <span class="p">{</span>
|
||
<span class="o">...</span> <span class="s2">&quot;title&quot;</span><span class="p">:</span> <span class="s2">&quot;todo&quot;</span><span class="p">,</span>
|
||
<span class="o">...</span> <span class="s2">&quot;description&quot;</span><span class="p">:</span> <span class="s2">&quot;A list of my stuff to do&quot;</span><span class="p">,</span>
|
||
<span class="o">...</span> <span class="s2">&quot;fields&quot;</span><span class="p">:</span> <span class="p">[</span>
|
||
<span class="o">...</span> <span class="p">{</span>
|
||
<span class="o">...</span> <span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;item&quot;</span><span class="p">,</span>
|
||
<span class="o">...</span> <span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;string&quot;</span><span class="p">,</span>
|
||
<span class="o">...</span> <span class="s2">&quot;description&quot;</span><span class="p">:</span> <span class="s2">&quot;The item&quot;</span>
|
||
<span class="o">...</span> <span class="p">},</span>
|
||
<span class="o">...</span> <span class="p">{</span>
|
||
<span class="o">...</span> <span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;status&quot;</span><span class="p">,</span>
|
||
<span class="o">...</span> <span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;enum&quot;</span><span class="p">,</span>
|
||
<span class="o">...</span> <span class="s2">&quot;choices&quot;</span><span class="p">:</span> <span class="p">[</span>
|
||
<span class="o">...</span> <span class="s2">&quot;done&quot;</span><span class="p">,</span>
|
||
<span class="o">...</span> <span class="s2">&quot;todo&quot;</span>
|
||
<span class="o">...</span> <span class="p">],</span>
|
||
<span class="o">...</span> <span class="s2">&quot;description&quot;</span><span class="p">:</span> <span class="s2">&quot;is it done or not&quot;</span>
|
||
<span class="o">...</span> <span class="p">}</span>
|
||
<span class="o">...</span> <span class="p">]}</span>
|
||
<span class="o">&gt;&gt;&gt;</span> <span class="n">cl</span><span class="o">.</span><span class="n">put_definition</span><span class="p">(</span><span class="n">model_name</span><span class="o">=</span><span class="s1">&#39;todo&#39;</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">todo_def</span><span class="p">)</span>
|
||
<span class="o">&gt;&gt;&gt;</span> <span class="n">cl</span><span class="o">.</span><span class="n">post_data</span><span class="p">(</span><span class="n">model_name</span><span class="o">=</span><span class="s1">&#39;todo&#39;</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="nb">dict</span><span class="p">(</span><span class="n">item</span><span class="o">=</span><span class="s1">&#39;make it work&#39;</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="s1">&#39;todo&#39;</span><span class="p">))</span>
|
||
<span class="p">{</span><span class="sa">u</span><span class="s1">&#39;id&#39;</span><span class="p">:</span> <span class="sa">u</span><span class="s1">&#39;9f2c90c0529a442cfdc03c191b022cf7&#39;</span><span class="p">}</span>
|
||
<span class="o">&gt;&gt;&gt;</span> <span class="n">cl</span><span class="o">.</span><span class="n">get_data</span><span class="p">(</span><span class="n">model_name</span><span class="o">=</span><span class="s1">&#39;todo&#39;</span><span class="p">)</span>
|
||
</pre></div>
|
||
<p>Finally, we were out of cheese so everyone headed back to their respective
|
||
houses and cities.</p>
|
||
<p>Until next time?</p>
|
||
</div>
|
||
</content></entry><entry><title>Cheese & Code party: October 20-21</title><link href="https://blog.notmyidea.org/afpy-computer-camp-october-20-21.html" rel="alternate"></link><published>2012-09-20T00:00:00+02:00</published><updated>2012-09-20T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2012-09-20:/afpy-computer-camp-october-20-21.html</id><summary type="html"><p>At PyconFR, this week-end, a few ideas were thrown in the air and one of them
|
||
was to have a /dev/fort doing some python-related coding.</p>
|
||
<p>The concept of a /dev/fort is to put a bunch of hackers together and see what
|
||
comes out from it. Tarek is doing …</p></summary><content type="html"><p>At PyconFR, this week-end, a few ideas were thrown in the air and one of them
|
||
was to have a /dev/fort doing some python-related coding.</p>
|
||
<p>The concept of a /dev/fort is to put a bunch of hackers together and see what
|
||
comes out from it. Tarek is doing something related with the Afpy Computer
|
||
Camps at his house each year, I've been there twice and it really was a nice
|
||
experience.</p>
|
||
<p>At Djangocong 2012, in Montpellier (south of France), <a class="reference external" href="http://blog.mathieu-leplatre.info/">Mathieu Leplatre</a> and myself started to work on a model
|
||
validation and storage service, named <a class="reference external" href="https://github.com/spiral-project/daybed/">Daybed</a>.</p>
|
||
<p>I've talked about this project to some persons this week-end (I've even done <a class="reference external" href="http://alexis.notmyidea.org/lightning-daybed.html">a
|
||
lightning talk</a> about it)
|
||
and it gathered some interest from people in the python community, so we
|
||
thought about sprinting on this.</p>
|
||
<div class="section" id="and-21-october-a-computer-camp">
|
||
<h2>20 and 21 October - A Computer Camp !</h2>
|
||
<p>Add to this a few beers, and the sprint turns magically into a camp. We'll be
|
||
sprinting at the end of October (the 20 and 21) near Angers, on daybed.</p>
|
||
<p>We plan to have great food and wine, so if you feel like it and if you want to
|
||
come and work on some python-related things, you're invited!</p>
|
||
<p>Although, I can host a limited number of persons, so you'll need to contact me
|
||
before :), that's <cite>alexis at notmyidea dot org</cite>.</p>
|
||
<p>I've setup a wiki page to organize a bit everything (knowing how many people will
|
||
come by will allow me to handle things better), please put information there:</p>
|
||
<p><a class="reference external" href="http://wiki.python.org/moin/ComputerCampAlexis">http://wiki.python.org/moin/ComputerCampAlexis</a></p>
|
||
</div>
|
||
</content></entry><entry><title>Circus sprint at PyconFR</title><link href="https://blog.notmyidea.org/circus-sprint-at-pyconfr.html" rel="alternate"></link><published>2012-09-17T00:00:00+02:00</published><updated>2012-09-17T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2012-09-17:/circus-sprint-at-pyconfr.html</id><summary type="html"><p>Last Thursday to Sunday, <a class="reference external" href="http://pycon.fr">Pycon France</a> took place, in
|
||
Paris. It was the opportunity to meet a lot of people and to talk about python
|
||
awesomness in general.</p>
|
||
<p>We had three tracks this year, plus sprints the two first days. We sprinted on
|
||
<a class="reference external" href="http://circus.io">Circus</a>, the process and socket manager we're …</p></summary><content type="html"><p>Last Thursday to Sunday, <a class="reference external" href="http://pycon.fr">Pycon France</a> took place, in
|
||
Paris. It was the opportunity to meet a lot of people and to talk about python
|
||
awesomness in general.</p>
|
||
<p>We had three tracks this year, plus sprints the two first days. We sprinted on
|
||
<a class="reference external" href="http://circus.io">Circus</a>, the process and socket manager we're using
|
||
at Mozilla for some of our setups.</p>
|
||
<p>The project gathered some interest, and we ended up with 5 persons working on
|
||
it. Of course, we spent some time explaining what is Circus, how it had been
|
||
built, a lot of time talking about use-cases and possible improvements, but we
|
||
also managed to add new features.</p>
|
||
<p>Having people wanting to sprint on our projects is exciting because that's
|
||
when making things in the open unleashes its full potential. You can't imagine
|
||
how happy I was to have some friends come and work on this with us :)</p>
|
||
<p>Here is a wrap-up of the sprint:</p>
|
||
<div class="section" id="autocompletion-on-the-command-line">
|
||
<h2>Autocompletion on the command-line</h2>
|
||
<p><a class="reference external" href="http://natim.ionyse.com">Remy Hubscher</a> worked on the command-line
|
||
autocompletion. Now we have a fancy command-line interface which is able to
|
||
aucomplete if you're using bash. It seems that not that much work is needed to
|
||
make it happen on zsh as well :)</p>
|
||
<p><a class="reference external" href="https://github.com/mozilla-services/circus/blob/master/extras/circusctl_bash_completion">Have a look at the feature</a></p>
|
||
<p>On the same topic, we now have a cool shell for Circus. If you start the
|
||
<cite>circusctl</cite> command without any option, you'll end-up with a cool shell. Thanks
|
||
<a class="reference external" href="https://github.com/jojax">Jonathan Dorival</a> for the work on this! You can
|
||
have a look at <a class="reference external" href="https://github.com/mozilla-services/circus/pull/268">the pull request</a>.</p>
|
||
</div>
|
||
<div class="section" id="future-changes-to-the-web-ui">
|
||
<h2>Future changes to the web ui</h2>
|
||
<p><a class="reference external" href="https://twitter.com/rachbelaid">Rachid Belaid</a> had a deep look at the source
|
||
code and is much more familiarized to it now than before. We discussed the
|
||
possibility to change the implementation of the web ui, and I'm glad of this.
|
||
Currently, it's done with bottle.py and we want to switch to pyramid.</p>
|
||
<p>He fixed some issues that were in the tracker, so we now can have the age of
|
||
watchers in the webui, for instance.</p>
|
||
</div>
|
||
<div class="section" id="bug-and-doc-fixing">
|
||
<h2>Bug and doc fixing</h2>
|
||
<p>While reading the source code, we found some inconsistencies and fixed them,
|
||
with <a class="reference external" href="http://mathieu.agopian.info/">Mathieu Agopian</a>. We also tried to
|
||
improve the documentation at different levels.</p>
|
||
<p>Documentation still needs a lot of love, and I'm planning to spend some time on
|
||
this shortly. I've gathered a bunch of feedback on this</p>
|
||
</div>
|
||
<div class="section" id="circus-clustering-capabilities">
|
||
<h2>Circus clustering capabilities</h2>
|
||
<p>One feature I wanted to work on during this sprint was the clustering abilities
|
||
of Circus. Nick Pellegrino made an internship on this topic at Mozilla so we
|
||
spent some time to review his pull requests.</p>
|
||
<p>A lot of code was written for this so we discussed a bunch of things
|
||
regarding all of this. It took us more time than expected (and I still need to
|
||
spend more time on this to provide appropriate feedback), but it allowed us to
|
||
have a starting-point about what this clustering thing could be.</p>
|
||
<p>Remy wrote <a class="reference external" href="http://tech.novapost.fr/circus-clustering-management-en.html">a good summary about our brainstorming</a> so I'll not do
|
||
it again here, but feel free to contact us if you have ideas on this, they're
|
||
very welcome!</p>
|
||
</div>
|
||
<div class="section" id="project-management">
|
||
<h2>Project management</h2>
|
||
<p>We've had some inquiries telling us that's not as easy as it should to get
|
||
started with the Circus project. Some of the reasons are that we don't
|
||
have any release schedule, and that the documentation is hairy enough to lost
|
||
people, at some point :)</p>
|
||
<p>That's something we'll try to fix soon :)</p>
|
||
<p>PyconFR was a very enjoyable event. I'm looking forward to meet the community
|
||
again and discuss how Circus can evolve in ways that are interesting to
|
||
everyone.</p>
|
||
<p>Tarek and me are going to <a class="reference external" href="http://python.ie/pycon/2012/">Pycon ireland</a>, feel
|
||
free to reach us if you're going there, we'll be happy to meet and enjoy beers!</p>
|
||
</div>
|
||
</content></entry><entry><title>Refactoring Cornice</title><link href="https://blog.notmyidea.org/refactoring-cornice.html" rel="alternate"></link><published>2012-05-14T00:00:00+02:00</published><updated>2012-05-14T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2012-05-14:/refactoring-cornice.html</id><summary type="html"><p>After working for a while with <a class="reference external" href="http://cornice.readthedocs.com">Cornice</a> to
|
||
define our APIs at <a class="reference external" href="http://docs.services.mozilla.com">Services</a>, it turned
|
||
out that the current implementation wasn't flexible enough to allow us to do
|
||
what we wanted to do.</p>
|
||
<p>Cornice started as a toolkit on top of the <a class="reference external" href="http://docs.pylonsproject.org/en/latest/docs/pyramid.html">pyramid</a> routing system,
|
||
allowing to register services in …</p></summary><content type="html"><p>After working for a while with <a class="reference external" href="http://cornice.readthedocs.com">Cornice</a> to
|
||
define our APIs at <a class="reference external" href="http://docs.services.mozilla.com">Services</a>, it turned
|
||
out that the current implementation wasn't flexible enough to allow us to do
|
||
what we wanted to do.</p>
|
||
<p>Cornice started as a toolkit on top of the <a class="reference external" href="http://docs.pylonsproject.org/en/latest/docs/pyramid.html">pyramid</a> routing system,
|
||
allowing to register services in a simpler way. Then we added some niceties
|
||
such as the ability to automatically generate the services documentation or
|
||
returning the correct HTTP headers <a class="reference external" href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html">as defined by the HTTP specification</a> without the need
|
||
from the developer to deal with them nor to know them.</p>
|
||
<p>If you're not familiar with Cornice, here is how you define a simple service
|
||
with it:</p>
|
||
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">cornice.service</span> <span class="kn">import</span> <span class="n">Service</span>
|
||
<span class="n">bar</span> <span class="o">=</span> <span class="n">Service</span><span class="p">(</span><span class="n">path</span><span class="o">=</span><span class="s2">&quot;/bar&quot;</span><span class="p">)</span>
|
||
|
||
<span class="nd">@bar.get</span><span class="p">(</span><span class="n">validators</span><span class="o">=</span><span class="n">validators</span><span class="p">,</span> <span class="n">accept</span><span class="o">=</span><span class="s1">&#39;application/json&#39;</span><span class="p">)</span>
|
||
<span class="k">def</span> <span class="nf">get_drink</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
|
||
<span class="c1"># do something with the request (with moderation).</span>
|
||
</pre></div>
|
||
<p>This external API is quite cool, as it allows to do a bunch of things quite
|
||
easily. For instance, we've written our <a class="reference external" href="https://github.com/mozilla-services/tokenserver">token-server</a> code on top of this in a
|
||
blast.</p>
|
||
<div class="section" id="the-burden">
|
||
<h2>The burden</h2>
|
||
<p>The problem with this was that we were mixing internally the service
|
||
description logic with the route registration one. The way we were doing this
|
||
was via an extensive use of decorators internally.</p>
|
||
<p>The API of the <cite>cornice.service.Service</cite> class was as following (simplified so
|
||
you can get the gist of it).</p>
|
||
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Service</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
|
||
|
||
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">**</span><span class="n">service_kwargs</span><span class="p">):</span>
|
||
<span class="c1"># some information, such as the colander schemas (for validation),</span>
|
||
<span class="c1"># the defined methods that had been registered for this service and</span>
|
||
<span class="c1"># some other things were registered as instance variables.</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">schemas</span> <span class="o">=</span> <span class="n">service_kwargs</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">schema</span><span class="s1">&#39;, None)</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">defined_methods</span> <span class="o">=</span> <span class="p">[]</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">definitions</span> <span class="o">=</span> <span class="p">[]</span>
|
||
|
||
<span class="k">def</span> <span class="nf">api</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">**</span><span class="n">view_kwargs</span><span class="p">):</span>
|
||
<span class="sd">&quot;&quot;&quot;This method is a decorator that is being used by some alias</span>
|
||
<span class="sd"> methods.</span>
|
||
<span class="sd"> &quot;&quot;&quot;</span>
|
||
<span class="k">def</span> <span class="nf">wrapper</span><span class="p">(</span><span class="n">view</span><span class="p">):</span>
|
||
<span class="c1"># all the logic goes here. And when I mean all the logic, I</span>
|
||
<span class="c1"># mean it.</span>
|
||
<span class="c1"># 1. we are registering a callback to the pyramid routing</span>
|
||
<span class="c1"># system so it gets called whenever the module using the</span>
|
||
<span class="c1"># decorator is used.</span>
|
||
<span class="c1"># 2. we are transforming the passed arguments so they conform</span>
|
||
<span class="c1"># to what is expected by the pyramid routing system.</span>
|
||
<span class="c1"># 3. We are storing some of the passed arguments into the</span>
|
||
<span class="c1"># object so we can retrieve them later on.</span>
|
||
<span class="c1"># 4. Also, we are transforming the passed view before</span>
|
||
<span class="c1"># registering it in the pyramid routing system so that it</span>
|
||
<span class="c1"># can do what Cornice wants it to do (checking some rules,</span>
|
||
<span class="c1"># applying validators and filters etc.</span>
|
||
<span class="k">return</span> <span class="n">wrapper</span>
|
||
|
||
<span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
|
||
<span class="sd">&quot;&quot;&quot;A shortcut of the api decorator&quot;&quot;&quot;</span>
|
||
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">api</span><span class="p">(</span><span class="n">request_method</span><span class="o">=</span><span class="s2">&quot;GET&quot;</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
|
||
</pre></div>
|
||
<p>I encourage you to go read <a class="reference external" href="https://github.com/mozilla-services/cornice/blob/4e0392a2ae137b6a11690459bcafd7325e86fa9e/cornice/service.py#L44">the entire file</a>.
|
||
on github so you can get a better opinion on how all of this was done.</p>
|
||
<p>A bunch of things are wrong:</p>
|
||
<ul class="simple">
|
||
<li>first, we are not separating the description logic from the registration one.
|
||
This causes problems when we need to access the parameters passed to the
|
||
service, because the parameters you get are not exactly the ones you passed
|
||
but the ones that the pyramid routing system is expecting. For instance, if
|
||
you want to get the view <cite>get_drink</cite>, you will instead get a decorator which
|
||
contains this view.</li>
|
||
<li>second, we are using decorators as APIs we expose. Even if decorators are
|
||
good as shortcuts, they shouldn't be the default way to deal with an API. A
|
||
good example of this is <a class="reference external" href="https://github.com/mozilla-services/cornice/blob/4e0392a2ae137b6a11690459bcafd7325e86fa9e/cornice/resource.py#L56">how the resource module consumes this API</a>.
|
||
This is quite hard to follow.</li>
|
||
<li>Third, in the <cite>api</cite> method, a bunch of things are done regarding inheritance
|
||
of parameters that are passed to the service or to its decorator methods.
|
||
This leaves you with a really hard to follow path when it comes to add new
|
||
parameters to your API.</li>
|
||
</ul>
|
||
</div>
|
||
<div class="section" id="how-do-we-improve-this">
|
||
<h2>How do we improve this?</h2>
|
||
<p>Python is great because it allows you to refactor things in an easy way. What I
|
||
did isn't breaking our APIs, but make things way simpler to hack-on. One
|
||
example is that it allowed me to add features that we wanted to bring to
|
||
Cornice really quickly (a matter of minutes), without touching the API that much.</p>
|
||
<p>Here is the gist of the new architecture:</p>
|
||
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Service</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
|
||
<span class="c1"># we define class-level variables that will be the default values for</span>
|
||
<span class="c1"># this service. This makes things more extensible than it was before.</span>
|
||
<span class="n">renderer</span> <span class="o">=</span> <span class="s1">&#39;simplejson&#39;</span>
|
||
<span class="n">default_validators</span> <span class="o">=</span> <span class="n">DEFAULT_VALIDATORS</span>
|
||
<span class="n">default_filters</span> <span class="o">=</span> <span class="n">DEFAULT_FILTERS</span>
|
||
|
||
<span class="c1"># we also have some class-level parameters that are useful to know</span>
|
||
<span class="c1"># which parameters are supposed to be lists (and so converted as such)</span>
|
||
<span class="c1"># or which are mandatory.</span>
|
||
<span class="n">mandatory_arguments</span> <span class="o">=</span> <span class="p">(</span><span class="s1">&#39;renderer&#39;</span><span class="p">,)</span>
|
||
<span class="n">list_arguments</span> <span class="o">=</span> <span class="p">(</span><span class="s1">&#39;validators&#39;</span><span class="p">,</span> <span class="s1">&#39;filters&#39;</span><span class="p">)</span>
|
||
|
||
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">description</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="o">**</span><span class="n">kw</span><span class="p">):</span>
|
||
<span class="c1"># setup name, path and description as instance variables</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">path</span> <span class="o">=</span> <span class="n">path</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">description</span> <span class="o">=</span> <span class="n">description</span>
|
||
|
||
<span class="c1"># convert the arguments passed to something we want to store</span>
|
||
<span class="c1"># and then store them as attributes of the instance (because they</span>
|
||
<span class="c1"># were passed to the constructor</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">arguments</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_arguments</span><span class="p">(</span><span class="n">kw</span><span class="p">)</span>
|
||
<span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">arguments</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
|
||
<span class="nb">setattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
|
||
|
||
<span class="c1"># we keep having the defined_methods tuple and the list of</span>
|
||
<span class="c1"># definitions that are done for this service</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">defined_methods</span> <span class="o">=</span> <span class="p">[]</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">definitions</span> <span class="o">=</span> <span class="p">[]</span>
|
||
|
||
<span class="k">def</span> <span class="nf">get_arguments</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">conf</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
|
||
<span class="sd">&quot;&quot;&quot;Returns a dict of arguments. It does all the conversions for</span>
|
||
<span class="sd"> you, and uses the information that were defined at the instance</span>
|
||
<span class="sd"> level as fallbacks.</span>
|
||
<span class="sd"> &quot;&quot;&quot;</span>
|
||
|
||
<span class="k">def</span> <span class="nf">add_view</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">method</span><span class="p">,</span> <span class="n">view</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
|
||
<span class="sd">&quot;&quot;&quot;Add a view to this service.&quot;&quot;&quot;</span>
|
||
<span class="c1"># this is really simple and looks a lot like this</span>
|
||
<span class="n">method</span> <span class="o">=</span> <span class="n">method</span><span class="o">.</span><span class="n">upper</span><span class="p">()</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">definitions</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">method</span><span class="p">,</span> <span class="n">view</span><span class="p">,</span> <span class="n">args</span><span class="p">))</span>
|
||
<span class="k">if</span> <span class="n">method</span> <span class="ow">not</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">defined_methods</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">defined_methods</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">method</span><span class="p">)</span>
|
||
|
||
<span class="k">def</span> <span class="nf">decorator</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">method</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
|
||
<span class="sd">&quot;&quot;&quot;This is only another interface to the add_view method, exposing a</span>
|
||
<span class="sd"> decorator interface&quot;&quot;&quot;</span>
|
||
<span class="k">def</span> <span class="nf">wrapper</span><span class="p">(</span><span class="n">view</span><span class="p">):</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">add_view</span><span class="p">(</span><span class="n">method</span><span class="p">,</span> <span class="n">view</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
|
||
<span class="k">return</span> <span class="n">view</span>
|
||
<span class="k">return</span> <span class="n">wrapper</span>
|
||
</pre></div>
|
||
<p>So, the service is now only storing the information that's passed to it and
|
||
nothing more. No more route registration logic goes here. Instead, I added this
|
||
as another feature, even in a different module. The function is named
|
||
<cite>register_service_views</cite> and has the following signature:</p>
|
||
<div class="highlight"><pre><span></span><span class="n">register_service_views</span><span class="p">(</span><span class="n">config</span><span class="p">,</span> <span class="n">service</span><span class="p">)</span>
|
||
</pre></div>
|
||
<p>To sum up, here are the changes I made:</p>
|
||
<ol class="arabic simple">
|
||
<li>Service description is now separated from the route registration.</li>
|
||
<li><cite>cornice.service.Service</cite> now provides a <cite>hook_view</cite> method, which is not a
|
||
decorator. decorators are still present but they are optional (you don't
|
||
need to use them if you don't want to).</li>
|
||
<li>Everything has been decoupled as much as possible, meaning that you really
|
||
can use the <cite>Service</cite> class as a container of information about the services
|
||
you are describing. This is especially useful when generating documentation.</li>
|
||
</ol>
|
||
<p>As a result, it is now possible to use Cornice with other frameworks. It means
|
||
that you can stick with the service description but plug any other framework on
|
||
top of it. <cite>cornice.services.Service</cite> is now only a description tool. To
|
||
register routes, one would need to read the information contained into this
|
||
service and inject the right parameters into their preferred routing system.</p>
|
||
<p>However, no integration with other frameworks is done at the moment even if the
|
||
design allows it.</p>
|
||
<p>The same way, the sphinx description layer is now only a consumer of this
|
||
service description tool: it looks at what's described and build-up the
|
||
documentation from it.</p>
|
||
<p>The resulting branch is not merged yet. Still, you can <a class="reference external" href="https://github.com/mozilla-services/cornice/tree/refactor-the-world">have a look at it</a>.</p>
|
||
<p>Any suggestions are of course welcome :-)</p>
|
||
</div>
|
||
</content><category term="python"></category><category term="Cornice"></category><category term="refactoring"></category></entry><entry><title>Thoughts about a form generation service, GIS enabled</title><link href="https://blog.notmyidea.org/carto-forms.html" rel="alternate"></link><published>2012-02-04T00:00:00+01:00</published><updated>2012-02-04T00:00:00+01:00</updated><author><name>Alexis Métaireau, Mathieu Leplatre</name></author><id>tag:blog.notmyidea.org,2012-02-04:/carto-forms.html</id><summary type="html"><p>We have a plan. A &quot;fucking good&quot; one.</p>
|
||
<p>A bunch of friends asked me twice for quite the same thing: a webpage with a
|
||
form, tied to a map generation with some information filtering. They didn't
|
||
explicitly ask that but that's the gist of it.</p>
|
||
<p>This idea has been stuck …</p></summary><content type="html"><p>We have a plan. A &quot;fucking good&quot; one.</p>
|
||
<p>A bunch of friends asked me twice for quite the same thing: a webpage with a
|
||
form, tied to a map generation with some information filtering. They didn't
|
||
explicitly ask that but that's the gist of it.</p>
|
||
<p>This idea has been stuck in my head since then and I even think that we can
|
||
come out with something a little bit more flexible and useful. I've named it
|
||
<em>carto-forms</em> for now, but that's only the &quot;codename&quot;.</p>
|
||
<p>To put it shortly: what if we had a way to build forms, ala Google forms, but
|
||
with geographic information in them?</p>
|
||
<p>If you don't know Google forms, it means having an user-friendly way to build
|
||
forms and to use them to gather information from different users.</p>
|
||
<p>In my opinion, Google forms is missing two important things: first, it's not
|
||
open-source, so it's not possible to hack it or even to run it on your own
|
||
server. Second, it doesn't really know how to deal with geographic data, and
|
||
there is no way to filter the information more than in a spreadsheet.</p>
|
||
<p>I knew that <a class="reference external" href="http://blog.mathieu-leplatre.info/">Mathieu</a> and some folks at <a class="reference external" href="http://makina-corpus.com">Makina Corpus</a> would be interested
|
||
in this, so I started a discussion with him on IRC and we refined the details
|
||
of the project and its objectives.</p>
|
||
<p>Imagine the following:</p>
|
||
<ol class="arabic simple">
|
||
<li>For a mapping party, we choose a specific topic to map and design the form
|
||
(list of fields (i.e. tags) to be filled + description + type of the
|
||
information) ;</li>
|
||
<li>In situ, users fill the form fields with what they see. Geo fields can be
|
||
pre-populated using device geolocation ;</li>
|
||
<li>At the end of the day, we can see a map with all user contributions seized
|
||
through this particular form ;</li>
|
||
<li>If relevant, a script could eventually import the resulting dataset and
|
||
publish/merge with OpenStreetMap.</li>
|
||
</ol>
|
||
<div class="section" id="some-use-cases">
|
||
<h2>Some use cases</h2>
|
||
<p>I can see some use cases for this. The first one is a collaborative map, with
|
||
facet filtering. Let's draw a potential user flow:</p>
|
||
<ul>
|
||
<li><p class="first">An &quot;administrator&quot; goes to the website and creates a form to list all the
|
||
alternative-related events. He creates the following fields:</p>
|
||
<ul class="simple">
|
||
<li>Name: a plain text field containing the name of the event.</li>
|
||
<li>Category: the category of the event. Can be a finite list.</li>
|
||
<li>Location: The location of the event. It could be provided by selecting a
|
||
point on a map or by typing an address.</li>
|
||
<li>Date: the date of the event (a datepicker could do the trick)</li>
|
||
</ul>
|
||
<p>Each field in the form has semantic information associated with it (yes/no,
|
||
multiple selection, date-time, geocoding carto, carto selection etc)</p>
|
||
</li>
|
||
<li><p class="first">Once finished, the form is generated and the user gets an url (say
|
||
<a class="reference external" href="http://forms.notmyidea.org/alternatives">http://forms.notmyidea.org/alternatives</a>) for it.</p>
|
||
</li>
|
||
<li><p class="first">REST APIs allow third parties to get the form description and to
|
||
push/edit/get information from there.</p>
|
||
</li>
|
||
<li><p class="first">He can communicate the address in any way he wants to his community so they
|
||
can go to the page and add information to it.</p>
|
||
</li>
|
||
<li><p class="first">Then, it is possible to filter the results per location / date or category.
|
||
This can be done via API calls (useful for third parties) or via a nice
|
||
interface in the browser.</p>
|
||
</li>
|
||
</ul>
|
||
<p>So, as you may have noticed, this would allow us to create interactive maps really
|
||
easily. It's almost just a matter of some clicks to the users. If we also come
|
||
up with a nice Web API for this, we could do server-side validation and build
|
||
even phone applications easily.</p>
|
||
<p>To push the cursor a bit further, if we can come with a cool description format
|
||
for the forms, we could even build the forms dynamically on different platforms,
|
||
with generic clients.</p>
|
||
<p>As mentioned before, the idea of a simple tool to support collaborative mapping
|
||
fullfils a recurring necessity !</p>
|
||
<p>We envision a lot of example uses for this : recycling spots, accessible spots (wheelchairs,
|
||
etc.), trees identification, mushrooms picking areas, tracking of endangered species
|
||
(e.g. Bonelli's Eagle is currently tracked by sharing a spreadsheet), spotting of dangerous
|
||
species (e.g. asian predatory wasps), map advertisement boards (most cities do not track them!),
|
||
citizen reporting (e.g. graffiti, potholes, garbage, lightning like <a class="reference external" href="http://fixmystreet.ca">http://fixmystreet.ca</a>),
|
||
geocaching, trajectories (e.g hiking, runners, cyclists)...</p>
|
||
<p>Here are some other examples of where <em>carto-forms</em> could be useful:</p>
|
||
<div class="section" id="simple-gis-storage-backend">
|
||
<h3>Simple GIS storage backend</h3>
|
||
<p>Let's say you are a mobile developer, you don't want to bother with PostGIS
|
||
nor write a custom and insecure code to insert and retrieve your GIS data! You
|
||
need carto-forms! A simple API helps you design your models/forms and the
|
||
same API allows you to CRUD and query your data. Thus, you only need to focus
|
||
on your application, not on how GIS data will be handled.</p>
|
||
<p>We make a distinction between storage and widgets.</p>
|
||
<p>Besides, if you are a django / drupal / plomino... maintainer : you
|
||
can develop a module to &quot;plug&quot; your models (content types) and UI to carto-forms!
|
||
Carto forms are then exposed to your backoffice users (ex: drupal admin UI, django
|
||
adminsite), and likewise you can write your own HTML widgets that consume datasets
|
||
in frontend views (facets in JSON/XML, and map data in GeoJSON).</p>
|
||
</div>
|
||
<div class="section" id="visualization-tool">
|
||
<h3>Visualization tool</h3>
|
||
<p>Since data submission can be done programmatically using the API, you could use Carto-forms
|
||
results page as a visualization tool.</p>
|
||
<p>You can explore your dataset content using filters related to each form field. Facets filtering
|
||
is a great advantage, and a map shows the resulting features set. You feel like you're in
|
||
front of a decision support system!</p>
|
||
<p>Of course, filtered raw data can be downloaded (GeoJSON, XML) and a permalink allows to
|
||
share the page with the state of the filters and the zoom/location of the map.</p>
|
||
</div>
|
||
<div class="section" id="generic-forms-service">
|
||
<h3>Generic forms service</h3>
|
||
<p>If you want to generate a configuration file (or whatever, email messages, ...),
|
||
you will need a form and a template to inlay user submitted values and get the result.</p>
|
||
<p>A form service would be really useful to create forms programmatically and retrieve
|
||
cleaned and validated input values.</p>
|
||
<p>You could run a dedicated template service based on <em>carto-forms</em>! Parsing a template
|
||
content, this external service could create a form dynamically and bind them together.
|
||
The output of the form service (fields =&gt; values) would be bound to the input of a template
|
||
engine (variables =&gt; final result).</p>
|
||
<p>Note that for this use-case, there is no specific need of GIS data nor storage of records
|
||
for further retrieval.</p>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="what-s-out-in-the-wild-already">
|
||
<h2>What's out in the wild already?</h2>
|
||
<p>Of course, there is Google forms, which allows you to do these kind of things,
|
||
but it's closed and not exactly what we are describing here.</p>
|
||
<p>We've discovered the interesting <a class="reference external" href="https://webform.com/">https://webform.com/</a> which allows one to create
|
||
forms with a nice drag-n-drop flow. I would love to reproduce something similar
|
||
for the user experience. However, the project doesn't handle APIs and
|
||
geolocation information.</p>
|
||
<p>The idea of <a class="reference external" href="http://thoth.io">http://thoth.io</a> is very attractive : an extremely simple web API to store
|
||
and retrieve data. In addition, <em>carto-forms</em> would do datatype validation and have
|
||
basic GIS fields (point, line, polygon).</p>
|
||
<p><a class="reference external" href="http://mapbox.com">http://mapbox.com</a> also did an awesome work on cartography, but didn't take into
|
||
account the form aspect we're leveraging here.</p>
|
||
</div>
|
||
<div class="section" id="so-let-s-get-it-real">
|
||
<h2>So… Let's get it real!</h2>
|
||
<p>As you may have understood, this isn't a really complicated problem. We have
|
||
been sometimes chatting about that with Mathieu about what we would need and
|
||
how we could achieve this.</p>
|
||
<p>We can probably come with an elegant solution without too much pain. Mathieu is
|
||
used to work with GIS systems (which is really cool because I'm not at all) and
|
||
knows his subject, so that's an opportunity to learn ;-)</p>
|
||
<p>We will be at <a class="reference external" href="http://rencontres.django-fr.org">Djangocong</a> on April 14 and 15 and will probably have
|
||
a brainstorming session and a sprint on this, so if you are around and want to
|
||
help us, or just to discuss, feel free to join!</p>
|
||
<p>We don't know yet if we will be using django for this or something else. We
|
||
have been thinking about couchdb, couchapps and geocouch but nothing is written
|
||
in stone yet. Comments and proposals are welcome!</p>
|
||
<p>Here is the etherpad document we worked on so far:
|
||
<a class="reference external" href="http://framapad.org/carto-forms">http://framapad.org/carto-forms</a>. Don't hesitate to add your thoughts and edit
|
||
it, that's what it's made for!</p>
|
||
<p>Thanks to <a class="reference external" href="http://sneakernet.fr/">Arnaud</a> and <a class="reference external" href="http://qwerty.fuzz.me.uk/">Fuzzmz</a> for proof-reading and typo fixing.</p>
|
||
</div>
|
||
</content><category term="GIS"></category><category term="forms"></category></entry><entry><title>Using JPype to bridge python and Java</title><link href="https://blog.notmyidea.org/using-jpype-to-bridge-python-and-java.html" rel="alternate"></link><published>2011-11-06T00:00:00+01:00</published><updated>2011-11-06T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2011-11-06:/using-jpype-to-bridge-python-and-java.html</id><summary type="html"><p>Java provides some interesting libraries that have no exact equivalent in
|
||
python. In my case, the awesome boilerpipe library allows me to remove
|
||
uninteresting parts of HTML pages, like menus, footers and other &quot;boilerplate&quot;
|
||
contents.</p>
|
||
<p>Boilerpipe is written in Java. Two solutions then: using java from python or
|
||
reimplement boilerpipe …</p></summary><content type="html"><p>Java provides some interesting libraries that have no exact equivalent in
|
||
python. In my case, the awesome boilerpipe library allows me to remove
|
||
uninteresting parts of HTML pages, like menus, footers and other &quot;boilerplate&quot;
|
||
contents.</p>
|
||
<p>Boilerpipe is written in Java. Two solutions then: using java from python or
|
||
reimplement boilerpipe in python. I will let you guess which one I chosen, meh.</p>
|
||
<p>JPype allows to bridge python project with java libraries. It takes another
|
||
point of view than Jython: rather than reimplementing python in Java, both
|
||
languages are interfacing at the VM level. This means you need to start a VM
|
||
from your python script, but it does the job and stay fully compatible with
|
||
Cpython and its C extensions.</p>
|
||
<div class="section" id="first-steps-with-jpype">
|
||
<h2>First steps with JPype</h2>
|
||
<p>Once JPype installed (you'll have to hack a bit some files to integrate
|
||
seamlessly with your system) you can access java classes by doing something
|
||
like that:</p>
|
||
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">jpype</span>
|
||
<span class="n">jpype</span><span class="o">.</span><span class="n">startJVM</span><span class="p">(</span><span class="n">jpype</span><span class="o">.</span><span class="n">getDefaultJVMPath</span><span class="p">())</span>
|
||
|
||
<span class="c1"># you can then access to the basic java functions</span>
|
||
<span class="n">jpype</span><span class="o">.</span><span class="n">java</span><span class="o">.</span><span class="n">lang</span><span class="o">.</span><span class="n">System</span><span class="o">.</span><span class="n">out</span><span class="o">.</span><span class="n">println</span><span class="p">(</span><span class="s2">&quot;hello world&quot;</span><span class="p">)</span>
|
||
|
||
<span class="c1"># and you have to shutdown the VM at the end</span>
|
||
<span class="n">jpype</span><span class="o">.</span><span class="n">shutdownJVM</span><span class="p">()</span>
|
||
</pre></div>
|
||
<p>Okay, now we have a hello world, but what we want seems somehow more complex.
|
||
We want to interact with java classes, so we will have to load them.</p>
|
||
</div>
|
||
<div class="section" id="interfacing-with-boilerpipe">
|
||
<h2>Interfacing with Boilerpipe</h2>
|
||
<p>To install boilerpipe, you just have to run an ant script:</p>
|
||
<pre class="literal-block">
|
||
$ cd boilerpipe
|
||
$ ant
|
||
</pre>
|
||
<p>Here is a simple example of how to use boilerpipe in Java, from their sources</p>
|
||
<div class="highlight"><pre><span></span><span class="kn">package</span> <span class="nn">de.l3s.boilerpipe.demo</span><span class="o">;</span>
|
||
<span class="kn">import</span> <span class="nn">java.net.URL</span><span class="o">;</span>
|
||
<span class="kn">import</span> <span class="nn">de.l3s.boilerpipe.extractors.ArticleExtractor</span><span class="o">;</span>
|
||
|
||
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">Oneliner</span> <span class="o">{</span>
|
||
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="kd">final</span> <span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
|
||
<span class="kd">final</span> <span class="n">URL</span> <span class="n">url</span> <span class="o">=</span> <span class="k">new</span> <span class="n">URL</span><span class="o">(</span><span class="s">&quot;http://notmyidea.org&quot;</span><span class="o">);</span>
|
||
<span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">ArticleExtractor</span><span class="o">.</span><span class="na">INSTANCE</span><span class="o">.</span><span class="na">getText</span><span class="o">(</span><span class="n">url</span><span class="o">));</span>
|
||
<span class="o">}</span>
|
||
<span class="o">}</span>
|
||
</pre></div>
|
||
<p>To run it:</p>
|
||
<div class="highlight"><pre><span></span>$ javac -cp dist/boilerpipe-1.1-dev.jar:lib/nekohtml-1.9.13.jar:lib/xerces-2.9.1.jar src/demo/de/l3s/boilerpipe/demo/Oneliner.java
|
||
$ java -cp src/demo:dist/boilerpipe-1.1-dev.jar:lib/nekohtml-1.9.13.jar:lib/xerces-2.9.1.jar de.l3s.boilerpipe.demo.Oneliner
|
||
</pre></div>
|
||
<p>Yes, this is kind of ugly, sorry for your eyes.
|
||
Let's try something similar, but from python</p>
|
||
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">jpype</span>
|
||
|
||
<span class="c1"># start the JVM with the good classpaths</span>
|
||
<span class="n">classpath</span> <span class="o">=</span> <span class="s2">&quot;dist/boilerpipe-1.1-dev.jar:lib/nekohtml-1.9.13.jar:lib/xerces-2.9.1.jar&quot;</span>
|
||
<span class="n">jpype</span><span class="o">.</span><span class="n">startJVM</span><span class="p">(</span><span class="n">jpype</span><span class="o">.</span><span class="n">getDefaultJVMPath</span><span class="p">(),</span> <span class="s2">&quot;-Djava.class.path=</span><span class="si">%s</span><span class="s2">&quot;</span> <span class="o">%</span> <span class="n">classpath</span><span class="p">)</span>
|
||
|
||
<span class="c1"># get the Java classes we want to use</span>
|
||
<span class="n">DefaultExtractor</span> <span class="o">=</span> <span class="n">jpype</span><span class="o">.</span><span class="n">JPackage</span><span class="p">(</span><span class="s2">&quot;de&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">l3s</span><span class="o">.</span><span class="n">boilerpipe</span><span class="o">.</span><span class="n">extractors</span><span class="o">.</span><span class="n">DefaultExtractor</span>
|
||
|
||
<span class="c1"># call them !</span>
|
||
<span class="k">print</span> <span class="n">DefaultExtractor</span><span class="o">.</span><span class="n">INSTANCE</span><span class="o">.</span><span class="n">getText</span><span class="p">(</span><span class="n">jpype</span><span class="o">.</span><span class="n">java</span><span class="o">.</span><span class="n">net</span><span class="o">.</span><span class="n">URL</span><span class="p">(</span><span class="s2">&quot;http://blog.notmyidea.org&quot;</span><span class="p">))</span>
|
||
</pre></div>
|
||
<p>And you get what you want.</p>
|
||
<p>I must say I didn't thought it could work so easily. This will allow me to
|
||
extract text content from URLs and remove the <em>boilerplate</em> text easily
|
||
for infuse (my master thesis project), without having to write java code, nice!</p>
|
||
</div>
|
||
</content><category term="python"></category><category term="java"></category></entry><entry><title>How are you handling your shared expenses?</title><link href="https://blog.notmyidea.org/how-are-you-handling-your-shared-expenses.html" rel="alternate"></link><published>2011-10-15T00:00:00+02:00</published><updated>2011-10-15T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2011-10-15:/how-are-you-handling-your-shared-expenses.html</id><summary type="html"><p><strong>TL;DR:</strong> We're kick-starting a new application to manage your shared
|
||
expenses. Have a look at <a class="reference external" href="http://ihatemoney.notmyidea.org">http://ihatemoney.notmyidea.org</a></p>
|
||
<p>As a student, I lived in a lot of different locations, and the majority of them
|
||
had something in common: I lived with others. It usually was a great experience …</p></summary><content type="html"><p><strong>TL;DR:</strong> We're kick-starting a new application to manage your shared
|
||
expenses. Have a look at <a class="reference external" href="http://ihatemoney.notmyidea.org">http://ihatemoney.notmyidea.org</a></p>
|
||
<p>As a student, I lived in a lot of different locations, and the majority of them
|
||
had something in common: I lived with others. It usually was a great experience
|
||
(and I think I will continue to live with others). Most of the time, we had to
|
||
spend some time each month to compute who had to pay what to the others.</p>
|
||
<p>I wanted to create a pet project using flask, so I wrote a little
|
||
(~150 lines) flask application to handle this. It worked out pretty well for my
|
||
housemates and me, and as we had to move into different locations,
|
||
one of them asked me if he could continue to use it for the year to come.</p>
|
||
<p>I said yes and gave it some more thoughts: We probably aren't the only ones
|
||
interested by such kind of software. I decided to extend a bit more the
|
||
software to have a concept of projects and persons (the list of persons was
|
||
hard-coded in the first time, boooh!).</p>
|
||
<p>I then discussed with a friend of mine, who was excited about it and wanted to learn
|
||
python. Great! That's a really nice way to get started. Some more friends were also
|
||
interested in it and contributed some features and provided feedback (thanks
|
||
<a class="reference external" href="http://www.sneakernet.fr/">Arnaud</a> and Quentin!)</p>
|
||
<p>Since that, the project now support multiple languages and provides a REST API
|
||
(android and iphone apps in the tubes!), into other things.
|
||
There is no need to register for an account or whatnot, just enter a project name,
|
||
a secret code and a contact email, invite friends and that's it (this was inspired by
|
||
doodle)!</p>
|
||
<img alt="Capture d'écran du site." src="images/ihatemoney.png" />
|
||
<p>You can try the project at <a class="reference external" href="http://ihatemoney.notmyidea.org">http://ihatemoney.notmyidea.org</a> for now, and the
|
||
code lives at <a class="reference external" href="https://github.com/spiral-project/ihatemoney/">https://github.com/spiral-project/ihatemoney/</a>.</p>
|
||
<div class="section" id="features">
|
||
<h2>Features</h2>
|
||
<p>In the wild, currently, there already are some implementations of this shared
|
||
budget manager thing. The fact is that most of them are either hard to use, with
|
||
a too much fancy design or simply trying to do too much things at once.</p>
|
||
<p>No, I don't want my budget manager to make my shopping list, or to run a blog for
|
||
me, thanks. I want it to let me focus on something else. Keep out of my way.</p>
|
||
<div class="section" id="no-user-registration">
|
||
<h3>No user registration</h3>
|
||
<p>You don't need to register an account on the website to start using it. You
|
||
just have to create a project, set a secret code for it, and give both the url and
|
||
the code to the people you want to share it with (or the website can poke
|
||
them for you).</p>
|
||
</div>
|
||
<div class="section" id="keeping-things-simple">
|
||
<h3>Keeping things simple</h3>
|
||
<p>&quot;Keep It Simple, Stupid&quot; really matches our philosophy here: you want to add a
|
||
bill? Okay. Just do it. You just have to enter who paid, for who, how much,
|
||
and a description, like you would have done when you're back from the
|
||
farmer's market on raw paper.</p>
|
||
</div>
|
||
<div class="section" id="no-categories">
|
||
<h3>No categories</h3>
|
||
<p>Some people like to organise their stuff into different &quot;categories&quot;:
|
||
leisure, work, eating, etc. That's not something I want (at least to begin
|
||
with).</p>
|
||
<p>I want things to be simple. Got that? Great. Just add your bills!</p>
|
||
</div>
|
||
<div class="section" id="balance">
|
||
<h3>Balance</h3>
|
||
<p>One of the most useful thing is to know what's your &quot;balance&quot; compared to
|
||
others. In other words, if you're negative, you owe money, if you're positive,
|
||
you have to receive money. This allows you to dispatch who has to pay for the
|
||
next thing, in order to re-equilibrate the balance.</p>
|
||
<p>Additionally, the system is able to compute for you who has to give how
|
||
much to who, in order to reduce the number of transactions needed to restore
|
||
the balance.</p>
|
||
</div>
|
||
<div class="section" id="api">
|
||
<h3>API</h3>
|
||
<p>All of what's possible to do with the standard web interface is also available
|
||
through a REST API. I developed a simple REST toolkit for flask for this (and
|
||
I should release it!).</p>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="interested">
|
||
<h2>Interested?</h2>
|
||
<p>This project is open source. All of us like to share what we are doing and
|
||
would be happy to work with new people and implement new ideas. If you have
|
||
a nice idea about this, if you want to tweak it or to fill bugs. Don't hesitate
|
||
a second! The project lives at <a class="reference external" href="http://github.com/spiral-project/ihatemoney/">http://github.com/spiral-project/ihatemoney/</a></p>
|
||
</div>
|
||
</content><category term="python"></category><category term="flask"></category><category term="budget-manager"></category></entry><entry><title>Using dbpedia to get languages influences</title><link href="https://blog.notmyidea.org/using-dbpedia-to-get-languages-influences.html" rel="alternate"></link><published>2011-08-16T00:00:00+02:00</published><updated>2011-08-16T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2011-08-16:/using-dbpedia-to-get-languages-influences.html</id><summary type="html"><p>While browsing the Python's wikipedia page, I found information about the languages
|
||
influenced by python, and the languages that influenced python itself.</p>
|
||
<p>Well, that's kind of interesting to know which languages influenced others,
|
||
it could even be more interesting to have an overview of the connexion between
|
||
them, keeping python …</p></summary><content type="html"><p>While browsing the Python's wikipedia page, I found information about the languages
|
||
influenced by python, and the languages that influenced python itself.</p>
|
||
<p>Well, that's kind of interesting to know which languages influenced others,
|
||
it could even be more interesting to have an overview of the connexion between
|
||
them, keeping python as the main focus.</p>
|
||
<p>This information is available on the wikipedia page, but not in a really
|
||
exploitable format. Hopefully, this information is provided into the
|
||
information box present on the majority of wikipedia pages. And… guess what?
|
||
there is project with the goal to scrap and index all this information in
|
||
a more queriable way, using the semantic web technologies.</p>
|
||
<p>Well, you may have guessed it, the project in question in dbpedia, and exposes
|
||
information in the form of RDF triples, which are way more easy to work with
|
||
than simple HTML.</p>
|
||
<p>For instance, let's take the page about python:
|
||
<a class="reference external" href="http://dbpedia.org/page/Python_%28programming_language%29">http://dbpedia.org/page/Python_%28programming_language%29</a></p>
|
||
<p>The interesting properties here are &quot;Influenced&quot; and &quot;InfluencedBy&quot;, which
|
||
allows us to get a list of languages. Unfortunately, they are not really using
|
||
all the power of the Semantic Web here, and the list is actually a string with
|
||
coma separated values in it.</p>
|
||
<p>Anyway, we can use a simple rule: All wikipedia pages of programming languages
|
||
are either named after the name of the language itself, or suffixed with &quot;(
|
||
programming language)&quot;, which is the case for python.</p>
|
||
<p>So I've built <a class="reference external" href="https://github.com/ametaireau/experiments/blob/master/influences/get_influences.py">a tiny script to extract the information from dbpedia</a> and transform them into a shiny graph using graphviz.</p>
|
||
<p>After a nice:</p>
|
||
<pre class="literal-block">
|
||
$ python get_influences.py python dot | dot -Tpng &gt; influences.png
|
||
</pre>
|
||
<p>The result is the following graph (<a class="reference external" href="http://files.lolnet.org/alexis/influences.png">see it directly here</a>)</p>
|
||
<img alt="Graph des influances des langages les uns sur les autres." src="http://files.lolnet.org/alexis/influences.png" style="width: 800px;" />
|
||
<p>While reading this diagram, keep in mind that it is a) not listing all the
|
||
languages and b) keeping a python perspective.</p>
|
||
<p>This means that you can trust the scheme by following the arrows from python to
|
||
something and from something to python, it is not trying to get the matching
|
||
between all the languages at the same time to keep stuff readable.</p>
|
||
<p>It would certainly be possible to have all the connections between all
|
||
languages (and the resulting script would be easier) to do so, but the resulting
|
||
graph would probably be way less readable.</p>
|
||
<p>You can find the script <a class="reference external" href="https://github.com/ametaireau/experiments">on my github account</a>. Feel free to adapt it for
|
||
whatever you want if you feel hackish.</p>
|
||
</content><category term="dbpedia"></category><category term="sparql"></category><category term="python"></category></entry><entry><title>Pelican, 9 months later</title><link href="https://blog.notmyidea.org/pelican-9-months-later.html" rel="alternate"></link><published>2011-07-25T00:00:00+02:00</published><updated>2011-07-25T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2011-07-25:/pelican-9-months-later.html</id><summary type="html"><p>Back in October, I released <a class="reference external" href="http://docs.notmyidea.org/alexis/pelican">pelican</a>,
|
||
a little piece of code I wrote to power this weblog. I had simple needs: I wanted
|
||
to be able to use my text editor of choice (vim), a vcs (mercurial) and
|
||
restructured text. I started to write a really simple blog engine
|
||
in …</p></summary><content type="html"><p>Back in October, I released <a class="reference external" href="http://docs.notmyidea.org/alexis/pelican">pelican</a>,
|
||
a little piece of code I wrote to power this weblog. I had simple needs: I wanted
|
||
to be able to use my text editor of choice (vim), a vcs (mercurial) and
|
||
restructured text. I started to write a really simple blog engine
|
||
in something like a hundred python lines and released it on github.</p>
|
||
<p>And people started contributing. I wasn't at all expecting to see people
|
||
interested in such a little piece of code, but it turned out that they were.
|
||
I refactored the code to make it evolve a bit more by two times and eventually,
|
||
in 9 months, got 49 forks, 139 issues and 73 pull requests.</p>
|
||
<p><strong>Which is clearly awesome.</strong></p>
|
||
<p>I pulled features such as translations, tag
|
||
clouds, integration with different services such as twitter or piwik, import
|
||
from dotclear and rss, fixed
|
||
a number of mistakes and improved a lot the codebase. This was a proof that
|
||
there is a bunch of people that are willing to make better softwares just for
|
||
the sake of fun.</p>
|
||
<p>Thank you, guys, you're why I like open source so much.</p>
|
||
</content><category term="pelican"></category><category term="python"></category><category term="open source"></category><category term="nice story"></category></entry><entry><title>Introducing Cornice</title><link href="https://blog.notmyidea.org/introducing-cornice.html" rel="alternate"></link><published>2011-07-12T00:00:00+02:00</published><updated>2011-07-12T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2011-07-12:/introducing-cornice.html</id><summary type="html"><p>Wow, already my third working day at Mozilla. Since Monday, I've been working with
|
||
<a class="reference external" href="http://ziade.org">Tarek Ziadé</a>, on a pyramid REST-ish toolkit named <a class="reference external" href="https://github.com/mozilla-services/cornice">Cornice</a>.</p>
|
||
<p>Its goal is to take care for you of what you're usually missing so you can
|
||
focus on what's important. Cornice provides you facilities
|
||
for validation of …</p></summary><content type="html"><p>Wow, already my third working day at Mozilla. Since Monday, I've been working with
|
||
<a class="reference external" href="http://ziade.org">Tarek Ziadé</a>, on a pyramid REST-ish toolkit named <a class="reference external" href="https://github.com/mozilla-services/cornice">Cornice</a>.</p>
|
||
<p>Its goal is to take care for you of what you're usually missing so you can
|
||
focus on what's important. Cornice provides you facilities
|
||
for validation of any kind.</p>
|
||
<p>The goal is to simplify your work, but we don't want to reinvent the wheel, so
|
||
it is easily pluggable with validations frameworks, such as <a class="reference external" href="http://docs.pylonsproject.org/projects/colander/en/latest/">Colander</a>.</p>
|
||
<div class="section" id="handling-errors-and-validation">
|
||
<h2>Handling errors and validation</h2>
|
||
<p>Here is how it works:</p>
|
||
<div class="highlight"><pre><span></span><span class="n">service</span> <span class="o">=</span> <span class="n">Service</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s2">&quot;service&quot;</span><span class="p">,</span> <span class="n">path</span><span class="o">=</span><span class="s2">&quot;/service&quot;</span><span class="p">)</span>
|
||
|
||
|
||
<span class="k">def</span> <span class="nf">is_awesome</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
|
||
<span class="k">if</span> <span class="ow">not</span> <span class="s1">&#39;awesome&#39;</span> <span class="ow">in</span> <span class="n">request</span><span class="o">.</span><span class="n">GET</span><span class="p">:</span>
|
||
<span class="n">request</span><span class="o">.</span><span class="n">errors</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="s1">&#39;query&#39;</span><span class="p">,</span> <span class="s1">&#39;awesome&#39;</span><span class="p">,</span>
|
||
<span class="s1">&#39;the awesome parameter is required&#39;</span><span class="p">)</span>
|
||
|
||
|
||
<span class="nd">@service.get</span><span class="p">(</span><span class="n">validator</span><span class="o">=</span><span class="n">is_awesome</span><span class="p">)</span>
|
||
<span class="k">def</span> <span class="nf">get1</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="p">{</span><span class="s2">&quot;test&quot;</span><span class="p">:</span> <span class="s2">&quot;yay!&quot;</span><span class="p">}</span>
|
||
</pre></div>
|
||
<p>All the errors collected during the validation process, or after, are collected
|
||
before returning the request. If any, a error 400 is fired up, with the list of
|
||
problems encountered returned as a nice json list response (we plan to support
|
||
multiple formats in the future)</p>
|
||
<p>As you might have seen, <cite>request.errors.add</cite> takes three parameters: <strong>location</strong>,
|
||
<strong>name</strong> and <strong>description</strong>.</p>
|
||
<p><strong>location</strong> is where the error is located in the request. It can either be &quot;body&quot;,
|
||
&quot;query&quot;, &quot;headers&quot; or &quot;path&quot;. <strong>name</strong> is the name of the variable causing
|
||
problem, if any, and <strong>description</strong> contains a more detailed message.</p>
|
||
<p>Let's run this simple service and send some queries to it:</p>
|
||
<pre class="literal-block">
|
||
$ curl -v http://127.0.0.1:5000/service
|
||
&gt; GET /service HTTP/1.1
|
||
&gt; Host: 127.0.0.1:5000
|
||
&gt; Accept: */*
|
||
&gt;
|
||
* HTTP 1.0, assume close after body
|
||
&lt; HTTP/1.0 400 Bad Request
|
||
&lt; Content-Type: application/json; charset=UTF-8
|
||
[{&quot;location&quot;: &quot;query&quot;, &quot;name&quot;: &quot;awesome&quot;, &quot;description&quot;: &quot;You lack awesomeness!&quot;}
|
||
</pre>
|
||
<p>I've removed the extra clutter from the curl's output, but you got the general idea.</p>
|
||
<p>The content returned is in JSON, and I know exactly what I have to do: add an
|
||
&quot;awesome&quot; parameter in my query. Let's do it again:</p>
|
||
<pre class="literal-block">
|
||
$ curl http://127.0.0.1:5000/service?awesome=yeah
|
||
{&quot;test&quot;: &quot;yay!&quot;}
|
||
</pre>
|
||
<p>Validators can also convert parts of the request and store the converted value
|
||
in <cite>request.validated</cite>. It is a standard dict automatically attached to the
|
||
requests.</p>
|
||
<p>For instance, in our validator, we can chose to validate the parameter passed
|
||
and use it in the body of the webservice:</p>
|
||
<div class="highlight"><pre><span></span><span class="n">service</span> <span class="o">=</span> <span class="n">Service</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s2">&quot;service&quot;</span><span class="p">,</span> <span class="n">path</span><span class="o">=</span><span class="s2">&quot;/service&quot;</span><span class="p">)</span>
|
||
|
||
|
||
<span class="k">def</span> <span class="nf">is_awesome</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
|
||
<span class="k">if</span> <span class="ow">not</span> <span class="s1">&#39;awesome&#39;</span> <span class="ow">in</span> <span class="n">request</span><span class="o">.</span><span class="n">GET</span><span class="p">:</span>
|
||
<span class="n">request</span><span class="o">.</span><span class="n">errors</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="s1">&#39;query&#39;</span><span class="p">,</span> <span class="s1">&#39;awesome&#39;</span><span class="p">,</span>
|
||
<span class="s1">&#39;the awesome parameter is required&#39;</span><span class="p">)</span>
|
||
<span class="k">else</span><span class="p">:</span>
|
||
<span class="n">request</span><span class="o">.</span><span class="n">validated</span><span class="p">[</span><span class="s1">&#39;awesome&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="s1">&#39;awesome &#39;</span> <span class="o">+</span> <span class="n">request</span><span class="o">.</span><span class="n">GET</span><span class="p">[</span><span class="s1">&#39;awesome&#39;</span><span class="p">]</span>
|
||
|
||
|
||
<span class="nd">@service.get</span><span class="p">(</span><span class="n">validator</span><span class="o">=</span><span class="n">is_awesome</span><span class="p">)</span>
|
||
<span class="k">def</span> <span class="nf">get1</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="p">{</span><span class="s2">&quot;test&quot;</span><span class="p">:</span> <span class="n">request</span><span class="o">.</span><span class="n">validated</span><span class="p">[</span><span class="s1">&#39;awesome&#39;</span><span class="p">]}</span>
|
||
</pre></div>
|
||
<p>The output would look like this:</p>
|
||
<pre class="literal-block">
|
||
curl http://127.0.0.1:5000/service?awesome=yeah
|
||
{&quot;test&quot;: &quot;awesome yeah&quot;}
|
||
</pre>
|
||
</div>
|
||
<div class="section" id="dealing-with-accept-headers">
|
||
<h2>Dealing with &quot;Accept&quot; headers</h2>
|
||
<p>The HTTP spec defines a <strong>Accept</strong> header the client can send so the response
|
||
is encoded the right way. A resource, available at an URL, can be available in
|
||
different formats. This is especially true for web services.</p>
|
||
<p>Cornice can help you dealing with this. The services you define can tell which
|
||
<cite>Content-Type</cite> values they can deal with and this will be checked against the
|
||
<strong>Accept</strong> headers sent by the client.</p>
|
||
<p>Let's refine a bit our previous example, by specifying which content-types are
|
||
supported, using the <cite>accept</cite> parameter:</p>
|
||
<div class="highlight"><pre><span></span><span class="nd">@service.get</span><span class="p">(</span><span class="n">validator</span><span class="o">=</span><span class="n">is_awesome</span><span class="p">,</span> <span class="n">accept</span><span class="o">=</span><span class="p">(</span><span class="s2">&quot;application/json&quot;</span><span class="p">,</span> <span class="s2">&quot;text/json&quot;</span><span class="p">))</span>
|
||
<span class="k">def</span> <span class="nf">get1</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="p">{</span><span class="s2">&quot;test&quot;</span><span class="p">:</span> <span class="s2">&quot;yay!&quot;</span><span class="p">}</span>
|
||
</pre></div>
|
||
<p>Now, if you specifically ask for XML, Cornice will throw a 406 with the list of
|
||
accepted <cite>Content-Type</cite> values:</p>
|
||
<pre class="literal-block">
|
||
$ curl -vH &quot;Accept: application/xml&quot; http://127.0.0.1:5000/service
|
||
&gt; GET /service HTTP/1.1
|
||
&gt; Host: 127.0.0.1:5000
|
||
&gt; Accept: application/xml
|
||
&gt;
|
||
&lt; HTTP/1.0 406 Not Acceptable
|
||
&lt; Content-Type: application/json; charset=UTF-8
|
||
&lt; Content-Length: 33
|
||
&lt;
|
||
[&quot;application/json&quot;, &quot;text/json&quot;]
|
||
</pre>
|
||
</div>
|
||
<div class="section" id="building-your-documentation-automatically">
|
||
<h2>Building your documentation automatically</h2>
|
||
<p>writing documentation for web services can be painful, especially when your
|
||
services evolve. Cornice provides a sphinx directive to automatically document
|
||
your API in your docs.</p>
|
||
<div class="highlight"><pre><span></span><span class="p">..</span> <span class="ow">services</span><span class="p">::</span>
|
||
<span class="nc">:package:</span> <span class="nf">coolapp</span>
|
||
<span class="nc">:service:</span> <span class="nf">quote</span>
|
||
</pre></div>
|
||
<p>Here is an example of what a generated page looks like: <a class="reference external" href="http://packages.python.org/cornice/exampledoc.html">http://packages.python.org/cornice/exampledoc.html</a></p>
|
||
</div>
|
||
<div class="section" id="yay-how-can-i-get-it">
|
||
<h2>Yay! How can I get it?</h2>
|
||
<p>We just cut a 0.4 release, so it's available at <a class="reference external" href="http://pypi.python.org/pypi/cornice">http://pypi.python.org/pypi/cornice</a>
|
||
You can install it easily using <cite>pip</cite>, for instance:</p>
|
||
<pre class="literal-block">
|
||
$ pip install cornice
|
||
</pre>
|
||
<p>You can also have a look at the documentation at
|
||
<a class="reference external" href="http://packages.python.org/cornice/">http://packages.python.org/cornice/</a></p>
|
||
</div>
|
||
<div class="section" id="what-s-next">
|
||
<h2>What's next?</h2>
|
||
<p>We try to make our best to find how Cornice can help you build better
|
||
web services. Cool features we want for the future include the automatic
|
||
publication of a static definition of the services, so it can be used by clients
|
||
to discover services in a nice way.</p>
|
||
<p>Of course, we are open to all your ideas and patches! If you feel haskish and
|
||
want to see the sources, <a class="reference external" href="https://github.com/mozilla-services/cornice">go grab them on github</a>
|
||
, commit and send us a pull request!</p>
|
||
</div>
|
||
</content></entry><entry><title>Un coup de main pour mon mémoire!</title><link href="https://blog.notmyidea.org/un-coup-de-main-pour-mon-memoire-fr.html" rel="alternate"></link><published>2011-05-25T00:00:00+02:00</published><updated>2011-05-25T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2011-05-25:/un-coup-de-main-pour-mon-memoire-fr.html</id><summary type="html"><p>Ça y est, bientôt la fin. LA FIN. La fin des études, et le début du reste.
|
||
En attendant je bosse sur mon mémoire de fin d'études et j'aurais besoin d'un petit
|
||
coup de main.</p>
|
||
<p>Mon mémoire porte sur les systèmes de recommandation. Pour ceux qui connaissent
|
||
last.fm, je …</p></summary><content type="html"><p>Ça y est, bientôt la fin. LA FIN. La fin des études, et le début du reste.
|
||
En attendant je bosse sur mon mémoire de fin d'études et j'aurais besoin d'un petit
|
||
coup de main.</p>
|
||
<p>Mon mémoire porte sur les systèmes de recommandation. Pour ceux qui connaissent
|
||
last.fm, je fais quelque chose de similaire mais pour les sites internet: en me
|
||
basant sur ce que vous visitez quotidiennement et comment vous le visitez (quelles
|
||
horaires, quelle emplacement géographique, etc.) je souhaites proposer des liens
|
||
qui vous intéresseront potentiellement, en me basant sur l'avis des personnes qui
|
||
ont des profils similaires au votre.</p>
|
||
<p>Le projet est loin d'être terminé, mais la première étape est de récupérer des
|
||
données de navigation, idéalement beaucoup de données de navigation. Donc si
|
||
vous pouvez me filer un coup de main je vous en serais éternellement
|
||
reconnaissant (pour ceux qui font semblant de pas comprendre, entendez &quot;tournée
|
||
générale&quot;).</p>
|
||
<p>J'ai créé un petit site web (en anglais) qui résume un peu le concept, qui vous
|
||
propose de vous inscrire et de télécharger un plugin firefox qui m'enverra des
|
||
information sur les sites que vous visitez (si vous avez l'habitude d'utiliser
|
||
chrome vous pouvez considérer de switcher à firefox4 pour les deux prochains
|
||
mois pour me filer un coup de main). Il est possible de désactiver le plugin
|
||
d'un simple clic si vous souhaitez garder votre vie privée privée ;-)</p>
|
||
<p>Le site est par là: <a class="reference external" href="http://infuse.notmyidea.org">http://infuse.notmyidea.org</a>. Une fois le plugin téléchargé
|
||
et le compte créé il faut renseigner vos identifiants dans le plugin en
|
||
question, et c'est tout!</p>
|
||
<p>A votre bon cœur ! Je récupérerais probablement des données durant les 2
|
||
prochains mois pour ensuite les analyser correctement.</p>
|
||
<p>Merci pour votre aide !</p>
|
||
</content></entry><entry><title>Analyse users' browsing context to build up a web recommender</title><link href="https://blog.notmyidea.org/analyse-users-browsing-context-to-build-up-a-web-recommender.html" rel="alternate"></link><published>2011-04-01T00:00:00+02:00</published><updated>2011-04-01T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2011-04-01:/analyse-users-browsing-context-to-build-up-a-web-recommender.html</id><summary type="html"><p>No, this is not an april's fool ;)</p>
|
||
<p>Wow, it's been a long time. My year in Oxford is going really well. I realized
|
||
few days ago that the end of the year is approaching really quickly.
|
||
Exams are coming in one month or such and then I'll be working full …</p></summary><content type="html"><p>No, this is not an april's fool ;)</p>
|
||
<p>Wow, it's been a long time. My year in Oxford is going really well. I realized
|
||
few days ago that the end of the year is approaching really quickly.
|
||
Exams are coming in one month or such and then I'll be working full time on my dissertation topic.</p>
|
||
<p>When I learned we'll have about 6 month to work on something, I first thought
|
||
about doing a packaging related stuff, but finally decided to start something
|
||
new. After all, that's the good time to learn.</p>
|
||
<p>Since a long time, I'm being impressed by the <a class="reference external" href="http://last.fm">last.fm</a>
|
||
recommender system. They're <em>scrobbling</em> the music I listen to since something
|
||
like 5 years now and the recommendations they're doing are really nice and
|
||
accurate (I discovered <strong>a lot</strong> of great artists listening to the
|
||
&quot;neighbour radio&quot;.) (by the way, <a class="reference external" href="http://lastfm.com/user/akounet/">here is</a>
|
||
my lastfm account)</p>
|
||
<p>So I decided to work on recommender systems, to better understand what is it
|
||
about.</p>
|
||
<p>Recommender systems are usually used to increase the sales of products
|
||
(like Amazon.com does) which is not really what I'm looking for (The one who
|
||
know me a bit know I'm kind of sick about all this consumerism going on).</p>
|
||
<p>Actually, the most simple thing I thought of was the web: I'm browsing it quite
|
||
every day and each time new content appears. I've stopped to follow <a class="reference external" href="https://bitbucket.org/bruno/aspirator/">my feed
|
||
reader</a> because of the
|
||
information overload, and reduced drastically the number of people I follow <a class="reference external" href="http://twitter.com/ametaireau/">on
|
||
twitter</a>.</p>
|
||
<p>Too much information kills the information.</p>
|
||
<p>You shall got what will be my dissertation topic: a recommender system for
|
||
the web. Well, such recommender systems already exists, so I will try to add contextual
|
||
information to them: you're probably not interested by the same topics at different
|
||
times of the day, or depending on the computer you're using. We can also
|
||
probably make good use of the way you browse to create groups into the content
|
||
you're browsing (or even use the great firefox4 tab group feature).</p>
|
||
<p>There is a large part of concerns to have about user's privacy as well.</p>
|
||
<p>Here is my proposal (copy/pasted from the one I had to do for my master)</p>
|
||
<div class="section" id="introduction-and-rationale">
|
||
<h2>Introduction and rationale</h2>
|
||
<p>Nowadays, people surf the web more and more often. New web pages are created
|
||
each day so the amount of information to retrieve is more important as the time
|
||
passes. These users uses the web in different contexts, from finding cooking
|
||
recipes to technical articles.</p>
|
||
<p>A lot of people share the same interest to various topics, and the quantity of
|
||
information is such than it's really hard to triage them efficiently without
|
||
spending hours doing it. Firstly because of the huge quantity of information
|
||
but also because the triage is something relative to each person. Although, this
|
||
triage can be facilitated by fetching the browsing information of all
|
||
particular individuals and put the in perspective.</p>
|
||
<p>Machine learning is a branch of Artificial Intelligence (AI) which deals with how
|
||
a program can learn from data. Recommendation systems are a particular
|
||
application area of machine learning which is able to recommend things (links
|
||
in our case) to the users, given a particular database containing the previous
|
||
choices users have made.</p>
|
||
<p>This browsing information is currently available in browsers. Even if it is not
|
||
in a very usable format, it is possible to transform it to something useful.
|
||
This information gold mine just wait to be used. Although, it is not as simple as
|
||
it can seems at the first approach: It is important to take care of the context
|
||
the user is in while browsing links. For instance, It's more likely that during
|
||
the day, a computer scientist will browse computing related links, and that during
|
||
the evening, he browse cooking recipes or something else.</p>
|
||
<p>Page contents are also interesting to analyse, because that's what people
|
||
browse and what actually contain the most interesting part of the information.
|
||
The raw data extracted from the browsing can then be translated into
|
||
something more useful (namely tags, type of resource, visit frequency,
|
||
navigation context etc.)</p>
|
||
<p>The goal of this dissertation is to create a recommender system for web links,
|
||
including this context information.</p>
|
||
<p>At the end of the dissertation, different pieces of software will be provided,
|
||
from raw data collection from the browser to a recommendation system.</p>
|
||
</div>
|
||
<div class="section" id="background-review">
|
||
<h2>Background Review</h2>
|
||
<p>This dissertation is mainly about data extraction, analysis and recommendation
|
||
systems. Two different research area can be isolated: Data preprocessing and
|
||
Information filtering.</p>
|
||
<p>The first step in order to make recommendations is to gather some data. The
|
||
more data we have available, the better it is (T. Segaran, 2007). This data can
|
||
be retrieved in various ways, one of them is to get it directly from user's
|
||
browsers.</p>
|
||
<div class="section" id="data-preparation-and-extraction">
|
||
<h3>Data preparation and extraction</h3>
|
||
<p>The data gathered from browsers is basically URLs and additional information
|
||
about the context of the navigation. There is clearly a need to extract more
|
||
information about the meaning of the data the user is browsing, starting by the
|
||
content of the web pages.</p>
|
||
<p>Because the information provided on the current Web is not meant to be read by
|
||
machines (T. Berners Lee, 2001) there is a need of tools to extract meaning from
|
||
web pages. The information needs to be preprocessed before stored in a machine
|
||
readable format, allowing to make recommendations (Choochart et Al, 2004).</p>
|
||
<p>Data preparation is composed of two steps: cleaning and structuring (
|
||
Castellano et Al, 2007). Because raw data can contain a lot of un-needed text
|
||
(such as menus, headers etc.) and need to be cleaned prior to be stored.
|
||
Multiple techniques can be used here and belongs to boilerplate removal and
|
||
full text extraction (Kohlschütter et Al, 2010).</p>
|
||
<p>Then, structuring the information: category, type of content (news, blog, wiki)
|
||
can be extracted from raw data. This kind of information is not clearly defined
|
||
by HTML pages so there is a need of tools to recognise them.</p>
|
||
<p>Some context-related information can also be inferred from each resource. It can go
|
||
from the visit frequency to the navigation group the user was in while
|
||
browsing. It is also possible to determine if the user &quot;liked&quot; a resource, and
|
||
determine a mark for it, which can be used by information filtering a later
|
||
step (T. Segaran, 2007).</p>
|
||
<p>At this stage, structuring the data is required. Storing this kind of
|
||
information in RDBMS can be a bit tedious and require complex queries to get
|
||
back the data in an usable format. Graph databases can play a major role in the
|
||
simplification of information storage and querying.</p>
|
||
</div>
|
||
<div class="section" id="information-filtering">
|
||
<h3>Information filtering</h3>
|
||
<p>To filter the information, three techniques can be used (Balabanovic et
|
||
Al, 1997):</p>
|
||
<ul class="simple">
|
||
<li>The content-based approach states that if an user have liked something in the
|
||
past, he is more likely to like similar things in the future. So it's about
|
||
establishing a profile for the user and compare new items against it.</li>
|
||
<li>The collaborative approach will rather recommend items that other similar users
|
||
have liked. This approach consider only the relationship between users, and
|
||
not the profile of the user we are making recommendations to.</li>
|
||
<li>the hybrid approach, which appeared recently combine both of the previous
|
||
approaches, giving recommendations when items score high regarding user's
|
||
profile, or if a similar user already liked it.</li>
|
||
</ul>
|
||
<p>Grouping is also something to consider at this stage (G. Myatt, 2007).
|
||
Because we are dealing with huge amount of data, it can be useful to detect group
|
||
of data that can fit together. Data clustering is able to find such groups (T.
|
||
Segaran, 2007).</p>
|
||
<p>References:</p>
|
||
<ul class="simple">
|
||
<li>Balabanović, M., &amp; Shoham, Y. (1997). Fab: content-based, collaborative
|
||
recommendation. Communications of the ACM, 40(3), 66–72. ACM.
|
||
Retrieved March 1, 2011, from <a class="reference external" href="http://portal.acm.org/citation.cfm?id=245108.245124&amp;amp">http://portal.acm.org/citation.cfm?id=245108.245124&amp;amp</a>;.</li>
|
||
<li>Berners-Lee, T., Hendler, J., &amp; Lassila, O. (2001).
|
||
The semantic web: Scientific american. Scientific American, 284(5), 34–43.
|
||
Retrieved November 21, 2010, from <a class="reference external" href="http://www.citeulike.org/group/222/article/1176986">http://www.citeulike.org/group/222/article/1176986</a>.</li>
|
||
<li>Castellano, G., Fanelli, A., &amp; Torsello, M. (2007).
|
||
LODAP: a LOg DAta Preprocessor for mining Web browsing patterns. Proceedings of the 6th Conference on 6th WSEAS Int. Conf. on Artificial Intelligence, Knowledge Engineering and Data Bases-Volume 6 (p. 12–17). World Scientific and Engineering Academy and Society (WSEAS). Retrieved March 8, 2011, from <a class="reference external" href="http://portal.acm.org/citation.cfm?id=1348485.1348488">http://portal.acm.org/citation.cfm?id=1348485.1348488</a>.</li>
|
||
<li>Kohlschutter, C., Fankhauser, P., &amp; Nejdl, W. (2010). Boilerplate detection using shallow text features. Proceedings of the third ACM international conference on Web search and data mining (p. 441–450). ACM. Retrieved March 8, 2011, from <a class="reference external" href="http://portal.acm.org/citation.cfm?id=1718542">http://portal.acm.org/citation.cfm?id=1718542</a>.</li>
|
||
<li>Myatt, G. J. (2007). Making Sense of Data: A Practical Guide to Exploratory
|
||
Data Analysis and Data Mining.</li>
|
||
<li>Segaran, T. (2007). Collective Intelligence.</li>
|
||
</ul>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="privacy">
|
||
<h2>Privacy</h2>
|
||
<p>The first thing that's come to people minds when it comes to process their
|
||
browsing data is privacy. People don't want to be stalked. That's perfectly
|
||
right, and I don't either.</p>
|
||
<p>But such a system don't have to deal with people identities. It's completely
|
||
possible to process completely anonymous data, and that's probably what I'm
|
||
gonna do.</p>
|
||
<p>By the way, if you have interesting thoughts about that, if you do know
|
||
projects that do seems related, fire the comments !</p>
|
||
</div>
|
||
<div class="section" id="what-s-the-plan">
|
||
<h2>What's the plan ?</h2>
|
||
<p>There is a lot of different things to explore, especially because I'm
|
||
a complete novice in that field.</p>
|
||
<ul class="simple">
|
||
<li>I want to develop a firefox plugin, to extract the browsing informations (
|
||
still, I need to know exactly which kind of informations to retrieve). The
|
||
idea is to provide some <em>raw</em> browsing data, and then to transform it and to
|
||
store it in the better possible way.</li>
|
||
<li>Analyse how to store the informations in a graph database. What can be the
|
||
different methods to store this data and to visualize the relationship
|
||
between different pieces of data? How can I define the different contexts,
|
||
and add those informations in the db?</li>
|
||
<li>Process the data using well known recommendation algorithms. Compare the
|
||
results and criticize their value.</li>
|
||
</ul>
|
||
<p>There is plenty of stuff I want to try during this experimentation:</p>
|
||
<ul class="simple">
|
||
<li>I want to try using Geshi to visualize the connexion between the links,
|
||
and the contexts</li>
|
||
<li>Try using graph databases such as Neo4j</li>
|
||
<li>Having a deeper look at tools such as scikit.learn (a machine learning
|
||
toolkit in python)</li>
|
||
<li>Analyse web pages in order to categorize them. Processing their
|
||
contents as well, to do some keyword based classification will be done.</li>
|
||
</ul>
|
||
<p>Lot of work on its way, yay !</p>
|
||
</div>
|
||
</content><category term="recommendations"></category><category term="browsers"></category><category term="users"></category></entry><entry><title>Working directly on your server? How to backup and sync your dev environment with unison</title><link href="https://blog.notmyidea.org/working-directly-on-your-server-how-to-backup-and-sync-your-dev-environment-with-unison.html" rel="alternate"></link><published>2011-03-16T00:00:00+01:00</published><updated>2011-03-16T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2011-03-16:/working-directly-on-your-server-how-to-backup-and-sync-your-dev-environment-with-unison.html</id><summary type="html"><p>I have a server running freebsd since some time now, and was wondering about
|
||
the possibility to directly have a development environment ready to use when
|
||
I get a internet connexion, even if I'm not on my computer.</p>
|
||
<p>Since I use vim to code, and spend most of my time …</p></summary><content type="html"><p>I have a server running freebsd since some time now, and was wondering about
|
||
the possibility to directly have a development environment ready to use when
|
||
I get a internet connexion, even if I'm not on my computer.</p>
|
||
<p>Since I use vim to code, and spend most of my time in a console while
|
||
developing, it's possible to work via ssh, from everywhere.</p>
|
||
<p>The only problem is the synchronisation of the source code, config files etc.
|
||
from my machine to the server.</p>
|
||
<p>Unison provides an interesting way to synchronise two folders, even over
|
||
a network. So let's do it !</p>
|
||
<div class="section" id="creating-the-jail">
|
||
<h2>Creating the jail</h2>
|
||
<p>In case you don't use FreeBSD, you can skip this section.</p>
|
||
<div class="highlight"><pre><span></span><span class="c1"># I have a flavour jail named default</span>
|
||
$ ezjail-admin -f default workspace.notmyidea.org <span class="m">172</span>.19.1.6
|
||
$ ezjail-admin start workspace.notmyidea.org
|
||
</pre></div>
|
||
<p>In my case, because the &quot;default&quot; flavour contains already a lot of interesting
|
||
things, my jail come already setup with ssh, bash and vim for instance, but
|
||
maybe you'll need it in your case.</p>
|
||
<p>I want to be redirected to the ssh of the jail when I connect to the host with
|
||
the 20006 port. Add lines in <cite>/etc/pf.conf</cite>:</p>
|
||
<pre class="literal-block">
|
||
workspace_jail=&quot;172.19.1.6&quot;
|
||
rdr on $ext_if proto tcp from any to $ext_ip port 20006 -&gt; $workspace_jail port 22
|
||
</pre>
|
||
<p>Reload packet filter rules</p>
|
||
<div class="highlight"><pre><span></span>$ /etc/rc.d/pf reload
|
||
</pre></div>
|
||
</div>
|
||
<div class="section" id="working-with-unison">
|
||
<h2>Working with unison</h2>
|
||
<p>Now that we've set up the jail. Set up unison on the server and on your client.
|
||
Unison is available on the freebsd ports so just install it</p>
|
||
<div class="highlight"><pre><span></span>$ ssh notmyidea.org -p <span class="m">20006</span>
|
||
$ make -C /usr/ports/net/unison-nox11 config-recursive
|
||
$ make -C /usr/ports/net/unison-nox11 package-recursive
|
||
</pre></div>
|
||
<p>Install as well unison on your local machine. Double check to install the same
|
||
version on the client and on the server. Ubuntu contains the 2.27.57 as well as
|
||
the 2.32.52.</p>
|
||
<p>Check that unison is installed and reachable via ssh from your machine</p>
|
||
<div class="highlight"><pre><span></span>$ ssh notmyidea.org -p <span class="m">20006</span> unison -version
|
||
unison version <span class="m">2</span>.27.157
|
||
$ unison -version
|
||
unison version <span class="m">2</span>.27.57
|
||
</pre></div>
|
||
</div>
|
||
<div class="section" id="let-sync-our-folders">
|
||
<h2>Let sync our folders</h2>
|
||
<p>The first thing I want to sync is my vim configuration. Well, it's already <a class="reference external" href="http://github.com/ametaireau/dotfiles/">in
|
||
a git repository</a> but let's try to use
|
||
unison for it right now.</p>
|
||
<p>I have two machines then: <cite>workspace</cite>, the jail, and <cite>ecureuil</cite> my laptop.</p>
|
||
<div class="highlight"><pre><span></span>unison .vim ssh://notmyidea.org:20006/.vim
|
||
unison .vimrc ssh://notmyidea.org:20006/.vimrc
|
||
</pre></div>
|
||
<p>It is also possible to put all the informations in a config file, and then to
|
||
only run <cite>unison</cite>. (fire up <cite>vim ~/.unison/default.prf</cite>.</p>
|
||
<p>Here is my config:</p>
|
||
<pre class="literal-block">
|
||
root = /home/alexis
|
||
root = ssh://notmyidea.org:20006
|
||
|
||
path = .vimrc
|
||
path = dotfiles
|
||
path = dev
|
||
|
||
follow = Name *
|
||
</pre>
|
||
<p>My vimrc is in fact a symbolic link on my laptop, but I don't want to specify
|
||
each of the links to unison. That's why the <cite>follow = Name *</cite> is for.</p>
|
||
<p>The folders you want to synchronize are maybe a bit large. If so, considering
|
||
others options such as rsync for the first import may be a good idea (I enjoyed
|
||
my university huge upload bandwith to upload 2GB in 20mn ;)</p>
|
||
</div>
|
||
<div class="section" id="run-the-script-frequently">
|
||
<h2>Run the script frequently</h2>
|
||
<p>Once that done, you just need to run the unison command line some times when
|
||
you want to sync your two machines. I've wrote a tiny script to get some
|
||
feedback from the sync:</p>
|
||
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||
<span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
|
||
|
||
<span class="n">DEFAULT_LOGFILE</span> <span class="o">=</span> <span class="s2">&quot;~/unison.log&quot;</span>
|
||
<span class="n">PROGRAM_NAME</span> <span class="o">=</span> <span class="s2">&quot;Unison syncer&quot;</span>
|
||
|
||
<span class="k">def</span> <span class="nf">sync</span><span class="p">(</span><span class="n">logfile</span><span class="o">=</span><span class="n">DEFAULT_LOGFILE</span><span class="p">,</span> <span class="n">program_name</span><span class="o">=</span><span class="n">PROGRAM_NAME</span><span class="p">):</span>
|
||
<span class="c1"># init</span>
|
||
<span class="n">display_message</span> <span class="o">=</span> <span class="bp">True</span>
|
||
<span class="n">error</span> <span class="o">=</span> <span class="bp">False</span>
|
||
|
||
<span class="n">before</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
|
||
<span class="c1"># call unison to make the sync</span>
|
||
<span class="n">os</span><span class="o">.</span><span class="n">system</span><span class="p">(</span><span class="s1">&#39;unison -batch &gt; {0}&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">logfile</span><span class="p">))</span>
|
||
|
||
<span class="c1"># get the duration of the operation</span>
|
||
<span class="n">td</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">-</span> <span class="n">before</span>
|
||
<span class="n">delta</span> <span class="o">=</span> <span class="p">(</span><span class="n">td</span><span class="o">.</span><span class="n">microseconds</span> <span class="o">+</span> <span class="p">(</span><span class="n">td</span><span class="o">.</span><span class="n">seconds</span> <span class="o">+</span> <span class="n">td</span><span class="o">.</span><span class="n">days</span> <span class="o">*</span> <span class="mi">24</span> <span class="o">*</span> <span class="mi">3600</span><span class="p">)</span> <span class="o">*</span> <span class="mi">10</span><span class="o">**</span><span class="mi">6</span><span class="p">)</span> <span class="o">/</span> <span class="mi">10</span><span class="o">**</span><span class="mi">6</span>
|
||
|
||
<span class="c1"># check what was the last entry in the log</span>
|
||
<span class="n">log</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">expanduser</span><span class="p">(</span><span class="n">logfile</span><span class="p">))</span>
|
||
<span class="n">lines</span> <span class="o">=</span> <span class="n">log</span><span class="o">.</span><span class="n">readlines</span><span class="p">()</span>
|
||
<span class="k">if</span> <span class="s1">&#39;No updates to propagate&#39;</span> <span class="ow">in</span> <span class="n">lines</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]:</span>
|
||
<span class="n">display_message</span> <span class="o">=</span> <span class="bp">False</span>
|
||
<span class="k">else</span><span class="p">:</span>
|
||
<span class="n">output</span> <span class="o">=</span> <span class="p">[</span><span class="n">l</span> <span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="n">lines</span> <span class="k">if</span> <span class="s2">&quot;Synchronization&quot;</span> <span class="ow">in</span> <span class="n">l</span><span class="p">]</span>
|
||
|
||
<span class="n">message</span> <span class="o">=</span> <span class="n">output</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
|
||
<span class="n">message</span> <span class="o">+=</span> <span class="s2">&quot; It took {0}s.&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">delta</span><span class="p">)</span>
|
||
|
||
<span class="k">if</span> <span class="n">display_message</span><span class="p">:</span>
|
||
<span class="n">os</span><span class="o">.</span><span class="n">system</span><span class="p">(</span><span class="s1">&#39;notify-send -i {2} &quot;{0}&quot; &quot;{1}&quot;&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">program_name</span><span class="p">,</span> <span class="n">message</span><span class="p">,</span>
|
||
<span class="s1">&#39;error&#39;</span> <span class="k">if</span> <span class="n">error</span> <span class="k">else</span> <span class="s1">&#39;info&#39;</span><span class="p">))</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">&quot;__main__&quot;</span><span class="p">:</span>
|
||
<span class="n">sync</span><span class="p">()</span>
|
||
</pre></div>
|
||
<p>This is probably perfectible, but that does the job.</p>
|
||
<p>Last step is to tell you machine to run that frequently. That's what <cite>crontab</cite>
|
||
is made for, so let's <cite>crontab -e</cite>:</p>
|
||
<pre class="literal-block">
|
||
$ * */3 * * * . ~/.Xdbus; /usr/bin/python /home/alexis/dev/python/unison-syncer/sync.py
|
||
</pre>
|
||
<p>The <cite>~/.Xdbus</cite> allows cron to communicate with your X11 session. Here is its
|
||
content.</p>
|
||
<div class="highlight"><pre><span></span><span class="ch">#!/bin/bash</span>
|
||
|
||
<span class="c1"># Get the pid of nautilus</span>
|
||
<span class="nv">nautilus_pid</span><span class="o">=</span><span class="k">$(</span>pgrep -u <span class="nv">$LOGNAME</span> -n nautilus<span class="k">)</span>
|
||
|
||
<span class="c1"># If nautilus isn&#39;t running, just exit silently</span>
|
||
<span class="k">if</span> <span class="o">[</span> -z <span class="s2">&quot;</span><span class="nv">$nautilus_pid</span><span class="s2">&quot;</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
|
||
<span class="nb">exit</span> <span class="m">0</span>
|
||
<span class="k">fi</span>
|
||
|
||
<span class="c1"># Grab the DBUS_SESSION_BUS_ADDRESS variable from nautilus&#39;s environment</span>
|
||
<span class="nb">eval</span> <span class="k">$(</span>tr <span class="s1">&#39;\0&#39;</span> <span class="s1">&#39;\n&#39;</span> &lt; /proc/<span class="nv">$nautilus_pid</span>/environ <span class="p">|</span> grep <span class="s1">&#39;^DBUS_SESSION_BUS_ADDRESS=&#39;</span><span class="k">)</span>
|
||
|
||
<span class="c1"># Check that we actually found it</span>
|
||
<span class="k">if</span> <span class="o">[</span> -z <span class="s2">&quot;</span><span class="nv">$DBUS_SESSION_BUS_ADDRESS</span><span class="s2">&quot;</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
|
||
<span class="nb">echo</span> <span class="s2">&quot;Failed to find bus address&quot;</span> &gt;<span class="p">&amp;</span><span class="m">2</span>
|
||
<span class="nb">exit</span> <span class="m">1</span>
|
||
<span class="k">fi</span>
|
||
|
||
<span class="c1"># export it so that child processes will inherit it</span>
|
||
<span class="nb">export</span> DBUS_SESSION_BUS_ADDRESS
|
||
</pre></div>
|
||
<p>And it comes from <a class="reference external" href="http://ubuntuforums.org/showthread.php?p=10148738#post10148738">here</a>.</p>
|
||
<p>A sync takes about 20s + the upload time on my machine, which stay acceptable for
|
||
all of my developments.</p>
|
||
</div>
|
||
</content><category term="freebsd"></category><category term="unison"></category></entry><entry><title>Wrap up of the distutils2 paris' sprint</title><link href="https://blog.notmyidea.org/wrap-up-of-the-distutils2-paris-sprint.html" rel="alternate"></link><published>2011-02-08T00:00:00+01:00</published><updated>2011-02-08T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2011-02-08:/wrap-up-of-the-distutils2-paris-sprint.html</id><summary type="html"><p>Finally, thanks to a bunch of people that helped me to pay my train and bus
|
||
tickets, I've made it to paris for the distutils2 sprint.</p>
|
||
<p>They have been a bit more than 10 people to come during the sprint, and it was
|
||
very productive. Here's a taste of what …</p></summary><content type="html"><p>Finally, thanks to a bunch of people that helped me to pay my train and bus
|
||
tickets, I've made it to paris for the distutils2 sprint.</p>
|
||
<p>They have been a bit more than 10 people to come during the sprint, and it was
|
||
very productive. Here's a taste of what we've been working on:</p>
|
||
<ul class="simple">
|
||
<li>the <cite>datafiles</cite>, a way to specify and to handle the installation of files which
|
||
are not python-related (pictures, manpages and so on).</li>
|
||
<li><cite>mkgcfg</cite>, a tool to help you to create a setup.cfg in minutes (and with funny
|
||
examples)</li>
|
||
<li>converters from setup.py scripts. We do now have a piece of code which
|
||
reads your current <cite>setup.py</cite> file and fill in some fields in the <cite>setup.cfg</cite>
|
||
for you.</li>
|
||
<li>a compatibility layer for distutils1, so it can read the <cite>setup.cfg</cite> you will
|
||
wrote for distutils2 :-)</li>
|
||
<li>the uninstaller, so it's now possible to uninstall what have been installed
|
||
by distutils2 (see PEP 376)</li>
|
||
<li>the installer, and the setuptools compatibility layer, which will allow you
|
||
to rely on setuptools' based distributions (and there are plenty of them!)</li>
|
||
<li>The compilers, so they are more flexible than they were. Since that's an
|
||
obscure part of the code for distutils2 commiters (it comes directly from the
|
||
distutils1 ages), having some guys who understood the problematics here was
|
||
a must.</li>
|
||
</ul>
|
||
<p>Some people have also tried to port their packaging from distutils1 to
|
||
distutils2. They have spotted a number of bugs and made some improvements
|
||
to the code, to make it more friendly to use.</p>
|
||
<p>I'm really pleased to see how newcomers went trough the code, and started
|
||
hacking so fast. I must say it wasn't the case when we started to work on
|
||
distutils1 so that's a very good point: people now can hack the code quicker
|
||
than they could before.</p>
|
||
<p>Some of the features here are not <em>completely</em> finished yet, but are on the
|
||
tubes, and will be ready for a release (hopefully) at the end of the week.</p>
|
||
<p>Big thanks to logilab for hosting (and sponsoring my train ticket) and
|
||
providing us food, and to bearstech for providing some money for breakfast and
|
||
bears^Wbeers.</p>
|
||
<p>Again, a big thanks to all the people who gave me money to pay the transport,
|
||
I really wasn't expecting such thing to happen :-)</p>
|
||
</content></entry><entry><title>PyPI on CouchDB</title><link href="https://blog.notmyidea.org/pypi-on-couchdb.html" rel="alternate"></link><published>2011-01-20T00:00:00+01:00</published><updated>2011-01-20T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2011-01-20:/pypi-on-couchdb.html</id><summary type="html"><p>By now, there are two ways to retrieve data from PyPI (the Python Package
|
||
Index). You can both rely on xml/rpc or on the &quot;simple&quot; API. The simple
|
||
API is not so simple to use as the name suggest, and have several existing
|
||
drawbacks.</p>
|
||
<p>Basically, if you want to …</p></summary><content type="html"><p>By now, there are two ways to retrieve data from PyPI (the Python Package
|
||
Index). You can both rely on xml/rpc or on the &quot;simple&quot; API. The simple
|
||
API is not so simple to use as the name suggest, and have several existing
|
||
drawbacks.</p>
|
||
<p>Basically, if you want to use informations coming from the simple API, you will
|
||
have to parse web pages manually, to extract informations using some black
|
||
vodoo magic. Badly, magic have a price, and it's sometimes impossible to get
|
||
exactly the informations you want to get from this index. That's the technique
|
||
currently being used by distutils2, setuptools and pip.</p>
|
||
<p>On the other side, while XML/RPC is working fine, it's requiring extra work
|
||
to the python servers each time you request something, which can lead to
|
||
some outages from time to time. Also, it's important to point out that, even if
|
||
PyPI have a mirroring infrastructure, it's only for the so-called <em>simple</em> API,
|
||
and not for the XML/RPC.</p>
|
||
<div class="section" id="couchdb">
|
||
<h2>CouchDB</h2>
|
||
<p>Here comes CouchDB. CouchDB is a document oriented database, that
|
||
knows how to speak REST and JSON. It's easy to use, and provides out of the box
|
||
a replication mechanism.</p>
|
||
</div>
|
||
<div class="section" id="so-what">
|
||
<h2>So, what ?</h2>
|
||
<p>Hmm, I'm sure you got it. I've wrote a piece of software to link informations from
|
||
PyPI to a CouchDB instance. Then you can replicate all the PyPI index with only
|
||
one HTTP request on the CouchDB server. You can also access the informations
|
||
from the index directly using a REST API, speaking json. Handy.</p>
|
||
<p>So PyPIonCouch is using the PyPI XML/RPC API to get data from PyPI, and
|
||
generate records in the CouchDB instance.</p>
|
||
<p>The final goal is to avoid to rely on this &quot;simple&quot; API, and rely on a REST
|
||
insterface instead. I have set up a couchdb server on my server, which is
|
||
available at <a class="reference external" href="http://couchdb.notmyidea.org/_utils/database.html?pypi">http://couchdb.notmyidea.org/_utils/database.html?pypi</a>.</p>
|
||
<p>There is not a lot to
|
||
see there for now, but I've done the first import from PyPI yesterday and all
|
||
went fine: it's possible to access the metadata of all PyPI projects via a REST
|
||
interface. Next step is to write a client for this REST interface in
|
||
distutils2.</p>
|
||
</div>
|
||
<div class="section" id="example">
|
||
<h2>Example</h2>
|
||
<p>For now, you can use pypioncouch via the command line, or via the python API.</p>
|
||
<div class="section" id="using-the-command-line">
|
||
<h3>Using the command line</h3>
|
||
<p>You can do something like that for a full import. This <strong>will</strong> take long,
|
||
because it's fetching all the projects at pypi and importing their metadata:</p>
|
||
<pre class="literal-block">
|
||
$ pypioncouch --fullimport http://your.couchdb.instance/
|
||
</pre>
|
||
<p>If you already have the data on your couchdb instance, you can just update it
|
||
with the last informations from pypi. <strong>However, I recommend to just replicate
|
||
the principal node, hosted at http://couchdb.notmyidea.org/pypi/</strong>, to avoid
|
||
the duplication of nodes:</p>
|
||
<pre class="literal-block">
|
||
$ pypioncouch --update http://your.couchdb.instance/
|
||
</pre>
|
||
<p>The principal node is updated once a day by now, I'll try to see if it's
|
||
enough, and ajust with the time.</p>
|
||
</div>
|
||
<div class="section" id="using-the-python-api">
|
||
<h3>Using the python API</h3>
|
||
<p>You can also use the python API to interact with pypioncouch:</p>
|
||
<pre class="literal-block">
|
||
&gt;&gt;&gt; from pypioncouch import XmlRpcImporter, import_all, update
|
||
&gt;&gt;&gt; full_import()
|
||
&gt;&gt;&gt; update()
|
||
</pre>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="what-s-next">
|
||
<h2>What's next ?</h2>
|
||
<p>I want to make a couchapp, in order to navigate PyPI easily. Here are some of
|
||
the features I want to propose:</p>
|
||
<ul class="simple">
|
||
<li>List all the available projects</li>
|
||
<li>List all the projects, filtered by specifiers</li>
|
||
<li>List all the projects by author/maintainer</li>
|
||
<li>List all the projects by keywords</li>
|
||
<li>Page for each project.</li>
|
||
<li>Provide a PyPI &quot;Simple&quot; API equivalent, even if I want to replace it, I do
|
||
think it will be really easy to setup mirrors that way, with the out of the
|
||
box couchdb replication</li>
|
||
</ul>
|
||
<p>I also still need to polish the import mechanism, so I can directly store in
|
||
couchdb:</p>
|
||
<ul class="simple">
|
||
<li>The OPML files for each project</li>
|
||
<li>The upload_time as couchdb friendly format (list of int)</li>
|
||
<li>The tags as lists (currently it's only a string separated by spaces</li>
|
||
</ul>
|
||
<p>The work I've done by now is available on
|
||
<a class="reference external" href="https://bitbucket.org/ametaireau/pypioncouch/">https://bitbucket.org/ametaireau/pypioncouch/</a>. Keep in mind that it's still
|
||
a work in progress, and everything can break at any time. However, any feedback
|
||
will be appreciated !</p>
|
||
</div>
|
||
</content></entry><entry><title>Help me to go to the distutils2 paris' sprint</title><link href="https://blog.notmyidea.org/help-me-to-go-to-the-distutils2-paris-sprint.html" rel="alternate"></link><published>2011-01-15T00:00:00+01:00</published><updated>2011-01-15T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2011-01-15:/help-me-to-go-to-the-distutils2-paris-sprint.html</id><summary type="html"><p><strong>Edit: Thanks to logilab and some amazing people, I can make it to paris for the
|
||
sprint. Many thanks to them for the support!</strong></p>
|
||
<p>There will be a distutils2 sprint from the 27th to the 30th of january, thanks
|
||
to logilab which will host the event.</p>
|
||
<p>You can find more …</p></summary><content type="html"><p><strong>Edit: Thanks to logilab and some amazing people, I can make it to paris for the
|
||
sprint. Many thanks to them for the support!</strong></p>
|
||
<p>There will be a distutils2 sprint from the 27th to the 30th of january, thanks
|
||
to logilab which will host the event.</p>
|
||
<p>You can find more informations about the sprint on the wiki page of the event
|
||
(<a class="reference external" href="http://wiki.python.org/moin/Distutils/SprintParis">http://wiki.python.org/moin/Distutils/SprintParis</a>).</p>
|
||
<p>I really want to go there but I'm unfortunately blocked in UK for money reasons.
|
||
The cheapest two ways I've found is about £80, which I can't afford.
|
||
Following some advices on #distutils, I've set up a ChipIn account for that, so
|
||
if some people want to help me making it to go there, they can give me some
|
||
money that way.</p>
|
||
<p>I'll probably work on the installer (to support old distutils and
|
||
setuptools distributions) and on the uninstaller (depending on the first
|
||
task). If I can't make it to paris, I'll hang around on IRC to give some help
|
||
while needed.</p>
|
||
<p>If you want to contribute some money to help me go there, feel free to use this
|
||
chipin page: <a class="reference external" href="http://ametaireau.chipin.com/distutils2-sprint-in-paris">http://ametaireau.chipin.com/distutils2-sprint-in-paris</a></p>
|
||
<p>Thanks for your support !</p>
|
||
</content></entry><entry><title>How to reboot your bebox using the CLI</title><link href="https://blog.notmyidea.org/how-to-reboot-your-bebox-using-the-cli.html" rel="alternate"></link><published>2010-10-21T00:00:00+02:00</published><updated>2010-10-21T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2010-10-21:/how-to-reboot-your-bebox-using-the-cli.html</id><summary type="html"><p>I've an internet connection which, for some obscure reasons, tend to be very
|
||
slow from time to time. After rebooting the box (yes, that's a hard solution),
|
||
all the things seems to go fine again.</p>
|
||
<div class="section" id="edit-using-grep">
|
||
<h2>EDIT : Using grep</h2>
|
||
<p>After a bit of reflexion, that's also really easy to do using …</p></div></summary><content type="html"><p>I've an internet connection which, for some obscure reasons, tend to be very
|
||
slow from time to time. After rebooting the box (yes, that's a hard solution),
|
||
all the things seems to go fine again.</p>
|
||
<div class="section" id="edit-using-grep">
|
||
<h2>EDIT : Using grep</h2>
|
||
<p>After a bit of reflexion, that's also really easy to do using directly the
|
||
command line tools curl, grep and tail (but really harder to read).</p>
|
||
<div class="highlight"><pre><span></span>curl -X POST -u joel:joel http://bebox.config/cgi/b/info/restart/<span class="se">\?</span>be<span class="se">\=</span><span class="m">0</span><span class="se">\&amp;</span>l0<span class="se">\=</span><span class="m">1</span><span class="se">\&amp;</span>l1<span class="se">\=</span><span class="m">0</span><span class="se">\&amp;</span>tid<span class="se">\=</span>RESTART -d <span class="s2">&quot;0=17&amp;2=`curl -u joel:joel http://bebox.config/cgi/b/info/restart/\?be\=0\&amp;l0\=1\&amp;l1\=0\&amp;tid\=RESTART | grep -o &quot;</span><span class="nv">name</span><span class="o">=</span><span class="s1">&#39;2&#39;</span> <span class="nv">value</span><span class="o">=</span><span class="err">&#39;</span><span class="o">[</span><span class="m">0</span>-9<span class="o">]</span><span class="se">\+</span><span class="s2">&quot; | grep -o &quot;</span><span class="o">[</span><span class="m">0</span>-9<span class="o">]</span><span class="se">\+</span><span class="s2">&quot; | tail -n 1`&amp;1&quot;</span>
|
||
</pre></div>
|
||
</div>
|
||
<div class="section" id="the-python-version">
|
||
<h2>The Python version</h2>
|
||
<p>Well, that's not the optimal solution, that's a bit &quot;gruik&quot;, but it works.</p>
|
||
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">urllib2</span>
|
||
<span class="kn">import</span> <span class="nn">urlparse</span>
|
||
<span class="kn">import</span> <span class="nn">re</span>
|
||
<span class="kn">import</span> <span class="nn">argparse</span>
|
||
|
||
<span class="n">REBOOT_URL</span> <span class="o">=</span> <span class="s1">&#39;/b/info/restart/?be=0&amp;l0=1&amp;l1=0&amp;tid=RESTART&#39;</span>
|
||
<span class="n">BOX_URL</span> <span class="o">=</span> <span class="s1">&#39;http://bebox.config/cgi&#39;</span>
|
||
|
||
<span class="k">def</span> <span class="nf">open_url</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="p">):</span>
|
||
<span class="n">passman</span> <span class="o">=</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">HTTPPasswordMgrWithDefaultRealm</span><span class="p">()</span>
|
||
<span class="n">passman</span><span class="o">.</span><span class="n">add_password</span><span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="n">url</span><span class="p">,</span> <span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span>
|
||
<span class="n">authhandler</span> <span class="o">=</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">HTTPBasicAuthHandler</span><span class="p">(</span><span class="n">passman</span><span class="p">)</span>
|
||
|
||
<span class="n">opener</span> <span class="o">=</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">build_opener</span><span class="p">(</span><span class="n">authhandler</span><span class="p">)</span>
|
||
|
||
<span class="n">urllib2</span><span class="o">.</span><span class="n">install_opener</span><span class="p">(</span><span class="n">opener</span><span class="p">)</span>
|
||
|
||
<span class="k">return</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">url</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
|
||
|
||
<span class="k">def</span> <span class="nf">reboot</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="p">):</span>
|
||
<span class="n">data</span> <span class="o">=</span> <span class="n">open_url</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span>
|
||
<span class="n">token</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s2">&quot;name\=</span><span class="se">\\</span><span class="s2">&#39;2</span><span class="se">\\</span><span class="s2">&#39; value=</span><span class="se">\\</span><span class="s2">&#39;([0-9]+)</span><span class="se">\\</span><span class="s2">&#39;&quot;</span><span class="p">,</span> <span class="n">data</span><span class="p">)[</span><span class="mi">1</span><span class="p">]</span>
|
||
<span class="n">urllib2</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">urllib2</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="n">url</span><span class="o">=</span><span class="n">url</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="s1">&#39;0=17&amp;2=</span><span class="si">%s</span><span class="s1">&amp;1&#39;</span> <span class="o">%</span> <span class="n">token</span><span class="p">))</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__file__</span> <span class="o">==</span> <span class="s1">&#39;__main__&#39;</span><span class="p">:</span>
|
||
<span class="n">parser</span> <span class="o">=</span> <span class="n">argparse</span><span class="o">.</span><span class="n">ArgumentParser</span><span class="p">(</span><span class="n">description</span><span class="o">=</span><span class="s2">&quot;&quot;&quot;Reboot your bebox !&quot;&quot;&quot;</span><span class="p">)</span>
|
||
|
||
<span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="n">dest</span><span class="o">=</span><span class="s1">&#39;user&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;username&#39;</span><span class="p">)</span>
|
||
<span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="n">dest</span><span class="o">=</span><span class="s1">&#39;password&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;password&#39;</span><span class="p">)</span>
|
||
<span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="n">boxurl</span><span class="o">=</span><span class="s1">&#39;boxurl&#39;</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="n">BOX_URL</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;Base box url. Default is </span><span class="si">%s</span><span class="s1">&#39;</span> <span class="o">%</span> <span class="n">BOX_URL</span><span class="p">)</span>
|
||
|
||
<span class="n">args</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">parse_args</span><span class="p">()</span>
|
||
<span class="n">url</span> <span class="o">=</span> <span class="n">urlparse</span><span class="o">.</span><span class="n">urljoin</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">boxurl</span><span class="p">,</span> <span class="n">REBOOT_URL</span><span class="p">)</span>
|
||
<span class="n">reboot</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">args</span><span class="o">.</span><span class="n">username</span><span class="p">,</span> <span class="n">args</span><span class="o">.</span><span class="n">password</span><span class="p">)</span>
|
||
</pre></div>
|
||
</div>
|
||
</content></entry><entry><title>Dynamically change your gnome desktop wallpaper</title><link href="https://blog.notmyidea.org/dynamically-change-your-gnome-desktop-wallpaper.html" rel="alternate"></link><published>2010-10-11T00:00:00+02:00</published><updated>2010-10-11T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2010-10-11:/dynamically-change-your-gnome-desktop-wallpaper.html</id><summary type="html"><p>In gnome, you can can use a XML file to have a dynamic wallpaper.
|
||
It's not so easy, and you can't just tell: use the pictures in this folder to do
|
||
so.</p>
|
||
<p>You can have a look to the git repository if you want: <a class="reference external" href="http://github.com/ametaireau/gnome-background-generator">http://github.com/ametaireau/gnome-background-generator</a></p>
|
||
<p>Some …</p></summary><content type="html"><p>In gnome, you can can use a XML file to have a dynamic wallpaper.
|
||
It's not so easy, and you can't just tell: use the pictures in this folder to do
|
||
so.</p>
|
||
<p>You can have a look to the git repository if you want: <a class="reference external" href="http://github.com/ametaireau/gnome-background-generator">http://github.com/ametaireau/gnome-background-generator</a></p>
|
||
<p>Some time ago, I've made a little python script to ease that, and you can now
|
||
use it too. It's named &quot;gnome-background-generator&quot;, and you can install it via
|
||
pip for instance.</p>
|
||
<div class="highlight"><pre><span></span>$ pip install gnome-background-generator
|
||
</pre></div>
|
||
<p>Then, you have just to use it this way:</p>
|
||
<div class="highlight"><pre><span></span>$ gnome-background-generator -p ~/Images/walls -s
|
||
/home/alexis/Images/walls/dynamic-wallpaper.xml generated
|
||
</pre></div>
|
||
<p>Here is a extract of the <cite>--help</cite>:</p>
|
||
<div class="highlight"><pre><span></span>$ gnome-background-generator --help
|
||
usage: gnome-background-generator <span class="o">[</span>-h<span class="o">]</span> <span class="o">[</span>-p PATH<span class="o">]</span> <span class="o">[</span>-o OUTPUT<span class="o">]</span>
|
||
<span class="o">[</span>-t TRANSITION_TIME<span class="o">]</span> <span class="o">[</span>-d DISPLAY_TIME<span class="o">]</span> <span class="o">[</span>-s<span class="o">]</span>
|
||
<span class="o">[</span>-b<span class="o">]</span>
|
||
|
||
A simple <span class="nb">command</span> line tool to generate an XML file to use <span class="k">for</span> gnome
|
||
wallpapers, to have dynamic walls
|
||
|
||
optional arguments:
|
||
-h, --help show this <span class="nb">help</span> message and <span class="nb">exit</span>
|
||
-p PATH, --path PATH Path to look <span class="k">for</span> the pictures. If no output is
|
||
specified, will be used too <span class="k">for</span> outputing the dynamic-
|
||
wallpaper.xml file. Default value is the current
|
||
directory <span class="o">(</span>.<span class="o">)</span>
|
||
-o OUTPUT, --output OUTPUT
|
||
Output filename. If no filename is specified, a
|
||
dynamic-wallpaper.xml file will be generated in the
|
||
path containing the pictures. You can also use <span class="s2">&quot;-&quot;</span> to
|
||
display the xml in the stdout.
|
||
-t TRANSITION_TIME, --transition-time TRANSITION_TIME
|
||
Time <span class="o">(</span>in seconds<span class="o">)</span> transitions must last <span class="o">(</span>default value
|
||
is <span class="m">2</span> seconds<span class="o">)</span>
|
||
-d DISPLAY_TIME, --display-time DISPLAY_TIME
|
||
Time <span class="o">(</span>in seconds<span class="o">)</span> a picture must be displayed. Default
|
||
value is <span class="m">900</span> <span class="o">(</span>15mn<span class="o">)</span>
|
||
-s, --set-background <span class="s1">&#39;&#39;</span><span class="err">&#39;</span>try to <span class="nb">set</span> the background using gnome-appearance-
|
||
properties
|
||
-b, --debug
|
||
</pre></div>
|
||
</content></entry><entry><title>How to install NGINX + PHP 5.3 on FreeBSD.</title><link href="https://blog.notmyidea.org/how-to-install-nginx-php-53-on-freebsd.html" rel="alternate"></link><published>2010-10-10T00:00:00+02:00</published><updated>2010-10-10T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2010-10-10:/how-to-install-nginx-php-53-on-freebsd.html</id><summary type="html"><p>I've not managed so far to get completely rid of php, so here's a simple
|
||
reminder about how to install php on NGINX, for FreeBSD. Nothing hard, but
|
||
that's worse to have the piece of configuration somewhere !</p>
|
||
<pre class="literal-block">
|
||
# update the ports
|
||
$ portsnap fetch update
|
||
|
||
# install php5 port
|
||
$ make config-recursive -C /usr …</pre></summary><content type="html"><p>I've not managed so far to get completely rid of php, so here's a simple
|
||
reminder about how to install php on NGINX, for FreeBSD. Nothing hard, but
|
||
that's worse to have the piece of configuration somewhere !</p>
|
||
<pre class="literal-block">
|
||
# update the ports
|
||
$ portsnap fetch update
|
||
|
||
# install php5 port
|
||
$ make config-recursive -C /usr/ports/lang/php5-extensions
|
||
$ make package-recursive -C /usr/ports/lang/php5-extensions
|
||
|
||
# install nginx
|
||
$ make config-recursive -C /usr/ports/www/nginx-devel
|
||
$ make package-recursive -C /usr/ports/www/nginx-devel
|
||
</pre>
|
||
<p>Now we have all the dependencies installed, we need to configure a bit the
|
||
server.</p>
|
||
<p>That's a simple thing in fact, but it could be good to have something that will
|
||
work without effort over time.</p>
|
||
<p>Here's a sample of my configuration:</p>
|
||
<pre class="literal-block">
|
||
server {
|
||
server_name ndd;
|
||
set $path /path/to/your/files;
|
||
root $path;
|
||
|
||
location / {
|
||
index index.php;
|
||
}
|
||
|
||
location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico|xml)$ {
|
||
access_log off;
|
||
expires 30d;
|
||
}
|
||
|
||
location ~ .php$ {
|
||
fastcgi_param SCRIPT_FILENAME $path$fastcgi_script_name;
|
||
fastcgi_pass backend;
|
||
include fastcgi_params;
|
||
}
|
||
}
|
||
|
||
upstream backend {
|
||
server 127.0.0.1:9000;
|
||
}
|
||
</pre>
|
||
<p>And that's it !</p>
|
||
</content></entry><entry><title>Pelican, a simple static blog generator in python</title><link href="https://blog.notmyidea.org/pelican-a-simple-static-blog-generator-in-python.html" rel="alternate"></link><published>2010-10-06T00:00:00+02:00</published><updated>2010-10-06T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2010-10-06:/pelican-a-simple-static-blog-generator-in-python.html</id><summary type="html"><p>Those days, I've wrote a little python application to fit my blogging needs.
|
||
I'm an occasional blogger, a vim lover, I like restructured text and DVCSes, so
|
||
I've made a little tool that makes good use of all that.</p>
|
||
<p><a class="reference external" href="http://docs.getpelican.com">Pelican</a> (for calepin) is just a simple tool to generate your …</p></summary><content type="html"><p>Those days, I've wrote a little python application to fit my blogging needs.
|
||
I'm an occasional blogger, a vim lover, I like restructured text and DVCSes, so
|
||
I've made a little tool that makes good use of all that.</p>
|
||
<p><a class="reference external" href="http://docs.getpelican.com">Pelican</a> (for calepin) is just a simple tool to generate your blog as static
|
||
files, letting you using your editor of choice (vim!). It's easy to extend,
|
||
and has a template support (via jinja2).</p>
|
||
<p>I've made it to fit <em>my</em> needs. I hope it will fit yours, but maybe it wont, and
|
||
it have not be designed to feet everyone's needs.</p>
|
||
<p>Need an example ? You're looking at it ! This weblog is using pelican to be
|
||
generated, also for the atom feeds.</p>
|
||
<p>I've released it under AGPL, since I want all the modifications to be profitable
|
||
to all the users.</p>
|
||
<p>You can find a repository to fork at <a class="reference external" href="https://github.com/getpelican/pelican/">https://github.com/getpelican/pelican/</a>.
|
||
feel free to hack it !</p>
|
||
<p>If you just want to get started, use your installer of choice (pip, easy_install, …)
|
||
And then have a look to the help (<cite>pelican --help</cite>)</p>
|
||
<div class="highlight"><pre><span></span>$ pip install pelican
|
||
</pre></div>
|
||
<div class="section" id="usage">
|
||
<h2>Usage</h2>
|
||
<p>Here's a sample usage of pelican</p>
|
||
<div class="highlight"><pre><span></span>$ pelican .
|
||
writing /home/alexis/projets/notmyidea.org/output/index.html
|
||
writing /home/alexis/projets/notmyidea.org/output/tags.html
|
||
writing /home/alexis/projets/notmyidea.org/output/categories.html
|
||
writing /home/alexis/projets/notmyidea.org/output/archives.html
|
||
writing /home/alexis/projets/notmyidea.org/output/category/python.html
|
||
writing
|
||
/home/alexis/projets/notmyidea.org/output/pelican-a-simple-static-blog-generator-in-python.html
|
||
Done !
|
||
</pre></div>
|
||
<p>You also can use the <cite>--help</cite> option for the command line to get more
|
||
informations</p>
|
||
<div class="highlight"><pre><span></span><span class="nv">$pelican</span> --help
|
||
usage: pelican <span class="o">[</span>-h<span class="o">]</span> <span class="o">[</span>-t TEMPLATES<span class="o">]</span> <span class="o">[</span>-o OUTPUT<span class="o">]</span> <span class="o">[</span>-m MARKUP<span class="o">]</span> <span class="o">[</span>-s SETTINGS<span class="o">]</span> <span class="o">[</span>-b<span class="o">]</span>
|
||
path
|
||
|
||
A tool to generate a static blog, with restructured text input files.
|
||
|
||
positional arguments:
|
||
path Path where to find the content files <span class="o">(</span>default is
|
||
<span class="s2">&quot;content&quot;</span><span class="o">)</span>.
|
||
|
||
optional arguments:
|
||
-h, --help show this <span class="nb">help</span> message and <span class="nb">exit</span>
|
||
-t TEMPLATES, --templates-path TEMPLATES
|
||
Path where to find the templates. If not specified,
|
||
will uses the ones included with pelican.
|
||
-o OUTPUT, --output OUTPUT
|
||
Where to output the generated files. If not specified,
|
||
a directory will be created, named <span class="s2">&quot;output&quot;</span> in the
|
||
current path.
|
||
-m MARKUP, --markup MARKUP
|
||
the markup language to use. Currently only
|
||
ReSTreucturedtext is available.
|
||
-s SETTINGS, --settings SETTINGS
|
||
the settings of the application. Default to None.
|
||
-b, --debug
|
||
</pre></div>
|
||
<p>Enjoy :)</p>
|
||
</div>
|
||
</content></entry><entry><title>An amazing summer of code working on distutils2</title><link href="https://blog.notmyidea.org/an-amazing-summer-of-code-working-on-distutils2.html" rel="alternate"></link><published>2010-08-16T00:00:00+02:00</published><updated>2010-08-16T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2010-08-16:/an-amazing-summer-of-code-working-on-distutils2.html</id><summary type="html"><p>The <a class="reference external" href="http://code.google.com/soc/">Google Summer of Code</a> I've
|
||
spent working on <a class="reference external" href="http://hg.python.org/distutils2/">distutils2</a>
|
||
is over. It was a really amazing experience, for many reasons.</p>
|
||
<p>First of all, we had a very good team, we were 5 students working
|
||
on distutils2: <a class="reference external" href="http://zubin71.wordpress.com">Zubin</a>,
|
||
<a class="reference external" href="http://wokslog.wordpress.com/">Éric</a>,
|
||
<a class="reference external" href="http://gsoc.djolonga.com/">Josip</a>,
|
||
<a class="reference external" href="http://konryd.blogspot.com/">Konrad</a> and me. In addition,
|
||
<a class="reference external" href="http://mouadino.blogspot.com/">Mouad</a> have worked on the …</p></summary><content type="html"><p>The <a class="reference external" href="http://code.google.com/soc/">Google Summer of Code</a> I've
|
||
spent working on <a class="reference external" href="http://hg.python.org/distutils2/">distutils2</a>
|
||
is over. It was a really amazing experience, for many reasons.</p>
|
||
<p>First of all, we had a very good team, we were 5 students working
|
||
on distutils2: <a class="reference external" href="http://zubin71.wordpress.com">Zubin</a>,
|
||
<a class="reference external" href="http://wokslog.wordpress.com/">Éric</a>,
|
||
<a class="reference external" href="http://gsoc.djolonga.com/">Josip</a>,
|
||
<a class="reference external" href="http://konryd.blogspot.com/">Konrad</a> and me. In addition,
|
||
<a class="reference external" href="http://mouadino.blogspot.com/">Mouad</a> have worked on the PyPI
|
||
testing infrastructure. You could find what each person have done
|
||
on
|
||
<a class="reference external" href="http://bitbucket.org/tarek/distutils2/wiki/GSoC_2010_teams">the wiki page of distutils2</a>.</p>
|
||
<p>We were in contact with each others really often, helping us when
|
||
possible (in #distutils), and were continuously aware of the state
|
||
of the work of each participant. This, in my opinion, have bring us
|
||
in a good shape.</p>
|
||
<p>Then, I've learned a lot. Python packaging was completely new to me
|
||
at the time of the GSoC start, and I was pretty unfamiliar with
|
||
python good practices too, as I've been introducing myself to
|
||
python in the late 2009.</p>
|
||
<p>I've recently looked at some python code I wrote just three months
|
||
ago, and I was amazed to think about many improvements to made on
|
||
it. I guess this is a good indicator of the path I've traveled
|
||
since I wrote it.</p>
|
||
<p>This summer was awesome because I've learned about python good
|
||
practices, now having some strong
|
||
<a class="reference external" href="http://mercurial.selenic.com/">mercurial</a> knowledge, and I've
|
||
seen a little how the python community works.</p>
|
||
<p>Then, I would like to say a big thanks to all the mentors that have
|
||
hanged around while needed, on IRC or via mail, and especially my
|
||
mentor for this summer, <a class="reference external" href="http://tarek.ziade.org">Tarek Ziadé</a>.</p>
|
||
<p>Thanks a lot for your motivation, your leadership and your
|
||
cheerfulness, even with a new-born and a new work!</p>
|
||
<div class="section" id="why">
|
||
<h2>Why ?</h2>
|
||
<p>I wanted to work on python packaging because, as the time pass, we
|
||
were having a sort of complex tools in this field. Each one wanted
|
||
to add features to distutils, but not in a standard way.</p>
|
||
<p>Now, we have PEPs that describes some format we agreed on (see PEP
|
||
345), and we wanted to have a tool on which users can base their
|
||
code on, that's <a class="reference external" href="http://hg.python.org/distutils2/">distutils2</a>.</p>
|
||
</div>
|
||
<div class="section" id="my-job">
|
||
<h2>My job</h2>
|
||
<p>I had to provide a way to crawl the PyPI indexes in a simple way,
|
||
and do some installation / uninstallation scripts.</p>
|
||
<p>All the work done is available in
|
||
<a class="reference external" href="http://bitbucket.org/ametaireau/distutils2/">my bitbucket repository</a>.</p>
|
||
<div class="section" id="crawling-the-pypi-indexes">
|
||
<h3>Crawling the PyPI indexes</h3>
|
||
<p>There are two ways of requesting informations from the indexes:
|
||
using the &quot;simple&quot; index, that is a kind of REST index, and using
|
||
XML-RPC.</p>
|
||
<p>I've done the two implementations, and a high level API to query
|
||
those twos. Basically, this supports the mirroring infrastructure
|
||
defined in PEP 381. So far, the work I've done is gonna be used in
|
||
pip (they've basically copy/paste the code, but this will change as
|
||
soon as we get something completely stable for distutils2), and
|
||
that's a good news, as it was the main reason for what I've done
|
||
that.</p>
|
||
<p>I've tried to have an unified API for the clients, to switch from
|
||
one to another implementation easily. I'm already thinking of
|
||
adding others crawlers to this stuff, and it was made to be
|
||
extensible.</p>
|
||
<p>If you want to get more informations about the crawlers/PyPI
|
||
clients, please refer to the distutils2 documentation, especially
|
||
<a class="reference external" href="http://distutils2.notmyidea.org/library/distutils2.index.html">the pages about indexes</a>.</p>
|
||
<p>You can find the changes I made about this in the
|
||
<a class="reference external" href="http://hg.python.org/distutils2/">distutils2</a> source code .</p>
|
||
</div>
|
||
<div class="section" id="installation-uninstallation-scripts">
|
||
<h3>Installation / Uninstallation scripts</h3>
|
||
<p>Next step was to think about an installation script, and an
|
||
uninstaller. I've not done the uninstaller part, and it's a smart
|
||
part, as it's basically removing some files from the system, so
|
||
I'll probably do it in a near future.</p>
|
||
<p><a class="reference external" href="http://hg.python.org/distutils2/">distutils2</a> provides a way to
|
||
install distributions, and to handle dependencies between releases.
|
||
For now, this support is only about the last version of the
|
||
METADATA (1.2) (See, the PEP 345), but I'm working on a
|
||
compatibility layer for the old metadata, and for the informations
|
||
provided via PIP requires.txt, for instance.</p>
|
||
</div>
|
||
<div class="section" id="extra-work">
|
||
<h3>Extra work</h3>
|
||
<p>Also, I've done some extra work. this includes:</p>
|
||
<ul class="simple">
|
||
<li>working on the PEP 345, and having some discussion about it
|
||
(about the names of some fields).</li>
|
||
<li>writing a PyPI server mock, useful for tests. you can find more
|
||
information about it on the
|
||
<a class="reference external" href="http://distutils.notmyidea.org">documentation</a>.</li>
|
||
</ul>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="futures-plans">
|
||
<h2>Futures plans</h2>
|
||
<p>As I said, I've enjoyed working on distutils2, and the people I've
|
||
met here are really pleasant to work with. So I <em>want</em> to continue
|
||
contributing on python, and especially on python packaging, because
|
||
there is still a lot of things to do in this scope, to get
|
||
something really usable.</p>
|
||
<p>I'm not plainly satisfied by the work I've done, so I'll probably
|
||
tweak it a bit: the installer part is not yet completely finished,
|
||
and I want to add support for a real
|
||
<a class="reference external" href="http://en.wikipedia.org/wiki/Representational_State_Transfer">REST</a>
|
||
index in the future.</p>
|
||
<p>We'll talk again of this in the next months, probably, but we
|
||
definitely need a real
|
||
<a class="reference external" href="http://en.wikipedia.org/wiki/Representational_State_Transfer">REST</a>
|
||
API for <a class="reference external" href="http://pypi.python.org">PyPI</a>, as the &quot;simple&quot; index
|
||
<em>is</em> an ugly hack, in my opinion. I'll work on a serious
|
||
proposition about this, maybe involving
|
||
<a class="reference external" href="http://couchdb.org">CouchDB</a>, as it seems to be a good option
|
||
for what we want here.</p>
|
||
</div>
|
||
<div class="section" id="issues">
|
||
<h2>Issues</h2>
|
||
<p>I've encountered some issues during this summer. The main one is
|
||
that's hard to work remotely, especially being in the same room
|
||
that we live, with others. I like to just think about a project
|
||
with other people, a paper and a pencil, no computers. This have
|
||
been not so possible at the start of the project, as I needed to
|
||
read a lot of code to understand the codebase, and then to
|
||
read/write emails.</p>
|
||
<p>I've finally managed to work in an office, so good point for
|
||
home/office separation.</p>
|
||
<p>I'd not planned there will be so a high number of emails to read,
|
||
in order to follow what's up in the python world, and be a part of
|
||
the community seems to takes some times to read/write emails,
|
||
especially for those (like me) that arent so confortable with
|
||
english (but this had brought me some english fu !).</p>
|
||
</div>
|
||
<div class="section" id="thanks">
|
||
<h2>Thanks !</h2>
|
||
<p>A big thanks to <a class="reference external" href="http://www.graine-libre.fr/">Graine Libre</a> and
|
||
<a class="reference external" href="http://www.makina-corpus.com/">Makina Corpus</a>, which has offered
|
||
me to come into their offices from time to time, to share they
|
||
cheerfulness ! Many thanks too to the Google Summer of Code program
|
||
for setting up such an initiative. If you're a student, if you're
|
||
interested about FOSS, dont hesitate any second, it's a really good
|
||
opportunity to work on interesting projects!</p>
|
||
</div>
|
||
</content></entry><entry><title>Introducing the distutils2 index crawlers</title><link href="https://blog.notmyidea.org/introducing-the-distutils2-index-crawlers.html" rel="alternate"></link><published>2010-07-06T00:00:00+02:00</published><updated>2010-07-06T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2010-07-06:/introducing-the-distutils2-index-crawlers.html</id><summary type="html"><p>I'm working for about a month for distutils2, even if I was being a
|
||
bit busy (as I had some class courses and exams to work on)</p>
|
||
<p>I'll try do sum-up my general feelings here, and the work I've made
|
||
so far. You can also find, if you're interested, my …</p></summary><content type="html"><p>I'm working for about a month for distutils2, even if I was being a
|
||
bit busy (as I had some class courses and exams to work on)</p>
|
||
<p>I'll try do sum-up my general feelings here, and the work I've made
|
||
so far. You can also find, if you're interested, my weekly
|
||
summaries in
|
||
<a class="reference external" href="http://wiki.notmyidea.org/distutils2_schedule">a dedicated wiki page</a>.</p>
|
||
<div class="section" id="general-feelings">
|
||
<h2>General feelings</h2>
|
||
<p>First, and it's a really important point, the GSoC is going very
|
||
well, for me as for other students, at least from my perspective.
|
||
It's a pleasure to work with such enthusiast people, as this make
|
||
the global atmosphere very pleasant to live.</p>
|
||
<p>First of all, I've spent time to read the existing codebase, and to
|
||
understand what we're going to do, and what's the rationale to do
|
||
so.</p>
|
||
<p>It's really clear for me now: what we're building is the
|
||
foundations of a packaging infrastructure in python. The fact is
|
||
that many projects co-exists, and comes all with their good
|
||
concepts. Distutils2 tries to take the interesting parts of all,
|
||
and to provide it in the python standard libs, respecting the
|
||
recently written PEP about packaging.</p>
|
||
<p>With distutils2, it will be simpler to make &quot;things&quot; compatible. So
|
||
if you think about a new way to deal with distributions and
|
||
packaging in python, you can use the Distutils2 APIs to do so.</p>
|
||
</div>
|
||
<div class="section" id="tasks">
|
||
<h2>Tasks</h2>
|
||
<p>My main task while working on distutils2 is to provide an
|
||
installation and an un-installation command, as described in PEP
|
||
376. For this, I first need to get informations about the existing
|
||
distributions (what's their version, name, metadata, dependencies,
|
||
etc.)</p>
|
||
<p>The main index, you probably know and use, is PyPI. You can access
|
||
it at <a class="reference external" href="http://pypi.python.org">http://pypi.python.org</a>.</p>
|
||
</div>
|
||
<div class="section" id="pypi-index-crawling">
|
||
<h2>PyPI index crawling</h2>
|
||
<p>There is two ways to get these informations from PyPI: using the
|
||
simple API, or via xml-rpc calls.</p>
|
||
<p>A goal was to use the version specifiers defined
|
||
in`PEP 345 &lt;<a class="reference external" href="http://www.python.org/dev/peps/pep-0345/">http://www.python.org/dev/peps/pep-0345/</a>&gt;`_ and to
|
||
provides a way to sort the grabbed distributions depending our
|
||
needs, to pick the version we want/need.</p>
|
||
<div class="section" id="using-the-simple-api">
|
||
<h3>Using the simple API</h3>
|
||
<p>The simple API is composed of HTML pages you can access at
|
||
<a class="reference external" href="http://pypi.python.org/simple/">http://pypi.python.org/simple/</a>.</p>
|
||
<p>Distribute and Setuptools already provides a crawler for that, but
|
||
it deals with their internal mechanisms, and I found that the code
|
||
was not so clear as I want, that's why I've preferred to pick up
|
||
the good ideas, and some implementation details, plus re-thinking
|
||
the global architecture.</p>
|
||
<p>The rules are simple: each project have a dedicated page, which
|
||
allows us to get informations about:</p>
|
||
<ul class="simple">
|
||
<li>the distribution download locations (for some versions)</li>
|
||
<li>homepage links</li>
|
||
<li>some other useful informations, as the bugtracker address, for
|
||
instance.</li>
|
||
</ul>
|
||
<p>If you want to find all the distributions of the &quot;EggsAndSpam&quot;
|
||
project, you could do the following (do not take so attention to
|
||
the names here, as the API will probably change a bit):</p>
|
||
<div class="highlight"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">index</span> <span class="o">=</span> <span class="n">SimpleIndex</span><span class="p">()</span>
|
||
<span class="o">&gt;&gt;&gt;</span> <span class="n">index</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s2">&quot;EggsAndSpam&quot;</span><span class="p">)</span>
|
||
<span class="p">[</span><span class="n">EggsAndSpam</span> <span class="mf">1.1</span><span class="p">,</span> <span class="n">EggsAndSpam</span> <span class="mf">1.2</span><span class="p">,</span> <span class="n">EggsAndSpam</span> <span class="mf">1.3</span><span class="p">]</span>
|
||
</pre></div>
|
||
<p>We also could use version specifiers:</p>
|
||
<div class="highlight"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">index</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s2">&quot;EggsAndSpam (&lt; =1.2)&quot;</span><span class="p">)</span>
|
||
<span class="p">[</span><span class="n">EggsAndSpam</span> <span class="mf">1.1</span><span class="p">,</span> <span class="n">EggsAndSpam</span> <span class="mf">1.2</span><span class="p">]</span>
|
||
</pre></div>
|
||
<p>Internally, what's done here is the following:</p>
|
||
<ul class="simple">
|
||
<li>it process the
|
||
<a class="reference external" href="http://pypi.python.org/simple/FooBar/">http://pypi.python.org/simple/FooBar/</a>
|
||
page, searching for download URLs.</li>
|
||
<li>for each found distribution download URL, it creates an object,
|
||
containing informations about the project name, the version and the
|
||
URL where the archive remains.</li>
|
||
<li>it sort the found distributions, using version numbers. The
|
||
default behavior here is to prefer source distributions (over
|
||
binary ones), and to rely on the last &quot;final&quot; distribution (rather
|
||
than beta, alpha etc. ones)</li>
|
||
</ul>
|
||
<p>So, nothing hard or difficult here.</p>
|
||
<p>We provides a bunch of other features, like relying on the new PyPI
|
||
mirroring infrastructure or filter the found distributions by some
|
||
criterias. If you're curious, please browse the
|
||
<a class="reference external" href="http://distutils2.notmyidea.org/">distutils2 documentation</a>.</p>
|
||
</div>
|
||
<div class="section" id="using-xml-rpc">
|
||
<h3>Using xml-rpc</h3>
|
||
<p>We also can make some xmlrpc calls to retreive informations from
|
||
PyPI. It's a really more reliable way to get informations from from
|
||
the index (as it's just the index that provides the informations),
|
||
but cost processes on the PyPI distant server.</p>
|
||
<p>For now, this way of querying the xmlrpc client is not available on
|
||
Distutils2, as I'm working on it. The main pieces are already
|
||
present (I'll reuse some work I've made from the SimpleIndex
|
||
querying, and
|
||
<a class="reference external" href="http://github.com/ametaireau/pypiclient">some code already set up</a>),
|
||
what I need to do is to provide a xml-rpc PyPI mock server, and
|
||
that's on what I'm actually working on.</p>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="processes">
|
||
<h2>Processes</h2>
|
||
<p>For now, I'm trying to follow the &quot;documentation, then test, then
|
||
code&quot; path, and that seems to be really needed while working with a
|
||
community. Code is hard to read/understand, compared to
|
||
documentation, and it's easier to change.</p>
|
||
<p>While writing the simple index crawling work, I must have done this
|
||
to avoid some changes on the API, and some loss of time.</p>
|
||
<p>Also, I've set up
|
||
<a class="reference external" href="http://wiki.notmyidea.org/distutils2_schedule">a schedule</a>, and
|
||
the goal is to be sure everything will be ready in time, for the
|
||
end of the summer. (And now, I need to learn to follow schedules
|
||
...)</p>
|
||
</div>
|
||
</content></entry><entry><title>Sprinting on distutils2 in Tours</title><link href="https://blog.notmyidea.org/sprinting-on-distutils2-in-tours.html" rel="alternate"></link><published>2010-07-06T00:00:00+02:00</published><updated>2010-07-06T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2010-07-06:/sprinting-on-distutils2-in-tours.html</id><summary type="html"><p>Yesterday, as I was traveling to Tours, I've took some time to
|
||
visit Éric, another student who's working on distutils2 this
|
||
summer, as a part of the GSoC. Basically, it was to take a drink,
|
||
discuss a bit about distutils2, our respective tasks and general
|
||
feelings, and to put a …</p></summary><content type="html"><p>Yesterday, as I was traveling to Tours, I've took some time to
|
||
visit Éric, another student who's working on distutils2 this
|
||
summer, as a part of the GSoC. Basically, it was to take a drink,
|
||
discuss a bit about distutils2, our respective tasks and general
|
||
feelings, and to put a face on a pseudonym. I'd really enjoyed this
|
||
time, because Éric knows a lot of things about mercurial and python
|
||
good practices, and I'm eager to learn about those. So, we have
|
||
discussed about things, have not wrote so much code, but have some
|
||
things to propose so far, about documentation, and I also provides
|
||
here some bribes of conversations we had.</p>
|
||
<div class="section" id="documentation">
|
||
<h2>Documentation</h2>
|
||
<p>While writing the PyPI simple index crawler documentation, I
|
||
realized that we miss some structure, or how-to about the
|
||
documentation. Yep, you read well. We lack documentation on how to
|
||
make documentation. Heh. We're missing some rules to follow, and
|
||
this lead to a not-so-structured final documentation. We probably
|
||
target three type of publics, and we can split the documentation
|
||
regarding those:</p>
|
||
<ul class="simple">
|
||
<li><strong>Packagers</strong> who want to distribute their softwares.</li>
|
||
<li><strong>End users</strong> who need to understand how to use end user
|
||
commands, like the installer/uninstaller</li>
|
||
<li><strong>packaging coders</strong> who <em>use</em> distutils2, as a base for
|
||
building a package manager.</li>
|
||
</ul>
|
||
<p>We also need to discuss about a pattern to follow while writing
|
||
documentation. How many parts do we need ? Where to put the API
|
||
description ? etc. That's maybe seems to be not so important, but I
|
||
guess the readers would appreciate to have the same structure all
|
||
along distutils2 documentation.</p>
|
||
</div>
|
||
<div class="section" id="mercurial">
|
||
<h2>Mercurial</h2>
|
||
<p>I'm really <em>not</em> a mercurial power user. I use it on daily basis,
|
||
but I lack of basic knowledge about it. Big thanks Éric for sharing
|
||
yours with me, you're of a great help. We have talked about some
|
||
mercurial extensions that seems to make the life simpler, while
|
||
used the right way. I've not used them so far, so consider this as
|
||
a personal note.</p>
|
||
<ul class="simple">
|
||
<li>hg histedit, to edit the history</li>
|
||
<li>hg crecord, to select the changes to commit</li>
|
||
</ul>
|
||
<p>We have spent some time to review a merge I made sunday, to
|
||
re-merge it, and commit the changes as a new changeset. Awesome.
|
||
These things make me say I <strong>need</strong> to read
|
||
<a class="reference external" href="http://hgbook.red-bean.com/read/">the hg book</a>, and will do as
|
||
soon as I got some spare time: mercurial seems to be simply great.
|
||
So ... Great. I'm a powerful merger now !</p>
|
||
</div>
|
||
<div class="section" id="on-using-tools">
|
||
<h2>On using tools</h2>
|
||
<p>Because we <em>also</em> are <em>hackers</em>, we have shared a bit our ways to
|
||
code, the tools we use, etc. Both of us were using vim, and I've
|
||
discovered vimdiff and hgtk, which will completely change the way I
|
||
navigate into the mercurial history. We aren't &quot;power users&quot;, so we
|
||
have learned from each other about vim tips. You can find
|
||
<a class="reference external" href="http://github.com/ametaireau/dotfiles">my dotfiles on github</a>,
|
||
if it could help. They're not perfect, and not intended to be,
|
||
because changing all the time, as I learn. Don't hesitate to have a
|
||
look, and to propose enhancements if you have !</p>
|
||
</div>
|
||
<div class="section" id="on-being-pythonic">
|
||
<h2>On being pythonic</h2>
|
||
<p>My background as an old Java user disserves me so far, as the
|
||
paradigms are not the same while coding in python. Hard to find the
|
||
more pythonic way to do, and sometimes hard to unlearn my way to
|
||
think about software engineering. Well, it seems that the only
|
||
solution is to read code, and to re-read import this from times to
|
||
times !
|
||
<a class="reference external" href="http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html">Coding like a pythonista</a>
|
||
seems to be a must-read, so, I know what to do.</p>
|
||
</div>
|
||
<div class="section" id="conclusion">
|
||
<h2>Conclusion</h2>
|
||
<p>It was really great. Next time, we'll need to focus a bit more on
|
||
distutils2, and to have a bullet list of things to do, but days
|
||
like this one are opportunities to catch ! We'll probably do
|
||
another sprint in a few weeks, stay tuned !</p>
|
||
</div>
|
||
</content></entry><entry><title>Use Restructured Text (ReST) to power your presentations</title><link href="https://blog.notmyidea.org/use-restructured-text-rest-to-power-your-presentations.html" rel="alternate"></link><published>2010-06-25T00:00:00+02:00</published><updated>2010-06-25T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2010-06-25:/use-restructured-text-rest-to-power-your-presentations.html</id><summary type="html"><p>Wednesday, we give a presentation, with some friends, about the
|
||
CouchDB Database, to
|
||
<a class="reference external" href="http://www.toulibre.org">the Toulouse local LUG</a>. Thanks a lot
|
||
to all the presents for being there, it was a pleasure to talk
|
||
about this topic with you. Too bad the season is over now an I quit
|
||
Toulouse next …</p></summary><content type="html"><p>Wednesday, we give a presentation, with some friends, about the
|
||
CouchDB Database, to
|
||
<a class="reference external" href="http://www.toulibre.org">the Toulouse local LUG</a>. Thanks a lot
|
||
to all the presents for being there, it was a pleasure to talk
|
||
about this topic with you. Too bad the season is over now an I quit
|
||
Toulouse next year.</p>
|
||
<p>During our brainstorming about the topic, we
|
||
used some paper, and we wanted to make a presentation the simpler
|
||
way. First thing that come to my mind was using
|
||
<a class="reference external" href="http://docutils.sourceforge.net/rst.html">restructured text</a>, so
|
||
I've wrote a simple file containing our different bullet points. In
|
||
fact, there is quite nothing to do then, to have a working
|
||
presentation.</p>
|
||
<p>So far, I've used
|
||
<a class="reference external" href="http://code.google.com/p/rst2pdf/">the rst2pdf program</a>, and a
|
||
simple template, to generate output. It's probably simple to have
|
||
similar results using latex + beamer, I'll try this next time, but
|
||
as I'm not familiar with latex syntax, restructured text was a
|
||
great option.</p>
|
||
<p>Here are
|
||
<a class="reference external" href="http://files.lolnet.org/alexis/rst-presentations/couchdb/couchdb.pdf">the final PDF output</a>,
|
||
<a class="reference external" href="http://files.lolnet.org/alexis/rst-presentations/couchdb/couchdb.rst">Rhe ReST source</a>,
|
||
<a class="reference external" href="http://files.lolnet.org/alexis/rst-presentations/slides.style">the theme used</a>,
|
||
and the command line to generate the PDF:</p>
|
||
<pre class="literal-block">
|
||
rst2pdf couchdb.rst -b1 -s ../slides.style
|
||
</pre>
|
||
</content></entry><entry><title>first week working on distutils2</title><link href="https://blog.notmyidea.org/first-week-working-on-distutils2.html" rel="alternate"></link><published>2010-06-04T00:00:00+02:00</published><updated>2010-06-04T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2010-06-04:/first-week-working-on-distutils2.html</id><summary type="html"><p>As I've been working on
|
||
<a class="reference external" href="http://hg.python.org/distutils2/">Distutils2</a> during the past
|
||
week, taking part of the
|
||
<a class="reference external" href="http://code.google.com/intl/fr/soc/">GSOC</a> program, here is a
|
||
short summary of what I've done so far.</p>
|
||
<p>As my courses are not over yet, I've not worked as much as I
|
||
wanted, and this will continues until the end of …</p></summary><content type="html"><p>As I've been working on
|
||
<a class="reference external" href="http://hg.python.org/distutils2/">Distutils2</a> during the past
|
||
week, taking part of the
|
||
<a class="reference external" href="http://code.google.com/intl/fr/soc/">GSOC</a> program, here is a
|
||
short summary of what I've done so far.</p>
|
||
<p>As my courses are not over yet, I've not worked as much as I
|
||
wanted, and this will continues until the end of June. My main
|
||
tasks are about making installation and uninstallation commands, to
|
||
have a simple way to install distributions via
|
||
<a class="reference external" href="http://hg.python.org/distutils2/">Distutils2</a>.</p>
|
||
<p>To do this, we need to rely on informations provided by the Python
|
||
Package Index (<a class="reference external" href="http://pypi.python.org/">PyPI</a>), and there is at
|
||
least two ways to retreive informations from here: XML-RPC and the
|
||
&quot;simple&quot; API.</p>
|
||
<p>So, I've been working on porting some
|
||
<a class="reference external" href="http://bitbucket.org/tarek/distribute/">Distribute</a> related
|
||
stuff to <a class="reference external" href="http://hg.python.org/distutils2/">Distutils2</a>, cutting
|
||
off all non distutils' things, as we do not want to depend from
|
||
Distribute's internals. My main work has been about reading the
|
||
whole code, writing tests about this and making those tests
|
||
possible.</p>
|
||
<p>In fact, there was a need of a pypi mocked server, and, after
|
||
reading and introducing myself to the distutils behaviors and code,
|
||
I've taken some time to improve the work
|
||
<a class="reference external" href="http://bitbucket.org/konrad">Konrad</a> makes about this mock.</p>
|
||
<div class="section" id="a-pypi-server-mock">
|
||
<h2>A PyPI Server mock</h2>
|
||
<p>The mock is embeded in a thread, to make it available during the
|
||
tests, in a non blocking way. We first used
|
||
<a class="reference external" href="http://wsgi.org">WSGI</a> and
|
||
<a class="reference external" href="http://docs.python.org/library/wsgiref.html">wsgiref</a> in order
|
||
control what to serve, and to log the requests made to the server,
|
||
but finally realised that
|
||
<a class="reference external" href="http://docs.python.org/library/wsgiref.html">wsgiref</a> is not
|
||
python 2.4 compatible (and we <em>need</em> to be python 2.4 compatible in
|
||
Distutils2).</p>
|
||
<p>So, we switched to
|
||
<a class="reference external" href="http://docs.python.org/library/basehttpserver.html">BaseHTTPServer</a>
|
||
and
|
||
<a class="reference external" href="http://docs.python.org/library/simplehttpserver.html">SimpleHTTPServer</a>,
|
||
and updated our tests accordingly. It's been an opportunity to
|
||
realize that <a class="reference external" href="http://wsgi.org">WSGI</a> has been a great step
|
||
forward for making HTTP servers, and expose a really simplest way
|
||
to discuss with HTTP !</p>
|
||
<p>You can find
|
||
<a class="reference external" href="http://bitbucket.org/ametaireau/distutils2/changesets">the modifications I made</a>,
|
||
and the
|
||
<a class="reference external" href="http://bitbucket.org/ametaireau/distutils2/src/tip/docs/source/test_framework.rst">related docs</a>
|
||
about this on
|
||
<a class="reference external" href="http://bitbucket.org/ametaireau/distutils2/">my bitbucket distutils2 clone</a>.</p>
|
||
</div>
|
||
<div class="section" id="the-pypi-simple-api">
|
||
<h2>The PyPI Simple API</h2>
|
||
<p>So, back to the main problematic: make a python library to access
|
||
and request information stored on PyPI, via the simple API. As I
|
||
said, I've just grabbed the work made from
|
||
<a class="reference external" href="http://bitbucket.org/tarek/distribute/">Distribute</a>, and played
|
||
a bit with, in order to view what are the different use cases, and
|
||
started to write the related tests.</p>
|
||
</div>
|
||
<div class="section" id="the-work-to-come">
|
||
<h2>The work to come</h2>
|
||
<p>So, once all use cases covered with tests, I'll rewrite a bit the
|
||
grabbed code, and do some software design work (to not expose all
|
||
things as privates methods, have a clear API, and other things like
|
||
this), then update the tests accordingly and write a documentation
|
||
to make this clear.</p>
|
||
<p>Next step is to a little client, as I've
|
||
<a class="reference external" href="http://github.com/ametaireau/pypiclient">already started here</a>
|
||
I'll take you updated !</p>
|
||
</div>
|
||
</content></entry><entry><title>A Distutils2 GSoC</title><link href="https://blog.notmyidea.org/a-distutils2-gsoc.html" rel="alternate"></link><published>2010-05-01T00:00:00+02:00</published><updated>2010-05-01T00:00:00+02:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2010-05-01:/a-distutils2-gsoc.html</id><summary type="html"><p>WOW. I've been accepted to be a part of the
|
||
<a class="reference external" href="http://code.google.com/intl/fr/soc/">Google Summer Of Code</a>
|
||
program, and will work on <a class="reference external" href="http://python.org/">python</a>
|
||
<a class="reference external" href="http://hg.python.org/distutils2/">distutils2</a>, with
|
||
<a class="reference external" href="http://pygsoc.wordpress.com/">a</a>
|
||
<a class="reference external" href="http://konryd.blogspot.com/">lot</a> <a class="reference external" href="http://ziade.org/">of</a>
|
||
(intersting!) <a class="reference external" href="http://zubin71.wordpress.com/">people</a>.</p>
|
||
<blockquote>
|
||
So, it's about building the successor of Distutils2, ie. &quot;the
|
||
python package manager&quot;. Today, there is too&nbsp;many ways to package a
|
||
python …</blockquote></summary><content type="html"><p>WOW. I've been accepted to be a part of the
|
||
<a class="reference external" href="http://code.google.com/intl/fr/soc/">Google Summer Of Code</a>
|
||
program, and will work on <a class="reference external" href="http://python.org/">python</a>
|
||
<a class="reference external" href="http://hg.python.org/distutils2/">distutils2</a>, with
|
||
<a class="reference external" href="http://pygsoc.wordpress.com/">a</a>
|
||
<a class="reference external" href="http://konryd.blogspot.com/">lot</a> <a class="reference external" href="http://ziade.org/">of</a>
|
||
(intersting!) <a class="reference external" href="http://zubin71.wordpress.com/">people</a>.</p>
|
||
<blockquote>
|
||
So, it's about building the successor of Distutils2, ie. &quot;the
|
||
python package manager&quot;. Today, there is too&nbsp;many ways to package a
|
||
python application (pip, setuptools, distribute, distutils, etc.)
|
||
so&nbsp;there is a huge effort to make in order to make all this
|
||
packaging stuff interoperable, as pointed out by
|
||
the&nbsp;<a class="reference external" href="http://www.python.org/dev/peps/pep-0376/">PEP 376</a>.</blockquote>
|
||
<p>In more details, I'm going to work on the Installer / Uninstaller
|
||
features of Distutils2, and on a PyPI XML-RPC client for distutils2.
|
||
Here are the already defined tasks:</p>
|
||
<ul class="simple">
|
||
<li>Implement Distutils2 APIs described in PEP 376.</li>
|
||
<li>Add the uninstall command.</li>
|
||
<li>think about a basic installer / uninstaller script. (with deps)
|
||
-- similar to pip/easy_install</li>
|
||
<li>in a pypi subpackage;</li>
|
||
<li>Integrate a module similar to setuptools' package_index'</li>
|
||
<li>PyPI XML-RPC client for distutils 2:
|
||
<a class="reference external" href="http://bugs.python.org/issue8190">http://bugs.python.org/issue8190</a></li>
|
||
</ul>
|
||
<p>As I'm relatively new to python, I'll need some extra work in order
|
||
to apply all good practice, among other things that can make a
|
||
developper-life joyful. I'll post here, each week, my advancement,
|
||
and my tought about python and especialy python packaging world.</p>
|
||
</content></entry><entry><title>Python ? go !</title><link href="https://blog.notmyidea.org/python-go-fr.html" rel="alternate"></link><published>2009-12-17T00:00:00+01:00</published><updated>2009-12-17T00:00:00+01:00</updated><author><name>Alexis Métaireau</name></author><id>tag:blog.notmyidea.org,2009-12-17:/python-go-fr.html</id><summary type="html"><p>Cela fait maintenant un peu plus d'un mois que je travaille sur un
|
||
projet en <a class="reference external" href="http://www.djangoproject.org">django</a>, et que,
|
||
nécessairement, je me forme à <a class="reference external" href="http://python.org/">Python</a>. Je
|
||
prends un plaisir non dissimulé à découvrir ce langage (et à
|
||
l'utiliser), qui ne cesse de me surprendre. Les premiers mots qui
|
||
me viennent à …</p></summary><content type="html"><p>Cela fait maintenant un peu plus d'un mois que je travaille sur un
|
||
projet en <a class="reference external" href="http://www.djangoproject.org">django</a>, et que,
|
||
nécessairement, je me forme à <a class="reference external" href="http://python.org/">Python</a>. Je
|
||
prends un plaisir non dissimulé à découvrir ce langage (et à
|
||
l'utiliser), qui ne cesse de me surprendre. Les premiers mots qui
|
||
me viennent à l'esprit à propos de Python, sont &quot;logique&quot; et
|
||
&quot;simple&quot;. Et pourtant puissant pour autant. Je ne manque d'ailleurs
|
||
pas une occasion pour faire un peu d'<em>évangélisation</em> auprès des
|
||
quelques personnes qui veulent bien m'écouter.</p>
|
||
<div class="section" id="the-zen-of-python">
|
||
<h2>The Zen of Python</h2>
|
||
<p>Avant toute autre chose, je pense utile de citer Tim Peters, et
|
||
<a class="reference external" href="http://www.python.org/dev/peps/pep-0020/">le PEP20</a>, qui
|
||
constituent une très bonne introduction au langage, qui prends la
|
||
forme d'un <em>easter egg</em> présent dans python</p>
|
||
<div class="highlight"><pre><span></span>&gt;&gt;&gt; import this
|
||
The Zen of Python, by Tim Peters
|
||
|
||
Beautiful is better than ugly.
|
||
Explicit is better than implicit.
|
||
Simple is better than complex.
|
||
Complex is better than complicated.
|
||
Flat is better than nested.
|
||
Sparse is better than dense.
|
||
Readability counts.
|
||
Special cases aren<span class="s1">&#39;t special enough to break the rules.</span>
|
||
<span class="s1">Although practicality beats purity.</span>
|
||
<span class="s1">Errors should never pass silently.</span>
|
||
<span class="s1">Unless explicitly silenced.</span>
|
||
<span class="s1">In the face of ambiguity, refuse the temptation to guess.</span>
|
||
<span class="s1">There should be one-- and preferably only one --obvious way to do it.</span>
|
||
<span class="s1">Although that way may not be obvious at first unless you&#39;</span>re Dutch.
|
||
Now is better than never.
|
||
Although never is often better than *right* now.
|
||
If the implementation is hard to explain, it<span class="s1">&#39;s a bad idea.</span>
|
||
<span class="s1">If the implementation is easy to explain, it may be a good idea.</span>
|
||
<span class="s1">Namespaces are one honking great idea -- let&#39;</span>s <span class="k">do</span> more of those!
|
||
</pre></div>
|
||
<p>J'ai la vague impression que c'est ce que j'ai toujours cherché à
|
||
faire en PHP, et particulièrement dans
|
||
<a class="reference external" href="http://www.spiral-project.org">le framework Spiral</a>, mais en
|
||
ajoutant ces concepts dans une sur-couche au langage. Ici, c'est
|
||
directement de <em>l'esprit</em> de python qu'il s'agit, ce qui signifie
|
||
que la plupart des bibliothèques python suivent ces concepts. Elle
|
||
est pas belle la vie ?</p>
|
||
</div>
|
||
<div class="section" id="comment-commencer-et-par-ou">
|
||
<h2>Comment commencer, et par ou ?</h2>
|
||
<p>Pour ma part, j'ai commencé par la lecture de quelques livres et
|
||
articles intéressants, qui constituent une bonne entrée en matière
|
||
sur le sujet (La liste n'est bien évidemment pas exhaustive et vos
|
||
commentaires sont les bienvenus) :</p>
|
||
<ul class="simple">
|
||
<li><a class="reference external" href="http://diveintopython.adrahon.org/">Dive into python</a></li>
|
||
<li><a class="reference external" href="http://www.swaroopch.com/notes/Python_fr:Table_des_Matières">A byte of python</a></li>
|
||
<li><a class="reference external" href="http://www.amazon.fr/Python-Petit-guide-lusage-développeur/dp/2100508830">Python: petit guide à l'usage du développeur agile</a>
|
||
de <a class="reference external" href="http://tarekziade.wordpress.com/">Tarek Ziadé</a></li>
|
||
<li><a class="reference external" href="http://docs.python.org/index.html">La documentation officielle python</a>,
|
||
bien sûr !</li>
|
||
<li><a class="reference external" href="http://video.pycon.fr/videos/pycon-fr-2009/">Les vidéos du pyconfr 2009</a>!</li>
|
||
<li>Un peu de temps, et une console python ouverte :)</li>
|
||
</ul>
|
||
<p>J'essaye par ailleurs de partager au maximum les ressources que je
|
||
trouve de temps à autres, que ce soit
|
||
<a class="reference external" href="http://www.twitter.com/ametaireau">via twitter</a> ou
|
||
<a class="reference external" href="http://delicious.com/ametaireau">via mon compte delicious</a>.
|
||
Allez jeter un œil
|
||
<a class="reference external" href="http://delicious.com/ametaireau/python">au tag python</a> sur mon
|
||
profil, peut être que vous trouverez des choses intéressantes, qui
|
||
sait!</p>
|
||
</div>
|
||
<div class="section" id="un-python-sexy">
|
||
<h2>Un python sexy</h2>
|
||
<p>Quelques fonctionnalités qui devraient vous mettre l'eau à la
|
||
bouche:</p>
|
||
<ul class="simple">
|
||
<li><a class="reference external" href="http://docs.python.org/library/stdtypes.html#comparisons">Le chaînage des opérateurs de comparaison</a>
|
||
est possible (a&lt;b &lt;c dans une condition)</li>
|
||
<li>Assignation de valeurs multiples (il est possible de faire a,b,c
|
||
= 1,2,3 par exemple)</li>
|
||
<li><a class="reference external" href="http://docs.python.org/tutorial/datastructures.html">Les listes</a>
|
||
sont simples à manipuler !</li>
|
||
<li>Les <a class="reference external" href="http://docs.python.org/tutorial/datastructures.html#list-comprehensions">list comprehension</a>,
|
||
ou comment faire des opérations complexes sur les listes, de
|
||
manière simple.</li>
|
||
<li>Les
|
||
<a class="reference external" href="http://docs.python.org/library/doctest.html?highlight=doctest">doctests</a>:
|
||
ou comment faire des tests directement dans la documentation de vos
|
||
classes, tout en la documentant avec de vrais exemples.</li>
|
||
<li>Les
|
||
<a class="reference external" href="http://www.python.org/doc/essays/metaclasses/meta-vladimir.txt">métaclasses</a>,
|
||
ou comment contrôler la manière dont les classes se construisent</li>
|
||
<li>Python est
|
||
<a class="reference external" href="http://wiki.python.org/moin/Why%20is%20Python%20a%20dynamic%20language%20and%20also%20a%20strongly%20typed%20language">un langage à typage fort dynamique</a>:
|
||
c'est ce qui m'agaçait avec PHP qui est un langage à typage faible
|
||
dynamique.</li>
|
||
</ul>
|
||
<p>Cous pouvez également aller regarder
|
||
<a class="reference external" href="http://video.pycon.fr/videos/free/53/">l'atelier donné par Victor Stinner durant le Pyconfr 09</a>.
|
||
Have fun !</p>
|
||
</div>
|
||
</content></entry></feed> |