Alexis' log

Pelican, 9 months later

2011-07-25T00:00:00+02:00

Back in October, I released pelican, a little piece of code I wrote to power this weblog. I had simple needs: I wanted to be able to use my text editor of choice (vim), a vcs (mercurial) and restructured text. I started to write a really simple blog engine in something like a hundred python lines and released it on github.

And people started contributing. I wasn't at all expecting to see people interested in such a little piece of code, but it turned out that they were. I refactored the code to make it evolve a bit more by two times and eventually, in 9 months, got 49 forks, 139 issues and 73 pull requests.

Which is clearly awesome.

I pulled features such as translations, tag clouds, integration with different services such as twitter or piwik, import from dotclear and rss, fixed a number of mistakes and improved a lot the codebase. This was a proof that there is a bunch of people that are willing to make better softwares just for the sake of fun.

Thank you, guys, you're why I like open source so much.

Using JPype to bridge python and Java

2011-06-11T00:00:00+02:00

Java provides some interesting libraries that have no exact equivalent in python. In my case, the awesome boilerpipe library allows me to remove uninteresting parts of HTML pages, like menus, footers and other "boilerplate" contents.

Boilerpipe is written in Java. Two solutions then: using java from python or reimplement boilerpipe in python. I will let you guess which one I chosen, meh.

JPype allows to bridge python project with java libraries. It takes another point of view than Jython: rather than reimplementing python in Java, both languages are interfacing at the VM level. This means you need to start a VM from your python script, but it does the job and stay fully compatible with Cpython and its C extensions.

First steps with JPype

Once JPype installed (you'll have to hack a bit some files to integrate seamlessly with your system) you can access java classes by doing something like that:

import jpype
jpype.startJVM(jpype.getDefaultJVMPath())

# you can then access to the basic java functions
jpype.java.lang.System.out.println("hello world")

# and you have to shutdown the VM at the end
jpype.shutdownJVM()

Okay, now we have a hello world, but what we want seems somehow more complex. We want to interact with java classes, so we will have to load them.

Interfacing with Boilerpipe

To install boilerpipe, you just have to run an ant script:

$ cd boilerpipe
$ ant

Here is a simple example of how to use boilerpipe in Java, from their sources

package de.l3s.boilerpipe.demo;
import java.net.URL;
import de.l3s.boilerpipe.extractors.ArticleExtractor;

public class Oneliner {
    public static void main(final String[] args) throws Exception {
        final URL url = new URL("http://notmyidea.org");
        System.out.println(ArticleExtractor.INSTANCE.getText(url));
    }
}

To run it:

$ javac -cp dist/boilerpipe-1.1-dev.jar:lib/nekohtml-1.9.13.jar:lib/xerces-2.9.1.jar src/demo/de/l3s/boilerpipe/demo/Oneliner.java
$ java -cp src/demo:dist/boilerpipe-1.1-dev.jar:lib/nekohtml-1.9.13.jar:lib/xerces-2.9.1.jar de.l3s.boilerpipe.demo.Oneliner

Yes, this is kind of ugly, sorry for your eyes. Let's try something similar, but from python

import jpype

# start the JVM with the good classpaths
classpath = "dist/boilerpipe-1.1-dev.jar:lib/nekohtml-1.9.13.jar:lib/xerces-2.9.1.jar"
jpype.startJVM(jpype.getDefaultJVMPath(), "-Djava.class.path=%s" % classpath)

# get the Java classes we want to use
DefaultExtractor = jpype.JPackage("de").l3s.boilerpipe.extractors.DefaultExtractor

# call them !
print DefaultExtractor.INSTANCE.getText(jpype.java.net.URL("http://blog.notmyidea.org"))

And you get what you want.

I must say I didn't thought it could work so easily. This will allow me to extract text content from URLs and remove the boilerplate text easily for infuse (my master thesis project), without having to write java code, nice!

Un coup de main pour mon mémoire!

2011-05-25T00:00:00+02:00

Ça y est, bientôt la fin. LA FIN. La fin des études, et le début du reste. En attendant je bosse sur mon mémoire de fin d'études et j'aurais besoin d'un petit coup de main.

Mon mémoire porte sur les systèmes de recommandation. Pour ceux qui connaissent last.fm, je fais quelque chose de similaire mais pour les sites internet: en me basant sur ce que vous visitez quotidiennement et comment vous le visitez (quelles horaires, quelle emplacement géographique, etc.) je souhaites proposer des liens qui vous intéresseront potentiellement, en me basant sur l'avis des personnes qui ont des profils similaires au votre.

Le projet est loin d'être terminé, mais la première étape est de récupérer des données de navigation, idéalement beaucoup de données de navigation. Donc si vous pouvez me filer un coup de main je vous en serais éternellement reconnaissant (pour ceux qui font semblant de pas comprendre, entendez "tournée générale").

J'ai créé un petit site web (en anglais) qui résume un peu le concept, qui vous propose de vous inscrire et de télécharger un plugin firefox qui m'enverra des information sur les sites que vous visitez (si vous avez l'habitude d'utiliser chrome vous pouvez considérer de switcher à firefox4 pour les deux prochains mois pour me filer un coup de main). Il est possible de désactiver le plugin d'un simple clic si vous souhaitez garder votre vie privée privée ;-)

Le site est par là: http://infuse.notmyidea.org. Une fois le plugin téléchargé et le compte créé il faut renseigner vos identifiants dans le plugin en question, et c'est tout!

A votre bon cœur ! Je récupérerais probablement des données durant les 2 prochains mois pour ensuite les analyser correctement.

Merci pour votre aide !

Analyse users' browsing context to build up a web recommender

2011-04-01T00:00:00+02:00

No, this is not an april's fool ;)

Wow, it's been a long time. My year in Oxford is going really well. I realized few days ago that the end of the year is approaching really quickly. Exams are coming in one month or such and then I'll be working full time on my dissertation topic.

When I learned we'll have about 6 month to work on something, I first thought about doing a packaging related stuff, but finally decided to start something new. After all, that's the good time to learn.

Since a long time, I'm being impressed by the last.fm recommender system. They're scrobbling the music I listen to since something like 5 years now and the recommendations they're doing are really nice and accurate (I discovered a lot of great artists listening to the "neighbour radio".) (by the way, here is my lastfm account)

So I decided to work on recommender systems, to better understand what is it about.

Recommender systems are usually used to increase the sales of products (like Amazon.com does) which is not really what I'm looking for (The one who know me a bit know I'm kind of sick about all this consumerism going on).

Actually, the most simple thing I thought of was the web: I'm browsing it quite every day and each time new content appears. I've stopped to follow my feed reader because of the information overload, and reduced drastically the number of people I follow on twitter.

Too much information kills the information.

You shall got what will be my dissertation topic: a recommender system for the web. Well, such recommender systems already exists, so I will try to add contextual information to them: you're probably not interested by the same topics at different times of the day, or depending on the computer you're using. We can also probably make good use of the way you browse to create groups into the content you're browsing (or even use the great firefox4 tab group feature).

There is a large part of concerns to have about user's privacy as well.

Here is my proposal (copy/pasted from the one I had to do for my master)

Introduction and rationale

Nowadays, people surf the web more and more often. New web pages are created each day so the amount of information to retrieve is more important as the time passes. These users uses the web in different contexts, from finding cooking recipes to technical articles.

A lot of people share the same interest to various topics, and the quantity of information is such than it's really hard to triage them efficiently without spending hours doing it. Firstly because of the huge quantity of information but also because the triage is something relative to each person. Although, this triage can be facilitated by fetching the browsing information of all particular individuals and put the in perspective.

Machine learning is a branch of Artificial Intelligence (AI) which deals with how a program can learn from data. Recommendation systems are a particular application area of machine learning which is able to recommend things (links in our case) to the users, given a particular database containing the previous choices users have made.

This browsing information is currently available in browsers. Even if it is not in a very usable format, it is possible to transform it to something useful. This information gold mine just wait to be used. Although, it is not as simple as it can seems at the first approach: It is important to take care of the context the user is in while browsing links. For instance, It's more likely that during the day, a computer scientist will browse computing related links, and that during the evening, he browse cooking recipes or something else.

Page contents are also interesting to analyse, because that's what people browse and what actually contain the most interesting part of the information. The raw data extracted from the browsing can then be translated into something more useful (namely tags, type of resource, visit frequency, navigation context etc.)

The goal of this dissertation is to create a recommender system for web links, including this context information.

At the end of the dissertation, different pieces of software will be provided, from raw data collection from the browser to a recommendation system.

Background Review

This dissertation is mainly about data extraction, analysis and recommendation systems. Two different research area can be isolated: Data preprocessing and Information filtering.

The first step in order to make recommendations is to gather some data. The more data we have available, the better it is (T. Segaran, 2007). This data can be retrieved in various ways, one of them is to get it directly from user's browsers.

Data preparation and extraction

The data gathered from browsers is basically URLs and additional information about the context of the navigation. There is clearly a need to extract more information about the meaning of the data the user is browsing, starting by the content of the web pages.

Because the information provided on the current Web is not meant to be read by machines (T. Berners Lee, 2001) there is a need of tools to extract meaning from web pages. The information needs to be preprocessed before stored in a machine readable format, allowing to make recommendations (Choochart et Al, 2004).

Data preparation is composed of two steps: cleaning and structuring ( Castellano et Al, 2007). Because raw data can contain a lot of un-needed text (such as menus, headers etc.) and need to be cleaned prior to be stored. Multiple techniques can be used here and belongs to boilerplate removal and full text extraction (Kohlschütter et Al, 2010).

Then, structuring the information: category, type of content (news, blog, wiki) can be extracted from raw data. This kind of information is not clearly defined by HTML pages so there is a need of tools to recognise them.

Some context-related information can also be inferred from each resource. It can go from the visit frequency to the navigation group the user was in while browsing. It is also possible to determine if the user "liked" a resource, and determine a mark for it, which can be used by information filtering a later step (T. Segaran, 2007).

At this stage, structuring the data is required. Storing this kind of information in RDBMS can be a bit tedious and require complex queries to get back the data in an usable format. Graph databases can play a major role in the simplification of information storage and querying.

Information filtering

To filter the information, three techniques can be used (Balabanovic et Al, 1997):

The content-based approach states that if an user have liked something in the past, he is more likely to like similar things in the future. So it's about establishing a profile for the user and compare new items against it.
The collaborative approach will rather recommend items that other similar users have liked. This approach consider only the relationship between users, and not the profile of the user we are making recommendations to.
the hybrid approach, which appeared recently combine both of the previous approaches, giving recommendations when items score high regarding user's profile, or if a similar user already liked it.

Grouping is also something to consider at this stage (G. Myatt, 2007). Because we are dealing with huge amount of data, it can be useful to detect group of data that can fit together. Data clustering is able to find such groups (T. Segaran, 2007).

References:

Balabanović, M., & Shoham, Y. (1997). Fab: content-based, collaborative recommendation. Communications of the ACM, 40(3), 66–72. ACM. Retrieved March 1, 2011, from http://portal.acm.org/citation.cfm?id=245108.245124&.
Berners-Lee, T., Hendler, J., & Lassila, O. (2001). The semantic web: Scientific american. Scientific American, 284(5), 34–43. Retrieved November 21, 2010, from http://www.citeulike.org/group/222/article/1176986.
Castellano, G., Fanelli, A., & Torsello, M. (2007). LODAP: a LOg DAta Preprocessor for mining Web browsing patterns. Proceedings of the 6th Conference on 6th WSEAS Int. Conf. on Artificial Intelligence, Knowledge Engineering and Data Bases-Volume 6 (p. 12–17). World Scientific and Engineering Academy and Society (WSEAS). Retrieved March 8, 2011, from http://portal.acm.org/citation.cfm?id=1348485.1348488.
Kohlschutter, C., Fankhauser, P., & Nejdl, W. (2010). Boilerplate detection using shallow text features. Proceedings of the third ACM international conference on Web search and data mining (p. 441–450). ACM. Retrieved March 8, 2011, from http://portal.acm.org/citation.cfm?id=1718542.
Myatt, G. J. (2007). Making Sense of Data: A Practical Guide to Exploratory Data Analysis and Data Mining.
Segaran, T. (2007). Collective Intelligence.

Privacy

The first thing that's come to people minds when it comes to process their browsing data is privacy. People don't want to be stalked. That's perfectly right, and I don't either.

But such a system don't have to deal with people identities. It's completely possible to process completely anonymous data, and that's probably what I'm gonna do.

By the way, if you have interesting thoughts about that, if you do know projects that do seems related, fire the comments !

What's the plan ?

There is a lot of different things to explore, especially because I'm a complete novice in that field.

I want to develop a firefox plugin, to extract the browsing informations ( still, I need to know exactly which kind of informations to retrieve). The idea is to provide some raw browsing data, and then to transform it and to store it in the better possible way.
Analyse how to store the informations in a graph database. What can be the different methods to store this data and to visualize the relationship between different pieces of data? How can I define the different contexts, and add those informations in the db?
Process the data using well known recommendation algorithms. Compare the results and criticize their value.

There is plenty of stuff I want to try during this experimentation:

I want to try using Geshi to visualize the connexion between the links, and the contexts
Try using graph databases such as Neo4j
Having a deeper look at tools such as scikit.learn (a machine learning toolkit in python)
Analyse web pages in order to categorize them. Processing their contents as well, to do some keyword based classification will be done.

Lot of work on its way, yay !

Wrap up of the distutils2 paris' sprint

2011-02-08T00:00:00+01:00

Finally, thanks to a bunch of people that helped me to pay my train and bus tickets, I've made it to paris for the distutils2 sprint.

They have been a bit more than 10 people to come during the sprint, and it was very productive. Here's a taste of what we've been working on:

the datafiles, a way to specify and to handle the installation of files which are not python-related (pictures, manpages and so on).
mkgcfg, a tool to help you to create a setup.cfg in minutes (and with funny examples)
converters from setup.py scripts. We do now have a piece of code which reads your current setup.py file and fill in some fields in the setup.cfg for you.
a compatibility layer for distutils1, so it can read the setup.cfg you will wrote for distutils2 :-)
the uninstaller, so it's now possible to uninstall what have been installed by distutils2 (see PEP 376)
the installer, and the setuptools compatibility layer, which will allow you to rely on setuptools' based distributions (and there are plenty of them!)
The compilers, so they are more flexible than they were. Since that's an obscure part of the code for distutils2 commiters (it comes directly from the distutils1 ages), having some guys who understood the problematics here was a must.

Some people have also tried to port their packaging from distutils1 to distutils2. They have spotted a number of bugs and made some improvements to the code, to make it more friendly to use.

I'm really pleased to see how newcomers went trough the code, and started hacking so fast. I must say it wasn't the case when we started to work on distutils1 so that's a very good point: people now can hack the code quicker than they could before.

Some of the features here are not completely finished yet, but are on the tubes, and will be ready for a release (hopefully) at the end of the week.

Big thanks to logilab for hosting (and sponsoring my train ticket) and providing us food, and to bearstech for providing some money for breakfast and bears^Wbeers.

Again, a big thanks to all the people who gave me money to pay the transport, I really wasn't expecting such thing to happen :-)

PyPI on CouchDB

2011-01-20T00:00:00+01:00

By now, there are two ways to retrieve data from PyPI (the Python Package Index). You can both rely on xml/rpc or on the "simple" API. The simple API is not so simple to use as the name suggest, and have several existing drawbacks.

Basically, if you want to use informations coming from the simple API, you will have to parse web pages manually, to extract informations using some black vodoo magic. Badly, magic have a price, and it's sometimes impossible to get exactly the informations you want to get from this index. That's the technique currently being used by distutils2, setuptools and pip.

On the other side, while XML/RPC is working fine, it's requiring extra work to the python servers each time you request something, which can lead to some outages from time to time. Also, it's important to point out that, even if PyPI have a mirroring infrastructure, it's only for the so-called simple API, and not for the XML/RPC.

CouchDB

Here comes CouchDB. CouchDB is a document oriented database, that knows how to speak REST and JSON. It's easy to use, and provides out of the box a replication mechanism.

So, what ?

Hmm, I'm sure you got it. I've wrote a piece of software to link informations from PyPI to a CouchDB instance. Then you can replicate all the PyPI index with only one HTTP request on the CouchDB server. You can also access the informations from the index directly using a REST API, speaking json. Handy.

So PyPIonCouch is using the PyPI XML/RPC API to get data from PyPI, and generate records in the CouchDB instance.

The final goal is to avoid to rely on this "simple" API, and rely on a REST insterface instead. I have set up a couchdb server on my server, which is available at http://couchdb.notmyidea.org/_utils/database.html?pypi.

There is not a lot to see there for now, but I've done the first import from PyPI yesterday and all went fine: it's possible to access the metadata of all PyPI projects via a REST interface. Next step is to write a client for this REST interface in distutils2.

Example

For now, you can use pypioncouch via the command line, or via the python API.

Using the command line

You can do something like that for a full import. This will take long, because it's fetching all the projects at pypi and importing their metadata:

$ pypioncouch --fullimport http://your.couchdb.instance/

If you already have the data on your couchdb instance, you can just update it with the last informations from pypi. However, I recommend to just replicate the principal node, hosted at http://couchdb.notmyidea.org/pypi/, to avoid the duplication of nodes:

$ pypioncouch --update http://your.couchdb.instance/

The principal node is updated once a day by now, I'll try to see if it's enough, and ajust with the time.

Using the python API

You can also use the python API to interact with pypioncouch:

>>> from pypioncouch import XmlRpcImporter, import_all, update
>>> full_import()
>>> update()

What's next ?

I want to make a couchapp, in order to navigate PyPI easily. Here are some of the features I want to propose:

List all the available projects
List all the projects, filtered by specifiers
List all the projects by author/maintainer
List all the projects by keywords
Page for each project.
Provide a PyPI "Simple" API equivalent, even if I want to replace it, I do think it will be really easy to setup mirrors that way, with the out of the box couchdb replication

I also still need to polish the import mechanism, so I can directly store in couchdb:

The OPML files for each project
The upload_time as couchdb friendly format (list of int)
The tags as lists (currently it's only a string separated by spaces

The work I've done by now is available on https://bitbucket.org/ametaireau/pypioncouch/. Keep in mind that it's still a work in progress, and everything can break at any time. However, any feedback will be appreciated !

Help me to go to the distutils2 paris' sprint

2011-01-15T00:00:00+01:00

Edit: Thanks to logilab and some amazing people, I can make it to paris for the sprint. Many thanks to them for the support!

There will be a distutils2 sprint from the 27th to the 30th of january, thanks to logilab which will host the event.

You can find more informations about the sprint on the wiki page of the event (http://wiki.python.org/moin/Distutils/SprintParis).

I really want to go there but I'm unfortunately blocked in UK for money reasons. The cheapest two ways I've found is about £80, which I can't afford. Following some advices on #distutils, I've set up a ChipIn account for that, so if some people want to help me making it to go there, they can give me some money that way.

I'll probably work on the installer (to support old distutils and setuptools distributions) and on the uninstaller (depending on the first task). If I can't make it to paris, I'll hang around on IRC to give some help while needed.

If you want to contribute some money to help me go there, feel free to use this chipin page: http://ametaireau.chipin.com/distutils2-sprint-in-paris

Thanks for your support !

How to reboot your bebox using the CLI

2010-10-21T00:00:00+02:00

I've an internet connection which, for some obscure reasons, tend to be very slow from time to time. After rebooting the box (yes, that's a hard solution), all the things seems to go fine again.

EDIT : Using grep

After a bit of reflexion, that's also really easy to do using directly the command line tools curl, grep and tail (but really harder to read).

curl -X POST -u joel:joel http://bebox.config/cgi/b/info/restart/\?be\=0\&l0\=1\&l1\=0\&tid\=RESTART -d "0=17&2=`curl -u joel:joel http://bebox.config/cgi/b/info/restart/\?be\=0\&l0\=1\&l1\=0\&tid\=RESTART | grep -o "name='2' value='[0-9]\+" | grep -o "[0-9]\+" | tail -n 1`&1"

The Python version

Well, that's not the optimal solution, that's a bit "gruik", but it works.

import urllib2
import urlparse
import re
import argparse

REBOOT_URL = '/b/info/restart/?be=0&l0=1&l1=0&tid=RESTART'
BOX_URL = 'http://bebox.config/cgi'

def open_url(url, username, password):
    passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
    passman.add_password(None, url, username, password)
    authhandler = urllib2.HTTPBasicAuthHandler(passman)

    opener = urllib2.build_opener(authhandler)

    urllib2.install_opener(opener)

    return urllib2.urlopen(url).read()

def reboot(url, username, password):
    data = open_url(url, username, password)
    token = re.findall("name\=\\'2\\' value=\\'([0-9]+)\\'", data)[1]
    urllib2.urlopen(urllib2.Request(url=url, data='0=17&2=%s&1' % token))

if __file__ == '__main__':
    parser = argparse.ArgumentParser(description="""Reboot your bebox !""")

    parser.add_argument(dest='user', help='username')
    parser.add_argument(dest='password', help='password')
    parser.add_argument(boxurl='boxurl', default=BOX_URL, help='Base box url.  Default is %s' % BOX_URL)

    args = parser.parse_args()
    url = urlparse.urljoin(args.boxurl, REBOOT_URL)
    reboot(url, args.username, args.password)

Dynamically change your gnome desktop wallpaper

2010-10-11T00:00:00+02:00

In gnome, you can can use a XML file to have a dynamic wallpaper. It's not so easy, and you can't just tell: use the pictures in this folder to do so.

You can have a look to the git repository if you want: http://github.com/ametaireau/gnome-background-generator

Some time ago, I've made a little python script to ease that, and you can now use it too. It's named "gnome-background-generator", and you can install it via pip for instance.

$ pip install gnome-background-generator

Then, you have just to use it this way:

$ gnome-background-generator -p ~/Images/walls -s
/home/alexis/Images/walls/dynamic-wallpaper.xml generated

Here is a extract of the --help:

$ gnome-background-generator --help
usage: gnome-background-generator [-h] [-p PATH] [-o OUTPUT]
                                  [-t TRANSITION_TIME] [-d DISPLAY_TIME] [-s]
                                  [-b]

A simple command line tool to generate an XML file to use for gnome
wallpapers, to have dynamic walls

optional arguments:
  -h, --help            show this help message and exit
  -p PATH, --path PATH  Path to look for the pictures. If no output is
                        specified, will be used too for outputing the dynamic-
                        wallpaper.xml file. Default value is the current
                        directory (.)
  -o OUTPUT, --output OUTPUT
                        Output filename. If no filename is specified, a
                        dynamic-wallpaper.xml file will be generated in the
                        path containing the pictures. You can also use "-" to
                        display the xml in the stdout.
  -t TRANSITION_TIME, --transition-time TRANSITION_TIME
                        Time (in seconds) transitions must last (default value
                        is 2 seconds)
  -d DISPLAY_TIME, --display-time DISPLAY_TIME
                        Time (in seconds) a picture must be displayed. Default
                        value is 900 (15mn)
  -s, --set-background  '''try to set the background using gnome-appearance-
                        properties
  -b, --debug

Pelican, a simple static blog generator in python

2010-10-06T00:00:00+02:00

Those days, I've wrote a little python application to fit my blogging needs. I'm an occasional blogger, a vim lover, I like restructured text and DVCSes, so I've made a little tool that makes good use of all that.

Pelican (for calepin) is just a simple tool to generate your blog as static files, letting you using your editor of choice (vim!). It's easy to extend, and has a template support (via jinja2).

I've made it to fit my needs. I hope it will fit yours, but maybe it wont, and it have not be designed to feet everyone's needs.

Need an example ? You're looking at it ! This weblog is using pelican to be generated, also for the atom feeds.

I've released it under AGPL, since I want all the modifications to be profitable to all the users.

You can find a mercurial repository to fork at http://hg.lolnet.org/pelican/, feel free to hack it !

If you just want to get started, use your installer of choice (pip, easy_install, …) And then have a look to the help (pelican --help)

$ pip install pelican

Usage

Here's a sample usage of pelican

$ pelican .
writing /home/alexis/projets/notmyidea.org/output/index.html
writing /home/alexis/projets/notmyidea.org/output/tags.html
writing /home/alexis/projets/notmyidea.org/output/categories.html
writing /home/alexis/projets/notmyidea.org/output/archives.html
writing /home/alexis/projets/notmyidea.org/output/category/python.html
writing
/home/alexis/projets/notmyidea.org/output/pelican-a-simple-static-blog-generator-in-python.html
Done !

You also can use the --help option for the command line to get more informations

$pelican --help
usage: pelican [-h] [-t TEMPLATES] [-o OUTPUT] [-m MARKUP] [-s SETTINGS] [-b]
               path

A tool to generate a static blog, with restructured text input files.

positional arguments:
  path                  Path where to find the content files (default is
                        "content").

optional arguments:
  -h, --help            show this help message and exit
  -t TEMPLATES, --templates-path TEMPLATES
                        Path where to find the templates. If not specified,
                        will uses the ones included with pelican.
  -o OUTPUT, --output OUTPUT
                        Where to output the generated files. If not specified,
                        a directory will be created, named "output" in the
                        current path.
  -m MARKUP, --markup MARKUP
                        the markup language to use. Currently only
                        ReSTreucturedtext is available.
  -s SETTINGS, --settings SETTINGS
                        the settings of the application. Default to None.
  -b, --debug

Enjoy :)

An amazing summer of code working on distutils2

2010-08-16T00:00:00+02:00

The Google Summer of Code I've spent working on distutils2 is over. It was a really amazing experience, for many reasons.

First of all, we had a very good team, we were 5 students working on distutils2: Zubin, Éric, Josip, Konrad and me. In addition, Mouad have worked on the PyPI testing infrastructure. You could find what each person have done on the wiki page of distutils2.

We were in contact with each others really often, helping us when possible (in #distutils), and were continuously aware of the state of the work of each participant. This, in my opinion, have bring us in a good shape.

Then, I've learned a lot. Python packaging was completely new to me at the time of the GSoC start, and I was pretty unfamiliar with python good practices too, as I've been introducing myself to python in the late 2009.

I've recently looked at some python code I wrote just three months ago, and I was amazed to think about many improvements to made on it. I guess this is a good indicator of the path I've traveled since I wrote it.

This summer was awesome because I've learned about python good practices, now having some strong mercurial knowledge, and I've seen a little how the python community works.

Then, I would like to say a big thanks to all the mentors that have hanged around while needed, on IRC or via mail, and especially my mentor for this summer, Tarek Ziadé.

Thanks a lot for your motivation, your leadership and your cheerfulness, even with a new-born and a new work!

Why ?

I wanted to work on python packaging because, as the time pass, we were having a sort of complex tools in this field. Each one wanted to add features to distutils, but not in a standard way.

Now, we have PEPs that describes some format we agreed on (see PEP 345), and we wanted to have a tool on which users can base their code on, that's distutils2.

My job

I had to provides a way to crawl the PyPI indexes in a simple way, and do some installation / uninstallation scripts.

All the work done is available in my bitbucket repository.

Crawling the PyPI indexes

There are two ways of requesting informations from the indexes: using the "simple" index, that is a kind of REST index, and using XML-RPC.

I've done the two implementations, and a high level API to query those twos. Basically, this supports the mirroring infrastructure defined in PEP 381. So far, the work I've done is gonna be used in pip (they've basically copy/paste the code, but this will change as soon as we get something completely stable for distutils2), and that's a good news, as it was the main reason for what I've done that.

I've tried to have an unified API for the clients, to switch from one to another implementation easily. I'm already thinking of adding others crawlers to this stuff, and it was made to be extensible.

If you want to get more informations about the crawlers/PyPI clients, please refer to the distutils2 documentation, especially the pages about indexes.

You can find the changes I made about this in the distutils2 source code .

Installation / Uninstallation scripts

Next step was to think about an installation script, and an uninstaller. I've not done the uninstaller part, and it's a smart part, as it's basically removing some files from the system, so I'll probably do it in a near future.

distutils2 provides a way to install distributions, and to handle dependencies between releases. For now, this support is only about the last version of the METADATA (1.2) (See, the PEP 345), but I'm working on a compatibility layer for the old metadata, and for the informations provided via PIP requires.txt, for instance.

Extra work

Also, I've done some extra work. this includes:

working on the PEP 345, and having some discussion about it (about the names of some fields).
writing a PyPI server mock, useful for tests. you can find more information about it on the documentation.

Futures plans

As I said, I've enjoyed working on distutils2, and the people I've met here are really pleasant to work with. So I want to continue contributing on python, and especially on python packaging, because there is still a lot of things to do in this scope, to get something really usable.

I'm not plainly satisfied by the work I've done, so I'll probably tweak it a bit: the installer part is not yet completely finished, and I want to add support for a real REST index in the future.

We'll talk again of this in the next months, probably, but we definitely need a real REST API for PyPI, as the "simple" index is an ugly hack, in my opinion. I'll work on a serious proposition about this, maybe involving CouchDB, as it seems to be a good option for what we want here.

Issues

I've encountered some issues during this summer. The main one is that's hard to work remotely, especially being in the same room that we live, with others. I like to just think about a project with other people, a paper and a pencil, no computers. This have been not so possible at the start of the project, as I needed to read a lot of code to understand the codebase, and then to read/write emails.

I've finally managed to work in an office, so good point for home/office separation.

I'd not planned there will be so a high number of emails to read, in order to follow what's up in the python world, and be a part of the community seems to takes some times to read/write emails, especially for those (like me) that arent so confortable with english (but this had brought me some english fu !).

Thanks !

A big thanks to Graine Libre and Makina Corpus, which has offered me to come into their offices from time to time, to share they cheerfulness ! Many thanks too to the Google Summer of Code program for setting up such an initiative. If you're a student, if you're interested about FOSS, dont hesitate any second, it's a really good opportunity to work on interesting projects!

Introducing the distutils2 index crawlers

2010-07-06T00:00:00+02:00

I'm working for about a month for distutils2, even if I was being a bit busy (as I had some class courses and exams to work on)

I'll try do sum-up my general feelings here, and the work I've made so far. You can also find, if you're interested, my weekly summaries in a dedicated wiki page.

General feelings

First, and it's a really important point, the GSoC is going very well, for me as for other students, at least from my perspective. It's a pleasure to work with such enthusiast people, as this make the global atmosphere very pleasant to live.

First of all, I've spent time to read the existing codebase, and to understand what we're going to do, and what's the rationale to do so.

It's really clear for me now: what we're building is the foundations of a packaging infrastructure in python. The fact is that many projects co-exists, and comes all with their good concepts. Distutils2 tries to take the interesting parts of all, and to provide it in the python standard libs, respecting the recently written PEP about packaging.

With distutils2, it will be simpler to make "things" compatible. So if you think about a new way to deal with distributions and packaging in python, you can use the Distutils2 APIs to do so.

Tasks

My main task while working on distutils2 is to provide an installation and an un-installation command, as described in PEP 376. For this, I first need to get informations about the existing distributions (what's their version, name, metadata, dependencies, etc.)

The main index, you probably know and use, is PyPI. You can access it at http://pypi.python.org.

PyPI index crawling

There is two ways to get these informations from PyPI: using the simple API, or via xml-rpc calls.

A goal was to use the version specifiers defined in`PEP 345 <http://www.python.org/dev/peps/pep-0345/>`_ and to provides a way to sort the grabbed distributions depending our needs, to pick the version we want/need.

Using the simple API

The simple API is composed of HTML pages you can access at http://pypi.python.org/simple/.

Distribute and Setuptools already provides a crawler for that, but it deals with their internal mechanisms, and I found that the code was not so clear as I want, that's why I've preferred to pick up the good ideas, and some implementation details, plus re-thinking the global architecture.

The rules are simple: each project have a dedicated page, which allows us to get informations about:

the distribution download locations (for some versions)
homepage links
some other useful informations, as the bugtracker address, for instance.

If you want to find all the distributions of the "EggsAndSpam" project, you could do the following (do not take so attention to the names here, as the API will probably change a bit):

>>> index = SimpleIndex()
>>> index.find("EggsAndSpam")
[EggsAndSpam 1.1, EggsAndSpam 1.2, EggsAndSpam 1.3]

We also could use version specifiers:

>>> index.find("EggsAndSpam (< =1.2)")
[EggsAndSpam 1.1, EggsAndSpam 1.2]

Internally, what's done here is the following:

it process the http://pypi.python.org/simple/FooBar/ page, searching for download URLs.
for each found distribution download URL, it creates an object, containing informations about the project name, the version and the URL where the archive remains.
it sort the found distributions, using version numbers. The default behavior here is to prefer source distributions (over binary ones), and to rely on the last "final" distribution (rather than beta, alpha etc. ones)

So, nothing hard or difficult here.

We provides a bunch of other features, like relying on the new PyPI mirroring infrastructure or filter the found distributions by some criterias. If you're curious, please browse the distutils2 documentation.

Using xml-rpc

We also can make some xmlrpc calls to retreive informations from PyPI. It's a really more reliable way to get informations from from the index (as it's just the index that provides the informations), but cost processes on the PyPI distant server.

For now, this way of querying the xmlrpc client is not available on Distutils2, as I'm working on it. The main pieces are already present (I'll reuse some work I've made from the SimpleIndex querying, and some code already set up), what I need to do is to provide a xml-rpc PyPI mock server, and that's on what I'm actually working on.

Processes

For now, I'm trying to follow the "documentation, then test, then code" path, and that seems to be really needed while working with a community. Code is hard to read/understand, compared to documentation, and it's easier to change.

While writing the simple index crawling work, I must have done this to avoid some changes on the API, and some loss of time.

Also, I've set up a schedule, and the goal is to be sure everything will be ready in time, for the end of the summer. (And now, I need to learn to follow schedules ...)

Sprinting on distutils2 in Tours

2010-07-06T00:00:00+02:00

Yesterday, as I was traveling to Tours, I've took some time to visit Éric, another student who's working on distutils2 this summer, as a part of the GSoC. Basically, it was to take a drink, discuss a bit about distutils2, our respective tasks and general feelings, and to put a face on a pseudonym. I'd really enjoyed this time, because Éric knows a lot of things about mercurial and python good practices, and I'm eager to learn about those. So, we have discussed about things, have not wrote so much code, but have some things to propose so far, about documentation, and I also provides here some bribes of conversations we had.

Documentation

While writing the PyPI simple index crawler documentation, I realized that we miss some structure, or how-to about the documentation. Yep, you read well. We lack documentation on how to make documentation. Heh. We're missing some rules to follow, and this lead to a not-so-structured final documentation. We probably target three type of publics, and we can split the documentation regarding those:

Packagers who want to distribute their softwares.
End users who need to understand how to use end user commands, like the installer/uninstaller
packaging coders who use distutils2, as a base for building a package manager.

We also need to discuss about a pattern to follow while writing documentation. How many parts do we need ? Where to put the API description ? etc. That's maybe seems to be not so important, but I guess the readers would appreciate to have the same structure all along distutils2 documentation.

Mercurial

I'm really not a mercurial power user. I use it on daily basis, but I lack of basic knowledge about it. Big thanks Éric for sharing yours with me, you're of a great help. We have talked about some mercurial extensions that seems to make the life simpler, while used the right way. I've not used them so far, so consider this as a personal note.

hg histedit, to edit the history
hg crecord, to select the changes to commit

We have spent some time to review a merge I made sunday, to re-merge it, and commit the changes as a new changeset. Awesome. These things make me say I need to read the hg book, and will do as soon as I got some spare time: mercurial seems to be simply great. So ... Great. I'm a powerful merger now !

On using tools

Because we also are hackers, we have shared a bit our ways to code, the tools we use, etc. Both of us were using vim, and I've discovered vimdiff and hgtk, which will completely change the way I navigate into the mercurial history. We aren't "power users", so we have learned from each other about vim tips. You can find my dotfiles on github, if it could help. They're not perfect, and not intended to be, because changing all the time, as I learn. Don't hesitate to have a look, and to propose enhancements if you have !

On being pythonic

My background as an old Java user disserves me so far, as the paradigms are not the same while coding in python. Hard to find the more pythonic way to do, and sometimes hard to unlearn my way to think about software engineering. Well, it seems that the only solution is to read code, and to re-read import this from times to times ! Coding like a pythonista seems to be a must-read, so, I know what to do.

Conclusion

It was really great. Next time, we'll need to focus a bit more on distutils2, and to have a bullet list of things to do, but days like this one are opportunities to catch ! We'll probably do another sprint in a few weeks, stay tuned !

Use Restructured Text (ReST) to power your presentations

2010-06-25T00:00:00+02:00

Wednesday, we give a presentation, with some friends, about the CouchDB Database, to the Toulouse local LUG. Thanks a lot to all the presents for being there, it was a pleasure to talk about this topic with you. Too bad the season is over now an I quit Toulouse next year.

During our brainstorming about the topic, we used some paper, and we wanted to make a presentation the simpler way. First thing that come to my mind was using restructured text, so I've wrote a simple file containing our different bullet points. In fact, there is quite nothing to do then, to have a working presentation.

So far, I've used the rst2pdf program, and a simple template, to generate output. It's probably simple to have similar results using latex + beamer, I'll try this next time, but as I'm not familiar with latex syntax, restructured text was a great option.

Here are the final PDF output, Rhe ReST source, the theme used, and the command line to generate the PDF:

rst2pdf couchdb.rst -b1 -s ../slides.style

first week working on distutils2

2010-06-04T00:00:00+02:00

As I've been working on Distutils2 during the past week, taking part of the GSOC program, here is a short summary of what I've done so far.

As my courses are not over yet, I've not worked as much as I wanted, and this will continues until the end of June. My main tasks are about making installation and uninstallation commands, to have a simple way to install distributions via Distutils2.

To do this, we need to rely on informations provided by the Python Package Index (PyPI), and there is at least two ways to retreive informations from here: XML-RPC and the "simple" API.

So, I've been working on porting some Distribute related stuff to Distutils2, cutting off all non distutils' things, as we do not want to depend from Distribute's internals. My main work has been about reading the whole code, writing tests about this and making those tests possible.

In fact, there was a need of a pypi mocked server, and, after reading and introducing myself to the distutils behaviors and code, I've taken some time to improve the work Konrad makes about this mock.

A PyPI Server mock

The mock is embeded in a thread, to make it available during the tests, in a non blocking way. We first used WSGI and wsgiref in order control what to serve, and to log the requests made to the server, but finally realised that wsgiref is not python 2.4 compatible (and we need to be python 2.4 compatible in Distutils2).

So, we switched to BaseHTTPServer and SimpleHTTPServer, and updated our tests accordingly. It's been an opportunity to realize that WSGI has been a great step forward for making HTTP servers, and expose a really simplest way to discuss with HTTP !

You can find the modifications I made, and the related docs about this on my bitbucket distutils2 clone.

The PyPI Simple API

So, back to the main problematic: make a python library to access and request information stored on PyPI, via the simple API. As I said, I've just grabbed the work made from Distribute, and played a bit with, in order to view what are the different use cases, and started to write the related tests.

The work to come

So, once all use cases covered with tests, I'll rewrite a bit the grabbed code, and do some software design work (to not expose all things as privates methods, have a clear API, and other things like this), then update the tests accordingly and write a documentation to make this clear.

Next step is to a little client, as I've already started here I'll take you updated !

A Distutils2 GSoC

2010-05-01T00:00:00+02:00

WOW. I've been accepted to be a part of the Google Summer Of Code program, and will work on python distutils2, with a lot of (intersting!) people.

So, it's about building the successor of Distutils2, ie. "the python package manager". Today, there is too many ways to package a python application (pip, setuptools, distribute, distutils, etc.) so there is a huge effort to make in order to make all this packaging stuff interoperable, as pointed out by the PEP 376.

In more details, I'm going to work on the Installer / Uninstaller features of Distutils2, and on a PyPI XML-RPC client for distutils2. Here are the already defined tasks:

Implement Distutils2 APIs described in PEP 376.
Add the uninstall command.
think about a basic installer / uninstaller script. (with deps) -- similar to pip/easy_install
in a pypi subpackage;
Integrate a module similar to setuptools' package_index'
PyPI XML-RPC client for distutils 2: http://bugs.python.org/issue8190

As I'm relatively new to python, I'll need some extra work in order to apply all good practice, among other things that can make a developper-life joyful. I'll post here, each week, my advancement, and my tought about python and especialy python packaging world.

Python ? go !

2009-12-17T00:00:00+01:00

Cela fait maintenant un peu plus d'un mois que je travaille sur un projet en django, et que, nécessairement, je me forme à Python. Je prends un plaisir non dissimulé à découvrir ce langage (et à l'utiliser), qui ne cesse de me surprendre. Les premiers mots qui me viennent à l'esprit à propos de Python, sont "logique" et "simple". Et pourtant puissant pour autant. Je ne manque d'ailleurs pas une occasion pour faire un peu d'évangélisation auprès des quelques personnes qui veulent bien m'écouter.

The Zen of Python

Avant toute autre chose, je pense utile de citer Tim Peters, et le PEP20, qui constituent une très bonne introduction au langage, qui prends la forme d'un easter egg présent dans python

>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

J'ai la vague impression que c'est ce que j'ai toujours cherché à faire en PHP, et particulièrement dans le framework Spiral, mais en ajoutant ces concepts dans une sur-couche au langage. Ici, c'est directement de l'esprit de python qu'il s'agit, ce qui signifie que la plupart des bibliothèques python suivent ces concepts. Elle est pas belle la vie ?

Comment commencer, et par ou ?

Pour ma part, j'ai commencé par la lecture de quelques livres et articles intéressants, qui constituent une bonne entrée en matière sur le sujet (La liste n'est bien évidemment pas exhaustive et vos commentaires sont les bienvenus) :

Dive into python
A byte of python
Python: petit guide à l'usage du développeur agile de Tarek Ziadé
La documentation officielle python, bien sûr !
Les vidéos du pyconfr 2009!
Un peu de temps, et une console python ouverte :)

J'essaye par ailleurs de partager au maximum les ressources que je trouve de temps à autres, que ce soit via twitter ou via mon compte delicious. Allez jeter un œil au tag python sur mon profil, peut être que vous trouverez des choses intéressantes, qui sait!

Un python sexy

Quelques fonctionnalités qui devraient vous mettre l'eau à la bouche:

Le chaînage des opérateurs de comparaison est possible (a<b <c dans une condition)
Assignation de valeurs multiples (il est possible de faire a,b,c = 1,2,3 par exemple)
Les listes sont simples à manipuler !
Les list comprehension, ou comment faire des opérations complexes sur les listes, de manière simple.
Les doctests: ou comment faire des tests directement dans la documentation de vos classes, tout en la documentant avec de vrais exemples.
Les métaclasses, ou comment contrôler la manière dont les classes se construisent
Python est un langage à typage fort dynamique: c'est ce qui m'agaçait avec PHP qui est un langage à typage faible dynamique.

Cous pouvez également aller regarder l'atelier donné par Victor Stinner durant le Pyconfr 09. Have fun !