diff --git a/content/python/caching-elastic-search.rst b/content/python/caching-elastic-search.rst index c95870c..fa3c7ec 100644 --- a/content/python/caching-elastic-search.rst +++ b/content/python/caching-elastic-search.rst @@ -3,10 +3,34 @@ How to cache Elastic Seach Queries? :status: other +These days, I'm working on Marketplace, the Mozilla's Appstore. Internally, +we're doing Elastic Search to do search, and after some load tests, we +eventually found out that Elastic Search was our bottleneck. + +So we want to reduce the number of requests we do to this server. Currently, +the requests are done trough HTTP. + The first thing to realize is what do we want to cache exactly? There is a fair bit of things we might want to cache. Let's start by the most queried pages: the home and the list of apps per category. +Different approaches +==================== + +You can put the cache in many different locations. The code powering +Marketplace is kinda fuzzy sometimes. The requests to Elastic Search are done +in a number of different parts of the code. They're done sometimes directly +with HTTP calls, sometimes using the ElasticUtils library, sometimes using some +other lib… + +That's kind of hard to get where and how to add the caching layer here. What +did we do? We started to work on an HTTP caching proxy. This proxy could + +Find a key +========== + + + Caching things ============== @@ -23,7 +47,6 @@ Back to business logic: I want to cache the request that are done on the homepage. The code currently looks like this:: popular = Webapp.popular(region=region, gaia=request.GAIA)[:10] - latest = Webapp.latest(region=region, gaia=request.GAIA)[:10] Nothing fancy going on here, we're displaying the list of popular and latest apps on the marketplace. I can cache the results for each region, and depending @@ -50,8 +73,8 @@ Which I can use like this:: popular = cache(Webapp.popular(region=region, gaia=request.GAIA)[:10], 'popular', *keys) -Caching is easy, invalidation is hard -===================================== +Invalidation is hard? +===================== Right, we got this cached, good. But what happens when the data we cached changes? Currently, nothing, we return the same data over and over again. @@ -65,8 +88,5 @@ We can use different strategies for this: * Invalidate manually the cache when something changes, for instance using signals. -Here, obviously, we don't want to invalidate the cache each time we're adding -a new rating; what we'll do is to invalidate the cache manually, when we -compute the new popularity of the addons. - -In this case, this is done by a command. +I started by having a look at how I wanted to invalidate all of this, case by +case. The problem diff --git a/content/python/cheese-and-code-result.rst b/content/python/cheese-and-code-result.rst new file mode 100644 index 0000000..60c04e2 --- /dev/null +++ b/content/python/cheese-and-code-result.rst @@ -0,0 +1,142 @@ +Cheese & code - Wrap-up +####################### + +:status: draft +:date: 2012-10-22 + +This week-end I hosted a *cheese & code* session in the country-side of Angers, +France. + +We were a bunch of python hackers and it rained a lot, wich forced us to stay +inside and to code. Bad. + +We were not enough to get rid of all the cheese and the awesome meals, but +well, we finally managed it pretty well. + +Here is a summary of what we worked on: + +Daybed +------ + +Daybed started some time ago, and intend to be a replacement to google forms, +in term of features, but backed as a REST web service, in python, and open +source. + +In case you wonder, daybed is effectively the name of a couch. We chose this +name because of the similarities (in the sound) with **db**, and because +we're using **CouchDB** as a backend. + +.. image:: images/daybed.jpg + :width: 400px + +We mainly hacked on daybed and are pretty close to the release of the first +version, meaning that we have something working. + +`The code `_ is available on github, +and we also wrote `a small documentation `_ for it. + +Mainly, we did a lot of cleanup, rewrote a bunch of tests so that it would be +easier to continue to work on the project, and implemented some minor features. +I'm pretty confidend that we now have really good basis for this project. + +Also, we will have a nice todolist application, with the backend **and** the +frontend, in javascript / html / css, you'll know more when it'll be ready :-) + +Once we have something good enough, we'll release the first version and I'll +host it somewhere so that people can play with it. + +Cornice +------- + +Daybed is built on top of `Cornice `_, a framework to +ease the creation of web-services. + +At Pycon France, we had the opportunity to attend a good presentation about `SPORE +`_. SPORE is a way to describe your +REST web services, as WSDL is for WS-* services. This allows to ease the +creation of generic SPORE clients, which are able to consume any REST API with +a SPORE endpoint. + +Here is how you can let cornice describe your web service for you + +.. code-block:: python + + from cornice.ext.spore import generate_spore_description + from cornice.service import Service, get_services + + spore = Service('spore', path='/spore', renderer='jsonp') + @spore.get + def get_spore(request): + services = get_services() + return generate_spore_description(services, 'Service name', + request.application_url, '1.0') + +And you'll get a definition of your service, in SPORE, available at `/spore`. + +Of course, you can use it to do other things, like generating the file locally +and exporting it wherever it makes sense to you, etc. + +I released today `Cornice 0.11 `_, which adds +into other things the support for SPORE, plus some other fixes we found on our +way. + +Respire +------- + +Once you have the description of the service, you can do generic clients +consuming them! + +We first wanted to contribute to `spyre `_ but +it was written in a way that wasn't supporting to `POST` data, and they +were using their own stack to handle HTTP. A lot of code that already exists in +other libraries. + +While waiting the train with `Rémy `_, we hacked +something together, named "Respire", a thin layer on top of the awesome +`Requests `_ library. + +We have a first version, feel free to have a look at it and provide +enhancements if you feel like it. We're still hacking on it so it may break +(for the better), but that had been working pretty well for us so far. + +You can `find the project on github +`_, but here is how to use it, really +quickly (these examples are how to interact with daybed) + +.. code-block:: python + + >>> from respire import client_from_url + + >>> # create the client from the SPORE definition + >>> cl = client_from_url('http://localhost:8000/spore') + + >>> # in daybed, create a new definition + >>> todo_def = { + ... "title": "todo", + ... "description": "A list of my stuff to do", + ... "fields": [ + ... { + ... "name": "item", + ... "type": "string", + ... "description": "The item" + ... }, + ... { + ... "name": "status", + ... "type": "enum", + ... "choices": [ + ... "done", + ... "todo" + ... ], + ... "description": "is it done or not" + ... } + ... ]} + >>> cl.put_definition(model_name='todo', data=todo_def) + >>> cl.post_data(model_name='todo', data=dict(item='make it work', status='todo')) + {u'id': u'9f2c90c0529a442cfdc03c191b022cf7'} + >>> cl.get_data(model_name='todo') + + +Finally, we were out of cheese so everyone headed back to their respective +houses and cities. + +Until next time?