mirror of
https://github.com/almet/notmyidea.git
synced 2025-04-28 11:32:39 +02:00
Remove useless files
This commit is contained in:
parent
615a90efee
commit
3b65203171
5 changed files with 1 additions and 263 deletions
|
|
@ -1,92 +0,0 @@
|
|||
How to cache Elastic Seach Queries?
|
||||
###################################
|
||||
|
||||
:status: other
|
||||
|
||||
These days, I'm working on Marketplace, the Mozilla's Appstore. Internally,
|
||||
we're doing Elastic Search to do search, and after some load tests, we
|
||||
eventually found out that Elastic Search was our bottleneck.
|
||||
|
||||
So we want to reduce the number of requests we do to this server. Currently,
|
||||
the requests are done trough HTTP.
|
||||
|
||||
The first thing to realize is what do we want to cache exactly? There is a fair
|
||||
bit of things we might want to cache. Let's start by the most queried pages:
|
||||
the home and the list of apps per category.
|
||||
|
||||
Different approaches
|
||||
====================
|
||||
|
||||
You can put the cache in many different locations. The code powering
|
||||
Marketplace is kinda fuzzy sometimes. The requests to Elastic Search are done
|
||||
in a number of different parts of the code. They're done sometimes directly
|
||||
with HTTP calls, sometimes using the ElasticUtils library, sometimes using some
|
||||
other lib…
|
||||
|
||||
That's kind of hard to get where and how to add the caching layer here. What
|
||||
did we do? We started to work on an HTTP caching proxy. This proxy could
|
||||
|
||||
Find a key
|
||||
==========
|
||||
|
||||
|
||||
|
||||
Caching things
|
||||
==============
|
||||
|
||||
Caching is easy, it's something like that, in term of python code::
|
||||
|
||||
if key in cache:
|
||||
value = cache.get(key)
|
||||
else:
|
||||
value = do_the_request()
|
||||
cache.set(key, value)
|
||||
return value
|
||||
|
||||
Back to business logic: I want to cache the request that are done on the
|
||||
homepage. The code currently looks like this::
|
||||
|
||||
popular = Webapp.popular(region=region, gaia=request.GAIA)[:10]
|
||||
|
||||
Nothing fancy going on here, we're displaying the list of popular and latest
|
||||
apps on the marketplace. I can cache the results for each region, and depending
|
||||
if `request.GAIA` is `True` or not. The key will look like `popular-{region}`,
|
||||
to which I will eventually append `-gaia` if it makes sense. More generally,
|
||||
I will turn a number of criterias into a single key, like this::
|
||||
|
||||
def cache(qs, *keys):
|
||||
"""Cache the given querystring"""
|
||||
key = '-'.join(keys)
|
||||
if key in cache:
|
||||
result = cache.get(key)
|
||||
else:
|
||||
result = qs
|
||||
cache.set(key, result)
|
||||
return result
|
||||
|
||||
Which I can use like this::
|
||||
|
||||
keys = [region.slug, ]
|
||||
if request.GAIA:
|
||||
keys.append('gaia')
|
||||
|
||||
popular = cache(Webapp.popular(region=region, gaia=request.GAIA)[:10],
|
||||
'popular', *keys)
|
||||
|
||||
Invalidation is hard?
|
||||
=====================
|
||||
|
||||
Right, we got this cached, good. But what happens when the data we cached
|
||||
changes? Currently, nothing, we return the same data over and over again.
|
||||
|
||||
We need to tell our application when this cache isn't good anymore; we need to
|
||||
*invalidate* it.
|
||||
|
||||
We can use different strategies for this:
|
||||
|
||||
* Set a timeout for the cache.
|
||||
* Invalidate manually the cache when something changes, for instance using
|
||||
signals.
|
||||
|
||||
I started by having a look at how I wanted to invalidate all of this, case by
|
||||
case. The problem
|
||||
|
|
@ -1,58 +0,0 @@
|
|||
API description using cornice and colander
|
||||
##########################################
|
||||
|
||||
:status: draft
|
||||
|
||||
One of the goals we want to accomplish with Cornice is to describe the web
|
||||
services in a nice and easy way, for the developer. This for different reasons.
|
||||
|
||||
Having web services described in some ways allows you do do interesting things,
|
||||
such as automatically generating a documentation, or a client, for instance.
|
||||
REST services are supposed to be discoverable, but in a lot of situations,
|
||||
that's not the case, and it can be a pain to implement a client for them, even
|
||||
if most of what is being done there is shared between a lot of web services.
|
||||
|
||||
In cornice, we already generate some documentation, but this one is incomplete in
|
||||
different ways. For instance, it is currently not possible to get information
|
||||
about the inputs you are waiting for, in the different locations (body,
|
||||
headers and query string).
|
||||
|
||||
For instance, if you want to describe that a *POST* on `/foobar` needs the
|
||||
*foo* and *bar* parameters, and that you eventually could add a *baz*
|
||||
parameter, you can't, or you have to describe it manually in the documentation
|
||||
of the web service.
|
||||
|
||||
Cornice, while not tied to `Colander`_, tries to make a good use of it when
|
||||
possible. So it is possible to do validation of the incoming data (in different
|
||||
locations) and, at the same time, documentation of the web service.
|
||||
|
||||
To do so, you have to use a particular `schema` keyword in the decorator. Here
|
||||
is an example of definition
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from cornice import Service
|
||||
from colander import MappingSchema, SchemaNode, String
|
||||
|
||||
foobar = Service(name="foobar", path="/foobar")
|
||||
|
||||
|
||||
class Foobar(MappingSchema):
|
||||
foo = SchemaNode(String(), location="body")
|
||||
bar = SchemaNode(String(), location="headers")
|
||||
baz = SchemaNode(String(), default=None)
|
||||
|
||||
|
||||
@foobar.get(scheme=Foobar)
|
||||
def foobar_get(request):
|
||||
# do something here
|
||||
|
||||
.. _Colander: http://docs.pylonsproject.org/projects/colander/en/latest/
|
||||
|
||||
The output documentation currently looks like this:
|
||||
|
||||
.. image:: images/cornice-exampledoc-validation.png
|
||||
|
||||
The plans for the future are to be able to put direct curl calls in the
|
||||
documentation to demonstrate how the service behaves for instance, and why not
|
||||
creating a tiny python client able to consume the cornice APIs.
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
L'heure pour un vote electronique à distance ?
|
||||
##############################################
|
||||
|
||||
Ce matin, je me fais reveiller par le reveil. La radio. Je suis chez des amis,
|
||||
à Paris, et le Front National à fait des scores incroyables. Du jamais vu.
|
||||
|
||||
En prenant le RER, je croise le regard de ces gens, probablements immigrés, qui
|
||||
semblent abrutis par la situation. Je suis le fils d'une émigrée et je partage
|
||||
leur rage, leur haine. Derrière ces beaux discours, le Front National souhaite
|
||||
fermer les frontières, controler qui rentre pour les laisser chez eux, ces
|
||||
personnes qui ont eu l'infortune de naitre ailleurs, la ou la terre est moins
|
||||
propice.
|
||||
|
||||
Je me sens coupable, ce matin. Coupable de ne pas avoir été voter hier. Dans ma
|
||||
bulle j'étais, absent. Cela fait plusieurs années que je ne vote plus, par
|
||||
manque de foi en la politique qui est menée actuellement, et par dépis. J'attends,
|
||||
mollement, le second tour pour exprimer mon suffrage. Pour ce qu'il vaut. Je ne
|
||||
crois pas en la representativité sous sa forme actuelle.
|
||||
|
||||
Mais ce matin, je m'en veux quand même. Même sans croire à ce système de
|
||||
representativité, j'aurais pu éviter que ces idées racistes et haineuses
|
||||
n'étendent leur pouvoir à d'autres régions.
|
||||
|
||||
Il est trop tard pour changer la donne, le mal est fait. Mais j'aurais pu aller
|
||||
voter. J'aurais du même.
|
||||
|
||||
La geule encore pateuse, au petit déjeuner, je me questionne sur mon
|
||||
attachement à un territoir, et aux consequences de ce non-attachement sur mon
|
||||
envie de voter.
|
||||
|
|
@ -1,83 +0,0 @@
|
|||
walint - tilt
|
||||
#############
|
||||
|
||||
:status: draft
|
||||
|
||||
Things start to fold together. I still need some time to be up to
|
||||
speed (new job / new city / new people), but the projects I'm working on are
|
||||
pretty exciting and I like what I'm doing. We've been working with Tarek on a
|
||||
tool about web services testing.
|
||||
|
||||
At Mozilla, we produce and consume web services. As an organisation defending
|
||||
web standards, providing services that match these specifications is the least
|
||||
we can do.
|
||||
|
||||
Web App Lint (WALint) is a simple tool to help you to test your web services
|
||||
against the HTTP specification. The goal is to detect potential problems a
|
||||
service could be subject to, and to provide a simple-to-configure tool to test
|
||||
your WS against HTTP as speced. The long term goal is to be exhaustive on the
|
||||
controllers we provide, but if you have special needs, you should be able to
|
||||
define tests in a simple way.
|
||||
|
||||
As everything we do at Mozilla, the tool is an opensource project. Our goal is
|
||||
to have something useful for web service consumers and providers. If you think
|
||||
you can be of any help, don't hesitate to drop me an email or fork on github.
|
||||
|
||||
Testing the HTTP specification
|
||||
==============================
|
||||
|
||||
A lot of web services are written on top of HTTP. The protocol is fully
|
||||
described in a specification and introduce some really interesting stuff we
|
||||
should care about when implementing our services. This tool wants to check this
|
||||
up and provide a ways to ensure that you are defining your web services the
|
||||
right way. It's not a replacement for manual service testing, thus: you still
|
||||
have to test that your web service is doing what you think it does. WALint is
|
||||
checking the stuff we almost forgot all of the time.
|
||||
|
||||
For instance, the HTTP specification specifies that if the client sends a
|
||||
request with a Content-Type not handled by the server, it should answer with a
|
||||
"406 Not Acceptable" and should return the list of accepted headers (so the
|
||||
client is able to send again the request with the right accept header). We also
|
||||
check that sending weird data (broken json objects or authentication headers)
|
||||
does not break the tested service.
|
||||
|
||||
Obviously, we don't have all the scenarios in mind, and they aren't all
|
||||
implemented. That's where being an open source project makes sense. You, as
|
||||
service providers and consumers, know what are the kind of mistake you are used
|
||||
to do / to deal with. It should make "tilt": fork the project and write some
|
||||
lines checking the particular behavior that's disturbing you, everyone will
|
||||
enjoy it. If you prefer, just open a ticket and we'll care about the
|
||||
implementation.
|
||||
|
||||
The documentation defines a list of all the implemented controllers:
|
||||
http://packages.python.org/walint/controllers.html
|
||||
|
||||
What does it looks like?
|
||||
========================
|
||||
|
||||
Here is an example of a configuration file: you describe the paths and
|
||||
signatures of your web services, the controllers you want to use and the test
|
||||
scenarios you want to run, that's it, just run "walint config.cfg" and you're
|
||||
good.
|
||||
|
||||
Here's a capture of the output of the command line invocation: you get all the
|
||||
paths tested with the different listed methods, and if needed what's not
|
||||
working. In the future, why not putting some documentation online with
|
||||
information about how to fix common mistakes?
|
||||
|
||||
We also provide a wizard so it's possible to describe your web service in a
|
||||
simple and easy way. Just invoke it with "walint --create config.cfg".
|
||||
|
||||
What's next ?
|
||||
=============
|
||||
|
||||
We plan to do some fun stuff with WALint. Here are some thoughts:
|
||||
|
||||
* service discovery: just provide an API root URI and we'll try to discover
|
||||
what's in there and how to generate configuration files
|
||||
|
||||
* unittest integration, so it's possible to have an unified way to describe web
|
||||
services with existing tools. (heh, we can do Web Services CI the easy way!)
|
||||
* integration with cornice description language and other web services description
|
||||
languages (still to be defined)
|
||||
* your ideas, so please fill issues and provide us feedback, it's useful!
|
||||
|
|
@ -23,7 +23,7 @@ LOCALE = "fr_FR.utf8"
|
|||
DEFAULT_DATE_FORMAT = ('%d %B %Y')
|
||||
LINKS = [
|
||||
('Brasserie du Vieux Singe', 'https://www.vieuxsinge.com'),
|
||||
('A propos', 'http://blog.notmyidea.org/pages/about.html')
|
||||
('A propos', 'http://blog.notmyidea.org/pages/about.html'),
|
||||
('Messages courts', 'https://twitter.com/ametaireau'),
|
||||
('Code', 'https://github.com/almet'),
|
||||
]
|
||||
|
|
|
|||
Loading…
Reference in a new issue