mirror of
https://github.com/almet/notmyidea.git
synced 2025-04-28 19:42:37 +02:00
Remove useless files
This commit is contained in:
parent
615a90efee
commit
3b65203171
5 changed files with 1 additions and 263 deletions
|
@ -1,92 +0,0 @@
|
||||||
How to cache Elastic Seach Queries?
|
|
||||||
###################################
|
|
||||||
|
|
||||||
:status: other
|
|
||||||
|
|
||||||
These days, I'm working on Marketplace, the Mozilla's Appstore. Internally,
|
|
||||||
we're doing Elastic Search to do search, and after some load tests, we
|
|
||||||
eventually found out that Elastic Search was our bottleneck.
|
|
||||||
|
|
||||||
So we want to reduce the number of requests we do to this server. Currently,
|
|
||||||
the requests are done trough HTTP.
|
|
||||||
|
|
||||||
The first thing to realize is what do we want to cache exactly? There is a fair
|
|
||||||
bit of things we might want to cache. Let's start by the most queried pages:
|
|
||||||
the home and the list of apps per category.
|
|
||||||
|
|
||||||
Different approaches
|
|
||||||
====================
|
|
||||||
|
|
||||||
You can put the cache in many different locations. The code powering
|
|
||||||
Marketplace is kinda fuzzy sometimes. The requests to Elastic Search are done
|
|
||||||
in a number of different parts of the code. They're done sometimes directly
|
|
||||||
with HTTP calls, sometimes using the ElasticUtils library, sometimes using some
|
|
||||||
other lib…
|
|
||||||
|
|
||||||
That's kind of hard to get where and how to add the caching layer here. What
|
|
||||||
did we do? We started to work on an HTTP caching proxy. This proxy could
|
|
||||||
|
|
||||||
Find a key
|
|
||||||
==========
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
Caching things
|
|
||||||
==============
|
|
||||||
|
|
||||||
Caching is easy, it's something like that, in term of python code::
|
|
||||||
|
|
||||||
if key in cache:
|
|
||||||
value = cache.get(key)
|
|
||||||
else:
|
|
||||||
value = do_the_request()
|
|
||||||
cache.set(key, value)
|
|
||||||
return value
|
|
||||||
|
|
||||||
Back to business logic: I want to cache the request that are done on the
|
|
||||||
homepage. The code currently looks like this::
|
|
||||||
|
|
||||||
popular = Webapp.popular(region=region, gaia=request.GAIA)[:10]
|
|
||||||
|
|
||||||
Nothing fancy going on here, we're displaying the list of popular and latest
|
|
||||||
apps on the marketplace. I can cache the results for each region, and depending
|
|
||||||
if `request.GAIA` is `True` or not. The key will look like `popular-{region}`,
|
|
||||||
to which I will eventually append `-gaia` if it makes sense. More generally,
|
|
||||||
I will turn a number of criterias into a single key, like this::
|
|
||||||
|
|
||||||
def cache(qs, *keys):
|
|
||||||
"""Cache the given querystring"""
|
|
||||||
key = '-'.join(keys)
|
|
||||||
if key in cache:
|
|
||||||
result = cache.get(key)
|
|
||||||
else:
|
|
||||||
result = qs
|
|
||||||
cache.set(key, result)
|
|
||||||
return result
|
|
||||||
|
|
||||||
Which I can use like this::
|
|
||||||
|
|
||||||
keys = [region.slug, ]
|
|
||||||
if request.GAIA:
|
|
||||||
keys.append('gaia')
|
|
||||||
|
|
||||||
popular = cache(Webapp.popular(region=region, gaia=request.GAIA)[:10],
|
|
||||||
'popular', *keys)
|
|
||||||
|
|
||||||
Invalidation is hard?
|
|
||||||
=====================
|
|
||||||
|
|
||||||
Right, we got this cached, good. But what happens when the data we cached
|
|
||||||
changes? Currently, nothing, we return the same data over and over again.
|
|
||||||
|
|
||||||
We need to tell our application when this cache isn't good anymore; we need to
|
|
||||||
*invalidate* it.
|
|
||||||
|
|
||||||
We can use different strategies for this:
|
|
||||||
|
|
||||||
* Set a timeout for the cache.
|
|
||||||
* Invalidate manually the cache when something changes, for instance using
|
|
||||||
signals.
|
|
||||||
|
|
||||||
I started by having a look at how I wanted to invalidate all of this, case by
|
|
||||||
case. The problem
|
|
|
@ -1,58 +0,0 @@
|
||||||
API description using cornice and colander
|
|
||||||
##########################################
|
|
||||||
|
|
||||||
:status: draft
|
|
||||||
|
|
||||||
One of the goals we want to accomplish with Cornice is to describe the web
|
|
||||||
services in a nice and easy way, for the developer. This for different reasons.
|
|
||||||
|
|
||||||
Having web services described in some ways allows you do do interesting things,
|
|
||||||
such as automatically generating a documentation, or a client, for instance.
|
|
||||||
REST services are supposed to be discoverable, but in a lot of situations,
|
|
||||||
that's not the case, and it can be a pain to implement a client for them, even
|
|
||||||
if most of what is being done there is shared between a lot of web services.
|
|
||||||
|
|
||||||
In cornice, we already generate some documentation, but this one is incomplete in
|
|
||||||
different ways. For instance, it is currently not possible to get information
|
|
||||||
about the inputs you are waiting for, in the different locations (body,
|
|
||||||
headers and query string).
|
|
||||||
|
|
||||||
For instance, if you want to describe that a *POST* on `/foobar` needs the
|
|
||||||
*foo* and *bar* parameters, and that you eventually could add a *baz*
|
|
||||||
parameter, you can't, or you have to describe it manually in the documentation
|
|
||||||
of the web service.
|
|
||||||
|
|
||||||
Cornice, while not tied to `Colander`_, tries to make a good use of it when
|
|
||||||
possible. So it is possible to do validation of the incoming data (in different
|
|
||||||
locations) and, at the same time, documentation of the web service.
|
|
||||||
|
|
||||||
To do so, you have to use a particular `schema` keyword in the decorator. Here
|
|
||||||
is an example of definition
|
|
||||||
|
|
||||||
.. code-block:: python
|
|
||||||
|
|
||||||
from cornice import Service
|
|
||||||
from colander import MappingSchema, SchemaNode, String
|
|
||||||
|
|
||||||
foobar = Service(name="foobar", path="/foobar")
|
|
||||||
|
|
||||||
|
|
||||||
class Foobar(MappingSchema):
|
|
||||||
foo = SchemaNode(String(), location="body")
|
|
||||||
bar = SchemaNode(String(), location="headers")
|
|
||||||
baz = SchemaNode(String(), default=None)
|
|
||||||
|
|
||||||
|
|
||||||
@foobar.get(scheme=Foobar)
|
|
||||||
def foobar_get(request):
|
|
||||||
# do something here
|
|
||||||
|
|
||||||
.. _Colander: http://docs.pylonsproject.org/projects/colander/en/latest/
|
|
||||||
|
|
||||||
The output documentation currently looks like this:
|
|
||||||
|
|
||||||
.. image:: images/cornice-exampledoc-validation.png
|
|
||||||
|
|
||||||
The plans for the future are to be able to put direct curl calls in the
|
|
||||||
documentation to demonstrate how the service behaves for instance, and why not
|
|
||||||
creating a tiny python client able to consume the cornice APIs.
|
|
|
@ -1,29 +0,0 @@
|
||||||
L'heure pour un vote electronique à distance ?
|
|
||||||
##############################################
|
|
||||||
|
|
||||||
Ce matin, je me fais reveiller par le reveil. La radio. Je suis chez des amis,
|
|
||||||
à Paris, et le Front National à fait des scores incroyables. Du jamais vu.
|
|
||||||
|
|
||||||
En prenant le RER, je croise le regard de ces gens, probablements immigrés, qui
|
|
||||||
semblent abrutis par la situation. Je suis le fils d'une émigrée et je partage
|
|
||||||
leur rage, leur haine. Derrière ces beaux discours, le Front National souhaite
|
|
||||||
fermer les frontières, controler qui rentre pour les laisser chez eux, ces
|
|
||||||
personnes qui ont eu l'infortune de naitre ailleurs, la ou la terre est moins
|
|
||||||
propice.
|
|
||||||
|
|
||||||
Je me sens coupable, ce matin. Coupable de ne pas avoir été voter hier. Dans ma
|
|
||||||
bulle j'étais, absent. Cela fait plusieurs années que je ne vote plus, par
|
|
||||||
manque de foi en la politique qui est menée actuellement, et par dépis. J'attends,
|
|
||||||
mollement, le second tour pour exprimer mon suffrage. Pour ce qu'il vaut. Je ne
|
|
||||||
crois pas en la representativité sous sa forme actuelle.
|
|
||||||
|
|
||||||
Mais ce matin, je m'en veux quand même. Même sans croire à ce système de
|
|
||||||
representativité, j'aurais pu éviter que ces idées racistes et haineuses
|
|
||||||
n'étendent leur pouvoir à d'autres régions.
|
|
||||||
|
|
||||||
Il est trop tard pour changer la donne, le mal est fait. Mais j'aurais pu aller
|
|
||||||
voter. J'aurais du même.
|
|
||||||
|
|
||||||
La geule encore pateuse, au petit déjeuner, je me questionne sur mon
|
|
||||||
attachement à un territoir, et aux consequences de ce non-attachement sur mon
|
|
||||||
envie de voter.
|
|
|
@ -1,83 +0,0 @@
|
||||||
walint - tilt
|
|
||||||
#############
|
|
||||||
|
|
||||||
:status: draft
|
|
||||||
|
|
||||||
Things start to fold together. I still need some time to be up to
|
|
||||||
speed (new job / new city / new people), but the projects I'm working on are
|
|
||||||
pretty exciting and I like what I'm doing. We've been working with Tarek on a
|
|
||||||
tool about web services testing.
|
|
||||||
|
|
||||||
At Mozilla, we produce and consume web services. As an organisation defending
|
|
||||||
web standards, providing services that match these specifications is the least
|
|
||||||
we can do.
|
|
||||||
|
|
||||||
Web App Lint (WALint) is a simple tool to help you to test your web services
|
|
||||||
against the HTTP specification. The goal is to detect potential problems a
|
|
||||||
service could be subject to, and to provide a simple-to-configure tool to test
|
|
||||||
your WS against HTTP as speced. The long term goal is to be exhaustive on the
|
|
||||||
controllers we provide, but if you have special needs, you should be able to
|
|
||||||
define tests in a simple way.
|
|
||||||
|
|
||||||
As everything we do at Mozilla, the tool is an opensource project. Our goal is
|
|
||||||
to have something useful for web service consumers and providers. If you think
|
|
||||||
you can be of any help, don't hesitate to drop me an email or fork on github.
|
|
||||||
|
|
||||||
Testing the HTTP specification
|
|
||||||
==============================
|
|
||||||
|
|
||||||
A lot of web services are written on top of HTTP. The protocol is fully
|
|
||||||
described in a specification and introduce some really interesting stuff we
|
|
||||||
should care about when implementing our services. This tool wants to check this
|
|
||||||
up and provide a ways to ensure that you are defining your web services the
|
|
||||||
right way. It's not a replacement for manual service testing, thus: you still
|
|
||||||
have to test that your web service is doing what you think it does. WALint is
|
|
||||||
checking the stuff we almost forgot all of the time.
|
|
||||||
|
|
||||||
For instance, the HTTP specification specifies that if the client sends a
|
|
||||||
request with a Content-Type not handled by the server, it should answer with a
|
|
||||||
"406 Not Acceptable" and should return the list of accepted headers (so the
|
|
||||||
client is able to send again the request with the right accept header). We also
|
|
||||||
check that sending weird data (broken json objects or authentication headers)
|
|
||||||
does not break the tested service.
|
|
||||||
|
|
||||||
Obviously, we don't have all the scenarios in mind, and they aren't all
|
|
||||||
implemented. That's where being an open source project makes sense. You, as
|
|
||||||
service providers and consumers, know what are the kind of mistake you are used
|
|
||||||
to do / to deal with. It should make "tilt": fork the project and write some
|
|
||||||
lines checking the particular behavior that's disturbing you, everyone will
|
|
||||||
enjoy it. If you prefer, just open a ticket and we'll care about the
|
|
||||||
implementation.
|
|
||||||
|
|
||||||
The documentation defines a list of all the implemented controllers:
|
|
||||||
http://packages.python.org/walint/controllers.html
|
|
||||||
|
|
||||||
What does it looks like?
|
|
||||||
========================
|
|
||||||
|
|
||||||
Here is an example of a configuration file: you describe the paths and
|
|
||||||
signatures of your web services, the controllers you want to use and the test
|
|
||||||
scenarios you want to run, that's it, just run "walint config.cfg" and you're
|
|
||||||
good.
|
|
||||||
|
|
||||||
Here's a capture of the output of the command line invocation: you get all the
|
|
||||||
paths tested with the different listed methods, and if needed what's not
|
|
||||||
working. In the future, why not putting some documentation online with
|
|
||||||
information about how to fix common mistakes?
|
|
||||||
|
|
||||||
We also provide a wizard so it's possible to describe your web service in a
|
|
||||||
simple and easy way. Just invoke it with "walint --create config.cfg".
|
|
||||||
|
|
||||||
What's next ?
|
|
||||||
=============
|
|
||||||
|
|
||||||
We plan to do some fun stuff with WALint. Here are some thoughts:
|
|
||||||
|
|
||||||
* service discovery: just provide an API root URI and we'll try to discover
|
|
||||||
what's in there and how to generate configuration files
|
|
||||||
|
|
||||||
* unittest integration, so it's possible to have an unified way to describe web
|
|
||||||
services with existing tools. (heh, we can do Web Services CI the easy way!)
|
|
||||||
* integration with cornice description language and other web services description
|
|
||||||
languages (still to be defined)
|
|
||||||
* your ideas, so please fill issues and provide us feedback, it's useful!
|
|
|
@ -23,7 +23,7 @@ LOCALE = "fr_FR.utf8"
|
||||||
DEFAULT_DATE_FORMAT = ('%d %B %Y')
|
DEFAULT_DATE_FORMAT = ('%d %B %Y')
|
||||||
LINKS = [
|
LINKS = [
|
||||||
('Brasserie du Vieux Singe', 'https://www.vieuxsinge.com'),
|
('Brasserie du Vieux Singe', 'https://www.vieuxsinge.com'),
|
||||||
('A propos', 'http://blog.notmyidea.org/pages/about.html')
|
('A propos', 'http://blog.notmyidea.org/pages/about.html'),
|
||||||
('Messages courts', 'https://twitter.com/ametaireau'),
|
('Messages courts', 'https://twitter.com/ametaireau'),
|
||||||
('Code', 'https://github.com/almet'),
|
('Code', 'https://github.com/almet'),
|
||||||
]
|
]
|
||||||
|
|
Loading…
Reference in a new issue