Remove useless files

This commit is contained in:
Alexis Métaireau 2017-07-03 21:32:24 +02:00
parent 615a90efee
commit 3b65203171
5 changed files with 1 additions and 263 deletions

View file

@ -1,92 +0,0 @@
How to cache Elastic Seach Queries?
###################################
:status: other
These days, I'm working on Marketplace, the Mozilla's Appstore. Internally,
we're doing Elastic Search to do search, and after some load tests, we
eventually found out that Elastic Search was our bottleneck.
So we want to reduce the number of requests we do to this server. Currently,
the requests are done trough HTTP.
The first thing to realize is what do we want to cache exactly? There is a fair
bit of things we might want to cache. Let's start by the most queried pages:
the home and the list of apps per category.
Different approaches
====================
You can put the cache in many different locations. The code powering
Marketplace is kinda fuzzy sometimes. The requests to Elastic Search are done
in a number of different parts of the code. They're done sometimes directly
with HTTP calls, sometimes using the ElasticUtils library, sometimes using some
other lib…
That's kind of hard to get where and how to add the caching layer here. What
did we do? We started to work on an HTTP caching proxy. This proxy could
Find a key
==========
Caching things
==============
Caching is easy, it's something like that, in term of python code::
if key in cache:
value = cache.get(key)
else:
value = do_the_request()
cache.set(key, value)
return value
Back to business logic: I want to cache the request that are done on the
homepage. The code currently looks like this::
popular = Webapp.popular(region=region, gaia=request.GAIA)[:10]
Nothing fancy going on here, we're displaying the list of popular and latest
apps on the marketplace. I can cache the results for each region, and depending
if `request.GAIA` is `True` or not. The key will look like `popular-{region}`,
to which I will eventually append `-gaia` if it makes sense. More generally,
I will turn a number of criterias into a single key, like this::
def cache(qs, *keys):
"""Cache the given querystring"""
key = '-'.join(keys)
if key in cache:
result = cache.get(key)
else:
result = qs
cache.set(key, result)
return result
Which I can use like this::
keys = [region.slug, ]
if request.GAIA:
keys.append('gaia')
popular = cache(Webapp.popular(region=region, gaia=request.GAIA)[:10],
'popular', *keys)
Invalidation is hard?
=====================
Right, we got this cached, good. But what happens when the data we cached
changes? Currently, nothing, we return the same data over and over again.
We need to tell our application when this cache isn't good anymore; we need to
*invalidate* it.
We can use different strategies for this:
* Set a timeout for the cache.
* Invalidate manually the cache when something changes, for instance using
signals.
I started by having a look at how I wanted to invalidate all of this, case by
case. The problem

View file

@ -1,58 +0,0 @@
API description using cornice and colander
##########################################
:status: draft
One of the goals we want to accomplish with Cornice is to describe the web
services in a nice and easy way, for the developer. This for different reasons.
Having web services described in some ways allows you do do interesting things,
such as automatically generating a documentation, or a client, for instance.
REST services are supposed to be discoverable, but in a lot of situations,
that's not the case, and it can be a pain to implement a client for them, even
if most of what is being done there is shared between a lot of web services.
In cornice, we already generate some documentation, but this one is incomplete in
different ways. For instance, it is currently not possible to get information
about the inputs you are waiting for, in the different locations (body,
headers and query string).
For instance, if you want to describe that a *POST* on `/foobar` needs the
*foo* and *bar* parameters, and that you eventually could add a *baz*
parameter, you can't, or you have to describe it manually in the documentation
of the web service.
Cornice, while not tied to `Colander`_, tries to make a good use of it when
possible. So it is possible to do validation of the incoming data (in different
locations) and, at the same time, documentation of the web service.
To do so, you have to use a particular `schema` keyword in the decorator. Here
is an example of definition
.. code-block:: python
from cornice import Service
from colander import MappingSchema, SchemaNode, String
foobar = Service(name="foobar", path="/foobar")
class Foobar(MappingSchema):
foo = SchemaNode(String(), location="body")
bar = SchemaNode(String(), location="headers")
baz = SchemaNode(String(), default=None)
@foobar.get(scheme=Foobar)
def foobar_get(request):
# do something here
.. _Colander: http://docs.pylonsproject.org/projects/colander/en/latest/
The output documentation currently looks like this:
.. image:: images/cornice-exampledoc-validation.png
The plans for the future are to be able to put direct curl calls in the
documentation to demonstrate how the service behaves for instance, and why not
creating a tiny python client able to consume the cornice APIs.

View file

@ -1,29 +0,0 @@
L'heure pour un vote electronique à distance ?
##############################################
Ce matin, je me fais reveiller par le reveil. La radio. Je suis chez des amis,
à Paris, et le Front National à fait des scores incroyables. Du jamais vu.
En prenant le RER, je croise le regard de ces gens, probablements immigrés, qui
semblent abrutis par la situation. Je suis le fils d'une émigrée et je partage
leur rage, leur haine. Derrière ces beaux discours, le Front National souhaite
fermer les frontières, controler qui rentre pour les laisser chez eux, ces
personnes qui ont eu l'infortune de naitre ailleurs, la ou la terre est moins
propice.
Je me sens coupable, ce matin. Coupable de ne pas avoir été voter hier. Dans ma
bulle j'étais, absent. Cela fait plusieurs années que je ne vote plus, par
manque de foi en la politique qui est menée actuellement, et par dépis. J'attends,
mollement, le second tour pour exprimer mon suffrage. Pour ce qu'il vaut. Je ne
crois pas en la representativité sous sa forme actuelle.
Mais ce matin, je m'en veux quand même. Même sans croire à ce système de
representativité, j'aurais pu éviter que ces idées racistes et haineuses
n'étendent leur pouvoir à d'autres régions.
Il est trop tard pour changer la donne, le mal est fait. Mais j'aurais pu aller
voter. J'aurais du même.
La geule encore pateuse, au petit déjeuner, je me questionne sur mon
attachement à un territoir, et aux consequences de ce non-attachement sur mon
envie de voter.

View file

@ -1,83 +0,0 @@
walint - tilt
#############
:status: draft
Things start to fold together. I still need some time to be up to
speed (new job / new city / new people), but the projects I'm working on are
pretty exciting and I like what I'm doing. We've been working with Tarek on a
tool about web services testing.
At Mozilla, we produce and consume web services. As an organisation defending
web standards, providing services that match these specifications is the least
we can do.
Web App Lint (WALint) is a simple tool to help you to test your web services
against the HTTP specification. The goal is to detect potential problems a
service could be subject to, and to provide a simple-to-configure tool to test
your WS against HTTP as speced. The long term goal is to be exhaustive on the
controllers we provide, but if you have special needs, you should be able to
define tests in a simple way.
As everything we do at Mozilla, the tool is an opensource project. Our goal is
to have something useful for web service consumers and providers. If you think
you can be of any help, don't hesitate to drop me an email or fork on github.
Testing the HTTP specification
==============================
A lot of web services are written on top of HTTP. The protocol is fully
described in a specification and introduce some really interesting stuff we
should care about when implementing our services. This tool wants to check this
up and provide a ways to ensure that you are defining your web services the
right way. It's not a replacement for manual service testing, thus: you still
have to test that your web service is doing what you think it does. WALint is
checking the stuff we almost forgot all of the time.
For instance, the HTTP specification specifies that if the client sends a
request with a Content-Type not handled by the server, it should answer with a
"406 Not Acceptable" and should return the list of accepted headers (so the
client is able to send again the request with the right accept header). We also
check that sending weird data (broken json objects or authentication headers)
does not break the tested service.
Obviously, we don't have all the scenarios in mind, and they aren't all
implemented. That's where being an open source project makes sense. You, as
service providers and consumers, know what are the kind of mistake you are used
to do / to deal with. It should make "tilt": fork the project and write some
lines checking the particular behavior that's disturbing you, everyone will
enjoy it. If you prefer, just open a ticket and we'll care about the
implementation.
The documentation defines a list of all the implemented controllers:
http://packages.python.org/walint/controllers.html
What does it looks like?
========================
Here is an example of a configuration file: you describe the paths and
signatures of your web services, the controllers you want to use and the test
scenarios you want to run, that's it, just run "walint config.cfg" and you're
good.
Here's a capture of the output of the command line invocation: you get all the
paths tested with the different listed methods, and if needed what's not
working. In the future, why not putting some documentation online with
information about how to fix common mistakes?
We also provide a wizard so it's possible to describe your web service in a
simple and easy way. Just invoke it with "walint --create config.cfg".
What's next ?
=============
We plan to do some fun stuff with WALint. Here are some thoughts:
* service discovery: just provide an API root URI and we'll try to discover
what's in there and how to generate configuration files
* unittest integration, so it's possible to have an unified way to describe web
services with existing tools. (heh, we can do Web Services CI the easy way!)
* integration with cornice description language and other web services description
languages (still to be defined)
* your ideas, so please fill issues and provide us feedback, it's useful!

View file

@ -23,7 +23,7 @@ LOCALE = "fr_FR.utf8"
DEFAULT_DATE_FORMAT = ('%d %B %Y') DEFAULT_DATE_FORMAT = ('%d %B %Y')
LINKS = [ LINKS = [
('Brasserie du Vieux Singe', 'https://www.vieuxsinge.com'), ('Brasserie du Vieux Singe', 'https://www.vieuxsinge.com'),
('A propos', 'http://blog.notmyidea.org/pages/about.html') ('A propos', 'http://blog.notmyidea.org/pages/about.html'),
('Messages courts', 'https://twitter.com/ametaireau'), ('Messages courts', 'https://twitter.com/ametaireau'),
('Code', 'https://github.com/almet'), ('Code', 'https://github.com/almet'),
] ]