django-cachalot¶

Should you use it?¶

Django-cachalot is the perfect speedup tool for most Django projects. It will speedup a website of 100 000 visits per month without any problem. In fact, the more visitors you have, the faster the website becomes. That’s because every possible SQL query on the project ends up being cached.

Django-cachalot is especially efficient in the Django administration website since it’s unfortunately badly optimised (use foreign keys in list_editable if you need to be convinced).

However, it’s not suited for projects where there is a high number of modifications per minute on each table, like a social network with more than a 50 messages per minute. Django-cachalot may still give a small speedup in such cases, but it may also slow things a bit (in the worst case scenario, a 20% slowdown, according to the benchmark). If you have a website like that, optimising your SQL database and queries is the number one thing you have to do.

There is also an obvious case where you don’t need django-cachalot: when the project is already fast enough (all pages load in less than 300 ms). Like any other dependency, django-cachalot is a potential source of problems (even though it’s currently bug free). Don’t use dependencies you can avoid, a “future you” may thank you for that.

Features¶

• Saves in cache the results of any SQL query generated by the Django ORM that reads data. These saved results are then returned instead of executing the same SQL query, which is faster.
• The first time a query is executed is about 10% slower, then the following times are way faster (7× faster being the average).
• Automatically invalidates saved results, so that you never get stale results.
• Invalidates per table, not per object: if you change an object, all the queries done on other objects of the same model are also invalidated. This is unfortunately technically impossible to make a reliable per-object cache. Don’t be fooled by packages pretending having that per-object feature, they are unreliable and dangerous for your data.
• Handles everything in the ORM. You can use the most advanced features from the ORM without a single issue, django-cachalot is extremely robust.
• An easy control thanks to Settings and a simple API. But that’s only required if you have a complex infrastructure. Most people will never use settings or the API.
• A few bonus features like a signal triggered at each database change (including bulk changes) and a template tag for a better template fragment caching.

Comparison with similar tools¶

This comparison was done in December 2015. It compares django-cachalot to the other popular automatic ORM caches at the moment: django-cache-machine & django-cacheops.

Requirements¶

• Django 1.8, 1.9 or 1.10
• Python 2.7, 3.3, 3.4, or 3.5
• a cache configured as 'default' with one of these backends:
  • django-redis
  • memcached (using either python-memcached or pylibmc)
  • filebased
  • locmem (but it’s not shared between processes, see locmem limits)
• one of these databases:
  • PostgreSQL
  • SQLite
  • MySQL (but on older versions like 5.5, django-cachalot has no effect, see MySQL limits)

Usage¶

1. pip install django-cachalot
2. Add 'cachalot', to your INSTALLED_APPS
3. If you use multiple servers with a common cache server, double check their clock synchronisation
4. If you modify data outside Django – typically after restoring a SQL database –, run ./manage.py invalidate_cachalot
5. Be aware of the few other limits
6. If you use django-debug-toolbar, you can add 'cachalot.panels.CachalotPanel', to your DEBUG_TOOLBAR_PANELS
7. Enjoy!

Settings¶

from django.conf import settings
from django.test.utils import override_settings

with override_settings(CACHALOT_ENABLED=False):
    # SQL queries are not cached in this block

@override_settings(CACHALOT_CACHE='another_alias')
def your_function():
    # What’s in this function uses another cache

# Globally disables SQL caching until you set it back to True
settings.CACHALOT_ENABLED = False

Template utils¶

Caching template fragments can be extremely powerful to speedup a Django application. However, it often means you have to adapt your models to get a relevant cache key, typically by adding a timestamp that refers to the last modification of the object.

But modifying your models and caching template fragments leads to stale contents most of the time. There’s a simple reason to that: we rarely only display the data from one model, we often want to display related data, such as the number of books written by someone, display a quote from a book of this author, display similar authors, etc. In such situations, it’s impossible to cache template fragments and avoid stale rendered data.

Fortunately, django-cachalot provides an easy way to fix this issue, by simply checking when was the last time data changed in the given models or tables. The API function get_last_invalidation does that, and we provided a get_last_invalidation template tag to directly use it in templates. It works exactly the same as the API function.

{% load cachalot cache %}

{% get_last_invalidation 'auth.User' 'library.Book' 'library.Author' as last_invalidation %}
{% cache 3600 short_user_profile last_invalidation %}
  {{ user }} has borrowed these books:
  {% for book in user.borrowed_books.all %}
    <div class="book">
      {{ book }} ({{ book.pages.count }} pages)
      <span class="authors">
        {% for author in book.authors.all %}
          {{ author }}{% if not forloop.last %},{% endif %}
        {% endfor %}
      </span>
    </div>
  {% endfor %}
{% endcache %}

{% cache get_last_invalidation('auth.User', 'library.Book', 'library.Author'),
         cache_key='short_user_profile', timeout=3600 %}
  {{ user }} has borrowed these books:
  {% for book in user.borrowed_books.all() %}
    <div class="book">
      {{ book }} ({{ book.pages.count() }} pages)
      <span class="authors">
        {% for author in book.authors.all() %}
          {{ author }}{% if not loop.last %},{% endif %}
        {% endfor %}
      </span>
    </div>
  {% endfor %}
{% endcache %}

Signal¶

cachalot.signals.post_invalidation is available if you need to do something just after a cache invalidation (when you modify something in a SQL table). sender is the name of the SQL table invalidated, and a keyword argument db_alias explains which database is affected by the invalidation. Be careful when you specify sender, as it is sensible to string type. To be sure, use Model._meta.db_table.

This signal is not directly triggered during transactions, it waits until the current transaction ends. This signal is also triggered when invalidating using the API or the manage.py command. Be careful when using multiple databases, if you invalidate all databases by simply calling invalidate(), this signal will be triggered one time for each database and for each model. If you have 3 databases and 20 models, invalidate() will trigger the signal 60 times.

Example:

from cachalot.signals import post_invalidation
from django.dispatch import receiver
from django.core.mail import mail_admins
from django.contrib.auth import *

# This prints a message to the console after each table invalidation
def invalidation_debug(sender, **kwargs):
    db_alias = kwargs['db_alias']
    print('%s was invalidated in the DB configured as %s'
          % (sender, db_alias))

post_invalidation.connect(invalidation_debug)

# Using the `receiver` decorator is just a nicer way
# to write the same thing as `signal.connect`.
# Here we specify `sender` so that the function is executed only if
# the table invalidated is the one specified.
# We also connect it several times to be executed for several senders.
@receiver(post_invalidation, sender=User.groups.through._meta.db_table)
@receiver(post_invalidation, sender=User.user_permissions.through._meta.db_table)
@receiver(post_invalidation, sender=Group.permissions.through._meta.db_table)
def warn_admin(sender, **kwargs):
    mail_admins('User permissions changed',
                'Someone probably gained or lost Django permissions.')

Do not use if:¶

• Your project runs on several servers and each server is connected to the same database and each server uses a different cache and each server cannot have access to all other caches. However if each server can have access to all other caches, simply specify all other caches as non-default in the CACHES setting. This way, django-cachalot will automatically invalidate all other caches when something changes in the database. Otherwise, django-cachalot has no way to know on a given server that another server triggered a database change, and it will therefore serve stale data because the cache of the given server was not invalidated.
• Your project has more than 50 database modifications per second on most of its tables. There will be no problem, but django-cachalot will become inefficient and will end up slowing your project instead of speeding it. Read the introduction for more details.

Redis¶

By default, Redis will not evict persistent cache keys (those with a None timeout) when the maximum memory has been reached. The cache keys created by django-cachalot are persistent by default, so if Redis runs out of memory, django-cachalot and all other cache.set will raise ResponseError: OOM command not allowed when used memory > 'maxmemory'. because Redis is not allowed to delete persistent keys.

To avoid this, 2 solutions:

• If you only store disposable data in Redis, you can change maxmemory-policy to allkeys-lru in your Redis configuration. Be aware that this setting is global; all your Redis databases will use it. If you don’t know what you’re doing, use the next solution or use another cache backend.
• Increase maxmemory in your Redis configuration. You can start by setting it to a high value (for example half of your RAM) then decrease it by looking at the Redis database maximum size using redis-cli info memory.

For more information, read Using Redis as a LRU cache.

Memcached¶

By default, memcached is configured for small servers. The maximum amount of memory used by memcached is 64 MB, and the maximum memory per cache key is 1 MB. This latter limit can lead to weird unhandled exceptions such as Error: error 37 from memcached_set: SUCCESS if you execute queries returning more than 1 MB of data.

To increase these limits, set the -I and -m arguments when starting memcached. If you use Ubuntu and installed the package, you can modify /etc/memcached.conf, add -I 10m on a newline to set the limit per cache key to 10 MB, and if you want increase the already existing -m 64 to something like -m 1000 to set the maximum cache size to 1 GB.

Locmem¶

Locmem is a just a dict stored in a single Python process. It’s not shared between processes, so don’t use locmem with django-cachalot in a multi-processes project, if you use RQ or Celery for instance.

Filebased¶

Filebased, a simple persistent cache implemented in Django, has a small bug (#25501): it cannot cache some objects, like psycopg2 ranges. If you use range fields from django.contrib.postgres and your Django version is affected by this bug, you need to add the tables using range fields to CACHALOT_UNCACHABLE_TABLES.

MySQL¶

This database software already provides by default something like django-cachalot: MySQL query cache. Unfortunately, this built-in query cache has no significant effect since at least MySQL 5.7. However, in MySQL 5.5 it was working so well that django-cachalot was not improving performance. So depending on the MySQL version, django-cachalot may be useless. See the current django-cachalot benchmark and compare it with an older run of the same benchmark to see the clear difference: MySQL became 4 × slower since then!

Raw SQL queries¶

By default, django-cachalot tries to invalidate its cache after a raw query. It detects if the raw query contains UPDATE, INSERT or DELETE, and then invalidates the tables contained in that query by comparing with models registered by Django.

This is quite robust, so if a query is not invalidated automatically by this system, please send a bug report. In the meantime, you can use the API to manually invalidate the tables where data has changed.

However, this simple system can be too efficient in some cases and lead to unwanted extra invalidations. In such cases, you may want to partially disable this behaviour by dynamically overriding settings to set CACHALOT_INVALIDATE_RAW to False. After that, use the API to manually invalidate the tables you modified.

Multiple servers clock synchronisation¶

Django-cachalot relies on the computer clock to handle invalidation. If you deploy the same Django project on multiple machines, but with a centralised cache server, all the machines serving Django need to have their clocks as synchronised as possible. Otherwise, invalidations will happen with a latency from one server to another. A difference of even a few seconds can be harmful, so double check this!

To keep your clocks synchronised, use the Network Time Protocol.

Replication server¶

If you use multiple databases where at least one is a replica of another, django-cachalot has no way to know that the replica is modified automatically, since it happens outside Django. The SQL queries cached for the replica will therefore not be invalidated, and you will see some stale queries results.

To fix this problem, you need to tell django-cachalot to also invalidate the replica when the primary database is invalidated. Suppose your primary database has the 'default' database alias in DATABASES, and your replica has the 'replica' alias. Use the signal and cachalot.api.invalidate() this way:

from cachalot.api import invalidate
from cachalot.signals import post_invalidation
from django.dispatch import receiver

@receiver(post_invalidation)
def invalidate_replica(sender, **kwargs):
    if kwargs['db_alias'] == 'default':
        invalidate(sender, db_alias='replica')

Introduction¶

This benchmark does not intend to be exhaustive nor fair to SQL. It shows how django-cachalot behaves on an unoptimised application. On an application using perfectly optimised SQL queries only, django-cachalot may not be useful. Unfortunately, most Django apps (including Django itself) use unoptimised queries. Of course, they often lack useful indexes (even though it only requires 20 characters per index…). But what you may not know is that the ORM currently generates totally unoptimised queries [1].

Conditions¶

In this benchmark, a small database is generated, and each test is executed 20 times under the following conditions:

Database results¶

• mysql is 1.2× slower then 4.1× faster
• postgresql is 1.1× slower then 8.9× faster
• sqlite is 1.1× slower then 6.0× faster

Cache results¶

• filebased is 1.1× slower then 6.4× faster
• locmem is 1.1× slower then 6.6× faster
• memcached is 1.1× slower then 6.4× faster
• pylibmc is 1.1× slower then 6.7× faster
• redis is 1.1× slower then 5.7× faster

Database detailed results¶

MySQL¶

PostgreSQL¶

SQLite¶

Cache detailed results¶

File-based¶

Locmem¶

Memcached (python-memcached)¶

Memcached (pylibmc)¶

Redis¶

[1]

The ORM fetches way too much data if you don’t restrict it using .only and .defer. You can divide the execution time of most queries by 2-3 by specifying what you want to fetch. But specifying which data we want for each query is very long and unmaintainable. An automation using field usage statistics is possible and would drastically improve performance. Other performance issues occur with slicing. You can often optimise a sliced query using a subquery, like YourModel.objects.filter(pk__in=YourModel.objects.filter(…)[10000:10050]).select_related(…) instead of YourModel.objects.filter(…).select_related(…)[10000:10050]. I’ll maybe work on these issues one day.

Reverse engineering¶

It’s a lot of Django reverse engineering combined with a strong test suite. Such a test suite is crucial for a reverse engineering project. If some important part of Django changes and breaks the expected behaviour, you can be sure that the test suite will fail.

Monkey patching¶

Django-cachalot modifies Django in place during execution to add a caching tool just before SQL queries are executed. When a SQL query reads data, we save the result in cache. If that same query is executed later, we fetch that result from cache. When we detect INSERT, UPDATE or DELETE, we know which tables are modified. All the previous cached queries can therefore be safely invalidated.

1.3.0¶

• Adds Django 1.10 support
• Drops Django 1.7 support
• Drops Python 3.2 support
• Adds a Jinja2 extension with a cache statement and the get_last_invalidation function.
• Adds a CACHALOT_TIMEOUT setting after dozens of private & public requests, but it’s not really useful
• Fixes a RuntimeError occurring if a DatabaseCache was used in a project, even if not used by django-cachalot
• Allows bytes raw queries (except on SQLite where it’s not supposed to work)
• Creates a Slack team to discuss, easier than using Google Groups

1.2.1¶

Mandatory update if you’re using django-cachalot 1.2.0.

This version reverts the cache keys hashing change from 1.2.0, as it was leading to a non-shared cache when Python used a random seed for hashing, which is the case by default on Python 3.3, 3.4, & 3.5, and also on 2.7 & 3.2 if you set PYTHONHASHSEED=random.

1.2.0¶

WARNING: This version is unsafe, it can lead to invalidation errors

• Adds Django 1.9 support
• Simplifies and speeds up cache keys hashing
• Documents how to use django-cachalot with a replica database
• Adds DummyCache to VALID_CACHE_BACKENDS
• Updates the comparison with django-cache-machine & django-cacheops by checking features and measuring performance instead of relying on their documentations and a 2-years-ago experience of them

1.1.0¶

Backwards incompatible changes:

• Adds Django 1.8 support and drops Django 1.6 & Python 2.6 support
• Merges the 3 API functions invalidate_all, invalidate_tables, & invalidate_models into a single invalidate function while optimising it

Other additions:

• Adds a get_last_invalidation function to the API and the equivalent template tag
• Adds a CACHALOT_ONLY_CACHABLE_TABLES setting in order to make a whitelist of the only table names django-cachalot can cache
• Caches queries with IP addresses, floats, or decimals in parameters
• Adds a Django check to ensure the project uses compatible cache and database backends
• Adds a lot of tests, especially to test django.contrib.postgres
• Adds a comparison with django-cache-machine and django-cacheops in the documentation

Fixed:

• Removes a useless extra invalidation during each write operation to the database, leading to a small speedup during data modification and tests

• The post_invalidation signal was triggered during transactions and was not triggered when using the API or raw write queries: both issues are now fixed

• Fixes a very unlikely invalidation issue occurring only when an error occurred in a transaction after a transaction of another database nested in the first transaction was committed, like this:

  from django.db import transaction
  
  assert list(YourModel.objects.using('another_db')) == []
  
  try:
      with transaction.atomic():
          with transaction.atomic('another_db'):
              obj = YourModel.objects.using('another_db').create(name='test')
          raise ZeroDivisionError
  except ZeroDivisionError:
      pass
  
  # Before django-cachalot 1.1.0, this assert was failing.
  assert list(YourModel.objects.using('another_db')) == [obj]
  

1.0.3¶

• Fixes an invalidation issue that could rarely occur when querying on a BinaryField with PostgreSQL, or with some geographic queries (there was a small chance that a same query with different parameters could erroneously give the same result as the previous one)
• Adds a CACHALOT_UNCACHABLE_TABLES setting
• Fixes a Django 1.7 migrations invalidation issue in tests (that was leading to this error half of the time: RuntimeError: Error creating new content types. Please make sure contenttypes is migrated before trying to migrate apps individually.)
• Optimises tests when using django-cachalot by avoid several useless cache invalidations

1.0.2¶

• Fixes an AttributeError occurring when excluding through a many-to-many relation on a child model (using multi-table inheritance)
• Stops caching queries with random subqueries – for example User.objects.filter(pk__in=User.objects.order_by('?'))
• Optimises automatic invalidation
• Adds a note about clock synchronisation

1.0.1¶

• Fixes an invalidation issue discovered by Helen Warren that was occurring when updating a ManyToManyField after executing using .exclude on that relation. For example, Permission.objects.all().delete() was not invalidating User.objects.exclude(user_permissions=None)
• Fixes a UnicodeDecodeError introduced with python-memcached 1.54
• Adds a post_invalidation signal

1.0.0¶

Fixes a bug occurring when caching a SQL query using a non-ascii table name.

1.0.0rc¶

Added:

• Adds an invalidate_cachalot command to invalidate django-cachalot from a script without having to clear the whole cache
• Adds the benchmark introduction, conditions & results to the documentation
• Adds a short guide on how to configure Redis as a LRU cache

Fixed:

• Fixes a rare invalidation issue occurring when updating a many-to-many table after executing a queryset generating a HAVING SQL statement – for example, User.objects.first().user_permissions.add(Permission.objects.first()) was not invalidating User.objects.annotate(n=Count('user_permissions')).filter(n__gte=1)

• Fixes an even rarer invalidation issue occurring when updating a many-to-many table after executing a queryset filtering nested subqueries by another subquery through that many-to-many table – for example:

  User.objects.filter(
      pk__in=User.objects.filter(
          pk__in=User.objects.filter(
              user_permissions__in=Permission.objects.all())))
  

• Avoids setting useless cache keys by using table names instead of Django-generated table alias

0.9.0¶

Added:

• Caches all queries implying Queryset.extra
• Invalidates raw queries
• Adds a simple API containing: invalidate_tables, invalidate_models, invalidate_all
• Adds file-based cache support for Django 1.7
• Adds a setting to choose if random queries must be cached
• Adds 2 settings to customize how cache keys are generated
• Adds a django-debug-toolbar panel
• Adds a benchmark

Fixed:

• Rewrites invalidation for a better speed & memory performance
• Fixes a stale cache issue occurring when an invalidation is done exactly during a SQL request on the invalidated table(s)
• Fixes a stale cache issue occurring after concurrent transactions
• Uses an infinite timeout

Removed:

• Simplifies cachalot_settings and forbids its use or modification

0.8.1¶

• Fixes an issue with pip if Django is not yet installed

0.8.0¶

• Adds multi-database support
• Adds invalidation when altering the DB schema using migrate, syncdb, flush, loaddata commands (also invalidates South, if you use it)
• Small optimizations & simplifications
• Adds several tests

0.7.0¶

• Adds thread-safety
• Optimizes the amount of cache queries during transaction

0.6.0¶

• Adds memcached support

0.5.0¶

• Adds CACHALOT_ENABLED & CACHALOT_CACHE settings
• Allows settings to be dynamically overridden using cachalot_settings
• Adds some missing tests

0.4.1¶

• Fixes pip install.

0.4.0 (install broken)¶

• Adds Travis CI and adds compatibility for:
  • Django 1.6 & 1.7
  • Python 2.6, 2.7, 3.2, 3.3, & 3.4
  • locmem & Redis
  • SQLite, PostgreSQL, MySQL

0.3.0¶

• Handles transactions
• Adds lots of tests for complex cases

0.2.0¶

• Adds a test suite
• Fixes invalidation for data creation/deletion
• Stops caching on queries defining select or where arguments with QuerySet.extra

0.1.0¶

Prototype simply caching all SQL queries reading the database and trying to invalidate them when SQL queries modify the database.

Has issues invalidating deletions and creations. Also caches QuerySet.extra queries but can’t reliably invalidate them. No transaction support, no test, no multi-database support, etc.