Django 1.6 and Celery 3.0 memory leaks

After upgrading Django to 1.6, my celery worker is eating up RAM.
Seems that the memory allocated for the workers isn’t released and grows after every task.

Related Settings:

# DB:
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'somedb',
        'USER': '',
        'PASSWORD': '',
        'HOST': 'localhost',
        'PORT': '',
    }
}


# CELERY SETTINGS:
CELERY_RESULT_BACKEND = 'redis://'
BROKER_URL = 'redis://'

Related package versions:

Django==1.6
celery==3.0.24
django-celery==3.0.23
billiard==2.7.3.34
kombu==2.5.16
redis==2.7.6

Happens in both my local env (with DEBUG=False) running the worker manually and in a staging environment where celery is running with Upstart.


Updates:

  1. Tried setting autocommit=False with no success.
  2. Could be it’s not related to the Django version upgrade, but to some setting or 3rd party package that I had to upgrade to make the switch to 1.6.

Best answer

It turns out the memory leak was not directly caused by the Django upgrade or Celery.

After a lot of digging around I found that, surprisingly, the celery worker memory leak happens because I upgraded django-debug-toolbar from 0.9.4 to 0.11.0 (which is needed for Django 1.6 compatibility).

Still no idea what exactly caused this issue, or why it only happens in the celery worker processes and not in the app server ones (Gunicorn).

Removing django-debug-toolbar from the installed apps and middleware solves the issue. At least temporarily.