Multiple django apps using celery and rabbitmq on one server
Raphael Kimmig, 31 Mär 2014
We recently needed a way to run multiple applications using celery + rabbitmq on the same server. Fortunately rabbitmq offers vhosts to isolate clients and setting them up is quite easy. The setup we are going for looks as follows
- Have one rabbitmq vhosts for each django application.
- Run an instance of celeryd for each django application.
For each of our apps djangoapp01 … djangoappN we do the following
Adding a vhost to rabbitmq ist simple
rabbitmqctl add_vhost /djangoapp01
Now we need to add a user and grant permissions to the vhost
rabbitmqctl add_user djangoapp01 <password>
rabbitmqctl set_permissions -p /djangoapp01 djangoapp01 ".*" ".*" ".*"
Now we need to set up celery. We use the init script supplied in celery extra (https://github.com/celery/celery/blob/3.1/extra/generic-init.d/celeryd) which has support for running multiple configurations by renaming it. We copy that file to /etc/init.d/celeryd_djangoapp01 and create an appropriate celery config file in /etd/default/celeryd_djangoapp01
In the settings.py for djangoapp01 we make sure to set
BROKER_URL = 'ampq://djangoapp01:<password>@localhost//djangoapp01'
When you want to run multiple django applications using celery on the same server you need some way to separate the work queues. And you should be good to go.
If you are interested, we use the ansible config below to set up rabbitmq and celery accordingly:
- name: set up rabbitmq vhost
rabbitmq_vhost: name="/{{ deploymentname }}" state=present
- rabbitmq_user: user="{{ deploymentname }}"
password="{{ rabbitmq_password }}"
vhost="/{{ deploymentname }}"
configure_priv=.*
read_priv=.*
write_priv=.*
state=present
- name: set up celeryd
template: src=celeryd_init dest="/etc/init.d/celeryd_{{ deploymentname }}" mode=755
- template: src=celeryd_default dest="/etc/default/celeryd_{{ deploymentname }}"
notify:
- restart celery