We re­cently needed a way to run mul­tiple ap­plic­a­tions us­ing cel­ery + rab­bit­mq on the same serv­er. For­tu­nately rab­bit­mq of­fers vhosts to isol­ate cli­ents and set­ting them up is quite easy. The setup we are go­ing for looks as fol­lows

  1. Have one rab­bit­mq vhosts for each django ap­plic­a­tion.
  2. Run an in­stance of cel­eryd for each django ap­plic­a­tion.

For each of our apps djan­goap­p01 … djan­goappN we do the fol­low­ing

Adding a vhost to rab­bit­mq ist simple

rabbitmqctl add_vhost /djangoapp01

Now we need to add a user and grant per­mis­sions to the vhost

 rabbitmqctl add_user djangoapp01 <password>

 rabbitmqctl set_permissions -p /djangoapp01 djangoapp01 ".*" ".*" ".*"

Now we need to set up cel­ery. We use the init script sup­plied in cel­ery ex­tra (ht­tps://git­hub.com/cel­ery/cel­ery/blob/3.1/ex­tra/gen­er­ic-init.d/cel­eryd) which has sup­port for run­ning mul­tiple con­fig­ur­a­tions by re­nam­ing it. We copy that file to /etc/init.d/cel­ery­d_d­jan­goap­p01 and cre­ate an ap­pro­pri­ate cel­ery con­fig file in /etd/de­fault/cel­ery­d_d­jan­goap­p01

In the set­tings.py for djan­goap­p01 we make sure to set

BROKER_URL = 'ampq://djangoapp01:<password>@localhost//djangoapp01'

When you want to run mul­tiple django ap­plic­a­tions us­ing cel­ery on the same serv­er you need some way to sep­ar­ate the work queues. And you should be good to go.

If you are in­ter­ested, we use the ans­ible con­fig be­low to set up rab­bit­mq and cel­ery ac­cord­ingly:

 - name: set up rabbitmq vhost
   rabbitmq_vhost: name="/{{ deploymentname }}" state=present
 - rabbitmq_user: user="{{ deploymentname }}"
                  password="{{ rabbitmq_password }}"
                  vhost="/{{ deploymentname }}"
                  configure_priv=.*
                  read_priv=.*
                  write_priv=.*
                  state=present
 - name: set up celeryd
   template: src=celeryd_init dest="/etc/init.d/celeryd_{{ deploymentname }}" mode=755
 - template: src=celeryd_default dest="/etc/default/celeryd_{{ deploymentname }}"
   notify:
     - restart celery