Python Background Jobs: Celery, RQ, Dramatiq, and APScheduler Compared

Sooner or later every Python web app needs to do work outside the request: send the email after returning the response, generate the report, process the upload, retry the flaky third-party call. Doing that work inline makes your app slow and fragile; doing it properly means a background job system. The trouble is choosing one Celery, RQ, Dramatiq, and APScheduler all "run background tasks," but they solve different problems. This guide compares them honestly so you pick the right tool the first time.

The shared mental model

Task queues all share the same shape. Your app enqueues a job (a function name plus arguments) into a broker usually Redis or RabbitMQ. Separate worker processes pull jobs off the queue and execute them, handling retries, failures, and results. Your web process stays fast because it only enqueues; the slow work happens elsewhere.

  web app ──enqueue──▶ [ BROKER: Redis/RabbitMQ ] ──▶ worker pool ──▶ done
   (fast)                  holds the queue            (does the slow work)

   Same shape for Celery, RQ, and Dramatiq.
   They differ in features, simplicity, and operational weight.

Celery: the feature-rich incumbent

Celery is the long-standing default the most powerful and the most widely deployed. It supports multiple brokers, complex workflows (chains, groups, chords), scheduled tasks via its beat scheduler, rate limiting, retries, and a deep configuration surface. That power is also its cost: Celery has a reputation for configuration complexity and operational sharp edges, and the breadth can be overwhelming for a simple need.

# tasks.py
from celery import Celery

app = Celery("app", broker="redis://localhost:6379/0", backend="redis://localhost:6379/1")

@app.task(bind=True, max_retries=3, autoretry_for=(ConnectionError,), retry_backoff=True)
def send_receipt(self, order_id: int) -> None:
    order = Order.objects.get(id=order_id)        # pass an ID, not the object
    email.send(order.email, render_receipt(order))

# Enqueue from your web app — returns instantly, work happens on a worker.
send_receipt.delay(order_id=42)

# Run workers:  celery -A tasks worker --concurrency=8
# Periodic:     celery -A tasks beat   (Celery's built-in scheduler)

RQ: simple by design

RQ (Redis Queue) is the antithesis of Celery's breadth: it is small, readable, and does one thing run Python functions on workers via Redis. If you can read its source in an afternoon (you nearly can), you can reason about its behaviour in production. The trade-off is fewer features and Redis-only, but for a large share of apps that is exactly enough.

from redis import Redis
from rq import Queue

# Any plain function is a job — no decorator required.
def send_receipt(order_id: int) -> None:
    order = Order.objects.get(id=order_id)
    email.send(order.email, render_receipt(order))

queue = Queue(connection=Redis())
queue.enqueue(send_receipt, 42, retry=Retry(max=3))   # enqueue and move on

# Run a worker:  rq worker

Dramatiq: the modern middle ground

Dramatiq was designed to keep most of Celery's usefulness while being simpler and more reliable by default. It offers sensible behaviour out of the box automatic retries, message age limits, and reliability features that you have to assemble or tune elsewhere. For teams who find Celery too heavy but RQ too bare, Dramatiq is often the sweet spot.

import dramatiq

# Retries and sane defaults are built in; you opt into specifics per actor.
@dramatiq.actor(max_retries=3, time_limit=30_000)
def send_receipt(order_id: int) -> None:
    order = Order.objects.get(id=order_id)
    email.send(order.email, render_receipt(order))

send_receipt.send(42)        # enqueue

# Run workers:  dramatiq tasks

APScheduler: a different tool for a different job

APScheduler is not a task queue it is a scheduler. Its job is to run code on a time-based trigger: every hour, every weekday at 9am, in thirty minutes. It is the in-process answer to "do this on a schedule." Importantly, it does not, by itself, give you a distributed worker pool with retries and queuing; for scheduled work that is also heavy or must scale, the common pattern is to let a scheduler simply enqueue jobs into a task queue.

from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.triggers.cron import CronTrigger

scheduler = BackgroundScheduler()

# Don't do the heavy work here — just drop a job onto the real task queue.
@scheduler.scheduled_job(CronTrigger(hour=2, minute=0))   # every day at 02:00
def nightly_reports():
    for tenant_id in active_tenant_ids():
        generate_report.delay(tenant_id)    # Celery/RQ/Dramatiq does the actual work

scheduler.start()

The decision guide

Operational truths for all of them

Whatever you choose, the same realities apply: make tasks idempotent, because they will occasionally run more than once; keep task arguments small and serializable (pass an ID, not a giant object); configure retries with backoff for transient failures and a dead-letter path for permanent ones; and monitor your queue depth, because a silently growing backlog is how background systems fail invisibly.

Practice on the go

Learn Python, the free Android app

Every topic in this series lives in the app too: bite-size lessons, runnable examples, quizzes, mini projects, and an offline Python playground that runs on your phone.

Get it on Google Play View the app page