Celery with Upstash Redis

You can use Celery with Upstash Redis to build scalable and serverless task queues. Celery is a Python library that manages asynchronous task execution, while Upstash Redis acts as both the broker (queue) and the result backend.

Setup#

Install Celery#

To get started, install the necessary libraries using pip:

pip install "celery[redis]"

Database Setup#

Create a Redis database using the Upstash Console. Export the UPSTASH_REDIS_HOST, UPSTASH_REDIS_PORT, and UPSTASH_REDIS_PASSWORD to your environment:

export UPSTASH_REDIS_HOST=<YOUR_HOST>export UPSTASH_REDIS_PORT=<YOUR_PORT>export UPSTASH_REDIS_PASSWORD=<YOUR_PASSWORD>

You can also use python-dotenv to load environment variables from a .env file:

UPSTASH_REDIS_HOST=<YOUR_HOST>UPSTASH_REDIS_PORT=<YOUR_PORT>UPSTASH_REDIS_PASSWORD=<YOUR_PASSWORD

Example Application#

Setting up Celery with Upstash Redis#

import osfrom celery import Celeryfrom dotenv import load_dotenvload_dotenv()# Configure Celery with Upstash RedisUPSTASH_REDIS_HOST = os.getenv("UPSTASH_REDIS_HOST")UPSTASH_REDIS_PORT = os.getenv("UPSTASH_REDIS_PORT")UPSTASH_REDIS_PASSWORD = os.getenv("UPSTASH_REDIS_PASSWORD")connection_link = f"rediss://:{UPSTASH_REDIS_PASSWORD}@{UPSTASH_REDIS_HOST}:{UPSTASH_REDIS_PORT}?ssl_cert_reqs=required"celery_app = Celery("tasks", broker=connection_link, backend=connection_link)@celery_app.taskdef add(x, y):    return x + y

Note that we should use the rediss:// protocol to connect to redis over TLS and set ssl_cert_reqs=required to enforce certificate validation.

Running the Worker#

Start the Celery worker to execute tasks:

celery -A tasks worker --loglevel=info

Using the Task#

You can now use the add task to perform background computations:

from tasks import addresult = add.delay(4, 6)print(f"Task state: {result.state}")  # Outputs 'PENDING' initially# Wait for the resultoutput = result.get(timeout=10)print(f"Task result: {output}")  # Outputs '10'

Billing Optimization#

Celery workers poll Redis continuously, even when there is no queue activity. This can incur extra costs because Upstash charges per request on the Pay-As-You-Go plan. With our Fixed plans, we recommend switching to a Fixed plan to avoid increased command count and high costs in Celery use cases.

Conclusion#

To see a more detailed example of using Celery with Upstash Redis, check out the Job Processor with Celery example on our website.

For more details on Celery, visit the Celery Documentation. For Upstash Redis, check out the Upstash Redis Documentation.