Timeouts and Retries

TL;DR In this post, I will walk through how to use timeouts and retries with Celery. The Code Refer this gist Code Walkthrough This part will cover some of the functions and their results Func: timeout_test This simple function shows how a timeout exception can be caught. Note that the timeout has been defined in... Continue Reading →

Playing with Chords – Celery

TL;DR In this post, I will walk through how to use "chord" structure in Celery. The Basics Chords is a complex workflow that is present in Celery. It is a combination of two things - Groups - Groups are a way of combining several independent tasks so that they run in parallel. In a previous post,... Continue Reading →

Celery – Groups, Loops & Parallelism

TL;DR In the post, I am exploring a few questions I had with Celery and parallelism on it using groups and implementing the same feature using blocking for loop and non-blocking for loop. I decided to do these experiments based on certain optimization challenges on an application I am working on. The Question I have... Continue Reading →

Using Celery with Flask

In this post, I will talk about using Celery with Flask. I'm going to assume that you have completed this tutorial on Celery. Basic Flask app structure Assume that we have written a nicely working Flask app which now needs to do some backend processing. Here's our basic Flask app structure: +--myapp_dir/ +--app/ +--static/ +--templates/ +--utils/ +--views/... Continue Reading →

Storing Results in Celery is a Bad Idea

For months I had sleepless nights over Celery backend system crashes. I was using RabbitMQ initially and then switched to Redis and it didn't change anything. The Problem My app would generate hundreds of data every hour after processing and RabbitMQ would run out of file descriptors to handle it and crash. With Redis, the... Continue Reading →

Powered by WordPress.com.

Up ↑