Timeouts and Retries

TL;DR In this post, I will walk through how to use timeouts and retries with Celery. The Code Refer this gist Code Walkthrough This part will cover some of the functions and their results Func: timeout_test This simple function shows how a timeout exception can be caught. Note that the timeout has been defined in... Continue Reading →

Playing with Chords – Celery

TL;DR In this post, I will walk through how to use "chord" structure in Celery. The Basics Chords is a complex workflow that is present in Celery. It is a combination of two things - Groups - Groups are a way of combining several independent tasks so that they run in parallel. In a previous post,... Continue Reading →

Celery – Groups, Loops & Parallelism

TL;DR In the post, I am exploring a few questions I had with Celery and parallelism on it using groups and implementing the same feature using blocking for loop and non-blocking for loop. I decided to do these experiments based on certain optimization challenges on an application I am working on. The Question I have... Continue Reading →

To Optimize or Not To Optimize

TL;DR Oft times, coding is trying to find a balance between multiple parameters. Some good factors we measure code upon are - Performance Efficiency Speed Correctness Occasionally, we do miss out on factors such as - Readability Ease of Troubleshooting In this post, I will walk you through an interesting case I encountered while developing... Continue Reading →

Using Celery with Flask

In this post, I will talk about using Celery with Flask. I'm going to assume that you have completed this tutorial on Celery. Basic Flask app structure Assume that we have written a nicely working Flask app which now needs to do some backend processing. Here's our basic Flask app structure: +--myapp_dir/ +--app/ +--static/ +--templates/ +--utils/ +--views/... Continue Reading →

Storing Results in Celery is a Bad Idea

For months I had sleepless nights over Celery backend system crashes. I was using RabbitMQ initially and then switched to Redis and it didn't change anything. The Problem My app would generate hundreds of data every hour after processing and RabbitMQ would run out of file descriptors to handle it and crash. With Redis, the... Continue Reading →

Using Fabric With Flask

I recently came across Fabric, a library that can be used for deployment or system administration tasks. To check out the most basic usage refer this tutorial. I'm going to assume that now you have a basic understanding of Fabric. You run fabric using the fab command on the terminal. This is efficient if you are using the terminal... Continue Reading →

Powered by WordPress.com.

Up ↑