Playing with Chords – Celery

TL;DR In this post, I will walk through how to use “chord” structure in Celery. The Basics Chords is a complex workflow that is present in Celery. It is a combination of two things – Groups – Groups are a way of combining several independent tasks so that they run in parallel. In a previous post,…

Celery – Groups, Loops & Parallelism

TL;DR In the post, I am exploring a few questions I had with Celery and parallelism on it using groups and implementing the same feature using blocking for loop and non-blocking for loop. I decided to do these experiments based on certain optimization challenges on an application I am working on. The Question I have…

To Optimize or Not To Optimize

TL;DR Oft times, coding is trying to find a balance between multiple parameters. Some good factors we measure code upon are – Performance Efficiency Speed Correctness Occasionally, we do miss out on factors such as – Readability Ease of Troubleshooting In this post, I will walk you through an interesting case I encountered while developing…

Dr Jekyll, Mr Hyde

I am super disappointed today. I spent a lot of time trying to move my site away from GitHub Pages to Wix. I love Wix and I find it extremely easy to create layouts in it. I had almost completed this process and I realized there were more limitations that I disliked. Jekyll & GitHub Pages Limitations…

Debugging ERR_CONNECTION_TIMED_OUT on Linux VM

I encountered a new scenario that I hadn’t in the past. The Problem I was trying to access Elasticsearch on a Linux VM and it wasn’t allowing me to connect to the port 9200. Check Ports and Processes I checked if the process was running and the correct ports were open – $ ps aux…

Developer Sharing (Giving Back to the Community)

Lately, I have been thinking about how I can share my experiences with the developer community. I am certain a lot of us try to re-invent the learning wheel when we are trying to understand a system or tool that we want to use. In this post I want to cover how to do it…

Using Celery with Flask

In this post, I will talk about using Celery with Flask. I’m going to assume that you have completed this tutorial on Celery. Basic Flask app structure Assume that we have written a nicely working Flask app which now needs to do some backend processing. Here’s our basic Flask app structure: +–myapp_dir/ +–app/ +–static/ +–templates/ +–utils/ +–views/…

Storing Results in Celery is a Bad Idea

For months I had sleepless nights over Celery backend system crashes. I was using RabbitMQ initially and then switched to Redis and it didn’t change anything. The Problem My app would generate hundreds of data every hour after processing and RabbitMQ would run out of file descriptors to handle it and crash. With Redis, the…

Using Fabric With Flask

I recently came across Fabric, a library that can be used for deployment or system administration tasks. To check out the most basic usage refer this tutorial. I’m going to assume that now you have a basic understanding of Fabric. You run fabric using the fab command on the terminal. This is efficient if you are using the terminal…

Heroku Environment Settings with Flask

I have to write this post now lest I forget what I have to say. I have been developing a website for a dear friend Patrick Joseph. While I am almost done with the most bits of the puzzle, I have two really cool things to share about Heroku. SparkPost plugin (I will be covering this in…

Retries using td-agent

At my current workplace, we use td-agent for data collection purposes. As with any system, network endpoints can break when there are issues on the server. Let’s take the following data flow – Considerations td-agent fetches data from service A and then sends an HTTP API call to service B. service A is highly reliable…

Using td-agent to archive data from SQS

Oft times we need to archive data for billing or viewing traffic trends and so on. This data can come from either one or multiple application servers. This post describes the use of fluentd/td-agent as a data collector. td-agent is a data collector tool invented by Treasure Data. It is open sourced and maintained by…