Agent Forwarding with Paramiko

TL;DR In this post, I will walk through a challenge I faced with agent forwarding on paramiko and how I solved it. Background Before we get started, it would be a good idea to go through SSH Keys, Agents and Linux. This post requires a good understanding of agent forwarding to be able to grasp the … Continue reading Agent Forwarding with Paramiko

SSH Keys, Agents and Linux

TL;DR In this post, I will walk through SSH, SSH Agents and a few key Linux concepts around them. Terminologies Client – Any machine that wishes to connect to a remote system. Server – Any machine that accepts connection from a client. User – The user as which the client will connect to a server. … Continue reading SSH Keys, Agents and Linux

Timeouts and Retries

TL;DR In this post, I will walk through how to use timeouts and retries with Celery. The Code Refer this gist Code Walkthrough This part will cover some of the functions and their results Func: timeout_test This simple function shows how a timeout exception can be caught. Note that the timeout has been defined in … Continue reading Timeouts and Retries

Playing with Chords – Celery

TL;DR In this post, I will walk through how to use “chord” structure in Celery. The Basics Chords is a complex workflow that is present in Celery. It is a combination of two things – Groups – Groups are a way of combining several independent tasks so that they run in parallel. In a previous post, … Continue reading Playing with Chords – Celery

Celery – Groups, Loops & Parallelism

TL;DR In the post, I am exploring a few questions I had with Celery and parallelism on it using groups and implementing the same feature using blocking for loop and non-blocking for loop. I decided to do these experiments based on certain optimization challenges on an application I am working on. The Question I have … Continue reading Celery – Groups, Loops & Parallelism

To Optimize or Not To Optimize

TL;DR Oft times, coding is trying to find a balance between multiple parameters. Some good factors we measure code upon are – Performance Efficiency Speed Correctness Occasionally, we do miss out on factors such as – Readability Ease of Troubleshooting In this post, I will walk you through an interesting case I encountered while developing … Continue reading To Optimize or Not To Optimize

Debugging ERR_CONNECTION_TIMED_OUT on Linux VM

I encountered a new scenario that I hadn’t in the past. The Problem I was trying to access Elasticsearch on a Linux VM and it wasn’t allowing me to connect to the port 9200. Check Ports and Processes I checked if the process was running and the correct ports were open – $ ps aux … Continue reading Debugging ERR_CONNECTION_TIMED_OUT on Linux VM

Developer Sharing (Giving Back to the Community)

Lately, I have been thinking about how I can share my experiences with the developer community. I am certain a lot of us try to re-invent the learning wheel when we are trying to understand a system or tool that we want to use. In this post I want to cover how to do it … Continue reading Developer Sharing (Giving Back to the Community)

Using Celery with Flask

In this post, I will talk about using Celery with Flask. I’m going to assume that you have completed this tutorial on Celery. Basic Flask app structure Assume that we have written a nicely working Flask app which now needs to do some backend processing. Here’s our basic Flask app structure: +–myapp_dir/ +–app/ +–static/ +–templates/ +–utils/ +–views/ … Continue reading Using Celery with Flask

Storing Results in Celery is a Bad Idea

For months I had sleepless nights over Celery backend system crashes. I was using RabbitMQ initially and then switched to Redis and it didn’t change anything. The Problem My app would generate hundreds of data every hour after processing and RabbitMQ would run out of file descriptors to handle it and crash. With Redis, the … Continue reading Storing Results in Celery is a Bad Idea