Agent Forwarding with Paramiko

TL;DR

In this post, I will walk through a challenge I faced with agent forwarding on paramiko and how I solved it.

Background

Before we get started, it would be a good idea to go through SSH Keys, Agents and Linux. This post requires a good understanding of agent forwarding to be able to grasp the problem and the solution.

The Goal

The example in my previous post was the goal.

Let’s say there are 3 machines – A (source host), B (jump host) and C (destination host). You need to perform a log collection on destination host but it can be accessed only via jump host.

We know the solution – source host connects to jump host which then connects to the destination and fetches the logs. In my case, I had already verified that my keys are authorized to access jump host and destination host.

This straightforward solution is what I set out to implement using paramiko.

The Problem

From the source host, I logged into the destination host via the jump host and fetched the logs manually. I ended up programming the same thing and tested it via ipython and it worked but the I minute I deployed it on a server, things started to break.

Any guesses why? Look at the partial code snippet yourself and try and find the issue –

from paramiko import AutoAddPolicy
from paramiko.client import SSHClient
from paramiko.agent import AgentRequestHandler


# Create an SSH client
client = SSHClient()

# Automatically add to known_hosts file if key fingerprints are not found
client.set_missing_host_key_policy(AutoAddPolicy())

# Load existing system host keys
client.load_system_host_keys()

# Connect to the machine
client.connect("192.168.0.4")

# Create a channel
session = client.get_transport().open_session()

# Attach agent to the channel
agent = AgentRequestHandler(session)

# Execute the session command
# NOTE: Channels cannot be reused once executed
session.exec_command("ssh 192.168.0.4 'less /var/log/file.log | grep pattern'")

recv_bytes = 1024

# Check if server is ready to host data or error and receive it
while not session.recv_ready() or not session.recv_stderr_ready():
    if session.recv_ready():
        res = ""
        data = session.recv(recv_bytes).decode("utf-8")
        while data:
            res = "{}{}".format(res, data)
            data = session.recv(recv_bytes).decode("utf-8")
        stdout = res
        break
    elif session.recv_stderr_ready():
        res = ""
        data = session.stderr_recv(recv_bytes).decode("utf-8")
        while data:
            res = "{}{}".format(res, data)
            data = session.stderr_recv(recv_bytes).decode("utf-8")
        stderr = res
        break

The error my code was throwing was this –

Exception in thread Thread-2246:
Traceback (most recent call last):
File "/usr/lib/python3.4/threading.py", line 920, in _bootstrap_inner
self.run()
File "/home/user/env/lib/python3.4/site-packages/paramiko/agent.py", line 16, in run
raise AuthenticationException("Unable to connect to SSH agent")
paramiko.ssh_exception.AuthenticationException: Unable to connect to SSH agent

Looks familiar to another error?

Could not open a connection to your authentication agent.

The APIs would call the correct functions but then they were unable to find the agent. And here began the quest of creating a new agent via paramiko.

Creating an Agent via paramiko

Paramiko allows you to control agents on your source host using the agent module. However, it turns out you cannot add keys to your Agent via paramiko.

Paramiko can create new agents or forward existing agents but it cannot add keys to your agent.

After browsing on the internet for hours and endlessly reading Stackoverflow answers, I came across this open issue.

The Solution

There wasn’t really a way for me to change my implementation because this was the only possible method to solve the business needs. So, I hacked around and created an agent using Python.

Here’s my code snippet –

import os
import subprocess

SSH_ADD_KEYS_CMD = "ssh-add"
SSH_AGENT_CREATE_CMD = "ssh-agent"
SSH_AGENT_KILL_CMD = "ssh-agent -k"
SSH_AUTH_SOCK_ENVVAR = "SSH_AUTH_SOCK"
SSH_AGENT_PID_ENVVAR = "SSH_AGENT_PID"
SSH_KEYS_DIR = "~/.ssh/"

def create_agent():
    """
    Function that creates the agent and sets the environment variables.
    """
    output = subprocess.check_output(SSH_AGENT_CREATE_CMD, universal_newlines=True)
    if output:
        output = output.strip().split("\n")
        for item in output[0:2]:
            envvar, val = item.split(";")[0].split("=")
            print("Setting environment variable: {}={}".format(envvar, val))
            os.environ[envvar] = val

def ssh_cleanup():
    """
    Function that kills the agents created so that there aren't too many agents lying around eating up resources.
    """
    # Kill the agent
    output = subprocess.check_output(SSH_AGENT_KILL_CMD, universal_newlines=True)
    print(output)
    # Print the values of the environment variables
    print(os.environ.get(SSH_AUTH_SOCK_ENVVAR))
    print(os.environ.get(SSH_AGENT_PID_ENVVAR))
    # Reset these values so that other function
    os.environ[SSH_AUTH_SOCK_ENVVAR] = ""
    os.environ[SSH_AGENT_PID_ENVVAR] = ""

def main():
    # Create an SSH client
    client = SSHClient()

    # Automatically add to known_hosts file if key fingerprints are not found
    client.set_missing_host_key_policy(AutoAddPolicy())

    # Load existing system host keys
    client.load_system_host_keys()

    # Connect to the machine
    client.connect("192.168.0.4")

    # Create a channel
    session = client.get_transport().open_session()

    if not os.environ.get(SSH_AUTH_SOCK_ENVVAR):
        create_agent()
        for keyfile in os.listdir(SSH_KEYS_DIR):
            add_key_cmd = "{} {}{}".format(SSH_ADD_KEYS_CMD, SSH_KEYS_DIR, keyfile)
            output = subprocess.check_output(add_key_cmd, universal_newlines=True)
            print(output)

    # Attach agent to the channel
    agent = AgentRequestHandler(session)

    ... # Execution and processing code comes here

This is the implementation of all the fundamentals in Linux discussed in my previous post.

Note: Do focus on cleanup. In the past I’ve run into issues where SSH agents are left unattended on the server – they hog resources and eventually make the system unresponsive so you have no option but to perform a hard restart on the server.

Conclusion

Paramiko is a great library which allows you to do a lot with SSH. It is also the foundation for libraries like Fabric. This was a great learning experience for me with respect to details around SSH.

Interestingly, while I was working on this problem, I ran into this Stackoverflow question. After reading it a few times, I answered it and had an epiphany that we were solving the exact same problem. Turns out, I was not the only one looking for the solution after all.

If you found this post helpful and feel that it answers the question asked, I humbly request you to upvote my answer (and even make it official, if you have that authority).

Thanks again!

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Powered by WordPress.com.

Up ↑