SSH Keys, Agents and Linux

TL;DR

In this post, I will walk through SSH, SSH Agents and a few key Linux concepts around them.

Terminologies

Client – Any machine that wishes to connect to a remote system.

Server – Any machine that accepts connection from a client.

User – The user as which the client will connect to a server.

SSH Basics

SSH (secure shell) is a cryptographic network protocol on the Application Layer of the TCP/IP or OSI models in network. It is used to connect a client to a server. You could use passwords or keys to authenticate yourself to the server. A typical SSH command looks like –

# Typical SSH command that prompts user for password (if enabled in SSH config)
ssh user@myhost.server.com

# Typical SSH command that uses keys
ssh -i ~/.ssh/id_rsa user@myhost.server.com

Note: I am going to restrict the conversation in this post to SSH keys because –

1. Passwords are not as secure as keys.
2. Passwords need to be re-entered every time a user is connecting to the machine.
3. Using keys, one can setup an agent and thus use agent forwarding which makes it super easy and powerful to use.
4. We will be deep diving on the “under the hood” fundamentals of agent forwarding.

SSH Agents and Agent Forwarding

SSH Agents are processes on the client that allow you to hold your keys while connecting to a server. They allow you to perform “Agent Forwarding”. In agent forwarding your agent can push your SSH keys from the client to the server and allow you to use the keys across a set of machines.

Example:

Let’s say there are 3 machines – A, B and C. You need to perform a log collection on machine C but it can be accessed only via machine B. How do you do it?

Assuming that A is the client, you can create an SSH agent on machine A and connect to machine B. Assuming that the agent has been forwarded to B, it becomes the client. Assuming that the keys on machine A allows you to connect to machine C, you can connect to machine C and perform the log collection.

If you continue beyond this point, you will understand the “under the hood” details of agent forwarding. We will revisit the example at the end.

SSH and Linux

In this section I’ll quickly walk through some fundamental things in Linux with regard to SSH.

~/.ssh – Directory on a client and server that holds SSH related information.

~/.ssh/config – A file that holds SSH configuration settings.

~/.ssh/authorized_keys – A file that is present on the server containing the public key of any user who is allowed to connect to the machine. Each line is a public key that is authorized to access the machine. If your public key is not present within this file (for that particular user), you cannot access the server.

~/.ssh/id_rsa (and other variants) – The default value of private key SSH will connect with. This can be overridden with the -i option on ssh.

~/.ssh/known_hosts – A file that is present on the client with the fingerprints of the machines it is familiar with. If you are connecting to a machine and your system is not familiar with it’s fingerprints, it will raise a prompt asking you to verify the fingerprint (with yes/no). Once verified, SSH stores them in this file and doesn’t prompt until the fingerprint changes.

SSH_AUTH_SOCK – This is an environment variable that is present on the client which contains the path of the linux file socket which SSH agent uses to communicate with other process.

SSH_AGENT_PID – This is an environment variable that is present on the client which points to the daemon of the SSH agent.

Deep Dive – SSH_AUTH_SOCK and SSH_AGENT_PID

I hope you are with me so far because things get a bit murky at this point. Remember when I said, the SSH_AUTH_SOCK and SSH_AGENT_PID environment variables are present in the client? This bit is a slightly untrue. They are not set by default and need to be set by running the SSH agent.

Running the following commands will validate if they are already set on the client or not.

$ echo "$SSH_AUTH_SOCK"

$ echo "$SSH_AGENT_PID"

SSH Agent can be run using the following command –

$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-QMeZ9mIl1uZq/agent.36307; export SSH_AUTH_SOCK;
SSH_AGENT_PID=36308; export SSH_AGENT_PID;
echo Agent pid 36308;

But running the above command only returns the variables and DOES NOT explicitly set them. To create and set them, you need to run the following –

$ eval $(ssh-agent)
Agent pid 36310

$ echo "$SSH_AUTH_SOCK"
/tmp/ssh-O3ntg6bK784b/agent.36309
$ echo "$SSH_AGENT_PID"
36310

Adding/Listing Keys in SSH agent

If you have followed thus far, let’s take this up a notch. Adding an agent doesn’t mean that your keys are automatically added to them. You can use the ssh-add command to check what keys have been added, add a key, remove a key or remove all keys etc.

$ ssh-add -L
The agent has no identities.

The above command tells you that the agent has no keys. If you need to add keys, you could do it using this command –

$ ssh-add ~/.ssh/my_key 
Identity added: /home/user/.ssh/my_key (/home/user/.ssh/my_key)

You can remove the above key by running this command –

$ ssh-add -d ~/.ssh/my_key 
Identity removed: /home/user/.ssh/my_key (/home/user/.ssh/my_key)
# All keys present in your agent can be removed using -D
$ ssh-add -D

There is one additional thing that ssh-add allows us to do. Remember how we ran the echo command on the environment variable to check if the agent was running or not? Turns out, ssh-add can be used to check if your agent is running or not.

If you run, ssh-add and your agent is not running, you will get the following message –

$ ssh-add
Could not open a connection to your authentication agent.

Passing Agents via SSH

If you are reading this, kudos! By now, you already know a great level of detail about SSH and we are heading into the “Advanced”/”Expert” mode now.

At the beginning of the blog post, we talked about agent forwarding. We are going to actually do that now. Assuming that you already have an agent running on your client and you have access to the server in the authorized_keys file, using the -A option in ssh allows you to forward your agent.

$ ssh -A user@myhost.server.com

That’s it!

Demystifying Agent Forwarding

Phew! We’ve covered a lot already but all that we did so far was actually a base for what is coming next. Remember when I said the SSH_AUTH_SOCK and SSH_AGENT_PID environment variables are present in the client (twice before)? I left out one more important detail in the mix.

The whole process of passing an existing SSH agent from the client to server using the -A option of ssh is so that the environment variables are created on the server. Thus, technically you would say it contains in your server as well but it is not true. By setting these variables, your server can act as a client for other machines and you can continue to connect via SSH.

Here are the commands I ran on a machine to test this –

# No agent forwarding
user@client$ ssh 192.168.0.3
# Server cannot act as a client as no agent is present
user@192.168.0.3$ echo "$SSH_AUTH_SOCK"

---
# Agent forwarding enabled
user@client$ ssh -A 192.168.0.3
# Server can now act as a client to connect to other machines as agent is present.
user@192.168.0.3$ echo "$SSH_AUTH_SOCK"
/tmp/ssh-5PlyyFpqub/agent.17101

Example Revisited:

The “under the hood” explanation of the example is –

Assuming that A is the client, you can create an SSH agent (using the ssh-agent and the ssh-add commands) on machine A and connect to machine B. Assuming that the agent has been forwarded to B, it acts as a client (because now the SSH_AUTH_SOCK and SSH_AGENT_PID variables are automatically set and there is an agent daemon process running). Assuming that the keys on machine A allows you to connect to machine C, you can connect to machine C and perform the log collection.

In the above example – machine A can be called the source host, machine C can be called the destination host and machine B can be called the jump host.

References and Additional Reading

[1] http://www.unixwiz.net/techtips/ssh-agent-forwarding.html

[2] http://blog.joncairns.com/2013/12/understanding-ssh-agent-and-ssh-add/