Managing Multiple SSH Keys on Both Ends

You might have multiple servers that you need to log into from one client machine.

Similarly, you might have multiple devices that need to all log into the same server.

You might even have multiple clients that all need to be able to log into multiple servers, all using SSH with public key authentication -- meaning no password.

This post will guide you through all of the above scenarios, and explain the "need to know" bits of ssh, ssh-keygen, and ssh-copy-id .

If you're already familiar with generating and copying keys, skip to the Multiple Keys on the Client section.

Prework

Note: These instructions assume you are on MacOS, Linux, or WSL

Make sure `ssh` is installed

First, execute the following command on both the client and the server to make sure ssh is installed:

which ssh

The command above should output a file path.

If not, install ssh:

sudo apt install openssh-client

after installing, you may need to restart your terminal for the system to recognize ssh as a command

`authorized_keys` file

On the server, we'll want to check for the ~/.ssh/authorized_keys file. If it doesn't exist, don't worry -- it will be automatically created with the ssh-copy-id command in the following steps

The authorized_keys file contains a list of public keys (one per line). The end of each line shows the username and hostname of the machine from which the public key was generated.

If the file doesn't exist, it just means that you have not yet copied any keys to this server with the ssh-copy-id command.

`.ssh` folder

On the client, this folder lives in the /home/<yourUserName> directory.

If this folder doesn't exist, it will be created when you generate the key pair with ssh-keygen.

Any private/public key pairs you generate will live in this folder.

Private/public key pairs share the same name; the private key has no extension, and the public key has a .pub extension.

Again, if there are no keys here don't worry.

However, if there are keys here, you may wish to back them up, as there is the potential to overwrite them in the following steps if we're not careful.

Backing up existing keys might look something like this:

mkdir backup_ssh_keys # make a backup folder
cp ~/.ssh -r backup_ssh_keys # copy the ~/.ssh directory to the backup folder
ls -la backup_ssh_keys/.ssh # confirm the keys were copied successfully

`ssh-keygen`

Generating a key pair is done via the ssh-keygen command.

If you have ssh installed, then you have access to ssh-keygen.

The public/private key pairs are generated using an encryption algorithm. The specifics of the algorithm aren't pertinent to the scope of this blog post. However, the algorithm you supply to the ssh-keygen command could come in handy, depending on your situation.

(More on that later).

The important thing to understand is the relationship between the public and private keys:

I like to think of the public key as a lock on a door, and the private key is what unlocks it.

Check the `~/.ssh` folder

ls -la ~/.ssh

If there are no keys in this folder, you can move to the next section.

If there are keys here, take note of the names of the keys. You'll need to reference them in a moment.

Generate the key

If the ssh-keygen command is run without any parameters, it will use the rsa algorithm by default, and generate the keys id_rsa and id_rsa.pub in your ~/.ssh folder.

The algorithm that is used to generate the key pair is also the default name of the key pair.

Using the default key name and location makes it easier to ssh into the server without any additional configuration.

This is why we took down the names of any keys that already existed in our ~/.ssh folder -- if there is already a key named id_rsa in that folder, then we may want to consider using a different algorithm so that we can keep the default name for an easier configuration.

If you're set on using the same algorithm, don't worry -- we'll cover that too.

Generate the key using either ssh-keygen on its own for the defaults, or use ssh-keygen -t <algorithm> to use a specific algorithm (man ssh-keygen to see all algorithm options).

You are then prompted to either accept the default name and path for the file or supply a new one:

Generating public/private rsa key pair.
Enter file in which to save the key (/home/<yourUsername>/.ssh/id_rsa):

If you already have a key named id_rsa in this folder, you'll want to provide a new name for the key here. Otherwise, the old key will be overwritten, and you'll no longer be able to access that server resource until you restore from the backup we created or a new key pair is generated.

If there are no conflict concerns, go ahead and accept the default.

`ssh-copy-id`

This command is used from the client side to copy the public key over to the authorized_keys file on the server. Just like ssh-keygen, you have access to this command as long as openssh-client is installed.

If you've only got one key pair in your .ssh folder, it's pretty easy.

If you've got more than one key pair, it's still pretty easy (ssh-copy-id uses the most recent key pair in your .ssh folder to determine which key to copy over)

if it's the most recently-generated key you're wanting to copy over (and you used the default key name and location), you can leave the command as-is
if not, you can specify the path to the file using the -i flag

Run the command with this syntax:

ssh-copy-id username@server

Again, if you're not wanting to use the most-recently-generated key pair (or need to specify the non-default key name you set), use the -i flag, specifying the file path of the public key like this:

ssh-copy-id -i /home/<yourUsername>/.ssh/<nameOfKey>.pub

If everything worked as expected, you will be prompted for the password for your user on the server, and get a message like this:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'user@server'"
and check to make sure that only the key(s) you wanted were added.

`ssh`

To doubly confirm that everything worked correctly, let's log out of the server and ssh back into it to make sure we aren't prompted for a password.

exit # log out of the server

# now back on client machine
ssh user@server

You should be logged right in, without being prompted for a password.

If you were prompted, log out of the server again and run the following to see what the issue is:

ssh user@server -v #verbose output for debugging

The most likely cause for this issue is that you need to specify the path to the key file because you supplied a custom filename to ssh-keygen instead of using the default. If that is the case, try running the command with the -i flag, and supply the path to the file:

ssh -i /home/.ssh/<yourCustomKey> user@server

If this works, but you don't want to have to type out the file path each time you access your server, we cover a few options in the next section.

Multiple Keys on the Client

You might need to access multiple servers from the same client machine. That means you need multiple keypairs on the client.

I prefer to be able to log into a server without having to remember the specific key (or path to the key) that I need to use. I want to be able to run ssh user@server every time, and have it work consistently.

For this to work, we have some options:

if the algorithm used for the key isn't important, simply generate keys for other servers with different algorithms
- this works if you only need to log into a small number of servers, as there are a limited number of algorithms
another option is to use a configuration file to specify which key to use for which server
- more flexible, but requires knowing how the configuration file works

Example with Different Algorithms

From my laptop, I need to access two servers:

a remote server hosted on vultr
a Raspberry Pi on my local network

NOTE: for the vultr instance we are using root to keep the example simple. Follow instructions for your cloud provider on setting up SSH keys for a non-root user and then disable root login and password login.

Using the `ed25519` algorithm for vultr

First I would generate the key pair for the vultr server using the ed25519 algorithm:

ssh-keygen -t ed25519

I press Enter through all the prompts to accept the default file name and location, choosing not to set a passphrase.

Now I copy the key over to my vultr server with ssh-copy-id

ssh-copy-id root@myvultrserver.com

I get the following output, and am then prompted for the root user's password:

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/idev/.ssh/id_ed25519.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@myvultrserver.com's password:

I enter my password for the server and get the following output:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@myvultrserver.com'"
and check to make sure that only the key(s) you wanted were added.

Because the default file path was used when generating the key, no file path needs to be specified with the ssh command and I can execute it just like it instructs me to in the output above:

ssh root@myvultrserver.com

I am successfully logged in using public key authentication since I was not prompted for a password.

Using the default `rsa` algorithm for the Raspberry Pi

Now that I've squared away my ssh configuration to my vultr instance, I generate the key pair for the Raspberry Pi, using the default rsa algorithm

ssh-keygen

Again I accept all the defaults at the prompts.

Since ssh-copy-id uses the most recently generated default identity file, we can use it just like we did for vultr, just replacing the server with the raspberry pi instead:

ssh-copy-id sharif@raspberrypi.local

We get similar output as before; the only difference is the username and host is for the raspberry pi instead of the vultr instance:

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/idev/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
sharif@raspberrypi.local's password:

I enter the password and get the message that the key was added, and am instructed to attempt the public key authentication:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'sharif@raspberrypi.local'"
and check to make sure that only the key(s) you wanted were added.

I type the command as instructed in the output

ssh sharif@raspberrypi.local

I am not prompted for a password once again, meaning the public key authentication was successful.

Example with Configuration File

Let's pretend that any keys generated must use the ed25519 algorithm.

For this example, we'll use the same two servers, but clear out the configuration so we can start fresh:

# as root on vultr server
rm ~/.ssh/authorized_keys

# as sharif on raspberry pi
rm ~/.ssh/authorized_keys

# as idev on client machine
rm ~/.ssh/rd_rsa
rm ~/.ssh/rd_rsa.pub
rm ~/.ssh/id_ed25519
rm ~/.ssh/id_ed25519.pub

Now we can begin generating the keys on the client machine.

Since we know ahead of time that

we want to use the ed25519 algorithm to generate both key pairs
the algorithm used to generate the key pair is the default name of the key pair

... we decide that we should not accept the default key pair name when prompted (otherwise the second key would overwrite the first), and give each key pair a more distinguishable name.

Configuration for vultr server

# on client machine
ssh-keygen -t ed25519

This time when we are prompted, instead of accepting the default, we provide a custom file name, but in the default folder:

Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/idev/.ssh/id_ed25519): /home/idev/.ssh/vultr_ed25519

Now we try to copy the key over to the vultr instance, but we get an error

ssh-copy-id root@myvultrserver.com
/usr/bin/ssh-copy-id: ERROR: No identities found #error

This is because we supplied a custom file name, and by default, ssh-copy-id only knows to look for id_rsa, id_ed25519, etc.

To fix this, we use -i and supply our custom key file. Since this is a one-time command, using the -i flag is fine.

ssh-copy-id -i /home/idev/.ssh/vultr_ed25519 root@myvultrserver.com

Now we see our familiar message, and are asked for root@myvultrserver.com's password, and are then instructed to attempt logging in via ssh

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@myvultrserver.com'"
and check to make sure that only the key(s) you wanted were added.

So we try to execute that command:

ssh root@myvultrserver.com

But we're prompted for a password. This is because we need to either:

supply the identity file with the -i flag again (remember that by default it only knows to look for the keys with the name of the algorithm)
or
use a configuration file

Since we are striving for consistency and ease of use, we opt for the configuration file (so we don't have to supply the path of the key want to use every time we ssh into the server).

First we create the configuration file and secure the permissions:

touch ~/.ssh/config && chmod 600 ~/.ssh/config

Now we can open ~/.ssh/config in a text editor and write our first configuration:

Host myvultrserver.com
    HostName myvultrserver.com
    User root
    IdentityFile ~/.ssh/vultr_ed25519

Once we save the file, we can ssh into our server like this:

ssh myvultrserver.com

Since everything is in the configuration file, we just set the user and the file path once, and now we have a convenient way to access the server.

Now we can do the same thing for our Raspberry Pi connection.

Configuration for Raspberry Pi

We follow the same process as we did for vultr, except we name the identity file something else this time.

# on client machine
ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/idev/.ssh/id_ed25519): /home/idev/.ssh/rpi_ed25519

Now we copy the key over to the Raspberry Pi.

ssh-copy-id -i /home/idev/.ssh/rpi_ed25519 sharif@raspberrypi.local

We get the expected output, along with one last password prompt

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/idev/.ssh/rpi_ed25519.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
sharif@raspberrypi.local's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'sharif@raspberrypi.local'"
and check to make sure that only the key(s) you wanted were added.

Now we can add to the configuration file:

# ~/.ssh/config

Host myvultrserver.com
    HostName myvultrserver.com
    User root
    IdentityFile ~/.ssh/vultr_ed25519

Host raspberrypi.local
    HostName raspberrypi.local
    User sharif
    IdentityFile ~/.ssh/rpi_ed25519

Finally, we access the Raspberry Pi with

ssh raspberrypi.local

Accessing one server from multiple clients

We've already done most of the heavy lifting, and at this point, we just need to connect the dots.

Connecting to multiple servers from one client (which we did in the last section) is more involved than accessing one server from multiple clients since we pretty much just do the same thing on each client.

In fact, if we use the same names for the keys when they are generated on the new client, we can just copy the existing configuration file on each new client.

Let's go through an example.

Example

I just got a new laptop, and I want to be able to access my vultr instance from it.

In the previous examples, the file paths looked like /home/idev/.ssh/...since idev was the name of the user on that computer.

Just for funsies, we'll use ihop as the user on this laptop.

Once I have openssh-client installed on the new laptop, I generate a key, and name it the same thing as I did for the other laptop

# on new laptop
ssh-keygen -t ed25519

# output
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/ihop/.ssh/id_ed25519): /home/ihop/.ssh/vultr_ed25519 # same key name, but on a different machine

Now our normal song and dance of copying the key over

ssh-copy-id -i /home/ihop/.ssh/vultr_ed25519 root@myvultrserver.com

Then we make our new config file and adjust the permissions

touch ~/.ssh/config && chmod 600 ~/.ssh/config

And now we can simply copy the same configuration from our other laptop, since the configuration is using relative paths, and does not specify a username

# ~/.ssh/config

Host myvultrserver.com
    HostName myvultrserver.com
    User root
    IdentityFile ~/.ssh/vultr_ed25519

Now we can use the same command (ssh myvultrserver.com) on both laptops and it works the same way.

This pattern can be followed for each new client machine that needs to access the server.

Access multiple servers from multiple clients

Similar to the last section, this sounds more complicated than it is. You may have already figured it out if you've read this far.

In this last section, we accessed the same server from a new client machine by generating a new key with the same name, then copying the existing configuration to the new client machine.

As it stands, we have two clients accessing the vultr server, but what if we also wanted the new laptop to have access to the Raspberry Pi?

Example

We follow the same steps as the last section, except we use the name of the first Raspberry Pi key we created instead of the vultr key:

# on new laptop
ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/ihop/.ssh/id_ed25519): /home/ihop/.ssh/rpi_ed25519

And copy the key

ssh-copy-id -i /home/ihop/.ssh/rpi_ed25519 sharif@raspberrypi.local

And update our configuration file (it's now identical to the one on the idev laptop):

# ~/.ssh/config

Host myvultrserver.com
    HostName myvultrserver.com
    User root
    IdentityFile ~/.ssh/vultr_ed25519

Host raspberrypi.local
    HostName raspberrypi.local
    User sharif
    IdentityFile ~/.ssh/rpi_ed25519

Apply this same pattern for as many clients and servers as you need.

Managing Multiple SSH Keys on Both Ends

Comments

More from this blog

On Side Projects

How to Run a Minecraft Server: Multiple Approaches

An Intro to Minecraft Mods

Installing Java on Ubuntu (without breaking your shell)

Learning chroot

Prework

Make sure `ssh` is installed

`authorized_keys` file

`.ssh` folder

`ssh-keygen`

Check the `~/.ssh` folder

Generate the key

`ssh-copy-id`

`ssh`

Multiple Keys on the Client

Example with Different Algorithms

Using the `ed25519` algorithm for vultr

Using the default `rsa` algorithm for the Raspberry Pi

Example with Configuration File

Configuration for vultr server

Configuration for Raspberry Pi

Accessing one server from multiple clients

Example

Access multiple servers from multiple clients

Example

Command Palette

Comments

More from this blog

Prework

Make sure ssh is installed

authorized_keys file

.ssh folder

ssh-keygen

Check the ~/.ssh folder

Generate the key

ssh-copy-id

ssh

Multiple Keys on the Client

Example with Different Algorithms

Using the ed25519 algorithm for vultr

Using the default rsa algorithm for the Raspberry Pi

Example with Configuration File

Configuration for vultr server

Configuration for Raspberry Pi

Accessing one server from multiple clients

Example

Access multiple servers from multiple clients

Example

Make sure `ssh` is installed

`authorized_keys` file

`.ssh` folder

`ssh-keygen`

Check the `~/.ssh` folder

`ssh-copy-id`

`ssh`

Using the `ed25519` algorithm for vultr

Using the default `rsa` algorithm for the Raspberry Pi