OpenSSH Public Key Rotation

User Keys | Host Keys | host_keys Scalability Problems

OpenSSH public key material should be rotated periodically, just as X.509 certificates are. This should be done yearly via a calendar invite, as this creates a predictable event around which the rotation tasks can be scheduled. Reasons to rotate keys include:

The Config::OpenSSH::Authkey Perl module offers a programatic interface to the OpenSSH authorized_keys file. This can help automate key user rotations.

User Keys

User keys can be updated via the ssh-keygen(1) command on the client, and the public key material copied via various means to any relevant servers. OpenSSH Public Key Authentication goes into more detail on setting up pulic key authentication. Public keys can also be maintained under version control or database, and installed automatically on the required hosts via custom software. This avoids the tedium and mistakes possible when a human manually copies the key material to many hosts.

While testing the new key material, note that shared connections may mask connectivity problems. Turn off connection sharing during testing, or write the rotation software such that a rollback is performed automatically should the user not be able to intervene.

$ ssh -S /dev/null server.example.org
Control socket connect(/dev/null): Socket operation on non-socket
ControlSocket /dev/null already exists, disabling multiplexing
?
server%  

Host Keys

The host keys should also be rotated. This requires a restart of the sshd process. Changing the host keys will also require updates to various known_hosts files on various systems, which may break automated jobs that assume a particular host key, or are set to ask should a host key mismatch occur.

#!/bin/sh
#
# Example SSH host key rotation.

# Different vendors hide the keys in /etc/ssh or /etc
DIR=/etc/ssh

# TODO This prompts to overwrite, and offers no
# backup for rollback. This also supports SSHv1,
# which should be disabled where possible.
while read type file; do
ssh-keygen -q -N '' -t $type -f "$DIR/$file"
done <<EOF
dsa ssh_host_dsa_key
rsa1 ssh_host_key
rsa ssh_host_rsa_key
EOF

Host keys may also be updated through host lease returns, or by retiring virtual servers. This ties the lifetime of the host SSH key to that of the system, so may not be viable for servers kept beyond the lifetime of the lease (ideally no more than a few years), or for long running virutal systems.

host_keys Scalability Problems

Larger organizations with many thousands of hosts will likely not support the known_hosts file, or will replace lookups from this flat file with some other code path, for example to a local cached database that is populated from a central source. If the known_hosts is not supported, then individual hosts will have random entries for host keys, depending on what systems they connect to, and how often. Programs at sites like this may need to disable the host key checks: this increases the risk of a man-in-the-middle attack, though decreases the risk that the program will fail due to an unknown host key being presented, should the host key change through normal key rotation, or when connecting to a new server for the service.

ssh -o CheckHostIP=no,StrictHostKeyChecking=no ?

As an alternative, programs that create SSH connections would use strict host key checking, though would behave properly should the connectivity be disrupted, and escalate the issue to a human to resolve. This will increase the complexity and time to write software that connects via SSH.