SSH known_hosts Management with CFEngine

Host Key Collection with ssh-keyscan | Distributing ssh_known_hosts

For small sites, a changed Secure Shell (SSH) host key poses only minor problems, as only a few systems might be affected. At larger sites, the host key data should be maintained with configuration management, so that any SSH host key changes can be propogated throughout the site. CFEngine can maintain a global ssh_known_hosts file for OpenSSH.

Alternatives inculde storing the host keys in a directory service that clients lookup when needed. With a directory service, servers could upload new host keys after being installed or rotating the host keys, and other systems would immediately see the new keys. On the other hand, should the directory service fail, so will new SSH connections. The following method requires an additional step, plus time for CFEngine to syncronize the new ssh_known_hosts file to all systems.

module:authkey can help automate authorized_keys file management.

Host Key Collection with ssh-keyscan

The global ssh_known_hosts file should be maintained under version control. I keep the file at masterfiles/ssh/ssh_known_hosts under the repository, and use a set of scripts around ssh-keyscan(1) to add or update host keys manually. When collecting host keys from other servers, care should be taken that no attacker has supplied a malicious key. The man-in-the-middle threat can be reduced by using an authoritative name server on the system running ssh-keyscan(1). Also, have each system compare its own keys with the keys the scanning system collected, and alert if differences appear.

Scripts to assist key collection include:

With a Makefile, make add or make update can be used to update the host key list:

# usage:
# echo hostname | make add
add:
@expand-hl | add-ssh-hostkeys ssh_known_hosts

update:
@update-ssh-hostkeys ssh_known_hosts

Alternative Collection Methods

If all systems at a site run cfservd, then the host keys could be copied between systems automatically. Other options include uploading the keys from client systems to a collection system via pre-established SSH trust or a HTTPS server.

Sites with thousands of systems should either use a directory service, or break the ssh_known_hosts down into different groups, as some systems may not need host keys for every other system at the site.

Distributing ssh_known_hosts

Use a copy statement to distribute the ssh_known_hosts file, with variables to account for vendor specific differences.

control:
macosx::
ssh_known_hosts = ( /etc/ssh_known_hosts )

freebsd|linux|openbsd::
ssh_known_hosts = ( /etc/ssh/ssh_known_hosts )

copy:
any::
${masterfiles}/etc/ssh/ssh_known_hosts
dest=${ssh_known_hosts}
mode=0444
server=${policyhost}
type=checksum backup=false

Users with old host keys in their ~/.ssh/known_hosts files will see key warnings. Old keys must be deleted in favor of the globally maintained keys. For systems not under configuration management, consider making the ssh_known_hosts information verifiable from a Transport Layer Security (TLS) encrypted website.