Why the RHEL8 system do not generate SSH host keys automatically when missing?

Question

On the RHEL 8 and previous it is usual, that the SSH host keys in /etc/ssh are generated automatically by sshd service when missing. Usually there should be:

/etc/ssh/ssh_host_ecdsa_key
/etc/ssh/ssh_host_ecdsa_key.pub
/etc/ssh/ssh_host_ed25519_key
/etc/ssh/ssh_host_ed25519_key.pub
/etc/ssh/ssh_host_rsa_key
/etc/ssh/ssh_host_rsa_key.pub

Restart of the node or even systemctl restart sshd should be sufficient.

But as of the minor version RHEL 8.7 this may not work any more and the sshd crashes complaining about missing host keys in the journal log. Why? How can I solve this?

Answer 1

The sshd service by default calls sshd-keygen.target, which checks availability of host keys in /etc/ssh directory and generates it when missing.

However this well known functionality can be blocked by the new version of cloud-init. As of cloud-init-22.1-5.el8.noarch there is new file:

/etc/systemd/system/sshd-[email protected]/disable-sshd-keygen-if-cloud-init-active.conf

with content:

# In some cloud-init enabled images the sshd-keygen template service may race
# with cloud-init during boot causing issues with host key generation.  This
# drop-in config adds a condition to [email protected] if it exists and
# prevents the sshd-keygen units from running *if* cloud-init is going to run.
#
[Unit]
ConditionPathExists=!/run/systemd/generator.early/multi-user.target.wants/cloud-init.target

So when you use the cloud-init you have 2 options now:

1. Generate host keys manually with ssh-keygen -A (see How to change a SSH host key? for more details and options.
2. Comment the condition

Simply put the # sign before ConditionPathExists...

[Unit]
#ConditionPathExists=!/run/systemd/generator.early/multi-user.target.wants/cloud-init.target

Then reload the systemd configuration with systemctl daemon-reload. The usual behavior should be working again.

Answer 2

If you are using cloud-init, then you can fix it by adding 'ssh' module in cloud-init config file in section "cloud_init_modules". Refer cloud-init docs

This will generate ssh host key during the firstboot.

You can test this in instance where you are having issue:

cloud-init config file: /etc/cloud/cloud.cfg
Check if you have 'ssh' module in "cloud_init_modules" section
Run this command to verify cloud-init action. NOTE: This will REBOOT your instance and run the cloud-init action from the scratch.
# cloud-init clean --reboot

Verify the ssh host key in /etc/ssh/ directory and sshd service status.

Answer 3

Try explicitly setting ssh key types in cloud.cfg

resize_rootfs_tmp: /dev
ssh_deletekeys:   1
ssh_genkeytypes: ['rsa', 'ecdsa', 'ed25519']