Permission denied (publickey) error when managing GCP machine via Ansible (running on GCP VM) despite successful SSH connection

Question

I deployed two machines to GCP via Terraform. Let's call them control-host and target-host. I want to manage the target-host via Ansible installed on the control-host. Unfortunately, I keep getting the following error no matter what I do:

10.128.100.3 | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: [email protected]: Permission denied (publickey).",
    "unreachable": true
}

The problem perplexes me because it is possible to ssh from control-host to target-host without any problem. I would assume that if "raw" ssh is possible, so is using it via Ansible.

Here's the list of things I did and tried:

1. Install Ansible on the control-host.
2. Generate a key pair on the control host. Copy the public key.
3. SSH on the target host. Open ./.ssh/authorized_keys. Paste the public key.
4. Execute ansible all -vvv -m ping. I got the error mentioned above.
5. Open /etc/ansible/ansible.cfg. Add the following:

[defaults]
remote_user = gcp_user
host_key_checking = False
ansible_ssh_common_args='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'
private_key_file = /home/gcp_user/.ssh/t_k

(t_k is the name of the key)

6. Run ansible all -vvv -m ping again. Same error.
7. Run ansible all -vvv -m ping --key-file=/home/gcp_user/.ssh/t_k. Same error.
8. Open /etc/ansible/hosts. Add the following:

10.128.100.3 ansible_ssh_private_key_file=/home/gcp_user/.ssh/t_k

Same story.

SSH logs on the server side say the following: Connection closed by authenticating user gcp_user 10.128.100.2 port 34470 [preauth]

I followed the recommendations from the following threads and none of them helped:

1. https://stackoverflow.com/questions/64681944/create-and-setup-gcp-vms-with-ansible-ssh-permission-denied-publickey
2. https://stackoverflow.com/questions/55897136/ansible-failed-to-connect-to-the-host-via-ssh-permission-denied-publickey
3. https://stackoverflow.com/questions/57424995/ansible-remote-user-root-ssh-permission-denied-publickey
4. https://stackoverflow.com/questions/33280244/ssh-error-permission-denied-publickey-password-in-ansible

---

Here's the output of the command ansible all -vvv -m ping -e 'ansible_ssh_extra_args="-vvv"':

1st part:

ansible [core 2.12.10]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/gcp_user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /home/gcp_user/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.8.10 (default, Mar 13 2023, 10:26:41) [GCC 9.4.0]
  jinja version = 2.10.1
  libyaml = True
Using /etc/ansible/ansible.cfg as config file
host_list declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
script declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
auto declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Parsed /etc/ansible/hosts inventory source with ini plugin
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.
META: ran handlers
<10.128.100.3> ESTABLISH SSH CONNECTION FOR USER: gcp_user
<10.128.100.3> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/gcp_user/.ssh/t_k"' -o KbdInteractiveAuthentication=no 
-o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="gcp_user"' -o ConnectTimeout=10 -vvv -o 
'ControlPath="/home/gcp_user/.ansible/cp/becfdd0705"' 10.128.100.3 '/bin/sh -c '"'"'echo ~gcp_user && sleep 0'"'"''
<10.128.100.3> (255, b'', b'OpenSSH_8.2p1 Ubuntu-4ubuntu0.5, OpenSSL 1.1.1f  31 Mar 2020\r\ndebug1:
 Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 19:
  Including file /etc/ssh/ssh_config.d/50-cloudimg-settings.conf depth 0\r\ndebug1: 
  Reading configuration data /etc/ssh/ssh_config.d/50-cloudimg-settings.conf\r\ndebug1: 
  /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.128.100.3 is address\r\ndebug1:
   auto-mux: Trying existing master\r\ndebug1: Control socket "/home/gcp_user/.ansible/cp/becfdd0705" does not exist\r\ndebug2: 
   ssh_connect_direct\r\ndebug1: Connecting to 10.128.100.3 [10.128.100.3] port 22.\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1:
    fd 3 clearing O_NONBLOCK\r\ndebug1: Connection established.\r\n
    debug3: timeout: 9998 ms remain after connect\r\n
    debug1: identity file /home/gcp_user/.ssh/t_k type 0\r\n
    debug1: identity file /home/gcp_user/.ssh/t_k-cert type -1\r\n
    debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5\r\n
    debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.5\r\n
    debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.5 pat OpenSSH* compat 0x04000000\r\n
    debug2: fd 3 setting O_NONBLOCK\r\ndebug1: Authenticating to 10.128.100.3:22 as \'gcp_user\'\r\n
    debug3: hostkeys_foreach: reading file "/home/gcp_user/.ssh/known_hosts"\r\ndebug3: record_hostkey:
     found key type RSA in file /home/gcp_user/.ssh/known_hosts:1\r\ndebug3: load_hostkeys: loaded 1 keys from 10.128.100.3\r\ndebug3
     : order_hostkeyalgs: prefer hostkeyalgs: send packet: type 20\r\ndebug1: SSH2_MSG_KEXINIT sent\r\ndebug3: receive packet: type 20\r\ndebug1:
      SSH2_MSG_KEXINIT received\r\ndebug2: local client KEXINIT proposal\r\ndebug2:
       KEX algorithms:  MACs stoc:  compression ctos: [email protected],zlib,none\r\ndebug2:
        compression stoc: [email protected],zlib,none\r\ndebug2: languages ctos: \r\ndebug2: languages stoc: \r\ndebug2:
         first_kex_follows 0 \r\ndebug2: reserved 0 \r\ndebug2: peer server KEXINIT proposal\r\n
         debug2: KEX algorithms:  host key algorithms: rsa-sha2-512,: ciphers ctos:  [email protected]\r\ndebug3: send packet: type 30\r\n
         debug1: expecting SSH2_MSG_KEX_ECDH_REPLY\r\ndebug3: receive packet: type 31\r\ndebug1: Server host key:
          ssh-rsa SHA256:***/***\r\ndebug3: hostkeys_foreach: 
          reading file "/home/gcp_user/.ssh/known_hosts"\r\ndebug3: record_hostkey: found key type RSA in file /home/gcp_user/.ssh/known_hosts:1\r\n
          debug3: load_hostkeys: loaded 1 keys from 10.128.100.3\r\ndebug1: Host \'10.128.100.3\' is known and matches the RSA host key.\r\n
          debug1: Found key in /home/gcp_user/.ssh/known_hosts:1\r\ndebug3: 
          send packet: type 21\r\ndebug2: set_newkeys: mode 1\r\ndebug1: rekey out after 134217728 blocks\r\ndebug1: SSH2_MSG_NEWKEYS sent\r\ndebug1:
           expecting SSH2_MSG_NEWKEYS\r\ndebug3: receive packet: type 21\r\ndebug1: SSH2_MSG_NEWKEYS received\r\ndebug2: set_newkeys: mode 0\r\ndebug1:
            rekey in after 134217728 blocks\r\ndebug1: Will attempt key: normal ECDSA SHA256:*** agent\r\ndebug1: 
            Will attempt key: /home/gcp_user/.ssh/t_k RSA SHA256:***/*** explicit\r\ndebug2: pubkey_prepare: done\r\n
            debug3: send packet: type 5\r\ndebug3: receive packet: type 7\r\ndebug1: SSH2_MSG_EXT_INFO received\r\ndebug1:
             kex_input_ext_info: server-sig-algs=<s\ndebug3: receive packet: type 6\r\ndebug2: service_accept: ssh-userauth\r\ndebug1:
              SSH2_MSG_SERVICE_ACCEPT received\r\ndebug3: send packet: type 50\r\ndebug3: receive packet: type 51\r\
              ndebug1: Authentications that can continue: publickey\r\ndebug3: start over, passed a different list publickey\r\ndebug3:
             authmethod_lookup publickey\r\ndebug3: remaining preferred: ,gssapi-keyex,hostbased,publickey\r\ndebug3: authmethod_is_enabled publickey\r\n
             debug1: Next authentication method: publickey\r\ndebug1: Offering public key:
              normal ECDSA SHA256:*** agent\r\n
              debug3: send packet: type 50\r\ndebug2: we sent a publickey packet, wait for reply\r\ndebug3: receive packet: type 51\r\ndebug1: 
              Authentications that can continue: publickey\r\ndebug1: Offering public key: /home/gcp_user/.ssh/t_k RSA SHA256:***/*** explicit\r\n
              debug3: send packet: type 50\r\ndebug2: we sent a publickey packet, wait for reply\r\ndebug3: receive packet: type 51\r\n
              debug1: Authentications that can continue: publickey\r\ndebug2: we did not send a packet, disable method\r\n
    debug1: No more authentication methods to try.\r\[email protected]: Permission denied (publickey).\r\n')

2nd part:

10.128.100.3 | UNREACHABLE! => {
  "changed": false,
  "msg": "Failed to connect to the host via ssh: OpenSSH_8.2p1 Ubuntu-4ubuntu0.5, OpenSSL 1.1.1f  31 Mar 2020\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3:
   /etc/ssh/ssh_config line 19: Including file /etc/ssh/ssh_config.d/50-cloudimg-settings.conf depth 0\r\ndebug1:
    Reading configuration data /etc/ssh/ssh_config.d/50-cloudimg-settings.conf\r\ndebug1: /etc/ssh/ssh_config line 21:
     Applying options for *\r\n
     debug2: resolve_canonicalize: hostname 10.128.100.3 is address \r\n
     debug1: auto-mux: Trying existing master\r\ndebug1:
     Control socket \"/home/gcp_user/.ansible/cp/becfdd0705\" does not exist\r\n
     debug2: ssh_connect_direct\r\n
     debug1: Connecting to 10.128.100.3 [10.128.100.3] port 22.\r\n
     debug2: fd 3 setting O_NONBLOCK\r\n
     debug1: fd 3 clearing O_NONBLOCK\r\n
     debug1: Connection established.\r\n
     debug3: timeout: 9998 ms remain after connect\r\n
     debug1: identity file /home/gcp_user/.ssh/t_k type 0\r\n
     debug1: identity file /home/gcp_user/.ssh/t_k-cert type -1\r\n
     debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5\r\n
     debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.5\r\n
     debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.5 pat OpenSSH* compat 0x04000000\r\n
     debug2: fd 3 setting O_NONBLOCK\r\n
     debug1: Authenticating to 10.128.100.3:22 as 'gcp_user'\r\n
     debug3: hostkeys_foreach: reading file \"/home/gcp_user/.ssh/known_hosts\"\r\n
     debug3:  record_hostkey: found key type RSA in file /home/gcp_user/.ssh/known_hosts:1\r\n
     debug3: load_hostkeys: loaded 1 keys from 10.128.100.3\r\n
     debug3: order_hostkeyalgs: prefer hostkeyalgs:[email protected],[email protected],[email protected],rsa-sha2-512,rsa-sha2-256,ssh-rsa\r\n
     debug3: send packet: type 20\r\n
     debug1: SSH2_MSG_KEXINIT sent\r\n
     debug3: receive packet: type 20\r\n
     debug1: SSH2_MSG_KEXINIT received\r\n
     debug2: local client KEXINIT proposal\r\n
     debug2: KEX algorithms: [email protected],[email protected],
     [email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1\r\n
     debug2: compression ctos: [email protected],zlib,none\r\n
     debug2: compression stoc: [email protected],zlib,none\r\n
     debug2: languages ctos: \r\ndebug2: languages stoc: \r\n
     debug2: first_kex_follows 0 \r\ndebug2: reserved 0 \r\ndebug2: peer server-sha2-512,hmac-sha1\r\ndebug2: compression ctos: none,[email protected]\r\n
      debug2: compression stoc: none,[email protected]\r\ndebug2: languages ctos: \r\ndebug2: languages stoc: \r\ndebug2: first_kex_follows 0 \r\nd
      ebug2: reserved 0 \r\ndebug1: kex: algorithm: curve25519-sha256\r\ndebug1: kex: host key algorithm: rsa-sha2-512\r\ndebug1: kex: server->client cipher: 
      [email protected] MAC: <implicit> compression: [email protected]\r\ndebug1: kex: client->server cipher: [email protected] MAC: 
      <implicit> compression: [email protected]\r\ndebug3: send packet: type 30\r\ndebug1: expecting SSH2_MSG_KEX_ECDH_REPLY\r\ndebug3: receive packet: type 31\r\ndebug1:
       Server host key: ssh-rsa SHA256:***\r\ndebug3: hostkeys_foreach: reading file
        \"/home/gcp_user/.ssh/known_hosts\"\r\ndebug3: record_hostkey: found key type RSA in file /home/gcp_user/.ssh/known_hosts:1\r\ndebug3:
         load_hostkeys: loaded 1 keys from 10.128.100.3\r\ndebug1: Host '10.128.100.3' is known and matches the RSA host key.\r\ndebug1: Found key in /home/gcp_user/.ssh/known_hosts:1\r\n
         debug3: send packet: type 21\r\ndebug2: set_newkeys: mode 1\r\ndebug1: rekey out after 134217728 blocks\r\ndebug1: SSH2_MSG_NEWKEYS sent\r\ndebug1: expecting SSH2_MSG_NEWKEYS\r\ndebug3:
          receive packet: type 21\r\ndebug1: SSH2_MSG_NEWKEYS received\r\ndebug2: set_newkeys: mode 0\r\ndebug1: rekey in after 134217728 blocks\r\ndebug1: 
          Will attempt key: normal ECDSA SHA256:*** agent\r\ndebug1: Will attempt key: /home/gcp_user/.ssh/t_k RSA SHA256:***/*** explicit\r\n
          debug2: pubkey_prepare: done\r\ndebug3: send packet: type 5\r\ndebug3: receive packet: type 7\r\ndebug1: SSH2_MSG_EXT_INFO received\r\ndebug1: kex_input_ext_info: server-sig-a
          debug2: service_accept: ssh-userauth\r\ndebug1: SSH2_MSG_SERVICE_ACCEPT received\r\ndebug3: send packet: type 50\r\ndebug3: receive packet: type 51\r\n
          debug1: Authentications that can continue: publickey\r\ndebug3: start over, passed a different list publickey\r\ndebug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey\r\n
          debug3: authmethod_lookup publickey\r\ndebug3: remaining preferred: ,gssapi-keyex,hostbased,publickey\r\ndebug3: authmethod_is_enabled publickey\r\n
          debug1: Next authentication method: publickey\r\ndebug1: Offering public key: normal ECDSA SHA256:*** agent\r\n
          debug3: send packet: type 50\r\ndebug2: we sent a publickey packet, wait for reply\r\ndebug3: receive packet: type 51\r\ndebug1: Authentications that can continue: publickey\r\n
          debug1: Offering public key: /home/gcp_user/.ssh/t_k RSA SHA256:***/*** explicit\r\ndebug3: send packet: type 50\r\n
          debug2: we sent a publickey packet, wait for reply\r\ndebug3: receive packet: type 51\r\ndebug1: Authentications that can continue: publickey\r\n
          debug2: we did not send a packet, disable method\r\ndebug1: No more authentication methods to try.\r\[email protected]: Permission denied (publickey).",
  "unreachable": true
}

Answer 1

I am wondering how it is possible to ssh from control-host to target-host without any problem. According to the output of ansible all -vvv -m ping -e 'ansible_ssh_extra_args="-vvv"', the target host is clearly refusing the public key supplied by the control host.

debug1: Authentications that can continue: publickey
debug1: Offering public key: /home/gcp_user/.ssh/t_k RSA SHA256:***/*** explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
[email protected]: Permission denied (publickey).

A SSH packet type 51 means SSH_MSG_USERAUTH_FAILURE, according to RFC4252:

These are the general authentication message codes:

  SSH_MSG_USERAUTH_REQUEST            50
  SSH_MSG_USERAUTH_FAILURE            51
  SSH_MSG_USERAUTH_SUCCESS            52
  SSH_MSG_USERAUTH_BANNER             53

In addition to the above, there is a range of message numbers (60 to 79) reserved for method-specific messages. These messages are only sent by the server (client sends only SSH_MSG_USERAUTH_REQUEST messages). Different authentication methods reuse the same message numbers.

Please, double check the content, ownership and permissions of /home/gcp_user/.ssh/authorized_keys on the target server. You may want to run a pure SSH authentication process for troubleshooting:

$ ssh -o BatchMode=yes -vvv -l gcp_user -i /home/gcp_user/.ssh/t_k 10.128.100.3

Also, on the target host, please temporarily set the LogLevel directive to DEBUG3 in /etc/ssh/sshd_config file.