Can't read ssh key when running remote action from 3.3 docker install

  1. Make a fresh install using docker.
    I can connect using ssh but remote commands can’t see the key.

stackstorm:
ST2_EXPOSE_HTTP=0.0.0.0:80 docker-compose up -d --force-recreate
Status: Downloaded newer image for stackstorm/st2web:3.3.0
Creating st2-docker_st2makesecrets_1 … done
Creating st2-docker_rabbitmq_1 … done
Creating st2-docker_mongo_1 … done
Creating st2-docker_redis_1 … done
Creating st2-docker_st2api_1 … done
Creating st2-docker_st2scheduler_1 … done
Creating st2-docker_st2stream_1 … done
Creating st2-docker_st2actionrunner_1 … done
Creating st2-docker_st2timersengine_1 … done
Creating st2-docker_st2notifier_1 … done
Creating st2-docker_st2auth_1 … done
Creating st2-docker_st2garbagecollector_1 … done
Creating st2-docker_st2workflowengine_1 … done
Creating st2-docker_st2rulesengine_1 … done
Creating st2-docker_st2sensorcontainer_1 … done
Creating st2-docker_st2client_1 … done
Creating st2-docker_st2web_1 … done

2.

start shell in docker instance

exec st2client bash
Welcome to StackStorm HA v3.3.0 (Ubuntu 18.04.5 LTS GNU/Linux x86_64)
** * Documentation: https://docs.stackstorm.com/**
** * Community: https://stackstorm.com/community-signup**
** * Forum: https://forum.stackstorm.com/**
** * Enterprise: StackStorm - StackStorm

** Here you can use StackStorm CLI. Examples:**
** st2 action list --pack=core**
** st2 run core.local cmd=date**
** st2 run core.local_sudo cmd=‘apt-get update’ --tail**
** st2 execution list**

3.

Create an SSH system user (default stanley user may already exist)

sudo useradd stanley
sudo mkdir -p /home/stanley/.ssh
sudo chmod 0700 /home/stanley/.ssh

4.

Generate ssh keys per directions. Ubuntu Xenial (16.04) — StackStorm 3.3.0 documentation

sudo ssh-keygen -f /home/stanley/.ssh/stanley_rsa -P “”

Authorize key-based access

sudo sh -c ‘cat /home/stanley/.ssh/stanley_rsa.pub >> /home/stanley/.ssh/authorized_keys’
sudo chown -R stanley:stanley /home/stanley/.ssh

Enable passwordless sudo

sudo sh -c ‘echo “stanley ALL=(ALL) NOPASSWD: SETENV: ALL” >> /etc/sudoers.d/st2’
sudo chmod 0440 /etc/sudoers.d/st2

Make sure Defaults requiretty is disabled in /etc/sudoers

sudo sed -i -r “s/^Defaults\s++?requiretty/# Defaults +requiretty/g” /etc/sudoers
5.
#copy authorized keys to remote host:
scp /home/stanley/.ssh/authorized_keys root@remotehost:/home/stanley/.ssh/
6.
#create user on remote host
#on the remote set permissions on remote host.
chown stanley.stanley /home/stanley/.ssh/authorized_keys
7.
#login to remote host using stanley ssh keys as the stanley user inside the docker instance.
su - stanley
ssh -i /home/stanley/.ssh/stanley_rsa stanley@remotehost
8.
#authenticate st2.
as root
st2 login st2admin -p 'Ch@ngeMe’

9.
#run test remote call
st2 run core.remote hosts=‘remotehost’ – uname -a
.
id: 5fb15374fb10759f07390a05
action.ref: core.remote
context.user: st2admin
parameters:
** cmd: uname -a**
** hosts: remotehost**
status: failed
start_timestamp: Sun, 15 Nov 2020 16:12:36 UTC
end_timestamp: Sun, 15 Nov 2020 16:12:36 UTC
result:
** error: "Unable to connect to any one of the hosts: [‘remotehost’].**

** connect_errors={**
** “remotehost”: {**
** “failed”: true,**
** “succeeded”: false,**
** “timeout”: false,**
** “return_code”: 255,**
** “stdout”: “”,**
** “stderr”: “”,**
** “error”: “Failed connecting to host remotehost. [Errno 2] No such file or directory: ‘/home/stanley/.ssh/stanley_rsa’”,**
** “traceback”: “Traceback (most recent call last):**
** File \”/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/runners/parallel_ssh.py\", line 260, in _connect**
** client.connect()**
** File \"/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/runners/paramiko_ssh.py\", line 152, in connect**
** self.client = self._connect(host=self.hostname, socket=self.bastion_socket)**
** File \"/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/runners/paramiko_ssh.py\", line 692, in _connect**
** passphrase_reqd = self._is_key_file_needs_passphrase(self.key_files)**
** File \"/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/runners/paramiko_ssh.py\", line 773, in _is_key_file_needs_passphrase**
** cls.from_private_key_file(file, password=None)**
** File \"/opt/stackstorm/st2/lib/python3.6/site-packages/paramiko/pkey.py\", line 235, in from_private_key_file**
** key = cls(filename=filename, password=password)**
** File \"/opt/stackstorm/st2/lib/python3.6/site-packages/paramiko/rsakey.py\", line 55, in init**
** self._from_private_key_file(filename, password)**
** File \"/opt/stackstorm/st2/lib/python3.6/site-packages/paramiko/rsakey.py\", line 175, in _from_private_key_file**
** data = self._read_private_key_file(\“RSA\”, filename, password)**
** File \"/opt/stackstorm/st2/lib/python3.6/site-packages/paramiko/pkey.py\", line 307, in _read_private_key_file**
** with open(filename, \“r\”) as f:**
FileNotFoundError: [Errno 2] No such file or directory: ‘/home/stanley/.ssh/stanley_rsa’*
"
** }
*
}"
** traceback: " File “/opt/stackstorm/st2/lib/python3.6/site-packages/st2actions/container/base.py”, line 114, in _do_run**
** runner.pre_run()**
** File “/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/runners/paramiko_ssh_runner.py”, line 186, in pre_run**
** self._parallel_ssh_client = ParallelSSHClient(client_kwargs)
** File “/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/runners/parallel_ssh.py”, line 76, in init**
** connect_results = self.connect(raise_on_any_error=raise_on_any_error)**
** File “/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/runners/parallel_ssh.py”, line 106, in connect**
** raise NoHostsConnectedToException(msg)**

10. make a connection just using paramiko
If i make a connection in virtualenv using paramiko it works perfectly,
running this sample python i just typed in the interpreter

import paramiko
cmd = 'echo i am connected to a remote host!'
k = paramiko.RSAKey.from_private_key_file("/home/stanley/.ssh/stanley_rsa")
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
print (“connecting”)
ssh.connect( hostname = “moo.moo.moo.com.remotehost.”, username = “stanley”, pkey = k )
print (“connected”)
stdin, stdout, ssh_stderr = ssh.exec_command(cmd)
out = stdout.read()
stdin.flush()
ssh.close()
print (out)

It gives the following output:

root@bf5778d641f0:/opt/stackstorm# source /opt/stackstorm/st2/bin/activate
(st2) root@bf5778d641f0:/opt/stackstorm# python3.6
Python 3.6.9 (default, Oct 8 2020, 12:12:24)
[GCC 8.4.0] on linux
Type “help”, “copyright”, “credits” or “license” for more information.
>>> import paramiko
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
print (“connecting”)
ssh.connect( hostname = “hmswebhd01.testhms.lan”, username = “stanley”, pkey = k )
print (“connected”)
stdin, stdout, ssh_stderr = ssh.exec_command(cmd)
out = stdout.read()
stdin.flush()
ssh.close()
print (out)>>> cmd = 'echo i am connected to a remote host!'
>>> k = paramiko.RSAKey.from_private_key_file("/home/stanley/.ssh/stanley_rsa")
>>> ssh = paramiko.SSHClient()
>>> ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
>>> print (“connecting”)
connecting
>>> ssh.connect( hostname = “remotehost.com.com”, username = “stanley”, pkey = k )
>>> print (“connected”)
connected
>>> stdin, stdout, ssh_stderr = ssh.exec_command(cmd)
>>> out = stdout.read()
>>> stdin.flush()
>>> ssh.close()
>>> print (out)
b’i am connected to a remote host!\n’
>>>

ok. So i wiped host. Reload all the docker images from scratch, and got to the same error.
If I pass it in as private_key material by doing a cat of the private key it works.

this:
core.remote private_key="cat /home/stanley/.ssh/stanley_rsa"

root@e3fb5cca0aaa:/home/stanley/.ssh# st2 run core.remote private_key="cat /home/stanley/.ssh/stanley_rsa" hosts=remotehost.foo.foo – hostname
.
id: 5fb5fad43f642eff04dc8c6d
action.ref: core.remote
context.user: st2admin
parameters:
** cmd: hostname**
** hosts: remotehost.foo.foo**
** private_key: '
status: succeeded
start_timestamp: Thu, 19 Nov 2020 04:55:48 UTC
end_timestamp: Thu, 19 Nov 2020 04:55:48 UTC
result:
** remotehost.foo.foo:

** failed: false

** return_code: 0

** stderr: ‘’**
** stdout: remotehost.foo.foo**
** succeeded: true**