Stackstorm in combination with ansible-container

Hi!

I’m a big fan of both ansible and stackstorm. Have been able to setup a working stackstorm system using ansible-container, st2 version 2.3 . But now when I try to use version 2.7 mistral stopped working. Everything seem to be working except the mistral part. And it’s looks like it has something to do with the mistral authentication. Don’t know how to move forward from here. Anyone that can help me or point me in the correct direction?

Self-check output:

/opt/stackstorm/st2/bin/st2-self-check
Attempting Test tests.test_inquiry_chain…OK!
Attempting Test tests.test_inquiry_mistral…ERROR!
Attempting Test tests.test_key_triggers…OK!
Attempting Test tests.test_packs_pack…OK!
Attempting Test tests.test_quickstart…ERROR!
Attempting Test tests.test_quickstart_key…OK!
Attempting Test tests.test_quickstart_local_script_actions…OK!
Attempting Test tests.test_quickstart_passive_sensor…OK!
Attempting Test tests.test_quickstart_polling_sensor…OK!
Attempting Test tests.test_quickstart_python_actions…OK!
Attempting Test tests.test_quickstart_remote_script_actions…ERROR!
Attempting Test tests.test_quickstart_rules…OK!
Attempting Test tests.test_quickstart_trace…OK!
Attempting Test tests.test_run_pack_tests_tool…OK!
Attempting Test tests.test_timer_rule…OK!
Skipping tests.test_windows_runners…
Attempting Example examples.mistral_examples…ERROR!
SELF CHECK FAILED!

Output from examples-mistral_examples

id: 5adf3fa3585f530a3ff4f190
action.ref: examples.mistral_examples
parameters: None
status: failed (32s elapsed)
result_task: mistral-basic
result:
error: ‘1’
traceback: " File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/container/base.py”, line 119, in _do_run
(status, result, context) = runner.run(action_params)
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py”, line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py”, line 206, in call
return attempt.get(self._wrap_exception)
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py”, line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py”, line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File “/opt/stackstorm/runners/mistral_v2/mistral_v2/mistral_v2.py”, line 247, in run
result = self.start_workflow(action_parameters=action_parameters)
File “/opt/stackstorm/runners/mistral_v2/mistral_v2/mistral_v2.py”, line 289, in start_workflow
**options)
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/mistralclient/api/v2/executions.py”, line 65, in create
return self._create(’/executions’, data)
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/mistralclient/api/base.py”, line 100, in _create
self._raise_api_exception(resp)
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/mistralclient/api/base.py”, line 160, in _raise_api_exception
error_message=error_data)
"

Running tests.test_inquiry_mistral manually result in the following output:

id: 5adf4105585f530a3ff4f195
action.ref: tests.test_inquiry_mistral
parameters: None
status: failed (2s elapsed)
result_task: execute_inquiry_workflow
result:
failed: true
return_code: 1
stderr: ‘’
stdout: 'ERROR: 401 Client Error: Unauthorized
MESSAGE: Unauthorized - One of Token or API key required. for url: http://127.0.0.1:9101/actions/examples.mistral-ask-basic
’
succeeded: false
start_timestamp: Tue, 24 Apr 2018 14:36:53 UTC
end_timestamp: Tue, 24 Apr 2018 14:36:55 UTC
±-------------------------±--------------------±-------------------------±-----------±------------------------------+
| id | status | task | action | start_timestamp |
±-------------------------±--------------------±-------------------------±-----------±------------------------------+
| 5adf4106585f530a12e4313c | failed (1s elapsed) | execute_inquiry_workflow | core.local | Tue, 24 Apr 2018 14:36:54 UTC |
±-------------------------±--------------------±-------------------------±-----------±------------------------------+

Trying to run mistral manually:

mistral --debug -v run-action std.echo ‘{“output”: “Hello”}’
DEBUG (extension) found extension EntryPoint.parse(‘table = cliff.formatters.table:TableFormatter’)
DEBUG (extension) found extension EntryPoint.parse(‘json = cliff.formatters.json_format:JSONFormatter’)
DEBUG (extension) found extension EntryPoint.parse(‘shell = cliff.formatters.shell:ShellFormatter’)
DEBUG (extension) found extension EntryPoint.parse(‘value = cliff.formatters.value:ValueFormatter’)
DEBUG (extension) found extension EntryPoint.parse(‘yaml = cliff.formatters.yaml_format:YAMLFormatter’)
DEBUG (command) run(Namespace(columns=[], fit_width=False, formatter=‘table’, input=’{“output”: “Hello”}’, max_width=0, name=‘std.echo’, noindent=False, prefix=’’, print_empty=False, run_sync=False, save_result=False, target=None, variables=[]))
DEBUG (connectionpool) Starting new HTTP connection (1): 127.0.0.1
DEBUG (connectionpool) http://127.0.0.1:8989 “POST /v2/action_executions HTTP/1.1” 500 62
DEBUG (httpclient) HTTP POST http://127.0.0.1:8989/v2/action_executions 500
ERROR (app) 1
Traceback (most recent call last):
File “/opt/stackstorm/mistral/local/lib/python2.7/site-packages/cliff/app.py”, line 400, in run_subcommand
result = cmd.run(parsed_args)
File “/opt/stackstorm/mistral/local/lib/python2.7/site-packages/osc_lib/command/command.py”, line 41, in run
return super(Command, self).run(parsed_args)
File “/opt/stackstorm/mistral/local/lib/python2.7/site-packages/cliff/display.py”, line 116, in run
column_names, data = self.take_action(parsed_args)
File “/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistralclient/commands/v2/action_executions.py”, line 165, in take_action
**params
File “/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistralclient/api/v2/action_executions.py”, line 47, in create
self._raise_api_exception(resp)
File “/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistralclient/api/base.py”, line 160, in _raise_api_exception
error_message=error_data)
APIException: 1
Traceback (most recent call last):
File “/opt/stackstorm/mistral/bin/mistral”, line 10, in
sys.exit(main())
File “/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistralclient/shell.py”, line 767, in main
return MistralShell().run(argv)
File “/opt/stackstorm/mistral/local/lib/python2.7/site-packages/cliff/app.py”, line 279, in run
result = self.run_subcommand(remainder)
File “/opt/stackstorm/mistral/local/lib/python2.7/site-packages/cliff/app.py”, line 400, in run_subcommand
result = cmd.run(parsed_args)
File “/opt/stackstorm/mistral/local/lib/python2.7/site-packages/osc_lib/command/command.py”, line 41, in run
return super(Command, self).run(parsed_args)
File “/opt/stackstorm/mistral/local/lib/python2.7/site-packages/cliff/display.py”, line 116, in run
column_names, data = self.take_action(parsed_args)
File “/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistralclient/commands/v2/action_executions.py”, line 165, in take_action
**params
File “/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistralclient/api/v2/action_executions.py”, line 47, in create
self._raise_api_exception(resp)
File “/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistralclient/api/base.py”, line 160, in _raise_api_exception
error_message=error_data)
mistralclient.api.base.APIException: 1

If anyone is wondering why I’m not just using docker-compose, which would be much simpler. It’s because I want to use ansible roles to configure my pack settings depending on which system I am deploying it to just using ansible var-files. Also I can encrypt my password.

My container.yaml if anyone is interested

version: “2”
defaults:
BASE_IMAGE: grossws/centos
BASE_IMAGE_VERSION: 7
MONGO_VERSION: 3.4
MONGO_HOST: mongo
MONGO_PORT: 27017
MONGO_DB: st2_db
POSTGRES_VERSION: 9.6
POSTGRES_HOST: postgres
POSTGRES_PORT: 5432
POSTGRES_DB: mistral
RABBITMQ_VERSION: 3.6-management
RABBITMQ_HOST: rabbitmq
RABBITMQ_PORT: 5672
REDIS_VERSION: 4.0
REDIS_HOST: redis
REDIS_PORT: 6379
ST2_VERSION: 2.7.0

settings:
conductor:
base: centos:7

project_name: test

services:
mongo:
from: mongo:3.4
volumes:
- /data/configdb
- mongo-volume:/data/db
environment:
- MONGO_HOST=mongo
- MONGO_PORT=27017

rabbitmq:
from: ‘rabbitmq:{{ RABBITMQ_VERSION }}’
volumes:
- rabbitmq-volume:/var/lib/rabbitmq
environment:
- RABBITMQ_DEFAULT_USER=admin
- RABBITMQ_DEFAULT_PASS=mistral-user
- RABBITMQ_HOST=rabbitmq
- RABBITMQ_PORT=5672

postgres:
from: ‘postgres:{{ POSTGRES_VERSION }}’
volumes:
- postgres-volume:/var/lib/postgresql/data
environment:
- POSTGRES_USER=mistral-user
- POSTGRES_PASSWORD=mistral-user
- POSTGRES_HOST=postgres
- POSTGRES_PORT=5432
- POSTGRES_DB=mistral
redis:
from: ‘redis:{{ REDIS_VERSION }}’
volumes:
- redis-volume:/data
environment:
- REDIS_PASSWORD=redis_pass
- REDIS_HOST=redis
- REDIS_PORT=6379
command: [
“bash”, “-c”,
’
docker-entrypoint.sh
–requirepass “redis_pass”
’
]
stackstorm:
from: “stackstorm/stackstorm:{{ ST2_VERSION }}”
priviliged: True
volumes:
- /home/user/workspace/test/packs.dev:/opt/stackstorm/packs.dev
- /home/user/workspace/st2-docker/runtime/entrypoint.d:/st2-docker/entrypoint.d
- /home/user/workspace/workspace/st2-docker/runtime/st2.d:/st2-docker/st2.d
- /home/user/workspace/workspace/st2-docker/conf/stackstorm.env:/st2-docker/env
- stackstorm-volume:/var/log
ports: # Make sure that the web interface isn’t accessible, until we have verified that we can have it.
- “443:4443”
- “80:8080”
environment:
- MONGO_HOST=mongo
- MONGO_PORT=27017
- RABBITMQ_DEFAULT_USER=admin
- RABBITMQ_DEFAULT_PASS=mistral-user
- RABBITMQ_HOST=rabbitmq
- RABBITMQ_PORT=5672
- POSTGRES_USER=mistral-user
- POSTGRES_PASSWORD=mistral-user
- POSTGRES_HOST=postgres
- POSTGRES_PORT=5432
- POSTGRES_DB=mistral
- REDIS_PASSWORD= redis_pass
- REDIS_HOST=redis
- REDIS_PORT=6379
- ST2_USER=st2admin
- ST2_PASSWORD=H8WHTIFe

registries: {}

volumes:
mongo-volume:
docker: {}
rabbitmq-volume:
docker: {}
postgres-volume:
docker: {}
redis-volume:
docker: {}
stackstorm-volume:
docker: {}

Hi @smeds !

This may be a red herring, but I’d appreciate if you would confirm whether [coordination] url begins with “redis://:” in /etc/st2/st2.conf?

See mistral doesn't work in Swarm mode · Issue #83 · StackStorm/st2-docker · GitHub for more detail. Your issue may be related.

Yes it does.

Here’s my st2.conf gile:

#System-wide configuration

[api]
#Host and port to bind the API server.
host = 127.0.0.1
port = 9101
logging = /etc/st2/logging.api.conf
mask_secrets = True
#allow_origin is required for handling CORS in st2 web UI.
#allow_origin = http://myhost1.example.com:3000,http://myhost2.example.com:3000

[stream]
logging = /etc/st2/logging.stream.conf

[sensorcontainer]
logging = /etc/st2/logging.sensorcontainer.conf

[rulesengine]
logging = /etc/st2/logging.rulesengine.conf

[actionrunner]
logging = /etc/st2/logging.actionrunner.conf
virtualenv_opts = --always-copy

[resultstracker]
logging = /etc/st2/logging.resultstracker.conf

[notifier]
logging = /etc/st2/logging.notifier.conf

[exporter]
logging = /etc/st2/logging.exporter.conf

[garbagecollector]
logging = /etc/st2/logging.garbagecollector.conf

[auth]
host = 127.0.0.1
port = 9100
use_ssl = False
debug = False
enable = True
logging = /etc/st2/logging.auth.conf

mode = standalone

#Note: Settings below are only used in “standalone” mode
backend = flat_file
backend_kwargs = {“file_path”: “/etc/st2/htpasswd”}

#Base URL to the API endpoint excluding the version (e.g. http://myhost.net:9101/)
api_url =http://127.0.0.1:9101

[system]
base_path = /opt/stackstorm

[webui]
#webui_base_url = https://mywebhost.domain

[syslog]
host = 127.0.0.1
port = 514
facility = local7
protocol = udp

[log]
excludes = requests,paramiko
redirect_stderr = False
mask_secrets = True

[system_user]
user = stanley
ssh_key_file = /home/stanley/.ssh/stanley_rsa

[messaging]
url = amqp://admin:mistral-user@rabbitmq:5672

[ssh_runner]
remote_dir = /tmp

[keyvalue]
encryption_key_path = /etc/st2/keys/datastore_key.json

[mistral]
api_url = http://127.0.0.1:9101
v2_base_url = http://127.0.0.1:8989/v2

[coordination]
url = redis://:redis_pass@redis:6379

[database]
host = mongo
port = 27017

[content]
packs_base_paths = /opt/stackstorm/packs.dev

Could it be a mistral configuration that I’m missing? What kind of authentication do I need to setup for mistral to work? Everything except executing workflows seem to work.

I found the cause of my problem. It’s a dns lookup problem, with ansibe-container I can’t set “dns_search: .” which is set in the docker-compose file. Mistral is trying to do a lookup using the name of my container and that fails.