Cannot start mistral workflow in ST2 3.1

We upgraded to st2 3.1 and we cannot execute mistral workflows. The actions work fine. The logs show me this error:

id: 5d66f520daf15a00f9054c98
action.ref: audit_components.generic_template_audit_workflow
parameters:
audit_name: SNMP_NX7000
node_names:

  • foo
    status: failed (787s elapsed)
    start_timestamp: Wed, 28 Aug 2019 21:41:52 UTC
    end_timestamp: Wed, 28 Aug 2019 21:54:59 UTC
    result:
    error: (‘Connection aborted.’, BadStatusLine("’’",))
    traceback: " File “/opt/stackstorm/st2/lib/python2.7/site-packages/st2actions/container/base.py”, line 127, in _do_run
    (status, result, context) = runner.run(action_params)
    File “/opt/stackstorm/st2/lib/python2.7/site-packages/retrying.py”, line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
    File “/opt/stackstorm/st2/lib/python2.7/site-packages/retrying.py”, line 212, in call
    raise attempt.get()
    File “/opt/stackstorm/st2/lib/python2.7/site-packages/retrying.py”, line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
    File “/opt/stackstorm/st2/lib/python2.7/site-packages/retrying.py”, line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
    File “/opt/stackstorm/st2/lib/python2.7/site-packages/mistral_v2/mistral_v2.py”, line 251, in run
    result = self.start_workflow(action_parameters=action_parameters)
    File “/opt/stackstorm/st2/lib/python2.7/site-packages/mistral_v2/mistral_v2.py”, line 288, in start_workflow
    **options)
    File “/opt/stackstorm/st2/lib/python2.7/site-packages/mistralclient/api/v2/executions.py”, line 65, in create
    return self._create(’/executions’, data)
    File “/opt/stackstorm/st2/lib/python2.7/site-packages/mistralclient/api/base.py”, line 95, in _create
    resp = self.http_client.post(url, data)
    File “/opt/stackstorm/st2/lib/python2.7/site-packages/mistralclient/api/httpclient.py”, line 54, in decorator
    resp = func(self, *args, **kwargs)
    File “/opt/stackstorm/st2/lib/python2.7/site-packages/mistralclient/api/httpclient.py”, line 120, in post
    return self.session.post(self.base_url + url, data=body, **options)
    File “/opt/stackstorm/st2/lib/python2.7/site-packages/requests/sessions.py”, line 581, in post
    return self.request(‘POST’, url, data=data, json=json, **kwargs)
    File “/opt/stackstorm/st2/lib/python2.7/site-packages/requests/sessions.py”, line 533, in request
    resp = self.send(prep, **send_kwargs)
    File “/opt/stackstorm/st2/lib/python2.7/site-packages/requests/sessions.py”, line 646, in send
    r = adapter.send(request, **kwargs)
    File “/opt/stackstorm/st2/lib/python2.7/site-packages/requests/adapters.py”, line 498, in send
    raise ConnectionError(err, request=request)

Please check Unable to run Mistral workflows and it has some steps to fix psql and some useful links related to debug.

1 Like

What was the old st2 version before upgrading to latest v3.1?

Did you follow Upgrade Procedure instructions? Upgrades — StackStorm 3.1.0 documentation
and Upgrade Notes: Upgrade Notes — StackStorm 3.1.0 documentation

We went from 2.9 to 3.1.

A few more important facts. We are running in K8s, and we have deployed successfully to a DEV and INT environment, using same exact procedure (Helm Charts, etc). Both previous env’s worked without issue. When we deployed to our PROD env, that’s where things went south. Last night i followed the procedure here: Upgrades — StackStorm 3.1.0 documentation
bringing down all ST2 services and running this:
/opt/stackstorm/mistral/bin/mistral-db-manage --config-file /etc/mistral/mistral.conf upgrade head
/opt/stackstorm/mistral/bin/mistral-db-manage --config-file /etc/mistral/mistral.conf populate | grep -v -e openstack -e keystone -e ironicclient

Then st2ctl reload, to no avail. I will try to get some better error messages this morning.