Cannot start mistral workflow in ST2 3.1

We upgraded to st2 3.1 and we cannot execute mistral workflows. The actions work fine. The logs show me this error:

id: 5d66f520daf15a00f9054c98
action.ref: audit_components.generic_template_audit_workflow
parameters:
audit_name: SNMP_NX7000
node_names:

• foo
  status: failed (787s elapsed)
  start_timestamp: Wed, 28 Aug 2019 21:41:52 UTC
  end_timestamp: Wed, 28 Aug 2019 21:54:59 UTC
  result:
  error: (‘Connection aborted.’, BadStatusLine("’’",))
  traceback: " File “/opt/stackstorm/st2/lib/python2.7/site-packages/st2actions/container/base.py”, line 127, in _do_run
  (status, result, context) = runner.run(action_params)
  File “/opt/stackstorm/st2/lib/python2.7/site-packages/retrying.py”, line 49, in wrapped_f
  return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File “/opt/stackstorm/st2/lib/python2.7/site-packages/retrying.py”, line 212, in call
  raise attempt.get()
  File “/opt/stackstorm/st2/lib/python2.7/site-packages/retrying.py”, line 247, in get
  six.reraise(self.value[0], self.value[1], self.value[2])
  File “/opt/stackstorm/st2/lib/python2.7/site-packages/retrying.py”, line 200, in call
  attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File “/opt/stackstorm/st2/lib/python2.7/site-packages/mistral_v2/mistral_v2.py”, line 251, in run
  result = self.start_workflow(action_parameters=action_parameters)
  File “/opt/stackstorm/st2/lib/python2.7/site-packages/mistral_v2/mistral_v2.py”, line 288, in start_workflow
  **options)
  File “/opt/stackstorm/st2/lib/python2.7/site-packages/mistralclient/api/v2/executions.py”, line 65, in create
  return self._create(’/executions’, data)
  File “/opt/stackstorm/st2/lib/python2.7/site-packages/mistralclient/api/base.py”, line 95, in _create
  resp = self.http_client.post(url, data)
  File “/opt/stackstorm/st2/lib/python2.7/site-packages/mistralclient/api/httpclient.py”, line 54, in decorator
  resp = func(self, *args, **kwargs)
  File “/opt/stackstorm/st2/lib/python2.7/site-packages/mistralclient/api/httpclient.py”, line 120, in post
  return self.session.post(self.base_url + url, data=body, **options)
  File “/opt/stackstorm/st2/lib/python2.7/site-packages/requests/sessions.py”, line 581, in post
  return self.request(‘POST’, url, data=data, json=json, **kwargs)
  File “/opt/stackstorm/st2/lib/python2.7/site-packages/requests/sessions.py”, line 533, in request
  resp = self.send(prep, **send_kwargs)
  File “/opt/stackstorm/st2/lib/python2.7/site-packages/requests/sessions.py”, line 646, in send
  r = adapter.send(request, **kwargs)
  File “/opt/stackstorm/st2/lib/python2.7/site-packages/requests/adapters.py”, line 498, in send
  raise ConnectionError(err, request=request)

Please check Unable to run Mistral workflows and it has some steps to fix psql and some useful links related to debug.

What was the old st2 version before upgrading to latest v3.1?

Did you follow Upgrade Procedure instructions? Upgrades — StackStorm 3.1.0 documentation
and Upgrade Notes: Upgrade Notes — StackStorm 3.1.0 documentation

We went from 2.9 to 3.1.

A few more important facts. We are running in K8s, and we have deployed successfully to a DEV and INT environment, using same exact procedure (Helm Charts, etc). Both previous env’s worked without issue. When we deployed to our PROD env, that’s where things went south. Last night i followed the procedure here: Upgrades — StackStorm 3.1.0 documentation
bringing down all ST2 services and running this:
/opt/stackstorm/mistral/bin/mistral-db-manage --config-file /etc/mistral/mistral.conf upgrade head
/opt/stackstorm/mistral/bin/mistral-db-manage --config-file /etc/mistral/mistral.conf populate | grep -v -e openstack -e keystone -e ironicclient

Then st2ctl reload, to no avail. I will try to get some better error messages this morning.