Unable to run Mistral workflows

Hi All,
I’ve had a production instance of Stackstorm running for quite a while untouched, and I have gone to try and run a Mistral workflow on it, and I am getting a failure with the following error.

{
  "traceback": "  File \"/opt/stackstorm/st2/lib/python2.7/site-packages/st2actions/container/base.py\", line 127, in _do_run
    (status, result, context) = runner.run(action_params)
  File \"/opt/stackstorm/st2/lib/python2.7/site-packages/retrying.py\", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File \"/opt/stackstorm/st2/lib/python2.7/site-packages/retrying.py\", line 206, in call
    return attempt.get(self._wrap_exception)
  File \"/opt/stackstorm/st2/lib/python2.7/site-packages/retrying.py\", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File \"/opt/stackstorm/st2/lib/python2.7/site-packages/retrying.py\", line 200, in call\n    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File \"/opt/stackstorm/runners/mistral_v2/mistral_v2/mistral_v2.py\", line 247, in run
    result = self.start_workflow(action_parameters=action_parameters)
  File \"/opt/stackstorm/runners/mistral_v2/mistral_v2/mistral_v2.py\", line 253, in start_workflow
    self._client.workflows.list()\n  File \"/opt/stackstorm/st2/lib/python2.7/site-packages/mistralclient/api/v2/workflows.py\", line 102, in list
    response_key='workflows',
  File \"/opt/stackstorm/st2/lib/python2.7/site-packages/mistralclient/api/base.py\", line 125, in _list\n    self._raise_api_exception(resp)
  File \"/opt/stackstorm/st2/lib/python2.7/site-packages/mistralclient/api/base.py\", line 160, in _raise_api_exception
    error_message=error_data)
",
  "error": "(psycopg2.ProgrammingError) column workflow_definitions_v2.namespace does not exist
LINE 1: ...2.is_system AS workflow_definitions_v2_is_system, workflow_d...
                                                             ^
 [SQL: 'SELECT workflow_definitions_v2.scope AS workflow_definitions_v2_scope, workflow_definitions_v2.project_id AS workflow_definitions_v2_project_id, workflow_definitions_v2.created_at AS workflow_definitions_v2_created_at, workflow_definitions_v2.updated_at AS workflow_definitions_v2_updated_at, workflow_definitions_v2.id AS workflow_definitions_v2_id, workflow_definitions_v2.name AS workflow_definitions_v2_name, workflow_definitions_v2.definition AS workflow_definitions_v2_definition, workflow_definitions_v2.spec AS workflow_definitions_v2_spec, workflow_definitions_v2.tags AS workflow_definitions_v2_tags, workflow_definitions_v2.is_system AS workflow_definitions_v2_is_system, workflow_definitions_v2.namespace AS workflow_definitions_v2_namespace \
FROM workflow_definitions_v2 \
WHERE workflow_definitions_v2.project_id = %(project_id_1)s OR workflow_definitions_v2.scope = %(scope_1)s ORDER BY workflow_definitions_v2.created_at ASC'] [parameters: {'project_id_1': '<default-project>', 'scope_1': 'public'}] (Background on this error at: http://sqlalche.me/e/f405)"
}

I understand the error, however I am not sure how solve. I can’t seem to connect to the database using the password that is set in /etc/mistral/mistral.conf, but all the services appear to come up and run OK.

Also, it is worth noting:

• I am not sure if mistral workflows have ever worked on this server - I’ve never run them on there before.

• If this can’t be resolved, I can always uninstall everything (st2, postgres, mongo, etc) and start from scratch

Update:
I got into psql as the mistral user, and I had to run the following commands to stop the errors:

alter table workflow_definitions_v2 add column namespace VARCHAR;
alter table workflow_executions_v2 add column workflow_namespace VARCHAR;
alter table workflow_executions_v2 add column root_execution_id VARCHAR;
alter table task_executions_v2 add column workflow_namespace VARCHAR;

Workflows now “run”, but they don’t actually do anything - they just sit there - I suspect I added the columns as the incorrect types

Update 2:
Looks like it had been updated, without the update having scripts been run properly.

I’ve run the upgrade migration scripts and it looks like it is broken worse - now the webUI isn’t working.

I think I’ll just blow this install away and start from scratch

Before using Mistral you have to apply DB migrations. Depending on your installation method this may already happened, but could fail due to unknown now reasons. What’s the environment and installation method initially used?

I’d suggest to drop the database and do Mistral DB init/upgrade again, as described here: Mistral Issues — StackStorm 2.9.1 documentation and Upgrades — StackStorm 2.9.1 documentation (Mistral DB upgrade section).

Hi Eugen,
I’ve since done that, and the errors have gone away. I am able to execute mistral workflows fine now.

The only issue I am dealing with since running the upgrade scripts is the web UI is no longer working - it just loads a page with this as the source:

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>StackStorm Web UI</title>

  <link rel="shortcut icon" href="favicon.ico" type="image/x-icon">
  <link rel="icon" href="favicon.ico" type="image/x-icon">

  <!-- buils:styles -->
  <link rel="stylesheet" href="css/style.css">
  <!-- endbuild -->

  <script type="text/javascript">
    window.angular = {
      module: (module) => ({
        constant: (constant, value) => {
          if (module !== 'main') {
            throw new Error('The st2web angular-config-polyfill only supports the "main" module.');
          }

          if (constant !== 'st2Config') {
            throw new Error('The st2web angular-config-polyfill only supports the "st2Config" constant.');
          }

          window.st2constants = window.st2constants || {};
          window.st2constants.st2Config = value;
        },
        run: (fn) => {
          if (module !== 'main') {
            throw new Error('The st2web angular-config-polyfill only supports the "main" module.');
          }

          window.st2constants = window.st2constants || {};
          window.st2constants.st2Config = window.st2constants.st2Config || {};

          fn(window.st2constants.st2Config);
        },
      }),
    };
  </script>
</head>

<body>
  <div class="wrapper" id="container"></div>
  <script src="config.js"></script>
  <script src="js/main.js"></script>
</body>

</html>

Take a look at the browser developer console - it might shed some light on what’s going on.