Workflow executions are canceled automatically

Since v3.1, we introduce a garbage collection routine for cleaning up orquesta workflow executions. In cases where there is unexpected failure in st2 services, such as prolong connection errors to rabbitMQ and etc., the workflow execution could be left in a running state. The purpose of this garbage collection routine is to cleanup and cancel these workflow executions.

Unfortunately, there is a bug in the clean up routine that misidentified and cancels long running workflow executions. We are diligently working on a bug fix and apologizes for any inconvenience.

In the meantime, if any user encountered this problem, the garbage collection can be disabled by setting the gc_max_idle_sec option to a value of zero in the workflow_engine of the /etc/st2/st2.conf file and restarting st2.

Example:

[workflow_engine]
gc_max_idle_sec = 0

The fix has been released in the new st2 3.1.0-2 package. We found the logical error when querying for these workflow executions. Also, the config option gc_max_idle_sec is now set to zero by default in the fix and the garbage collection for orphaned workflow executions is not enabled. If you want to enable this specific garbage collection routine, you can set gc_max_idle_sec to any integer value greater than 0. This value is the maximum number of seconds to allow workflow execution to be idled before it is identified as orphaned and cancelled by the garbage collector. The recommended value is 900 seconds. If the value is too low, then there is a greater risk of identifying false positives.

On Ubuntu, you can run the following to check the installed st2 packages.

$ sudo apt list --installed | grep st2
st2/xenial,now 3.1.0-2 amd64 [installed]
st2chatops/xenial,now 3.1.0-2 amd64 [installed]
st2mistral/xenial,now 3.1.0-1 amd64 [installed]
st2web/xenial,now 3.1.0-1 amd64 [installed]

On CentOS or Red Hat, you can run the following to check the installed st2 packages.

$ yum list installed | grep st2
st2.x86_64                       3.1.0-2                        @StackStorm_stable
st2chatops.x86_64                3.1.0-2                        @StackStorm_stable
st2mistral.x86_64                3.1.0-1                        @StackStorm_stable
st2web.x86_64                    3.1.0-1                        @StackStorm_stable