Actions getting hung

We deployed StackStorm newer version 2.9.x after deploying the code, we started testing our actions, although actions got hung and were not completing. We had to restart the StackStorm then the actions worked fine.
It shows that the coding on our part is absolutely fine as it is fine after a restart but StackStorm itself is having some issues. This is recurring and repeating after every other day.
The previous version was working absolutely fine with the same actions/code.

What could be possible cause of “actions hung” as it i doesn’t seem to be a coding issue and moreover there is no load also on the server. A Single action is even not completing without a restart.

Strangely After a restart application works fine for few hours a day or so and then starting getting these issues and then we we have to Restart again.

Please help us in this matter.

Please provide more info and do more extensive debugging.

• What OS and environment do you use for running StackStorm?
• Are you sure actions are not working or is it workflow? If so is it Mistral, Action-chain or Orquesta?
• Are you having these issues with any specific actions or all actions? Any actions used from StackStorm Exchange or they’re all self-written? Which action runner was used?

What means “action hung”. Is the action process running and never stops or it exits with error code? If it never stops, at which part of the action code it’s freezed? I’d suggest at that point to use some debugging tools like strace to understand the problem deeper.

There should be some samples that would help us to reproduce the problem. Once you’ll have it, - we can fill a bug in st2 core and see how to fix that.

1.OS is Linux
2.It is a mistral workflow
3.We are calling a sub workflow from the main workflow and then returning it back to the main workflow in the end dont know what it is called a Action-chain or Orquesta?.
4.We are facing this issue in all workflows
5. The workflow is not completing in the end and waiting for something.

further providing you more information on this we got below error in in mistral* log in run time and that seems to be the issue to me.

ERROR mistral.notifiers.default_notifier : Unable to process event for publisher: Exception:
“faultstring”: “Internal Server Error”

workflow A can call workflow B however it is not coming back from B to A to complete something is stuck in between and status keeps in “RUNNING” state.
This is for all Mistral workflow we are running.

Please look in to the issue.

Sounds like you found which exactly part of the workflow is getting stuck. Is it consistent? You should know if you’re calling from Mistral workflows Action-chain or Orequesta workflows as currently you can’t call Orquesta from mistral.

So if you have it consistent hanging, try to minimize your entire workflow and isolate the exact part which is failing.

To understand if it’s bug or not, - please include that isolated workflow code. You’ll need to make sure the code has only failing part and nothing else for easier investigation from st2 side.

There’s a limited amount that other people can do here. Without a reproducible test case, it is very hard for someone without access to your system to understand what’s going on.