Orquesta workflows take ages to start the first action

Complex orquesta workflows take ages to start the first action. I have seen workflows taking as much as 45 min to start the first action. Although it runs smoothly after that. Is there a way to improve this performance issue.

We didn’t see anything dramatic (45mins!) like that before.

There seems to be something very specific about your environment. Please provide more info about the environment you’re in like CPU/Memory resources, load, number of workflows, workflow logic, load, OS, cloud, scaling strategy, eg what’s specific about your config.

I am doing POC for orquesta and for that I am using stand alone stackstorm set up on a single server (centos 7 - 4 core 32 gb ram) with no other process running on it. Also as of now i don’t have any other workflow running on this system.
Its not possible for me to share workflow here but the workflow has approx 70 tasks with multiple branches such that a successful path can have around 40 tasks to execute.
We have been using the mistral flow for the similar process (same number of tasks) very effectively with multiple flows running in parallel but with orquesta i am facing this strange issue

Try to work on isolating the issue if it’s a problem with the specific orquestra workflow part or is it with the system. For example, such waits could be related for example to RabbitMQ issues.

We have a similar situation. The workflow has approximately 20 tasks and sub workflows and a combination of mistral/Orquesta workflows. Noticed it takes anywhere from 15-20 mins to start the workflow and proceeds fine after that. Other smaller workflows are working fine with little or no wait time.

By moving some actions in the workflow to a separate action (about 10 of them), i was able to reduce the start time from 20 mins to 1 1/2 mins. There is definitely an issue.

@pvyaka01 f you could put together an example workflow filled with dummy actions for us to reproduce the issue, - please report it and share in GitHub - StackStorm/orquesta: Orquesta is a graph based workflow engine for StackStorm. Questions? https://forum.stackstorm.com so that could be investigated and fixed.
Thanks!

yeah, i’ll report it as soon as i can modify the workflow with dummy actions

@armab all this time is taken up by conducting.py file in orquesta where the workflow graphs and specs are serialized.

It might be related to this issue (although it shows resolved on github)

Just to confirm, make sure to use the latest st2 v3.2.0 version with the most recent fixes.

If you believe you found a performance bug, - please report it in GitHub - StackStorm/orquesta: Orquesta is a graph based workflow engine for StackStorm. Questions? https://forum.stackstorm.com providing as much details as possible, including an example workflow to reproduce the problem.