St2-docker: Actions are stuck in "Requested"


I have setup a docker environment with the stackstorm/stackstorm:2.10.4 image. For a while it was able to execute actions and I was able to see results, but for some unknown reason they now always stuck in “Requested”. Once in a while they actually execute fully, but not often. No amount of docker-compose down -v and restarting the container helps.

How do I debug this? Tracing doesn’t provide much information

$ st2 execution get     5ecfc9c6266dd002d9050e20
id: 5ecfc9c6266dd002d9050e20
status: requested
  log_level: DEBUG
result: None

Rabbitmq queues are empty and the logs aren’t of much help, but maybe I’m grepping for the wrong thing. tail -F -n 0 /var/log/st2/*.log | grep ERROR keeps including the parameters sent to the api and makes the logs unreadable.
How do I get all the logs related to one execution?

Looks like a Rabbitmq Problem for me. Restarting it helps in our case. We usually check it by RabbitMQ webui. Any other receipts guys?

1 Like

That was indeed the problem, thank you! :+1:

I was running 2 stackstorm containers pointing to the same rabbitmq container with the same user. After creating a rabbitmq container per stackstorm instance (didn’t want to mess waste time finding out how to create users on rabbitmq) it was OK.

It would’ve been nice to have some kind of logs in stackstorm.

Have a good day :slight_smile: