Executions stuck in "Scheduled" or "Requested" state when using mongoDB replica set


Recently, I’m working on deploying high availability environment of StackStorm with mongoDB replica set, but the executions often stuck in “Scheduled” or “Requested” state. After switching to standalone mongod, it looks well again.

st2actionrunner.log shows Successfully connected to database, so I didn’t know where the problem is.

2020-06-21 18:16:31,721 140452299783216 INFO __init__ [-] Connecting to database "st2" @ "ip1:27017,ip2:27017,ip3:27017 (replica set)" as user "None".
2020-06-21 18:16:31,778 140452299783216 INFO __init__ [-] Successfully connected to database "st2" @ "ip1:27017,ip2:27017,ip3:27017 (replica set)" as user "None".
2020-06-21 18:16:34,815 140452299783216 INFO actionrunner [-] (PID=23536) Worker started.
2020-06-21 18:16:34,816 140452299783216 INFO consumers [-] Starting ActionExecutionDispatcher...
2020-06-21 18:16:34,859 140451979676400 INFO mixins [-] Connected to amqp://guest:**@st2-multi-node-controller:5672//


It seems wrong with RabbitMQ, there was a large number of connection_closed_abruptly in RabbitMQ’s log but didn’t know why.

How did you solve the problem with RabbitMQ? And it wasn’t Mongo after all?

By using the RabbitMQ cluster, it works well. I didn’t know the reason. :sweat:

FWIW it ended up being this for me:

=INFO REPORT==== 1-Oct-2020::02:39:15 ===
Disk free space sufficient. Free bytes:50294784 Limit:50000000

=WARNING REPORT==== 1-Oct-2020::02:39:15 ===
disk resource limit alarm cleared on node 'rabbit@stackstorm-vf'

=INFO REPORT==== 1-Oct-2020::02:39:15 ===
Disk free space insufficient. Free bytes:49590272 Limit:50000000