Stackstorm HA: Multiple PollingSensor Cause Duplicated Actions

Hello,
Did anyone try to deploy Stackstorm HA on Virtual Linux directly? I try to search in this forum, but it look like few people have the same problem as me.
Anyway, I have Stackstorm HA deploy on 3 Oracle Linux 8. But I’m facing many Duplicated Actions issue:

  • First, I’m facing duplicated st2timersengine.

According to High Availability Deployment — StackStorm 3.5.0 documentation,

[timer]
enable = False

I set 2 false and 1 true on 3 vms, but it don’t work for me, very strange.
Anyway I solve this issue by using keeplived, As mentioned as external monitoring

  • Now, I find that multiple PollingSensor also cause duplicated actions

There are 3 same rule/sensor of cause(each Stackstorm has one st2rulesengine/st2sensorcontainer), because they load the same pack.

Obviously, 3 sensor polling the same system, get 3 same result and generate 3 same payloads, and then trigger 3 same actions.

But I can’t kill 2 of them like st2timersengine, becasue external webhook send requset to upper Nginx, and nginx transfer the request to one of 3 stackstorm. They will cause webhook out of work.

What should I do?
Otherwise, I have to use timer action instead of PollingSensor.

I think I find answer here:
https://docs.stackstorm.com/reference/ha.html#st2sensorcontainer
I need more investigation, and update result here

1 Like

Config st2.conf as following:
https://docs.stackstorm.com/reference/sensor_partitioning.html
I’m using the fourth method [Hash], divide 4294967296 evenly into three
If I use [Key-Value], then:

  1. Should I store all sensors in databse, divide them into three parts? I’m afraid there is no all the rest logic.
  2. So if I develop a new sensor, I have to
    • run packs.load
    • reset key by st2 key set
    • restart st2sensorcontainer by st2ctl restart-component st2sensorcontainer

I don’t use the config to disable it. Under the hood st2ctl just runs systemd services. So I keep st2sensorcontainer and st2timersengine disabled and stopped on all the other nodes.

I also customized the st2ctl script to understand which nodes are set as primary and secondary. That way when you run something like st2ctl start it will only start those two services on the marked primary node.

Eventually I will have triggers to automate the failover.