K8s PV issue for stackStorm HA

I have configured kubernetes with Multinode with one master and two worker node with Kubeadm .
I have created 6 directories with full access in /mnt/resource directory whose name is below
mongo1 mongo2 mongo3 rabbit1 rabbit2 rabbit3

i have created 6 pv …3 for rabbitmq and 3 for mongoDB each and i have created storageclass with standardclass

for mongodb pv creation

apiVersion: v1
kind: PersistentVolume
metadata:
name: mongodb1
spec:
capacity:
storage: 10Gi
accessModes:

  • ReadWriteOnce
    persistentVolumeReclaimPolicy: Retain
    storageClassName: my-local-storage
    local:
    path: /mnt/resource/mongo1
    nodeAffinity:
    required:
    nodeSelectorTerms:

for RabbitMQ pv creation

apiVersion: v1
kind: PersistentVolume
metadata:
name: rabbit1
spec:
capacity:
storage: 10Gi
accessModes:

  • ReadWriteOnce
    persistentVolumeReclaimPolicy: Retain
    storageClassName: my-local-storage
    local:
    path: /mnt/resource/rabbit1
    nodeAffinity:
    required:
    nodeSelectorTerms:

PV is not bound for RAbbitMQ and MongoDB .Please help me how we can proceed as all pods are going in crashloop
one of crashed pods of stackStorm

[root@k8smaster1 /]# kubectl logs stackstorm-ha-1580637351-st2actionrunner-577648bcd-4kxzv
2020-02-02 10:36:30,219 DEBUG [-] Using Python: 2.7.12 (/opt/stackstorm/st2/bin/python)
2020-02-02 10:36:30,219 DEBUG [-] Using config files: /etc/st2/st2.conf,/etc/st2/st2.docker.conf,/etc/st2/st2.user.conf
2020-02-02 10:36:30,219 DEBUG [-] Using logging config: /etc/st2/logging.actionrunner.conf
2020-02-02 10:36:30,224 INFO [-] Connecting to database “st2” @ “stackstorm-ha-1580637351-mongodb-ha-0.stackstorm-ha-1580637351-mongodb-ha:27017,stackstorm-ha-1580637351-mongodb-ha-1.stackstorm-ha-1580637351-mongodb-ha:27017,stackstorm-ha-1580637351-mongodb-ha-2.stackstorm-ha-1580637351-mongodb-ha:27017 (replica set)” as user “admin”.
2020-02-02 10:36:33,230 ERROR [-] Failed to connect to database “st2” @ “stackstorm-ha-1580637351-mongodb-ha-0.stackstorm-ha-1580637351-mongodb-ha:27017,stackstorm-ha-1580637351-mongodb-ha-1.stackstorm-ha-1580637351-mongodb-ha:27017,stackstorm-ha-1580637351-mongodb-ha-2.stackstorm-ha-1580637351-mongodb-ha:27017 (replica set)” as user “admin”: No replica set members found yet
2020-02-02 10:36:33,231 ERROR [-] (PID=1) Worker quit due to exception.
Traceback (most recent call last):
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/cmd/actionrunner.py”, line 94, in main
_setup()
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/cmd/actionrunner.py”, line 55, in _setup
register_signal_handlers=True, service_registry=True, capabilities=capabilities)
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/service_setup.py”, line 160, in setup
db_setup()
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/database_setup.py”, line 56, in db_setup
connection = db_init.db_setup_with_retry(**db_cfg)
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/db_init.py”, line 75, in db_setup_with_retry
ssl_match_hostname=ssl_match_hostname)
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/db_init.py”, line 58, in db_func_with_retry
return retrying_obj.call(db_func, *args, **kwargs)
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py”, line 206, in call
return attempt.get(self._wrap_exception)
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py”, line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py”, line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/init.py”, line 169, in db_setup
ssl_match_hostname=ssl_match_hostname)
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/init.py”, line 151, in _db_connect
raise e

Stackstorm pods fail in crashloop trying to connect to alive Mongo and Rabbitmq clusters, which cant start because of issues with K8s configuration around persistent volumes.

Your #1 objective is to get Rabbitmq and Mongodb clusters up & running.
Try to use respective Helm charts directly without stackstorm first. That’ll get you a deployment fast to reproduce and iterate. See GitHub - StackStorm/stackstorm-ha: K8s Helm Chart (βeta!) that codifies StackStorm (aka "IFTTT for Ops" https://stackstorm.com/) Highly Availability fleet as a simple to use reproducible infrastructure-as-code app for reference and charts/stable/mongodb-replicaset at master · helm/charts · GitHub for detailed instructions about official Helm charts we use, their deployment details and configuration settings.

Reated issues that might give more pointers and ideas to google for: