This site is in read only mode. Please continue to browse, but replying, likes, and other actions are disabled for now.

⚠️ We've moved!

Hi there!

To reduce project dependency on 3rd party paid services the StackStorm TSC has decided to move the Q/A from this forum to Github Discussions. This will make user experience better integrated with the native Github flow, as well as the questions closer to the community where they can provide answers.

Use 🔗 Github Discussions to ask your questions.

K8s PV issue for stackStorm HA

Operations
,
#1

I have configured kubernetes with Multinode with one master and two worker node with Kubeadm .
I have created 6 directories with full access in /mnt/resource directory whose name is below
mongo1 mongo2 mongo3 rabbit1 rabbit2 rabbit3

i have created 6 pv …3 for rabbitmq and 3 for mongoDB each and i have created storageclass with standardclass

for mongodb pv creation

apiVersion: v1
kind: PersistentVolume
metadata:
name: mongodb1
spec:
capacity:
storage: 10Gi
accessModes:

  • ReadWriteOnce
    persistentVolumeReclaimPolicy: Retain
    storageClassName: my-local-storage
    local:
    path: /mnt/resource/mongo1
    nodeAffinity:
    required:
    nodeSelectorTerms:

for RabbitMQ pv creation

apiVersion: v1
kind: PersistentVolume
metadata:
name: rabbit1
spec:
capacity:
storage: 10Gi
accessModes:

  • ReadWriteOnce
    persistentVolumeReclaimPolicy: Retain
    storageClassName: my-local-storage
    local:
    path: /mnt/resource/rabbit1
    nodeAffinity:
    required:
    nodeSelectorTerms:

PV is not bound for RAbbitMQ and MongoDB .Please help me how we can proceed as all pods are going in crashloop
one of crashed pods of stackStorm

[root@k8smaster1 /]# kubectl logs stackstorm-ha-1580637351-st2actionrunner-577648bcd-4kxzv
2020-02-02 10:36:30,219 DEBUG [-] Using Python: 2.7.12 (/opt/stackstorm/st2/bin/python)
2020-02-02 10:36:30,219 DEBUG [-] Using config files: /etc/st2/st2.conf,/etc/st2/st2.docker.conf,/etc/st2/st2.user.conf
2020-02-02 10:36:30,219 DEBUG [-] Using logging config: /etc/st2/logging.actionrunner.conf
2020-02-02 10:36:30,224 INFO [-] Connecting to database “st2” @ “stackstorm-ha-1580637351-mongodb-ha-0.stackstorm-ha-1580637351-mongodb-ha:27017,stackstorm-ha-1580637351-mongodb-ha-1.stackstorm-ha-1580637351-mongodb-ha:27017,stackstorm-ha-1580637351-mongodb-ha-2.stackstorm-ha-1580637351-mongodb-ha:27017 (replica set)” as user “admin”.
2020-02-02 10:36:33,230 ERROR [-] Failed to connect to database “st2” @ “stackstorm-ha-1580637351-mongodb-ha-0.stackstorm-ha-1580637351-mongodb-ha:27017,stackstorm-ha-1580637351-mongodb-ha-1.stackstorm-ha-1580637351-mongodb-ha:27017,stackstorm-ha-1580637351-mongodb-ha-2.stackstorm-ha-1580637351-mongodb-ha:27017 (replica set)” as user “admin”: No replica set members found yet
2020-02-02 10:36:33,231 ERROR [-] (PID=1) Worker quit due to exception.
Traceback (most recent call last):
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/cmd/actionrunner.py”, line 94, in main
_setup()
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/cmd/actionrunner.py”, line 55, in _setup
register_signal_handlers=True, service_registry=True, capabilities=capabilities)
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/service_setup.py”, line 160, in setup
db_setup()
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/database_setup.py”, line 56, in db_setup
connection = db_init.db_setup_with_retry(**db_cfg)
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/db_init.py”, line 75, in db_setup_with_retry
ssl_match_hostname=ssl_match_hostname)
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/db_init.py”, line 58, in db_func_with_retry
return retrying_obj.call(db_func, *args, **kwargs)
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py”, line 206, in call
return attempt.get(self._wrap_exception)
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py”, line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py”, line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/init.py”, line 169, in db_setup
ssl_match_hostname=ssl_match_hostname)
File “/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/init.py”, line 151, in _db_connect
raise e

#2

Stackstorm pods fail in crashloop trying to connect to alive Mongo and Rabbitmq clusters, which cant start because of issues with K8s configuration around persistent volumes.

Your #1 objective is to get Rabbitmq and Mongodb clusters up & running.
Try to use respective Helm charts directly without stackstorm first. That’ll get you a deployment fast to reproduce and iterate. See GitHub - StackStorm/stackstorm-ha: K8s Helm Chart (βeta!) that codifies StackStorm (aka "IFTTT for Ops" https://stackstorm.com/) Highly Availability fleet as a simple to use reproducible infrastructure-as-code app for reference and charts/stable/mongodb-replicaset at master · helm/charts · GitHub for detailed instructions about official Helm charts we use, their deployment details and configuration settings.

Reated issues that might give more pointers and ideas to google for: