Hi Armab,
Thanks for your comments, We have restarted the server with the following command:
“helm install stackstorm/stackstorm-ha --wait --timeout 5000”
The result was the same, when I tried the command -
“kubectl get pods”. I did check the logs of the st2 api and found that it could not connect to MongoDB.
Pods containing Mongodb and RabbitMQ had no logs.
I wanted to explore more about the description of RabbitMQ and the MongoDB pods, observed that their
status was alternating between pending and CrashLoopBackOff, have added outputs of their respective
-describe- commands. Request you to guide - on how to fix RabbitMQ and Mongodb pods(with configuring
replicas):
indent preformatted text by 4 spaces
kubectl get pods
NAME READY STATUS RESTARTS AGE
fancy-lizard-etcd-78964d45d7-qn5jd 1/1 Running 0 147m
fancy-lizard-mongodb-ha-0 0/1 Pending 0 147m
fancy-lizard-rabbitmq-ha-0 0/1 Pending 0 147m
fancy-lizard-st2actionrunner-9f7bdb447-mhsv4 0/1 CrashLoopBackOff 29 147m
fancy-lizard-st2actionrunner-9f7bdb447-nt8hw 0/1 CrashLoopBackOff 29 147m
fancy-lizard-st2actionrunner-9f7bdb447-s6wbx 0/1 CrashLoopBackOff 29 147m
fancy-lizard-st2actionrunner-9f7bdb447-th7lw 0/1 CrashLoopBackOff 29 147m
fancy-lizard-st2actionrunner-9f7bdb447-zncbm 1/1 Running 30 147m
fancy-lizard-st2api-555cb7db85-d66k6 0/1 CrashLoopBackOff 30 147m
fancy-lizard-st2api-555cb7db85-s4crl 1/1 Running 30 147m
fancy-lizard-st2auth-8dfcbb44-9hktq 0/1 CrashLoopBackOff 29 147m
fancy-lizard-st2auth-8dfcbb44-db6gb 0/1 CrashLoopBackOff 29 147m
fancy-lizard-st2client-778b496fcf-pbjgl 1/1 Running 0 147m
fancy-lizard-st2garbagecollector-7d5887665f-q9mm7 1/1 Running 30 147m
fancy-lizard-st2notifier-7648c659d7-cn6fr 0/1 Error 30 147m
fancy-lizard-st2notifier-7648c659d7-dmsgx 1/1 Running 30 147m
fancy-lizard-st2rulesengine-677685f6c6-qvttc 0/1 CrashLoopBackOff 29 147m
fancy-lizard-st2rulesengine-677685f6c6-tx844 0/1 CrashLoopBackOff 29 147m
fancy-lizard-st2scheduler-cd8578bb4-47cdf 0/1 CrashLoopBackOff 29 147m
fancy-lizard-st2scheduler-cd8578bb4-qg4wh 1/1 Running 30 147m
fancy-lizard-st2sensorcontainer-76f5f77697-gbf9h 0/1 CrashLoopBackOff 29 147m
fancy-lizard-st2stream-7874dfdf4d-766mk 0/1 CrashLoopBackOff 29 147m
fancy-lizard-st2stream-7874dfdf4d-8knlh 0/1 CrashLoopBackOff 29 147m
fancy-lizard-st2timersengine-66f57f88fc-d8njp 1/1 Running 30 147m
fancy-lizard-st2web-7d6c8c5d7c-7n244 1/1 Running 0 147m
fancy-lizard-st2web-7d6c8c5d7c-wgcpq 1/1 Running 0 147m
fancy-lizard-st2workflowengine-559b946546-k8tx7 0/1 CrashLoopBackOff 29 147m
fancy-lizard-st2workflowengine-559b946546-pzk6j 0/1 CrashLoopBackOff 29 147m
[root@k8s-master ~]# kubectl logs po/fancy-lizard-st2api-555cb7db85-d66k6
2019-01-19 04:55:19,757 DEBUG [-] Using Python: 2.7.12 (/opt/stackstorm/st2/bin/python)
2019-01-19 04:55:19,758 DEBUG [-] Using config files: /etc/st2/st2.conf,/etc/st2/st2.docker.conf,/etc/st2/st2.user.conf
2019-01-19 04:55:19,758 DEBUG [-] Using logging config: /etc/st2/logging.api.conf
2019-01-19 04:55:19,765 INFO [-] Connecting to database "st2" @ "fancy-lizard-mongodb-ha:27017" as user "None".
2019-01-19 04:55:49,783 ERROR [-] Failed to connect to database "st2" @ "fancy-lizard-mongodb-ha:27017" as user "None": No replica set members found yet
2019-01-19 04:55:49,783 ERROR [-] (PID=1) ST2 API quit due to exception.
Traceback (most recent call last):
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2api/cmd/api.py", line 73, in main
_setup()
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2api/cmd/api.py", line 47, in _setup
register_signal_handlers=True, register_internal_trigger_types=True)
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/service_setup.py", line 130, in setup
db_setup()
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/database_setup.py", line 57, in db_setup
connection = db_init.db_setup_with_retry(**db_cfg)
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/db_init.py", line 76, in db_setup_with_retry
ssl_match_hostname=ssl_match_hostname)
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/db_init.py", line 59, in db_func_with_retry
return retrying_obj.call(db_func, *args, **kwargs)
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py", line 206, in call
return attempt.get(self._wrap_exception)
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 159, in db_setup
ssl_match_hostname=ssl_match_hostname)
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 141, in _db_connect
raise e
ServerSelectionTimeoutError: No replica set members found yet
[root@k8s-master ~]# kubectl logs po/fancy-lizard-mongodb-ha-0
[root@k8s-master ~]# kubectl logs po/fancy-lizard-rabbitmq-ha-0
[root@k8s-master ~]# kubectl logs po/fancy-lizard-mongodb-ha-0
[root@k8s-master ~]# kubectl logs po/fancy-lizard-rabbitmq-ha-0
[root@k8s-master ~]# kubectl describe po/fancy-lizard-rabbitmq-ha-0
Name: fancy-lizard-rabbitmq-ha-0
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: <none>
Labels: app=rabbitmq-ha
controller-revision-hash=fancy-lizard-rabbitmq-ha-578c4f5c6
release=fancy-lizard
statefulset.kubernetes.io/pod-name=fancy-lizard-rabbitmq-ha-0
Annotations: checksum/config: f0e6787b570ecb396db600fa333ea81dd16f84792e9fed699e6c9d8529affab7
Status: Pending
IP:
Controlled By: StatefulSet/fancy-lizard-rabbitmq-ha
Init Containers:
copy-rabbitmq-config:
Image: busybox
Port: <none>
Host Port: <none>
Command:
sh
-c
cp /configmap/* /etc/rabbitmq; rm -f /var/lib/rabbitmq/.erlang.cookie
Environment: <none>
Mounts:
/configmap from configmap (rw)
/etc/rabbitmq from config (rw)
/var/lib/rabbitmq from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from fancy-lizard-rabbitmq-ha-token-pw7zg (ro)
Containers:
rabbitmq-ha:
Image: rabbitmq:3.7-alpine
Ports: 4369/TCP, 5672/TCP, 15672/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Liveness: exec [rabbitmqctl status] delay=120s timeout=5s period=10s #success=1 #failure=6
Readiness: exec [rabbitmqctl status] delay=10s timeout=3s period=5s #success=1 #failure=3
Environment:
MY_POD_NAME: fancy-lizard-rabbitmq-ha-0 (v1:metadata.name)
RABBITMQ_USE_LONGNAME: true
RABBITMQ_NODENAME: rabbit@$(MY_POD_NAME).fancy-lizard-rabbitmq-ha-discovery.default.svc.cluster.local
K8S_HOSTNAME_SUFFIX: .fancy-lizard-rabbitmq-ha-discovery.default.svc.cluster.local
K8S_SERVICE_NAME: fancy-lizard-rabbitmq-ha-discovery
RABBITMQ_ERLANG_COOKIE: <set to the key 'rabbitmq-erlang-cookie' in secret 'fancy-lizard-rabbitmq-ha'> Optional: false
RABBITMQ_DEFAULT_USER: admin
RABBITMQ_DEFAULT_PASS: <set to the key 'rabbitmq-password' in secret 'fancy-lizard-rabbitmq-ha'> Optional: false
RABBITMQ_DEFAULT_VHOST: /
Mounts:
/etc/rabbitmq from config (rw)
/var/lib/rabbitmq from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from fancy-lizard-rabbitmq-ha-token-pw7zg (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-fancy-lizard-rabbitmq-ha-0
ReadOnly: false
config:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
configmap:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: fancy-lizard-rabbitmq-ha
Optional: false
fancy-lizard-rabbitmq-ha-token-pw7zg:
Type: Secret (a volume populated by a Secret)
SecretName: fancy-lizard-rabbitmq-ha-token-pw7zg
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
Warning FailedScheduling 29m (x2 over 29m) default-scheduler pod has unbound immediate PersistentVolumeClaims
kubectl describe po/fancy-lizard-mongodb-ha-0
Name: fancy-lizard-mongodb-ha-0
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: <none>
Labels: app=mongodb-ha
controller-revision-hash=fancy-lizard-mongodb-ha-864d5f6749
release=fancy-lizard
statefulset.kubernetes.io/pod-name=fancy-lizard-mongodb-ha-0
Annotations: prometheus.io/path: /metrics
prometheus.io/port: 9216
prometheus.io/scrape: true
Status: Pending
IP:
Controlled By: StatefulSet/fancy-lizard-mongodb-ha
Init Containers:
copy-config:
Image: busybox
Port: <none>
Host Port: <none>
Command:
sh
Args:
-c
set -e
set -x
cp /configdb-readonly/mongod.conf /data/configdb/mongod.conf
Environment: <none>
Mounts:
/configdb-readonly from config (rw)
/data/configdb from configdir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cr842 (ro)
/work-dir from workdir (rw)
> Blockquote
install:
Image: k8s.gcr.io/mongodb-install:0.6
Port: <none>
Host Port: <none>
Args:
--work-dir=/work-dir
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cr842 (ro)
/work-dir from workdir (rw)
bootstrap:
Image: mongo:3.4
Port: <none>
Host Port: <none>
Command:
/work-dir/peer-finder
Args:
-on-start=/init/on-start.sh
-service=fancy-lizard-mongodb-ha
Environment:
POD_NAMESPACE: default (v1:metadata.namespace)
REPLICA_SET: rs0
Mounts:
/data/configdb from configdir (rw)
/data/db from datadir (rw)
/init from init (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cr842 (ro)
/work-dir from workdir (rw)
Containers:
mongodb-ha:
Image: mongo:3.4
Port: 27017/TCP
Host Port: 0/TCP
Command:
mongod
Args:
--config=/data/configdb/mongod.conf
--dbpath=/data/db
--replSet=rs0
--port=27017
--bind_ip=0.0.0.0
Liveness: exec [mongo --eval db.adminCommand('ping')] delay=30s timeout=5s period=10s #success=1 #failure=3
Readiness: exec [mongo --eval db.adminCommand('ping')] delay=5s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/data/configdb from configdir (rw)
/data/db from datadir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cr842 (ro)
/work-dir from workdir (rw)
Conditions:
Type Status
PodScheduled False
Volumes:
datadir:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: datadir-fancy-lizard-mongodb-ha-0
ReadOnly: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: fancy-lizard-mongodb-ha-mongodb
Optional: false
init:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: fancy-lizard-mongodb-ha-init
Optional: false
workdir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
configdir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
default-token-cr842:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-cr842
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
