Some pods are not in ready state

Hello Team,

I’m trying to install StackStorm on Kubernetes. Found that a few pods are not running. Please help me to fix it.

[root@ip-172-31-89-90 tmp]# kubectl get pods
NAME READY STATUS RESTARTS AGE
invincible-toucan-etcd-0 1/1 Running 0 23m
invincible-toucan-etcd-1 0/1 CrashLoopBackOff 9 23m
invincible-toucan-etcd-2 1/1 Running 0 23m
invincible-toucan-job-st2-apikey-load-m425s 0/1 Completed 0 23m
invincible-toucan-job-st2-key-load-ppxsw 0/1 Completed 0 23m
invincible-toucan-job-st2-register-content-7d6nc 0/1 Completed 2 23m
invincible-toucan-mongodb-ha-0 1/1 Running 0 23m
invincible-toucan-mongodb-ha-1 1/1 Running 0 21m
invincible-toucan-mongodb-ha-2 1/1 Running 0 20m
invincible-toucan-rabbitmq-ha-0 1/1 Running 0 23m
invincible-toucan-rabbitmq-ha-1 1/1 Running 0 21m
invincible-toucan-rabbitmq-ha-2 1/1 Running 0 20m
invincible-toucan-st2actionrunner-74f97ff545-2tqqb 1/1 Running 2 23m
invincible-toucan-st2actionrunner-74f97ff545-2zhvx 1/1 Running 2 23m
invincible-toucan-st2actionrunner-74f97ff545-dk5qd 1/1 Running 2 23m
invincible-toucan-st2actionrunner-74f97ff545-k797s 1/1 Running 2 23m
invincible-toucan-st2actionrunner-74f97ff545-v7x8w 1/1 Running 2 23m
invincible-toucan-st2api-84d4cdd8bc-l99t4 1/1 Running 2 23m
invincible-toucan-st2api-84d4cdd8bc-r2ngq 1/1 Running 2 23m
invincible-toucan-st2auth-598df86996-lgk5t 1/1 Running 2 23m
invincible-toucan-st2auth-598df86996-qzr49 1/1 Running 2 23m
invincible-toucan-st2client-7f5c848bdd-wzsvd 1/1 Running 0 23m
invincible-toucan-st2garbagecollector-66c9888c4f-bhvxf 1/1 Running 2 23m
invincible-toucan-st2notifier-657f7c97f7-p89hl 1/1 Running 2 23m
invincible-toucan-st2notifier-657f7c97f7-rsh4j 1/1 Running 2 23m
invincible-toucan-st2rulesengine-68c778d99b-l8mwl 1/1 Running 2 23m
invincible-toucan-st2rulesengine-68c778d99b-mtnx9 1/1 Running 3 23m
invincible-toucan-st2scheduler-ddf475d84-gdc97 1/1 Running 2 23m
invincible-toucan-st2scheduler-ddf475d84-m8sjk 1/1 Running 2 23m
invincible-toucan-st2sensorcontainer-6c588bc8db-c9wc5 1/1 Running 2 23m
invincible-toucan-st2stream-6d7f5f785b-49m2s 1/1 Running 2 23m
invincible-toucan-st2stream-6d7f5f785b-hgpsd 1/1 Running 2 23m
invincible-toucan-st2timersengine-cb97c8c55-bqrt7 1/1 Running 3 23m
invincible-toucan-st2web-54cbddd6f6-4z7tx 1/1 Running 0 23m
invincible-toucan-st2web-54cbddd6f6-lphg9 1/1 Running 0 23m
invincible-toucan-st2workflowengine-6bb6854b66-shzq4 1/1 Running 2 23m
invincible-toucan-st2workflowengine-6bb6854b66-w74hd 1/1 Running 2 23m
[root@ip-172-31-89-90 tmp]#

[root@ip-172-31-89-90 tmp]# kubectl logs invincible-toucan-etcd-1
Waiting for invincible-toucan-etcd-0.invincible-toucan-etcd to come up
Waiting for invincible-toucan-etcd-1.invincible-toucan-etcd to come up
ping: bad address ‘invincible-toucan-etcd-1.invincible-toucan-etcd’
Waiting for invincible-toucan-etcd-1.invincible-toucan-etcd to come up
ping: bad address ‘invincible-toucan-etcd-1.invincible-toucan-etcd’
Waiting for invincible-toucan-etcd-1.invincible-toucan-etcd to come up
ping: bad address ‘invincible-toucan-etcd-1.invincible-toucan-etcd’
Waiting for invincible-toucan-etcd-1.invincible-toucan-etcd to come up
ping: bad address ‘invincible-toucan-etcd-1.invincible-toucan-etcd’
Waiting for invincible-toucan-etcd-1.invincible-toucan-etcd to come up
ping: bad address ‘invincible-toucan-etcd-1.invincible-toucan-etcd’
Waiting for invincible-toucan-etcd-1.invincible-toucan-etcd to come up
ping: bad address ‘invincible-toucan-etcd-1.invincible-toucan-etcd’
Waiting for invincible-toucan-etcd-1.invincible-toucan-etcd to come up
ping: bad address ‘invincible-toucan-etcd-1.invincible-toucan-etcd’
Waiting for invincible-toucan-etcd-1.invincible-toucan-etcd to come up
ping: bad address ‘invincible-toucan-etcd-1.invincible-toucan-etcd’
Waiting for invincible-toucan-etcd-1.invincible-toucan-etcd to come up
ping: bad address ‘invincible-toucan-etcd-1.invincible-toucan-etcd’
Waiting for invincible-toucan-etcd-1.invincible-toucan-etcd to come up
ping: bad address ‘invincible-toucan-etcd-1.invincible-toucan-etcd’
Waiting for invincible-toucan-etcd-1.invincible-toucan-etcd to come up
ping: bad address ‘invincible-toucan-etcd-1.invincible-toucan-etcd’
Waiting for invincible-toucan-etcd-1.invincible-toucan-etcd to come up
ping: bad address ‘invincible-toucan-etcd-1.invincible-toucan-etcd’
Waiting for invincible-toucan-etcd-1.invincible-toucan-etcd to come up
ping: bad address ‘invincible-toucan-etcd-1.invincible-toucan-etcd’
Waiting for invincible-toucan-etcd-1.invincible-toucan-etcd to come up
ping: bad address ‘invincible-toucan-etcd-1.invincible-toucan-etcd’
Waiting for invincible-toucan-etcd-1.invincible-toucan-etcd to come up
ping: bad address ‘invincible-toucan-etcd-1.invincible-toucan-etcd’
Waiting for invincible-toucan-etcd-1.invincible-toucan-etcd to come up
ping: bad address ‘invincible-toucan-etcd-1.invincible-toucan-etcd’
Waiting for invincible-toucan-etcd-1.invincible-toucan-etcd to come up
ping: bad address ‘invincible-toucan-etcd-1.invincible-toucan-etcd’
Waiting for invincible-toucan-etcd-1.invincible-toucan-etcd to come up
ping: bad address ‘invincible-toucan-etcd-1.invincible-toucan-etcd’
Waiting for invincible-toucan-etcd-1.invincible-toucan-etcd to come up
Waiting for invincible-toucan-etcd-2.invincible-toucan-etcd to come up
Re-joining etcd member
cat: can’t open ‘/var/run/etcd/member_id’: No such file or directory
[root@ip-172-31-89-90 tmp]#

[root@ip-172-31-89-90 tmp]# kubectl logs invincible-toucan-job-st2-apikey-load-m425s
2019-06-14 06:52:15,262 WARNING - Auth API server is not available, skipping authentication.
No api key found in /etc/st2/apikeys.yaml.
[root@ip-172-31-89-90 tmp]#
[root@ip-172-31-89-90 tmp]#
[root@ip-172-31-89-90 tmp]# kubectl logs invincible-toucan-job-st2-key-load-ppxsw
2019-06-14 06:52:28,874 WARNING - Auth API server is not available, skipping authentication.
No matching items found
[root@ip-172-31-89-90 tmp]#
[root@ip-172-31-89-90 tmp]#
[root@ip-172-31-89-90 tmp]# kubectl logs invincible-toucan-job-st2-register-content-7d6nc
2019-06-14 06:53:33,724 INFO [-] Connecting to database “st2” @ “invincible-toucan-mongodb-ha:27017” as user “admin”.
2019-06-14 06:53:33,776 INFO [-] Successfully connected to database “st2” @ “invincible-toucan-mongodb-ha:27017” as user “admin”.
2019-06-14 06:53:37,750 INFO [-] =========================================================
2019-06-14 06:53:37,751 INFO [-] ############## Registering triggers #####################
2019-06-14 06:53:37,751 INFO [-] =========================================================
2019-06-14 06:53:38,002 INFO [-] Registered 0 triggers.
2019-06-14 06:53:38,003 INFO [-] =========================================================
2019-06-14 06:53:38,004 INFO [-] ############## Registering sensors ######################
2019-06-14 06:53:38,004 INFO [-] =========================================================
2019-06-14 06:53:38,112 INFO [-] Registered 1 sensors.
2019-06-14 06:53:38,113 INFO [-] =========================================================
2019-06-14 06:53:38,113 INFO [-] ############## Registering runners ######################
2019-06-14 06:53:38,113 INFO [-] =========================================================
2019-06-14 06:53:41,538 INFO [-] Registered 15 runners.
2019-06-14 06:53:41,540 INFO [-] =========================================================
2019-06-14 06:53:41,540 INFO [-] ############## Registering actions ######################
2019-06-14 06:53:41,540 INFO [-] =========================================================
2019-06-14 06:53:44,165 INFO [-] Registered 54 actions.
2019-06-14 06:53:44,165 INFO [-] =========================================================
2019-06-14 06:53:44,165 INFO [-] ############## Registering rules ########################
2019-06-14 06:53:44,165 INFO [-] =========================================================
2019-06-14 06:53:44,228 INFO [-] Registered 1 rules.
2019-06-14 06:53:44,229 INFO [-] =========================================================
2019-06-14 06:53:44,229 INFO [-] ############## Registering aliases ######################
2019-06-14 06:53:44,229 INFO [-] =========================================================
2019-06-14 06:53:44,326 INFO [-] Registered 4 aliases.
2019-06-14 06:53:44,326 INFO [-] =========================================================
2019-06-14 06:53:44,326 INFO [-] ############## Registering policy types #################
2019-06-14 06:53:44,326 INFO [-] =========================================================
2019-06-14 06:53:44,361 INFO [-] Registered 3 policy types.
2019-06-14 06:53:44,361 INFO [-] =========================================================
2019-06-14 06:53:44,361 INFO [-] ############## Registering policies #####################
2019-06-14 06:53:44,361 INFO [-] =========================================================
2019-06-14 06:53:44,362 INFO [-] Registered 0 policies.
2019-06-14 06:53:44,362 INFO [-] =========================================================
2019-06-14 06:53:44,362 INFO [-] ############## Registering configs ######################
2019-06-14 06:53:44,362 INFO [-] =========================================================
2019-06-14 06:53:44,370 INFO [-] Registered 1 configs.
[root@ip-172-31-89-90 tmp]#
[root@ip-172-31-89-90 tmp]#

One pod from etcd backend is indeed in a failing state and we observe similar issues from time to time during our e2e tests, see: etcd deployment is unstable · Issue #75 · StackStorm/stackstorm-ha · GitHub, - something that will be fixed in future versions. But because 2/3 of etcd pods are running, - you’re good there as it’s enough for having etcd alive.

Those 3 Completed pods were one-time jobs triggered on initial cluster setup. They shouldn’t be always running.

Did you get output with ASCII art ST2 HA OK and instructions how to login to st2client after running helm install?
If yes, your StackStorm cluster deployed successfully and in a working state.

Hello Eugen,

Yes, I have received “ST2 OK HA” during the installation.

Cool, so you’re good there as cluster is alive & working :+1:
You can follow instructions there and bash exec into st2client pod to interact with StackStorm HA cluster via CLI.