st2scheduler pods are always going to “CrashLoopBackOff” and then recreating then going to “CrashLoopBackOff”. Please find “kubectl logs” command output here.
# kubectl -n stackstorm logs dd-prod-st2scheduler-5cc46584f4-fw258
2019-08-20 11:28:32,697 DEBUG [-] Using Python: 2.7.12 (/opt/stackstorm/st2/bin/python)
2019-08-20 11:28:32,697 DEBUG [-] Using config files: /etc/st2/st2.conf,/etc/st2/st2.docker.conf,/etc/st2/st2.user.conf
2019-08-20 11:28:32,698 DEBUG [-] Using logging config: /etc/st2/logging.scheduler.conf
2019-08-20 11:28:32,714 INFO [-] Connecting to database "st2" @ "dd-prod-mongodb-ha:27017" as user "admin".
2019-08-20 11:29:02,916 ERROR [-] Failed to connect to database "st2" @ "dd-prod-mongodb-ha:27017" as user "admin": No replica set members match selector "Primary()"
2019-08-20 11:29:02,917 ERROR [-] (PID=1) Scheduler quit due to exception.
Traceback (most recent call last):
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/cmd/scheduler.py", line 124, in main
_setup()
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/cmd/scheduler.py", line 53, in _setup
register_signal_handlers=True, service_registry=True, capabilities=capabilities)
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/service_setup.py", line 160, in setup
db_setup()
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/database_setup.py", line 56, in db_setup
connection = db_init.db_setup_with_retry(**db_cfg)
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/db_init.py", line 75, in db_setup_with_retry
ssl_match_hostname=ssl_match_hostname)
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/db_init.py", line 58, in db_func_with_retry
return retrying_obj.call(db_func, *args, **kwargs)
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py", line 206, in call
return attempt.get(self._wrap_exception)
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 158, in db_setup
ssl_match_hostname=ssl_match_hostname)
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 140, in _db_connect
raise e
ServerSelectionTimeoutError: No replica set members match selector "Primary()"