Both st2scheduler Pods are not running

st2scheduler pods are always going to “CrashLoopBackOff” and then recreating then going to “CrashLoopBackOff”. Please find “kubectl logs” command output here.

# kubectl -n stackstorm logs dd-prod-st2scheduler-5cc46584f4-fw258
2019-08-20 11:28:32,697 DEBUG [-] Using Python: 2.7.12 (/opt/stackstorm/st2/bin/python)
2019-08-20 11:28:32,697 DEBUG [-] Using config files: /etc/st2/st2.conf,/etc/st2/st2.docker.conf,/etc/st2/st2.user.conf
2019-08-20 11:28:32,698 DEBUG [-] Using logging config: /etc/st2/logging.scheduler.conf
2019-08-20 11:28:32,714 INFO [-] Connecting to database "st2" @ "dd-prod-mongodb-ha:27017" as user "admin".
2019-08-20 11:29:02,916 ERROR [-] Failed to connect to database "st2" @ "dd-prod-mongodb-ha:27017" as user "admin": No replica set members match selector "Primary()"
2019-08-20 11:29:02,917 ERROR [-] (PID=1) Scheduler quit due to exception.
Traceback (most recent call last):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/cmd/scheduler.py", line 124, in main
    _setup()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/cmd/scheduler.py", line 53, in _setup
    register_signal_handlers=True, service_registry=True, capabilities=capabilities)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/service_setup.py", line 160, in setup
    db_setup()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/database_setup.py", line 56, in db_setup
    connection = db_init.db_setup_with_retry(**db_cfg)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/db_init.py", line 75, in db_setup_with_retry
    ssl_match_hostname=ssl_match_hostname)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/db_init.py", line 58, in db_func_with_retry
    return retrying_obj.call(db_func, *args, **kwargs)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py", line 206, in call
    return attempt.get(self._wrap_exception)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 158, in db_setup
    ssl_match_hostname=ssl_match_hostname)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 140, in _db_connect
    raise e
ServerSelectionTimeoutError: No replica set members match selector "Primary()"

Sounds like you need to fix your MongoDB cluster, as pod unable to connect to primary MongoDB node.

Quick google search found this for you:
https://groups.google.com/forum/#!topic/mongodb-user/921m7vUY1H0