ST2API Timeout after install - API never comes online

Doing a fresh install of ST2 via Ansible to a Centos 7 system.

The install tries to get the list of packs install and hangs. In digging into this, it is cause the API is not online so the st2 pack list -j command fails.

In the st2api.log file, seeing this:

2020-01-16 16:23:23,610 140642781823536 INFO __init__ [-] Successfully connected to database "st2" @ "127.0.0.1:27017" as user "None".

[2020-01-16 16:23:33 +0000] [1582] [CRITICAL] WORKER TIMEOUT (pid:8512)
[2020-01-16 16:23:33 +0000] [8512] [INFO] Worker exiting (pid: 8512)
[2020-01-16 16:23:33 +0000] [8534] [INFO] Booting worker with pid: 8534
2020-01-16 16:23:43,795 140642996422464 INFO app [-] Creating st2api: StackStorm v3.1.0 as OpenAPI app.
2020-01-16 16:23:53,861 140642781823536 INFO init [-] Connecting to database “st2” @ “127.0.0.1:27017” as user “None”.
2020-01-16 16:23:53,866 140642781823536 INFO init [-] Successfully connected to database “st2” @ “127.0.0.1:27017” as user “None”.
[2020-01-16 16:24:03 +0000] [1582] [CRITICAL] WORKER TIMEOUT (pid:8534)
2020-01-16 16:24:03,353 140642781823536 ERROR base [-] Publish failed.
Traceback (most recent call last):
File “/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/persistence/base.py”, line 193, in add_or_update
cls.publish_update(model_object)
File “/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/persistence/base.py”, line 274, in publish_update
publisher.publish_update(model_object)
File “/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/transport/publishers.py”, line 122, in publish_update
self._publisher.publish(payload, self._exchange, UPDATE_RK)
File “/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/transport/publishers.py”, line 84, in publish
retry_wrapper.run(connection=connection, wrapped_callback=do_publish)
File “/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/transport/connection_retry_wrapper.py”, line 144, in run
eventlet.sleep(wait)
File “/opt/stackstorm/st2/lib/python2.7/site-packages/eventlet/greenthread.py”, line 36, in sleep
hub.switch()
File “/opt/stackstorm/st2/lib/python2.7/site-packages/eventlet/hubs/hub.py”, line 297, in switch
return self.greenlet.switch()
File “/opt/stackstorm/st2/lib/python2.7/site-packages/eventlet/hubs/hub.py”, line 349, in run
self.wait(sleep_time)
File “/opt/stackstorm/st2/lib/python2.7/site-packages/eventlet/hubs/poll.py”, line 76, in wait
time.sleep(seconds)
File “/opt/stackstorm/st2/lib/python2.7/site-packages/gunicorn/workers/base.py”, line 196, in handle_abort
sys.exit(1)
SystemExit: 1

I’m guessing that something is up with communications to RabbitMQ, but that’s only a theory, and I can’t figure out how to get around it. Has anyone run into this before or have any ideas how to get around it?

Was able to resolve this. Turns out there was a bad Primary DNS server configured, so DNS lookups were taking a long time to complete. Something in this process was causing these to time out. As soon as the DNS issue was resolved, everything worked as expected.

1 Like