Unable to resolve issue related to ' Connection refused /St2 Services stopped '

#1

I have install Stackstorm on VM , it worked fine for couple of days. I restarted VM and after that most of the services stopped working.

When I run command st2 pack install kafka or login using st2 login commands, it throws below error

ERROR: HTTPConnectionPool(host=‘127.0.0.1’, port=9101): Max retries exceeded with url: /v1/packs/index/search (Caused by NewConnectionError(’<urllib3.connection.HTTPConnection object at 0x7f41a45abe50>: Failed to establish a new connection: [Errno 111] Connection refused’,))

When I run command St2 reload or St2 restart it does not start all the services

st2 components status

st2actionrunner PID: 24162
st2actionrunner PID: 24164
st2actionrunner PID: 24167
st2actionrunner PID: 24170
st2actionrunner PID: 24172
st2actionrunner PID: 24175
st2actionrunner PID: 24180
st2actionrunner PID: 24183
st2actionrunner PID: 24187
st2actionrunner PID: 24192
st2api is not running.
st2stream is not running.
st2auth is not running.
st2garbagecollector PID: 1542
st2notifier is not running.
st2resultstracker PID: 1540
st2rulesengine PID: 1533
st2sensorcontainer is not running.
st2chatops is not running.
st2timersengine is not running.
st2workflowengine PID: 1622
st2scheduler PID: 1541

After that I uninstalled ST2 and tried to install it but services does not start.

#2

Is mongoDB running on the same server? What is the status of the mongod service?

#3

Both MongoDB and RabbitMQ services are running

#4

I would:

  • Check the service status of the failed services
  • check the ST2 logs in /var/log/st2 for the services that failed to start, in particular I am wondering if they are failing to establish connection to mongodb

And then if that doesn’t help then please supply:

  • What version of StackStorm and what O/S
  • Installation method used - e.g. bash installer, ansible, manual etc
  • Are you running all services in same VM, e.g. mongod and rabbitmq etc?
  • If you are using the one-line installer and you performed an uninstall, did you follow the uninstall instructions and delete all the supplementary services, directories, repos etc?
#5

• What version of StackStorm and what O/S – We are using RHEL 7 and St2 V3.3 versions
• Installation method used - Manual
• Are you running all services in same VM- All services are on same VM
• If you are using the one-line installer and you performed an uninstall, did you follow the uninstall instructions and delete all the supplementary services, directories, repos etc
• - Yes I have followed uninstall instructions provided in Uninstall — StackStorm 3.3.0 documentation
•
I tried to filter out some of the errors from the /var/logs/st2/*.log files. Below are some of the repetitive errors found.

File “/opt/stackstorm/st2/lib/python2.7/site-packages/mongoengine/document.py”, line 411, in save
raise NotUniqueError(message % six.text_type(err))
NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.runner_type_d_b index: uid_1 dup key: { : “runner_type:python-script” })
2020-12-04 08:13:49,651 140178653154352 ERROR runnersregistrar [-] Unable to register runner type python-script.

File “/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/bootstrap/runnersregistrar.py”, line 101, in register_runner
runner_type_db = RunnerType.add_or_update(runner_type_model)
File “/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/persistence/base.py”, line 186, in add_or_update
model_object=model_object)
StackStormDBObjectConflictError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.runner_type_d_b index: uid_1 dup key: { : “runner_type:python-script” })
2020-12-04 08:13:49,896 140178653154352 ERROR base [-] Conflict while trying to save in DB.
Traceback (most recent call last):

File “/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/models/db/init.py”, line 466, in add_or_update
instance.save(validate=validate)
File “/opt/stackstorm/st2/lib/python2.7/site-packages/mongoengine/document.py”, line 411, in save
raise NotUniqueError(message % six.text_type(err))
NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.runner_type_d_b index: uid_1 dup key: { : “runner_type:winrm-ps-script” })
2020-12-04 08:13:48,063 140533716778032 ERROR runnersregistrar [-] Unable to register runner type winrm-ps-script.
Traceback (most recent call last):

File “/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/models/db/init.py”, line 466, in add_or_update
instance.save(validate=validate)
File “/opt/stackstorm/st2/lib/python2.7/site-packages/mongoengine/document.py”, line 411, in save
raise NotUniqueError(message % six.text_type(err))
NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.runner_type_d_b index: uid_1 dup key: { : “runner_type:noop” })
2020-12-04 08:13:50,456 140533716778032 ERROR runnersregistrar [-] Unable to register runner type noop.
Traceback (most recent call last):
File “/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/bootstrap/runnersregistrar.py”, line 101, in register_runner
runner_type_db = RunnerType.add_or_update(runner_type_model)
File “/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/persistence/base.py”, line 186, in add_or_update
model_object=model_object)
StackStormDBObjectConflictError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.runner_type_d_b index: uid_1 dup key: { : “runner_type:noop” })
2020-12-04 08:13:50,834 140533716778032 E

#6

What’s the MongoDB version you’re using?

#7

I am using Mongo DB version v4.0.21

#8

Any specific reason you went through Manual install, did the default curl|bash quick installer (Installation — StackStorm 3.3.0 documentation) work for you?

#9

I did try quick install on a newly built VM but it is failing with below errors

20201208T091457-0600 Error: Package: erlang-erts-R16B-03.18.el7.x86_64 (epel)
20201208T091457-0600            Requires: lksctp-tools
20201208T091457-0600  You could try using --skip-broken to work around the problem
20201208T091510-0600  You could try running: rpm -Va --nofiles --nodigest
20201208T091510-0600 ############### ERROR ###############
20201208T091510-0600 # Failed on Install st2 dependencies #
#10

Please help me to resolve this issue.

#11

I procured a VM and did manual ST2 install on it couple of days back. One of the developer of my team informed me that he is not able to login on St2 GUI. When I checked GUI it is showing error
‘Nginx is unable to reach st2auth. Make sure service is running.’
I also checked /var/log/st2/st2auth.log but did not find any error which can tell me why nginx authentication failed.

When I tried to login using CLI, it throws below error. Both MongoDB and RabbitMQ services running

st2 login st2admin

Password:
ERROR: Failed to log in as st2admin: HTTPConnectionPool(host=‘127.0.0.1’, port=9100): Max retries exceeded with url: /tokens (Caused by NewConnectionError(’<urllib3.connection.HTTPConnection object at 0x7f04581d5190>: Failed to establish a new connection: [Errno 111] Connection refused’,))

#12

What is outcome of:

  • st2ctl status
  • systemctl status mongod
  • systemctl status rabbitmq-server
  • systemctl status nginx
  • systemctl status st2auth

Did st2auth manage to connect to mongod ok? (There should be message similar to following in the /var/log/st2/st2auth.log around the time the service started…
Successfully connected to database “st2”
)