Can't setup stackstorm with mongodb SSL and ReplicaSet

Hello

I try to setup stackstorm when the DB is on Alas cloud and consists of 3 noes in replicaset.

The st2 actually succeeds to start up and even command like st2ctl reload shows that everything ok but in the api log and auth log i see every 5 minutes error 鈥渃annot connect to database default :
maximum recursion depth exceeded鈥

I definitely see that stackstorm can connect to mongo and even it succeeded to create collection there but i can鈥檛 login to stackstorm and things don鈥檛 work with all mentioned errors in the logs.

If i change the same instance and connect it to standalone mongo db on another vm by changing paramters in st2.conf , everything fine.
the issue exists only with replica set on atlas mongo :frowning:
anybody can help ? :slightly_smiling_face:

We hadn鈥檛 chance to try StackStorm with Atlas MongoDB yet as a backend.

What version of MongoDB your Atlas cluster is running on?
StackStorm supports v3.6-v4.0 at the moment.

We use version 4.0.

But if i think about it, it doesn鈥檛 matter where mongo db runs. On prem or in the cloud. It is exposed to stackstorm as regular mongo like in prem, on the same ports without any differenc. So it鈥檚 strange. I personally suspect that somehow related to wrong settings for replica set in st2.conf.
I really can鈥檛 find any example or use case how to config it properly.

We鈥檝e seen somewhat similar issues when in-Cloud offering is in fact, differs by functionality and behaves differently comparing to OSS MongoDB.

For st2.conf MongoDB ReplicaSet, check the Documentation: Config details for connecting to external PostgreSQL MongoDB and RabbitMQ Clusters

Ok , i succeeded to reproduce this issue on premises.

The problem is not related to replica set but to ssl.
I succeeded to connect a stackstorm server to on premises mongo replica set without ssl.
But once i require the connection to be protected by ssl, the same issue is appearing.
without ssl all seems working perfectly with or without replica set.

I tried all combination of paramters, i tried to remove part of them but always the same issue is appearing - the stackstorm is succeeded to start up and even to create the DB and collections in the DB ( so the initial connection is ok) but after, every few seconds in st2api.log and st2auth.log i see this:

2019-12-13 15:22:58,551 139656589406280 INFO (unknown file) [-] Connecting to database 鈥渟t5鈥 @ 鈥渄-stackstorm-control-box-vm:27017,d-stackstorm1-vm:27017 (replica set)鈥 as user 鈥渁lex鈥.
2019-12-13 15:22:58,553 139656589406280 WARNING (unknown file) [-] Retry on ConnectionError - Cannot connect to database default :
maximum recursion depth exceeded.

So it鈥檚 definitely some bug with ssl implementation that leads the exception : maximum recursion depth exceeded.

currently my settings as following:
username=alex
password=alexalexalexalex
host=mongodb://some-host1,some-host2/?replicaSet=alexreplica
ssl=true
ssl_cert_reqs=optional
ssl_match_hostname=false
ssl_keyfile=/etc/st2/st2mongo.key
ssl_certfile=/etc/st2/st2mongo.crt
ssl_ca_certs=/etc/st2/rootCA.crt

I already tried to gray out 3 last settings or even part of them without any positive result. still the same error.
Only removing ssl requirement solves it but i can鈥檛 remove it forever. Noway to connect to mongo DB without ssl protection. unfortunately no way to use stackstorm now because of this limitation :frowning:

Information that you really wanted to configure RepliaSet and SSL was crucial here. Please try to provide more complete info for others to help.

It looks like MongoDB ReplicaSet and SSL connection settings were documented in Nginx and WSGI 鈥 StackStorm 3.1.0 documentation

For the SSL configuration in st2.conf, this was definitely tested before.

According to https://api.mongodb.com/python/current/examples/tls.html, can you try with just simplest setting first

[database]
ssl = True

and see any error messages?

Additionally, depending on OS and openssl version results might differ.
Try to enable debug settings in st2.conf for services, - there should be a better information in logs about why st2 can鈥檛 connect.

It worth checking the server logs as well for more information about the connected clients. Something like
https://stackoverflow.com/questions/35790287/self-signed-ssl-connection-using-pymongo/35967188 might help to understand where to look further.

And finally, check if mongo client can connect to remote server http://www.dba86.com/docs/mongo/3.0/tutorial/configure-ssl-clients.html with similar SSL settings. That might help you to indicate where the problem is. If mongo client will be able to connect with SSL configuration, - st2 should be able to connect too with the same settings.

Hey
I tried all but still without success :frowning:
Now i even destroyed again the replicaset and started up the mongo as standalone server and reconfugred the st2.conf file in this way
[database]
db_name=st5
username=alex
password=password
port=27017
host=d-stackstorm-control-box-vm
ssl=true

Unfortunately , the same error.
so it鈥檚 not a problem of replica set.
mongo client from stackstorm server connects to mongodb server (diffrent vm) over ssl without any problem .
No evidence for error in mongo db log,. Debug mode for staskstorm didnt鈥 bring any new information, but only the same error every few seconds.
Moreover, st2ctl reload works perfectly and connect to mongo over ssl without any issue. I see all collection were created in the DB. So it鈥檚 seems as not some underlying infrastracture issue in ssl/os/mongo. very strange .

It seems to me some bug in python code because the 鈥渕aximum recursion depth exceeded鈥 is definitely some python exception.

I tried to reproduce this issue in 2 different vm and in the both was that bug with ssl. I use ubuntu 18.04 and mongo 4 and installed exactly following to this instruction but with separation of mongo and stackstorm on different vms.:

https://ewc-docs.extremenetworks.com/install/u18.html

Just wondering if somebody succeeded to get it working in the same condititions and requirements.

Just to confirm the debug instructions for StackStorm configuration, make sure to follow this doc:
https://docs.stackstorm.com/troubleshooting/debug_mode.html
There should be something in StackStorm logs for services like st2api.

trust me. i spent enough time to troubleshoot this. I have enabled debug mode too but no useful information was found in the log files, just the same error every few seconds.

I also defined in mongod.conf not to require client certificate i.e allowConnectionsWithoutCertificates: true.

Any way i don;t see any connection attempts in the monogd logs. I think the stackstorm even doesn鈥檛 try to connect because this exception. I am sorry but it is definitely bug in the stackstorm. I pretty sure that it doen鈥檛 eork over ssl in my scenario .

Alright, please report bug in GitHub - StackStorm/st2: StackStorm (aka "IFTTT for Ops") is event-driven automation for auto-remediation, security responses, troubleshooting, deployments, and more. Includes rules engine, workflow, 160 integration packs with 6000+ actions (see https://exchange.stackstorm.org) and ChatOps. Installer at https://docs.stackstorm.com/install/index.html. Questions? https://forum.stackstorm.com/. then with more details as possible.
Someone will take a look at that closer.

Thank you for assistance :slight_smile:
I just have opened report bug here Connection to mongo db over SSL doesn't work 路 Issue #4832 路 StackStorm/st2 路 GitHub

Hope wiil be resolved :slight_smile:

Thanks for opening the issue.

I believe a change in https://github.com/StackStorm/st2/pull/4834 should fix it.

You can test that change by upgrading to staging unstable packages (https://packagecloud.io/StackStorm/staging-unstable) after they have been built (the PR has just been merged and the packages should be built and promoted by tomorrow EOD by the latest).

That鈥檚 of course at your own risk since staging unstable packages are based directly off master and are considered as an development version.

1 Like

I am not sure whether SSL on mongodb has any impact on K8 HA configuration. Currently doing wireshark analysis which shows there are several handshake request to mongodb which seem to fail. I will update the details next week. Meanwhile if someone has information on how to enable or disable mongodb SSL , please share the details here