Ghost2Logger - SensorContainer reset error

Hello,
Installed ghost2logger, rule is been created. Host and pattern is working, i can see in the syslog and entry matched, but in the st2sensorcontainer.log, I see connection reset error.

2020-06-15 20:54:55,369 140610254255064 WARNING mixins [-] Broker connection error, trying again in 8.0 seconds: ConnectionResetError(104, 'Connection reset by peer').
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/kombu/utils/functional.py", line 343, in retry_over_time
    return fun(*args, **kwargs)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/kombu/connection.py", line 283, in connect
    return self.connection
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/kombu/connection.py", line 837, in connection
    self._connection = self._establish_connection()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/kombu/connection.py", line 792, in _establish_connection
    conn = self.transport.establish_connection()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
    conn.connect()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/amqp/connection.py", line 317, in connect
    self.drain_events(timeout=self.connect_timeout)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/amqp/connection.py", line 505, in drain_events
    while not self.blocking_read(timeout):
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/amqp/connection.py", line 511, in blocking_read
    return self.on_inbound_frame(frame)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/amqp/method_framing.py", line 55, in on_frame
    callback(channel, method_sig, buf, None)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/amqp/connection.py", line 518, in on_inbound_method
    method_sig, payload, content,
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/amqp/abstract_channel.py", line 145, in dispatch_method
    listener(*args)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/amqp/connection.py", line 406, in _on_start
    login_response, self.locale),
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/amqp/abstract_channel.py", line 59, in send_method
    conn.frame_writer(1, self.channel_id, sig, args, content)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/amqp/method_framing.py", line 172, in write_frame
    write(view[:offset])
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/amqp/transport.py", line 284, in write
    self._write(s)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/eventlet/greenio/base.py", line 403, in sendall
    tail = self.send(data, flags)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/eventlet/greenio/base.py", line 397, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/eventlet/greenio/base.py", line 384, in _send_loop
    return send_method(data, *args)
ConnectionResetError: [Errno 104] Connection reset by peer

2020-06-15 20:55:03,413 140610254257040 INFO (unknown file) [-] [Ghost2logger]: Received Matched Syslog Event
2020-06-15 20:55:03,414 140610254257040 INFO (unknown file) [-] {'host': '192.168.1.1', 'hostpattern': '192.168.1.1', 'message': '<165>Jun 15 20:50:49 swdevice01 ConfigAgent: 4929190: %SYS-5-CONFIG_STARTUP: Startup config saved from system:/running-config by st2user on vty4 (10.11.11.72).', 'pattern': 'SYS-5-CONFIG_STARTUP'}

Hi,
Did you manage to resolve this issue?

I am in a similar position at the moment.

In the st2sensorcontainer.log did you have any messages after the connection reset?

For example if the sensor is getting the event but matching then you will have a message
“2020-07-02 16:32:20,219 140468067622232 INFO (unknown file) [-] [Ghost2logger]: Received Matched Syslog Event”

In my situation I also see the connection reset, but I then have the matched syslog event. I’m just debugging to see what happens when the sensor was dispatched to see what went wrong.

Thanks,

You may also want to check out the rabbitmq logs as that might be the connection problems.

@amanda11 thanks for the info, I checked the rabbitmq logs, It connects and then dissconnect, not sure if that is normal.

=ERROR REPORT==== 16-Sep-2020::10:53:16 ===
closing AMQP connection <0.21726.2> (127.0.0.1:58430 -> 127.0.0.1:5672):
{handshake_timeout,frame_header}

What version of python are you running? In the end my problem was due to a problem with stackstorm and python 3. That problem is now fixed in the stackstorm 3.3dev unstable packages.
So I could either use 3.3dev and python 3 or python 2 and stackstorm 3.2.

Hi,
I’m using stackstorm ver 3.2 with python 3. Should upgrade to 3.3beta? How easy is to install to 3.3beta?
Thank you

I’m not sure on your setup but I’ll give you some info on the 3.3dev release, so that you can decide how best to proceed

1. Documentation for the unstable release can be found at: StackStorm Documentation — StackStorm 3.3dev documentation
2. As it is a dev release we haven’t yet gone through our release testing process, so be aware of that. We do run CI/CD on our master branch, but we haven’t yet gone through release testing, which would involve help from the community so that it is tested on a wide range of setups. This would involve testing upgrade etc.
3. We were hoping to code freeze 3.3 dev last week, but we’ve a few loose-ends to tie up first. (always looking for any help anyone can help with this). So I can’t give a date yet for 3.3 to be released. After code-freeze we would then go through our release process which would include asking community to help test.
4. To install the 3.3dev release then you need to install the unstable repos. Each of the installation methods should allow that - and the help or documentation on the relevant install method should help. e.g. the -s on the one line installer indicates you want the stable release, to install a unstable release you can use the -u flag instead or use the -v flag and specify the version as 3.3dev. Other install methods such as ansible etc should also support these options.
5. The PR that contained the fix was Fixes: #4875 monkeypatch st2 sensor earlier by punkrokk · Pull Request #4976 · StackStorm/st2 · GitHub.
6. I did a fresh install of 3.3dev to resolve it, but there is info on how to upgrade at Upgrades — StackStorm 3.3dev documentation - please also read the upgrade_notes (Upgrade Notes — StackStorm 3.3dev documentation), and note that you would need to follow the comments about the “unstable” upgrade. However, as we’ve not yet gone through a release process then the upgrade from 3.2 to current unstable might not have been tested. I didn’t try this option at the time.
7. If it’s of any help my forum questions with the resolution is at CentOS 8 Problems with rabbitmq connections when sensor dispatches trigger (though I did go down a lot of red herrings trying to resolve it!)