Mistral connection issue


(Deepchandna) #1

Hi ,

My workflows are hung and i investigated to find below that mistral does not seem working fine.
I am seeing below error while running command -
[user@abc07323 workflows] mistral workflow-validate custom_workflow_common.yaml ERROR (app) ('Connection aborted.', BadStatusLine("''",)) [user@abc07323 workflows]

How to resolve this issue

Please suggest.


(Deepchandna) #2

It fixed when i restarted st2 with command - sudo st2ctl restart but happens again after a day.

Logs before fix -->

[2018-12-03 18:03:39 +0000] [2897] [ERROR] Socket error processing request.
Traceback (most recent call last):
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 191, in handle_request
    six.reraise(*sys.exc_info())
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/webob/dec.py", line 129, in __call__
    resp = self.call_func(req, *args, **kw)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/webob/dec.py", line 193, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/oslo_middleware/base.py", line 131, in __call__
    response = req.get_response(self.application)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/webob/request.py", line 1313, in send
    application, catch_exc_info=False)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/webob/request.py", line 1277, in call_application
    app_iter = application(self.environ, start_response)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/pecan/middleware/recursive.py", line 56, in __call__
    return self.application(environ, start_response)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/pecan/core.py", line 840, in __call__
    return super(Pecan, self).__call__(environ, start_response)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/pecan/core.py", line 683, in __call__
    self.invoke_controller(controller, args, kwargs, state)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/pecan/core.py", line 574, in invoke_controller
    result = controller(*args, **kwargs)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/api/controllers/v2/validation.py", line 32, in post
    definition = pecan.request.text
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/pecan/core.py", line 77, in __getattr__
    return getattr(obj, attr)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/pecan/core.py", line 52, in __getattribute__
    return WebObRequest.__getattribute__(self, name)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/webob/request.py", line 731, in _text__get
    body = self.body
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/pecan/core.py", line 52, in __getattribute__
    return WebObRequest.__getattribute__(self, name)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/webob/request.py", line 692, in body
    self.make_body_seekable() # we need this to have content_length
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/webob/request.py", line 928, in make_body_seekable
    self.copy_body()
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/webob/request.py", line 978, in copy_body
    fileobj = self.make_tempfile()
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/webob/request.py", line 1018, in make_tempfile
    return tempfile.TemporaryFile()
  File "/usr/lib64/python2.7/tempfile.py", line 489, in TemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags)
  File "/usr/lib64/python2.7/tempfile.py", line 239, in _mkstemp_inner
    fd = _os.open(file, flags, 0600)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/eventlet/green/os.py", line 109, in open
    fd = __original_open__(file, flags, mode)
OSError: [Errno 2] No such file or directory: '/tmp/tmpnxRFOx'

(Deepchandna) #3

I compared two of my servers running st2api . Hope below investigation will help to resolve issue ->

[user@abc1234 st2]$ sudo service st2api status
Redirecting to /bin/systemctl status st2api.service
● st2api.service - StackStorm service st2api
Loaded: loaded (/usr/lib/systemd/system/st2api.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2018-11-28 11:23:03 CST; 5 days ago
Main PID: 2158 (gunicorn)
CGroup: /system.slice/st2api.service
├─2158 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/gunicorn st2api.wsgi:application -k eventlet -b 127.0.0.1:9101 --wo…
└─2937 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/gunicorn st2api.wsgi:application -k eventlet -b 127.0.0.1:9101 --wo…

Dec 03 18:13:45 abc1234 gunicorn[2158]: File “/opt/stackstorm/st2/lib/python2.7/site-packages/webob/request.py”, line 1031, in make_tempfile
Dec 03 18:13:45 abc1234 gunicorn[2158]: return tempfile.TemporaryFile()
Dec 03 18:13:45 abc1234 gunicorn[2158]: File “/usr/lib64/python2.7/tempfile.py”, line 489, in TemporaryFile
Dec 03 18:13:45 abc1234 gunicorn[2158]: (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags)
Dec 03 18:13:45 abc1234 gunicorn[2158]: File “/usr/lib64/python2.7/tempfile.py”, line 239, in _mkstemp_inner
Dec 03 18:13:45 abc1234 gunicorn[2158]: fd = _os.open(file, flags, 0600)
Dec 03 18:13:45 abc1234 gunicorn[2158]: File “/opt/stackstorm/st2/lib/python2.7/site-packages/eventlet/green/os.py”, line 109, in open
Dec 03 18:13:45 abc1234 gunicorn[2158]: fd = original_open(file, flags, mode)
Dec 03 18:13:45 abc1234 gunicorn[2158]: OSError: [Errno 2] No such file or directory: ‘/tmp/tmp_VtDQL’ (_exception_data={},_excepti…VtDQL’")
Dec 03 18:13:45 abc1234 gunicorn[2158]: 2018-12-03 18:13:45,333 INFO [-] e958885a-9dba-4776-ba36-949256977cd1 - 500 46 14.719ms (co…d23e87’)

[user@xyz1234 workflows]$ sudo service st2api status
Redirecting to /bin/systemctl status st2api.service
● st2api.service - StackStorm service st2api
Loaded: loaded (/usr/lib/systemd/system/st2api.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2018-12-03 15:38:02 CST; 3h 56min ago
Main PID: 29397 (gunicorn)
CGroup: /system.slice/st2api.service
├─29397 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/gunicorn st2api.wsgi:application -k eventlet -b 127.0.0.1:9101 --w…
└─29520 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/gunicorn st2api.wsgi:application -k eventlet -b 127.0.0.1:9101 --w…

Dec 03 15:48:36 xyz1234 gunicorn[29397]: 2018-12-03 15:48:36,665 INFO [-] 68de636f-b56b-48fd-8df5-0322502cd312 - 200 72752 104.795m…utions’)
Dec 03 15:48:42 xyz1234 gunicorn[29397]: 2018-12-03 15:48:42,204 INFO [-] ae6894b9-5dc8-43ba-8d46-cff9f3bb9194 - GET /v1/executions with qu…
Dec 03 15:48:42 xyz1234 gunicorn[29397]: 2018-12-03 15:48:42,207 AUDIT [-] Token with id “5c05595d18daf30b6b8b3bbc” is validated.
Dec 03 15:48:42 xyz1234 gunicorn[29397]: 2018-12-03 15:48:42,251 INFO [-] ae6894b9-5dc8-43ba-8d46-cff9f3bb9194 - 200 31668 47.65ms …utions’)
Dec 03 15:48:44 xyz1234 gunicorn[29397]: 2018-12-03 15:48:44,038 INFO [-] af16b557-3aa1-46c6-ac75-7fe09a86efc4 - GET /v1/executions with qu…
Dec 03 15:48:44 xyz1234 gunicorn[29397]: 2018-12-03 15:48:44,041 AUDIT [-] Token with id “5c05595d18daf30b6b8b3bbc” is validated.
Dec 03 15:48:44 xyz1234 gunicorn[29397]: 2018-12-03 15:48:44,204 INFO [-] af16b557-3aa1-46c6-ac75-7fe09a86efc4 - 200 139624 166.32m…utions’)
Dec 03 15:48:52 xyz1234 gunicorn[29397]: 2018-12-03 15:48:52,619 INFO [-] 5d3c813b-7e2d-4f56-94a3-74bcf8be89c1 - GET /v1/executions…4a54b1’)
Dec 03 15:48:52 xyz1234 gunicorn[29397]: 2018-12-03 15:48:52,622 AUDIT [-] Token with id “5c05595d18daf30b6b8b3bbc” is validated.
Dec 03 15:48:52 xyz1234 gunicorn[29397]: 2018-12-03 15:48:52,686 INFO [-] 5d3c813b-7e2d-4f56-94a3-74bcf8be89c1 - 200 74595 66.991ms…4a54b1’)


(Deepchandna) #4

Can anyone please help here?


#5

Hey deepchandna, you’re a very active poster on these forums. We would be happy to help you with a support contract if you would like. I can put you in touch with our sales staff.


(Deepchandna) #6

Hi, I am not in a position to go for support contract. Need assistance here on portal to proceed further.


(Lindsay Hill) #7

This is the downside to ‘free’ support forums - there’s no guarantees that you can get free help. Sometimes someone can help, other times they can’t, or its so specific to your environment that it needs detailed investigation and troubleshooting, and that needs people who are being paid to do that sort of investigation.

It’s totally your business’s choice to decide to get, or not get a support contract. The code is open source, and you are entirely within your rights to freely use it. But if you can’t get the level of free help you need, repeatedly posting “Can anyone please help here?” isn’t going to make much difference. Your business will need to decide on the level of risk they are comfortable with.