I have been testing URL Encoded webhooks and I keep running into an error where Stackstorm is converting the URL Encoded text data into binary format which then causes it to fail when trying to encode it as JSON data.
The simple URL Encoded body data I have been testing with is:
MyVariableOne=ValueOne&MyVariableTwo=ValueTwo
When posting this to my webook URL using Postman, I get the an HTPP 500 error with an “Internal Server Error” message response for Stackstrom and the following debug output in the API logs:
[2020-01-27 11:49:57 +0000] [10770] [DEBUG] POST /v1/webhooks/valero_cpe
2020-01-27 11:49:57,579 140064340756784 DEBUG (unknown file) [-] Match path: /v1/webhooks/valero_cpe
2020-01-27 11:49:57,580 140064340756784 INFO (unknown file) [-] 684b74a5-5ea6-4602-9006-2a9a0033710c - POST /v1/webhooks/valero_cpe with query={} (method='POST',path='/v1/webhooks/valero_cpe',remote_addr='127.0.0.1',query={},request_id='684b74a5-5ea6-4602-9006-2a9a0033710c')
2020-01-27 11:49:57,580 140064340756784 DEBUG (unknown file) [-] Received call with WebOb: POST /v1/webhooks/valero_cpe HTTP/1.0
Accept: */*
Accept-Encoding: gzip, deflate, br
Cache-Control: no-cache
Content-Length: 46
Content-Type: application/x-www-form-urlencoded
Host: 81296fd8.ngrok.io,81296fd8.ngrok.io
Postman-Token: 00a125ee-11b4-46fe-9500-75f5d9a811d8
St2-Api-Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
User-Agent: PostmanRuntime/7.22.0
X-Forwarded-For: 68.187.78.177, 127.0.0.1
X-Forwarded-Proto: https
X-Real-Ip: 127.0.0.1
X-Request-Id: 684b74a5-5ea6-4602-9006-2a9a0033710c
MyVariableOne=ValueOne&MyVariableTwo=ValueTwo
2020-01-27 11:49:57,580 140064340756784 DEBUG (unknown file) [-] Match path: /v1/webhooks/valero_cpe
2020-01-27 11:49:57,581 140064340756784 DEBUG (unknown file) [-] Parsed endpoint: {'operationId': 'st2api.controllers.v1.webhooks:webhooks_controller.post', 'x-requirements': {'hook': '.*'}, 'description': 'Trigger a webhook.\n', 'parameters': [{'name': 'hook', 'in': 'path', 'description': 'Webhook path', 'type': 'string', 'required': True}, {'name': 'webhook_body_api', 'in': 'body', 'description': 'Webhook payload', 'schema': {'$ref': '#/definitions/WebhookBody'}}], 'x-parameters': [{'name': 'headers', 'in': 'request', 'description': 'List of headers attached to the request.'}, {'name': 'user', 'in': 'context', 'x-as': 'requester_user', 'description': 'User performing the operation.'}], 'responses': {'202': {'description': 'Single action being created', 'schema': {'$ref': '#/definitions/WebhookBody'}, 'examples': {'application/json': {'ref': 'core.local'}}}, 'default': {'description': 'Unexpected error', 'schema': {'$ref': '#/definitions/Error'}}}}
2020-01-27 11:49:57,581 140064340756784 DEBUG (unknown file) [-] Parsed path_vars: {'hook': 'valero_cpe'}
2020-01-27 11:49:57,583 140064340756784 AUDIT (unknown file) [-] API key with id "5ddbbb999e14ef34dced4bfc" is validated.
2020-01-27 11:49:57,587 140064340756784 DEBUG (unknown file) [-] Dispatching trigger {'uid': 'trigger:core:8e1eb20c-33f6-4e4e-b0e9-dc166b37be7d:f7c2bb36b9bae14671835604c4167e1c', 'ref': 'core.8e1eb20c-33f6-4e4e-b0e9-dc166b37be7d', 'name': '8e1eb20c-33f6-4e4e-b0e9-dc166b37be7d', 'pack': 'core', 'type': 'core.st2.webhook', 'parameters': {'url': '/valero_cpe'}, 'id': '5df13f759e14ef0c5f1bcc3a'} with payload {'headers': {'Host': '81296fd8.ngrok.io,81296fd8.ngrok.io', 'X-Real-Ip': '127.0.0.1', 'X-Forwarded-For': '68.187.78.177, 127.0.0.1', 'Content-Length': '46', 'St2-Api-Key': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx', 'Content-Type': 'application/x-www-form-urlencoded', 'User-Agent': 'PostmanRuntime/7.22.0', 'Accept': '*/*', 'Cache-Control': 'no-cache', 'Postman-Token': '00a125ee-11b4-46fe-9500-75f5d9a811d8', 'Accept-Encoding': 'gzip, deflate, br', 'X-Forwarded-Proto': 'https', 'X-Request-Id': '684b74a5-5ea6-4602-9006-2a9a0033710c'}, 'body': {b'MyVariableOne': [b'ValueOne'], b'MyVariableTwo': [b'ValueTwo\n']}}.
2020-01-27 11:49:57,590 140064340756784 DEBUG (unknown file) [-] Dispatching trigger (trigger={'uid': 'trigger:core:8e1eb20c-33f6-4e4e-b0e9-dc166b37be7d:f7c2bb36b9bae14671835604c4167e1c', 'ref': 'core.8e1eb20c-33f6-4e4e-b0e9-dc166b37be7d', 'name': '8e1eb20c-33f6-4e4e-b0e9-dc166b37be7d', 'pack': 'core', 'type': 'core.st2.webhook', 'parameters': {'url': '/valero_cpe'}, 'id': '5df13f759e14ef0c5f1bcc3a'},payload={'trigger': {'uid': 'trigger:core:8e1eb20c-33f6-4e4e-b0e9-dc166b37be7d:f7c2bb36b9bae14671835604c4167e1c', 'ref': 'core.8e1eb20c-33f6-4e4e-b0e9-dc166b37be7d', 'name': '8e1eb20c-33f6-4e4e-b0e9-dc166b37be7d', 'pack': 'core', 'type': 'core.st2.webhook', 'parameters': {'url': '/valero_cpe'}, 'id': '5df13f759e14ef0c5f1bcc3a'}, 'payload': {'headers': {'Host': '81296fd8.ngrok.io,81296fd8.ngrok.io', 'X-Real-Ip': '127.0.0.1', 'X-Forwarded-For': '68.187.78.177, 127.0.0.1', 'Content-Length': '46', 'St2-Api-Key': 'MzE0NDQ2MzI1YjhiZDFjNmRjZjE5NmUzODEyYTViYmZjNzY3YjYyNzVlNGMxMGQ3ODQ3NzI0YzE0YjNjNjVjNA', 'Content-Type': 'application/x-www-form-urlencoded', 'User-Agent': 'PostmanRuntime/7.22.0', 'Accept': '*/*', 'Cache-Control': 'no-cache', 'Postman-Token': '00a125ee-11b4-46fe-9500-75f5d9a811d8', 'Accept-Encoding': 'gzip, deflate, br', 'X-Forwarded-Proto': 'https', 'X-Request-Id': '684b74a5-5ea6-4602-9006-2a9a0033710c'}, 'body': {b'MyVariableOne': [b'ValueOne'], b'MyVariableTwo': [b'ValueTwo\n']}}, 'trace_context': <st2common.models.api.trace.TraceContext object at 0x7f634564e518>})
2020-01-27 11:49:57,591 140064340756784 DEBUG channel [-] using channel_id: 1
2020-01-27 11:49:57,596 140064340756784 DEBUG channel [-] Channel open
2020-01-27 11:49:57,600 140064340756784 DEBUG channel [-] Closed channel #1
2020-01-27 11:49:57,601 140064340756784 ERROR (unknown file) [-] Failed to call controller function "post" for operation "st2api.controllers.v1.webhooks:webhooks_controller.post": key b'MyVariableOne' is not a string
Traceback (most recent call last):
File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/router.py", line 515, in __call__
resp = func(**kw)
File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2api/controllers/v1/webhooks.py", line 172, in post
return Response(json=body, status=http_client.ACCEPTED)
File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/router.py", line 150, in __init__
body = json_encode(json_body).encode('UTF-8')
File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/util/jsonify.py", line 45, in json_encode
return json.dumps(obj, cls=GenericJSON, indent=indent)
File "/usr/lib/python3.6/json/__init__.py", line 238, in dumps
**kw).encode(obj)
File "/usr/lib/python3.6/json/encoder.py", line 201, in encode
chunks = list(chunks)
File "/usr/lib/python3.6/json/encoder.py", line 430, in _iterencode
yield from _iterencode_dict(o, _current_indent_level)
File "/usr/lib/python3.6/json/encoder.py", line 376, in _iterencode_dict
raise TypeError("key " + repr(key) + " is not a string")
TypeError: key b'MyVariableOne' is not a string
2020-01-27 11:49:57,601 140064340756784 ERROR (unknown file) [-] API call failed: key b'MyVariableOne' is not a string
Traceback (most recent call last):
File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/error_handling.py", line 48, in __call__
return self.app(environ, start_response)
File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/streaming.py", line 47, in __call__
return self.app(environ, start_response)
File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/router.py", line 594, in as_wsgi
resp = self(req)
File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/router.py", line 519, in __call__
raise e
File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/router.py", line 515, in __call__
resp = func(**kw)
File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2api/controllers/v1/webhooks.py", line 172, in post
return Response(json=body, status=http_client.ACCEPTED)
File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/router.py", line 150, in __init__
body = json_encode(json_body).encode('UTF-8')
File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/util/jsonify.py", line 45, in json_encode
return json.dumps(obj, cls=GenericJSON, indent=indent)
File "/usr/lib/python3.6/json/__init__.py", line 238, in dumps
**kw).encode(obj)
File "/usr/lib/python3.6/json/encoder.py", line 201, in encode
chunks = list(chunks)
File "/usr/lib/python3.6/json/encoder.py", line 430, in _iterencode
yield from _iterencode_dict(o, _current_indent_level)
File "/usr/lib/python3.6/json/encoder.py", line 376, in _iterencode_dict
raise TypeError("key " + repr(key) + " is not a string")
TypeError: key b'MyVariableOne' is not a string (_exception_class='TypeError',_exception_message="key b'MyVariableOne' is not a string",_exception_data={})
2020-01-27 11:49:57,606 140064340756784 DEBUG (unknown file) [-] Match path: /v1/webhooks/valero_cpe
2020-01-27 11:49:57,607 140064340756784 INFO (unknown file) [-] 684b74a5-5ea6-4602-9006-2a9a0033710c - 500 46 27.193ms (method='POST',path='/v1/webhooks/valero_cpe',remote_addr='127.0.0.1',status=500,runtime=27.193,content_length=46,request_id='684b74a5-5ea6-4602-9006-2a9a0033710c')
2020-01-27 11:49:57,608 140064340756784 DEBUG (unknown file) [-] 684b74a5-5ea6-4602-9006-2a9a0033710c - 500 46 27.193ms
b'{\n "faultstring": "Internal Server Error"\n}' (method='POST',path='/v1/webhooks/valero_cpe',remote_addr='127.0.0.1',status=500,runtime=27.193,content_length=46,request_id='684b74a5-5ea6-4602-9006-2a9a0033710c',result='b\'{\\n "faultstring": "Internal Server Error"\\n}\'')
[2020-01-27 11:49:57 +0000] [10770] [DEBUG] Closing connection.