Custom workflow unable to complete due to secret key decryption failing

Summary

On St2 test and corp, the custom workflow fails because of the error: “error”: “Non-hexadecimal digit found” on “task_id”: “get_z_hostid”

Steps to Reproduce the error.

  1. Login to St2 and run automation.zabbix.host.global_update action.

  2. Input a device’s FQDN and run the action.

What Is the Current Unexpected Behavior?

The automation.zabbix.host.global_update fails to complete due to an error with either Python, St2, or Zabbix.

What is the Correct Behavior?

The custom workflow in St2 completes, devices are updated, and correct in Netbox.

Relevant logs and/or Screenshots


{

  "output": null,

  "errors": [

    {

      "message": "Execution failed. See result for details.",

      "type": "error",

      "result": {

        "traceback": "  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/worker.py\", line 158, in _run_action

    result = self.container.dispatch(liveaction_db)

  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/container/base.py\", line 65, in dispatch

    runner = self._get_runner(runner_type_db, action_db, liveaction_db)

  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/container/base.py\", line 342, in _get_runner

    config = config_loader.get_config()

  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/util/config_loader.py\", line 82, in get_config

    config_db=config_db)

  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/util/config_loader.py\", line 94, in _get_values_for_config

    config = self._assign_dynamic_config_values(schema=schema_values, config=config)

  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/util/config_loader.py\", line 135, in _assign_dynamic_config_values

    parent_keys=parent_keys)

  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/util/config_loader.py\", line 150, in _assign_dynamic_config_values

    config_schema_item=schema_item)

  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/util/config_loader.py\", line 216, in _get_datastore_value_for_expression

    value = deserialize_key_value(value=value, secret=secret)

  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/services/config.py\", line 96, in deserialize_key_value

    value = symmetric_decrypt(KeyValuePairAPI.crypto_key, value)

  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/util/crypto.py\", line 189, in symmetric_decrypt

    return cryptography_symmetric_decrypt(decrypt_key=decrypt_key, ciphertext=ciphertext)

  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/util/crypto.py\", line 276, in cryptography_symmetric_decrypt

    ciphertext = binascii.unhexlify(ciphertext)

",

        "error": "Non-hexadecimal digit found"

      },

      "task_id": "get_z_hostid

Troubleshooting that has been tried.

  1. Because the error initially reported that a non-hexadecimal digit was found with a service account, A password change was completed to remove any non-hexadecimal characters. - This did not work.

  2. Configured the Zabbix pack to use credentials in plaintext and that worked but an unacceptable fix as the password is in plaintext.

  3. Installing pip and pycrypto, reloading St2 with the command sudo st2ctl restart - This did not work.

  4. Generated a token to update the Zabbix pack to version 0.3.0 - This did not work.

  5. Updating the gcc complier version to version 9 per Github issue from St2 Dev and Ubuntu commands to update gcc for Ubuntu 16.04 - This did not work.

Hi @Steve855. Can you cat /var/log/messages and /var/log/st2/st2api.log to see if there is anything else of interested at that time?

Alot of times I find a lower level error in one of those logs.

1 Like

Hey punkrokk,

Thanks for the response, I did some digging with the problematic workflow and all actions within the Zabbix pack are breaking with the same error above. From the st2api.log, here is what I found that looked interesting.

) {"action":"zabbix.test_credentials","parameters":{},"context":{"trace_context":{}}}
2020-02-06 21:53:59,498 140150516333936 DEBUG router [-] Match path: /v1/executions
2020-02-06 21:53:59,498 140150516333936 DEBUG router [-] Parsed endpoint: {'x-parameters': [{'in': 'context', 'x-as': 'requester_user', 'name': 'user', 'description': 'User performing the operat$
2020-02-06 21:53:59,498 140150516333936 DEBUG router [-] Parsed path_vars: {}
2020-02-06 21:53:59,500 140150516333936 AUDIT auth [-] Token with id "Token_ID" is validated.
2020-02-06 21:53:59,502 140150516333936 DEBUG router [-] Missing x-api-model definition for st2api.controllers.v1.actionexecutions:action_executions_controller.post, using generic Body model.
2020-02-06 21:53:59,504 140150516333936 DEBUG actionexecutions [-] User is: username
8) 2020-02-06 21:53:59,508 140150516333936 DEBUG config_loader [-] Attempting to get config for pack "zabbix" and user "username"
9) 2020-02-06 21:53:59,508 140150516333936 DEBUG config_loader [-] Pack and user found. Loading config.
10) 2020-02-06 21:53:59,512 140150516333936 DEBUG keyvalues [-] Lookup system kv: scope: st2kv.system and key: automation_user_password
11) 2020-02-06 21:53:59,516 140150516333936 DEBUG keyvalues [-] Got value secretpw from datastore.
12) 2020-02-06 21:53:59,516 140150516333936 INFO param [-] Failed to retrieve config for pack zabbix and user username: Non-hexadecimal digit found
2020-02-06 21:53:59,540 140150516333936 DEBUG channel [-] using channel_id: 1
2020-02-06 21:53:59,541 140150516333936 DEBUG channel [-] Channel open
2020-02-06 21:53:59,542 140150516333936 DEBUG channel [-] Closed channel #1
2020-02-06 21:53:59,542 140150516333936 DEBUG channel [-] using channel_id: 1
2020-02-06 21:53:59,543 140150516333936 DEBUG channel [-] Channel open
2020-02-06 21:53:59,544 140150516333936 DEBUG channel [-] Closed channel #1
2020-02-06 21:53:59,544 140150516333936 DEBUG channel [-] using channel_id: 1
2020-02-06 21:53:59,545 140150516333936 DEBUG channel [-] Channel open
2020-02-06 21:53:59,547 140150516333936 DEBUG channel [-] Closed channel #1
2020-02-06 21:53:59,547 140150516333936 AUDIT action [-] Action execution requested. LiveAction.id=5e3c8af7fada25371addae7d, ActionExecution.id=5e3c8af7fada25371addae7e
2020-02-06 21:53:59,550 140150516333936 DEBUG router [-] Using response spec "201" for endpoint st2api.controllers.v1.actionexecutions:action_executions_controller.post and status code 201
2020-02-06 21:53:59,551 140150516333936 DEBUG router [-] Match path: /v1/executions
2020-02-06 21:53:59,551 140150516333936 INFO logging [-] a9e89998-4b7d-48e4-a920-b6da13d1f4aa - 201 4009 53.943ms (content_length=4009,request_id='a9e89998-4b7d-48e4-a920-b6da13d1f4aa',runtime=5$[2020-02-06 21:53:59 +0000] [14106] [DEBUG] Closing connection

Lines 8 - 12 stick out as interesting, but I am not sure what they mean in the context of running an action/workflow.

Steve,

Can you share your /opt/stackstorm/configs/zabbix.yml file? My guess is that when you stored the secretpw in the datastore you didn’t encrypt it.

Did you maybe do:
st2 key set secretpw somepassword
instead of:
st2 key set secretpw somepassword --encrypt

Let me know if that helps fix it.

Hey punkrokk,

Your fix resolved my issue. Thanks for all the help.

-Steve

1 Like

Thanks steve, welcome to the community. Can you please mark my reply as the solution :slight_smile: