Passing output to slack from workflow


(gmnvn) #1

Hello,

I’m exploring stackstorm and getting used to it. I’m trying to automate to get top 5 cpu process when we get a cpu alert. I achieved that by following disk remediation example. However I’m facing issue with posting the command output directly to slack. Kindly requesting your help.

Here’s my workflow(actions/workflows/check_cpu.yaml):

---
version: '2.0'
name: ops.check_cpu

workflows:
  main:
    input:
      - hostname
      - event_id
      - alert_message
      - raw_payload

    tasks:
      check_process:
        action: core.remote
        input:
          hosts: <% $.hostname %>
          cmd: "ps -eo pid,pcpu,user,args --sort=-%cpu | head -6 | head -c 1000"
        on-error:
          - pagerduty_escalation
        on-success:
          - post_to_slack

      pagerduty_escalation:
        action: pagerduty.incident.create.rest_v2.simple
        input:
          service_id: "PxxxxxxN"
          title: "Stackstorm Check CPU failed on <% $.hostname %>"
          from_email: "xxxxx@xxxxx.com"
          details: "Stackstorm could not autoremediate high cpu usage on <% $.hostname %>. Alert: <% $.alert_message %>"

      post_to_slack:
        action: slack.post_message
        input:
          channel: "#stackstorm"
          message: "High CPU Usage triggered from: <% $.hostname %>. Here's the processes details:\n<% task(check_process).result.stdout %>"

Here’s the action chain(actions/chains/check_cpu.yaml):

---
  chain:
    -
      name: "check_process"
      ref: "core.remote"
      params:
        hosts: "{{hostname}}"
        cmd: "ps -eo pid,pcpu,user,args --sort=-%cpu | head -6"
      on-success: "post_to_slack"
      on-failure: "pagerduty_escalation"
    -
      name: "pagerduty_escalation"
      ref: "pagerduty.incident.create.rest_v2.simple"
      params:
        service_id: "PxxxxxN"
        title: "Stackstorm Check CPU failed on {{hostname}}"
        from_email: "xxxxxx@xxxxx.com"
        details: "Stackstorm could not autoremediate high cpu usage on {{hostname}}. Alert: {{alert_message}}"
    -
      name: "post_to_slack"
      ref: "slack.post_message"
      params:
        channel: "#stackstorm"
        message: "Stackstorm got a cpu alert & has got top 5 cpu consuming processes on {{hostname}}. Details:\n{{ check_process.stdout }}"
  default: "check_process"

Here’s the Error I’m seeing in the UI console:

{
  "tasks": [
    {
      "state_info": null,
      "name": "check_process",
      "workflow_name": "ops.check_cpu.main",
      "created_at": "2019-02-22 19:30:53",
      "updated_at": "2019-02-22 19:30:54",
      "workflow_execution_id": "47529098-87e4-49b8-beaa-ce320ab9f432",
      "state": "SUCCESS",
      "result": {
        "10.146.74.91": {
          "succeeded": true,
          "failed": false,
          "return_code": 0,
          "stderr": "",
          "stdout": "  PID %CPU USER     COMMAND\
 9032 61.1 root     /usr/bin/python2.7 /usr/bin/aws s3 sync /media/ebs/logs s3://dev.canopydata.com/logserver\
17787 23.9 root     /usr/bin/java -Xms1g -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /usr/lib/logstash-6.2.3/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/commons-compiler-3.0.8.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/google-java-format-1.5.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/guava-22.0.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/jackson-annotations-2.9."
        }
      },
      "published": {},
      "input": null,
      "id": "bc149571-1d49-4b68-9320-d87bd668fe1d"
    },
    {
      "state_info": "Failed to run task [error=Can not evaluate YAQL expression [expression=task(check_process).result.stdout, error=u'stdout', data={}], wf=ops.check_cpu.main, task=post_to_slack]:
Traceback (most recent call last):
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/engine/task_handler.py\", line 63, in run_task
    task.run()
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/osprofiler/profiler.py\", line 159, in wrapper
    result = f(*args, **kwargs)
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/engine/tasks.py\", line 374, in run
    self._run_new()
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/osprofiler/profiler.py\", line 159, in wrapper
    result = f(*args, **kwargs)
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/engine/tasks.py\", line 403, in _run_new
    self._schedule_actions()
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/engine/tasks.py\", line 470, in _schedule_actions
    input_dict = self._get_action_input()
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/osprofiler/profiler.py\", line 159, in wrapper
    result = f(*args, **kwargs)
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/engine/tasks.py\", line 500, in _get_action_input
    input_dict = self._evaluate_expression(self.task_spec.get_input(), ctx)
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/engine/tasks.py\", line 524, in _evaluate_expression
    ctx_view
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/__init__.py\", line 100, in evaluate_recursively
    data[key] = _evaluate_item(data[key], context)
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/__init__.py\", line 89, in _evaluate_item
    return evaluate_recursively(item, context)
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/__init__.py\", line 100, in evaluate_recursively
    data[key] = _evaluate_item(data[key], context)
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/__init__.py\", line 79, in _evaluate_item
    return evaluate(item, context)
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/__init__.py\", line 71, in evaluate
    return evaluator.evaluate(expression, context)
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py\", line 119, in evaluate
    cls).evaluate(trim_expr, data_context)
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py\", line 73, in evaluate
    \", data=%s]\" % (expression, str(e), data_context)
YaqlEvaluationException: Can not evaluate YAQL expression [expression=task(check_process).result.stdout, error=u'stdout', data={}]
",

#2

I suspect you might be missing a .result, try this:

<% task(check_process).result.result.stdout %>

If that doesn’t work, try to echo the task(check_process).result:

    tasks:
      check_process:
        action: core.remote
        input:
          hosts: <% $.hostname %>
          cmd: "ps -eo pid,pcpu,user,args --sort=-%cpu | head -6 | head -c 1000"
        on-error:
          - pagerduty_escalation
        on-success:
          - echo_result
          - post_to_slack

      pagerduty_escalation:
        action: pagerduty.incident.create.rest_v2.simple
        input:
          service_id: "PxxxxxxN"
          title: "Stackstorm Check CPU failed on <% $.hostname %>"
          from_email: "xxxxx@xxxxx.com"
          details: "Stackstorm could not autoremediate high cpu usage on <% $.hostname %>. Alert: <% $.alert_message %>"

      echo_result:
        action: core.echo
        input:
          message: "<% str(task(check_process).result) %>"

      post_to_slack:
        action: slack.post_message
        input:
          channel: "#stackstorm"
          message: "High CPU Usage triggered from: <% $.hostname %>. Here's the processes details:\n<% task(check_process).result.stdout %>"

then you should be able to see what task(check_process).result is in the StackStorm web UI.


(gmnvn) #3

Hi, Thanks for reply!

I tried <% task(check_process).result.result.stdout %> but that didn’t work, got the same error message as invalid YAQL expression:

      "state_info": "Failed to run task [error=Can not evaluate YAQL expression [expression=task(check_process).result.result.stdout, error=u'result', data={}], wf=ops.check_cpu.main, task=post_to_slack]:
Traceback (most recent call last):

As you suggested when i tried echo, i see the stdout is null.

{
  "tasks": [
    {
      "state_info": null,
      "name": "check_process",
      "workflow_name": "ops.check_cpu.main",
      "created_at": "2019-02-22 19:56:03",
      "updated_at": "2019-02-22 19:56:05",
      "workflow_execution_id": "9003c8de-1734-462e-bd36-dd4e4d50d634",
      "state": "SUCCESS",
      "result": {
        "10.146.74.91": {
          "succeeded": true,
          "failed": false,
          "return_code": 0,
          "stderr": "",
          "stdout": "  PID %CPU USER     COMMAND\
 9032 60.1 root     /usr/bin/python2.7 /usr/bin/aws s3 sync /media/ebs/logs s3://dev.canopydata.com/logserver\
17787 23.9 root     /usr/bin/java -Xms1g -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /usr/lib/logstash-6.2.3/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/commons-compiler-3.0.8.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/google-java-format-1.5.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/guava-22.0.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/jackson-annotations-2.9."
        }
      },
      "published": {},
      "input": null,
      "id": "18081169-a7b2-462b-b0d8-35a214287e62"
    },
    {
      "state_info": null,
      "name": "echo_result",
      "workflow_name": "ops.check_cpu.main",
      "created_at": "2019-02-22 19:56:03",
      "updated_at": "2019-02-22 19:56:04",
      "workflow_execution_id": "9003c8de-1734-462e-bd36-dd4e4d50d634",
      "state": "SUCCESS",
      "result": {
        "failed": false,
        "stderr": "",
        "return_code": 0,
        "succeeded": true,
        "stdout": []
      },
      "published": {},
      "input": null,
      "id": "08b8f123-fd10-41b2-be95-185bff49ec4f"
    },
    {
      "state_info": "Failed to run task [error=Can not evaluate YAQL expression [expression=task(check_process).result.stdout, error=u'stdout', data={}], wf=ops.check_cpu.main, task=post_to_slack]:
Traceback (most recent call last):

But the stdout is present in the task: check_process, how should i call that into slack message ?

Another naive question, I’ve added “echo_result” in workflow and tested, should i same module in chains/check_cpu.yaml as well, did that null value came due to no action for that in chain ?

Thanks!


(Lindsay Hill) #4

It’s not result.stdout - the result object is a list of results per host.


(Lindsay Hill) #5

Look at ChatOps w/ Slack not working for an example


(gmnvn) #6

Hi,

The link you mentioned talks about “action aliases” from chatops.

I tried:
workflows/check_cpu.yaml

      post_to_slack:
        action: slack.post_message
        input:
          channel: "#stackstorm"
          message: "Stackstorm got a cpu alert & has got top 5 cpu consuming processes on <% $.hostname %>. Details:\n<% task(check_process).$.hostname.result.stdout %>"

chains/check_cpu.yaml:

     name: "post_to_slack"
     ref: "slack.post_message"
     params:
       channel: "#stackstorm"
       message: "Stackstorm got a cpu alert & has got top 5 cpu consuming processes on {{hostname}}. Details:\n{{ execution.result[host].stdout }}"```

Still no luck. Still failing:
  "state_info": "Failed to run task [error=Can not evaluate YAQL expression [expression=task(check_process).$.hostname.result.stdout, error=No function \"#operator_.\" matches supplied arguments, data={}], wf=ops.check_cpu.main, task=post_to_slack]:

Traceback (most recent call last):


(Lindsay Hill) #7

Yes, that’s for ChatOps, but it’s a similar syntax for what you need. Note that the result object is a list of results per hostname.

Try this example st2cd/st2_get_installed_version.yaml at 2c29083494a84e55892475c7c31de56d07a209af · StackStorm/st2cd · GitHub


(gmnvn) #8

in this instead of <% task(check_process).$.hostname.result.stdout %> , <% task(check_process).result.get($.hostname).stdout %> has worked.

But have to say the output is really ugly in the slack :slightly_smiling_face:

anyway!! thanks a lot for helping out as always.


(Lindsay Hill) #9

Switch to Orquesta, that helps with some of the newline handling. And then just do some filtering of the output data.


(gmnvn) #10

Thanks for the suggestion! Since I’m a learner, would definitely give a try to Orquesta.