Passing output to slack from workflow

Hello,

I’m exploring stackstorm and getting used to it. I’m trying to automate to get top 5 cpu process when we get a cpu alert. I achieved that by following disk remediation example. However I’m facing issue with posting the command output directly to slack. Kindly requesting your help.

Here’s my workflow(actions/workflows/check_cpu.yaml):

---
version: '2.0'
name: ops.check_cpu

workflows:
  main:
    input:
      - hostname
      - event_id
      - alert_message
      - raw_payload

    tasks:
      check_process:
        action: core.remote
        input:
          hosts: <% $.hostname %>
          cmd: "ps -eo pid,pcpu,user,args --sort=-%cpu | head -6 | head -c 1000"
        on-error:
          - pagerduty_escalation
        on-success:
          - post_to_slack

      pagerduty_escalation:
        action: pagerduty.incident.create.rest_v2.simple
        input:
          service_id: "PxxxxxxN"
          title: "Stackstorm Check CPU failed on <% $.hostname %>"
          from_email: "[email protected]"
          details: "Stackstorm could not autoremediate high cpu usage on <% $.hostname %>. Alert: <% $.alert_message %>"

      post_to_slack:
        action: slack.post_message
        input:
          channel: "#stackstorm"
          message: "High CPU Usage triggered from: <% $.hostname %>. Here's the processes details:\n<% task(check_process).result.stdout %>"

Here’s the action chain(actions/chains/check_cpu.yaml):

---
  chain:
    -
      name: "check_process"
      ref: "core.remote"
      params:
        hosts: "{{hostname}}"
        cmd: "ps -eo pid,pcpu,user,args --sort=-%cpu | head -6"
      on-success: "post_to_slack"
      on-failure: "pagerduty_escalation"
    -
      name: "pagerduty_escalation"
      ref: "pagerduty.incident.create.rest_v2.simple"
      params:
        service_id: "PxxxxxN"
        title: "Stackstorm Check CPU failed on {{hostname}}"
        from_email: "[email protected]"
        details: "Stackstorm could not autoremediate high cpu usage on {{hostname}}. Alert: {{alert_message}}"
    -
      name: "post_to_slack"
      ref: "slack.post_message"
      params:
        channel: "#stackstorm"
        message: "Stackstorm got a cpu alert & has got top 5 cpu consuming processes on {{hostname}}. Details:\n{{ check_process.stdout }}"
  default: "check_process"

Here’s the Error I’m seeing in the UI console:

{
  "tasks": [
    {
      "state_info": null,
      "name": "check_process",
      "workflow_name": "ops.check_cpu.main",
      "created_at": "2019-02-22 19:30:53",
      "updated_at": "2019-02-22 19:30:54",
      "workflow_execution_id": "47529098-87e4-49b8-beaa-ce320ab9f432",
      "state": "SUCCESS",
      "result": {
        "10.146.74.91": {
          "succeeded": true,
          "failed": false,
          "return_code": 0,
          "stderr": "",
          "stdout": "  PID %CPU USER     COMMAND\
 9032 61.1 root     /usr/bin/python2.7 /usr/bin/aws s3 sync /media/ebs/logs s3://dev.canopydata.com/logserver\
17787 23.9 root     /usr/bin/java -Xms1g -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /usr/lib/logstash-6.2.3/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/commons-compiler-3.0.8.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/google-java-format-1.5.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/guava-22.0.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/jackson-annotations-2.9."
        }
      },
      "published": {},
      "input": null,
      "id": "bc149571-1d49-4b68-9320-d87bd668fe1d"
    },
    {
      "state_info": "Failed to run task [error=Can not evaluate YAQL expression [expression=task(check_process).result.stdout, error=u'stdout', data={}], wf=ops.check_cpu.main, task=post_to_slack]:
Traceback (most recent call last):
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/engine/task_handler.py\", line 63, in run_task
    task.run()
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/osprofiler/profiler.py\", line 159, in wrapper
    result = f(*args, **kwargs)
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/engine/tasks.py\", line 374, in run
    self._run_new()
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/osprofiler/profiler.py\", line 159, in wrapper
    result = f(*args, **kwargs)
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/engine/tasks.py\", line 403, in _run_new
    self._schedule_actions()
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/engine/tasks.py\", line 470, in _schedule_actions
    input_dict = self._get_action_input()
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/osprofiler/profiler.py\", line 159, in wrapper
    result = f(*args, **kwargs)
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/engine/tasks.py\", line 500, in _get_action_input
    input_dict = self._evaluate_expression(self.task_spec.get_input(), ctx)
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/engine/tasks.py\", line 524, in _evaluate_expression
    ctx_view
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/__init__.py\", line 100, in evaluate_recursively
    data[key] = _evaluate_item(data[key], context)
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/__init__.py\", line 89, in _evaluate_item
    return evaluate_recursively(item, context)
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/__init__.py\", line 100, in evaluate_recursively
    data[key] = _evaluate_item(data[key], context)
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/__init__.py\", line 79, in _evaluate_item
    return evaluate(item, context)
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/__init__.py\", line 71, in evaluate
    return evaluator.evaluate(expression, context)
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py\", line 119, in evaluate
    cls).evaluate(trim_expr, data_context)
  File \"/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py\", line 73, in evaluate
    \", data=%s]\" % (expression, str(e), data_context)
YaqlEvaluationException: Can not evaluate YAQL expression [expression=task(check_process).result.stdout, error=u'stdout', data={}]
",

I suspect you might be missing a .result, try this:

<% task(check_process).result.result.stdout %>

If that doesn’t work, try to echo the task(check_process).result:

    tasks:
      check_process:
        action: core.remote
        input:
          hosts: <% $.hostname %>
          cmd: "ps -eo pid,pcpu,user,args --sort=-%cpu | head -6 | head -c 1000"
        on-error:
          - pagerduty_escalation
        on-success:
          - echo_result
          - post_to_slack

      pagerduty_escalation:
        action: pagerduty.incident.create.rest_v2.simple
        input:
          service_id: "PxxxxxxN"
          title: "Stackstorm Check CPU failed on <% $.hostname %>"
          from_email: "[email protected]"
          details: "Stackstorm could not autoremediate high cpu usage on <% $.hostname %>. Alert: <% $.alert_message %>"

      echo_result:
        action: core.echo
        input:
          message: "<% str(task(check_process).result) %>"

      post_to_slack:
        action: slack.post_message
        input:
          channel: "#stackstorm"
          message: "High CPU Usage triggered from: <% $.hostname %>. Here's the processes details:\n<% task(check_process).result.stdout %>"

then you should be able to see what task(check_process).result is in the StackStorm web UI.

Hi, Thanks for reply!

I tried <% task(check_process).result.result.stdout %> but that didn’t work, got the same error message as invalid YAQL expression:

      "state_info": "Failed to run task [error=Can not evaluate YAQL expression [expression=task(check_process).result.result.stdout, error=u'result', data={}], wf=ops.check_cpu.main, task=post_to_slack]:
Traceback (most recent call last):

As you suggested when i tried echo, i see the stdout is null.

{
  "tasks": [
    {
      "state_info": null,
      "name": "check_process",
      "workflow_name": "ops.check_cpu.main",
      "created_at": "2019-02-22 19:56:03",
      "updated_at": "2019-02-22 19:56:05",
      "workflow_execution_id": "9003c8de-1734-462e-bd36-dd4e4d50d634",
      "state": "SUCCESS",
      "result": {
        "10.146.74.91": {
          "succeeded": true,
          "failed": false,
          "return_code": 0,
          "stderr": "",
          "stdout": "  PID %CPU USER     COMMAND\
 9032 60.1 root     /usr/bin/python2.7 /usr/bin/aws s3 sync /media/ebs/logs s3://dev.canopydata.com/logserver\
17787 23.9 root     /usr/bin/java -Xms1g -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /usr/lib/logstash-6.2.3/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/commons-compiler-3.0.8.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/google-java-format-1.5.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/guava-22.0.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/lib/logstash-6.2.3/logstash-core/lib/jars/jackson-annotations-2.9."
        }
      },
      "published": {},
      "input": null,
      "id": "18081169-a7b2-462b-b0d8-35a214287e62"
    },
    {
      "state_info": null,
      "name": "echo_result",
      "workflow_name": "ops.check_cpu.main",
      "created_at": "2019-02-22 19:56:03",
      "updated_at": "2019-02-22 19:56:04",
      "workflow_execution_id": "9003c8de-1734-462e-bd36-dd4e4d50d634",
      "state": "SUCCESS",
      "result": {
        "failed": false,
        "stderr": "",
        "return_code": 0,
        "succeeded": true,
        "stdout": []
      },
      "published": {},
      "input": null,
      "id": "08b8f123-fd10-41b2-be95-185bff49ec4f"
    },
    {
      "state_info": "Failed to run task [error=Can not evaluate YAQL expression [expression=task(check_process).result.stdout, error=u'stdout', data={}], wf=ops.check_cpu.main, task=post_to_slack]:
Traceback (most recent call last):

But the stdout is present in the task: check_process, how should i call that into slack message ?

Another naive question, I’ve added “echo_result” in workflow and tested, should i same module in chains/check_cpu.yaml as well, did that null value came due to no action for that in chain ?

Thanks!

It’s not result.stdout - the result object is a list of results per host.

Look at ChatOps w/ Slack not working for an example

Hi,

The link you mentioned talks about “action aliases” from chatops.

I tried:
workflows/check_cpu.yaml

      post_to_slack:
        action: slack.post_message
        input:
          channel: "#stackstorm"
          message: "Stackstorm got a cpu alert & has got top 5 cpu consuming processes on <% $.hostname %>. Details:\n<% task(check_process).$.hostname.result.stdout %>"

chains/check_cpu.yaml:

     name: "post_to_slack"
     ref: "slack.post_message"
     params:
       channel: "#stackstorm"
       message: "Stackstorm got a cpu alert & has got top 5 cpu consuming processes on {{hostname}}. Details:\n{{ execution.result[host].stdout }}"```

Still no luck. Still failing:

  "state_info": "Failed to run task [error=Can not evaluate YAQL expression [expression=task(check_process).$.hostname.result.stdout, error=No function \"#operator_.\" matches supplied arguments, data={}], wf=ops.check_cpu.main, task=post_to_slack]:

Traceback (most recent call last):

Yes, that’s for ChatOps, but it’s a similar syntax for what you need. Note that the result object is a list of results per hostname.

Try this example st2cd/st2_get_installed_version.yaml at 2c29083494a84e55892475c7c31de56d07a209af · StackStorm/st2cd · GitHub

in this instead of <% task(check_process).$.hostname.result.stdout %> , <% task(check_process).result.get($.hostname).stdout %> has worked.

But have to say the output is really ugly in the slack

anyway!! thanks a lot for helping out as always.

Switch to Orquesta, that helps with some of the newline handling. And then just do some filtering of the output data.

Thanks for the suggestion! Since I’m a learner, would definitely give a try to Orquesta.

@lhill @blag
I have a query regarding the output formatting in orquesta.

I would like to post a message as “Got the cja-id: 0000000”

In order to do that, I tried following the following possibilities, but all of them failed with “mapping values are not allowed” error:

publish: msg="Got the cja-id :<% task(get_cjaid).result.get(ctx(logserver)).stdout %>"
publish: msg="Got the cja-id : <% task(get_cjaid).result.get(ctx(logserver)).stdout %>"
publish: msg="Got the cja-id\: <% task(get_cjaid).result.get(ctx(logserver)).stdout %>"
publish: msg="Got the cja-id\\: <% task(get_cjaid).result.get(ctx(logserver)).stdout %>"
publish: msg="Got the cja-id \\: <% task(get_cjaid).result.get(ctx(logserver)).stdout %>"

And the error:

{
  "output": null,
  "errors": [
    {
      "message": "mapping values are not allowed here
  in \"<string>\", line 14, column 40:
     ...  publish: msg=\"Got the cja-id \\\\: <% task(get_cjaid).result.get( ... 
                                         ^"
    }
  ]
}

{
  "output": null,
  "errors": [
    {
      "message": "mapping values are not allowed here
  in \"<string>\", line 14, column 40:
     ...  publish: msg=\"Got the cja-id \\\\: <% task(get_cjaid).result.get( ... 
                                         ^"
    }
  ]
}

I’ve taken this as reference:

Requesting your help in how to format the message.

I would start by making your code look more like the example. Move the :

Hi, do you mean that, we should not use : Sorry I didn’t get your message.

As shown in the example, I written my task… here’s the full snippet.

  get_cjaid:
    action: core.remote hosts=<% ctx(logserver) %> cmd=<% ctx(grep_cjaid) %>
    next:
      - when: <% succeeded() %>
        publish: msg="Got the cja-id: <% task(get_cjaid).result.get(ctx(logserver)).stdout %>", cjaid=<% task(get_cjaid).result.get(ctx(logserver)).stdout %>
        do: post_to_slack, check_in_db
      - when: <% failed() %>
        publish: msg="Unable to get the cja id due to error <% result() %>"
        do: pagerduty_escalation

But still getting the same error.

This line doesn’t look right. Why are you doing <% task(get_cjaid).result.get(ctx(logserver)).stdout %> twice in one line? Is that intentional?

Hi,

I wanted to post a message to slack as well as declare one for a variable.

Got the cja-id: 0000
cja-id = 0000

PS: On the side note, even for the error function, getting same mapping error. Here there’s no clumsy variable declaration like the previous one.

      - when: <% failed() %>
        publish: msg="Unable to get the cja id due to error: <% result() %>"
        do: pagerduty_escalation

I would strip it back to the simplest possible thing you can - just publish the value you’re extracting, nothing else around it. Get that working, then add it more surrounding details, text messages, etc.