This site is in read only mode. Please continue to browse, but replying, likes, and other actions are disabled for now.

⚠️ We've moved!

Hi there!

To reduce project dependency on 3rd party paid services the StackStorm TSC has decided to move the Q/A from this forum to Github Discussions. This will make user experience better integrated with the native Github flow, as well as the questions closer to the community where they can provide answers.

Use 🔗 Github Discussions to ask your questions.

ChatOps with Hubot and Slack

Chatops
#1

I tried to set up Chatops with Slack and invited the bot to a channel successfully. It is successfully logging and connecting in as you can see below but Hubot (aka friday in my case) does not seem to respond to examples that I have tried from examples I see online. I’ve tried “@friday: help”, “!help”, “st2 list actions” and “!st2 list actions”

When debugging, I see the following:

root@b31433eae1069:~# [Wed Sep 05 2018 13:29:47 GMT+0000 (UTC)] DEBUG Loading adapter slack
[Wed Sep 05 2018 13:29:48 GMT+0000 (UTC)] INFO Logged in as friday of Opus Ops
[Wed Sep 05 2018 13:29:48 GMT+0000 (UTC)] INFO Slack client now connected
[Wed Sep 05 2018 13:29:48 GMT+0000 (UTC)] DEBUG Loading scripts from /root/scripts
[Wed Sep 05 2018 13:29:48 GMT+0000 (UTC)] DEBUG Loading scripts from /root/src/scripts
[Wed Sep 05 2018 13:57:32 GMT+0000 (UTC)] DEBUG Received message: 'friday help' in channel: chatops, from:  <myname>
[Wed Sep 05 2018 13:57:32 GMT+0000 (UTC)] DEBUG No listeners executed; falling back to catch-all
[Wed Sep 05 2018 13:58:08 GMT+0000 (UTC)] DEBUG Received message: '@friday: help' in channel: chatops, from:  <myname>
[Wed Sep 05 2018 13:58:08 GMT+0000 (UTC)] DEBUG No listeners executed; falling back to catch-all
warn: Reconnecting, on attempt 1
[Wed Sep 05 2018 14:39:51 GMT+0000 (UTC)] INFO Logged in as friday of <companyname>
[Wed Sep 05 2018 14:39:51 GMT+0000 (UTC)] INFO Slack client now connected
[Wed Sep 05 2018 17:29:14 GMT+0000 (UTC)] DEBUG Received message: '!help' in channel: chatops, from:  <myname>
[Wed Sep 05 2018 17:29:14 GMT+0000 (UTC)] DEBUG No listeners executed; falling back to catch-all
[Wed Sep 05 2018 17:29:53 GMT+0000 (UTC)] DEBUG Received message: '@friday!help' in channel: chatops, from: <myname>
[Wed Sep 05 2018 17:29:53 GMT+0000 (UTC)] DEBUG No listeners executed; falling back to catch-all
[Wed Sep 05 2018 17:32:50 GMT+0000 (UTC)] DEBUG Received message: 'st2 list actions' in channel: chatops, from:  <myname>
[Wed Sep 05 2018 17:32:50 GMT+0000 (UTC)] DEBUG No listeners executed; falling back to catch-all
[Wed Sep 05 2018 17:32:54 GMT+0000 (UTC)] DEBUG Received message: '!st2 list actions' in channel: chatops, from:  <myname>
[Wed Sep 05 2018 17:32:54 GMT+0000 (UTC)] DEBUG No listeners executed; falling back to catch-all

So beside the fact that I don’t know what I am doing, there are a number of possibilities including I missed a critical setup step and I misconfigured something.

Can anyone point me in the right direction to get this working?

#2

How have you configured your system? Did you install StackStorm using the install script, or is this a manual setup?

What have you configured in st2chatops.env? What does grep -v ^# /opt/stackstorm/chatops/st2chatops.env|grep -v ^$ give you? (Mask out any tokens)

#3

Doh! That’s the most important thing, isn’t it. It’s the standard Docker image (Docker — StackStorm 2.8.1 documentation). The only changes that I’ve made are running through the examples.

here’s the grep:

export NODE_TLS_REJECT_UNAUTHORIZED=0
export EXPRESS_PORT=8081
export HUBOT_LOG_LEVEL=debug
export HUBOT_NAME=friday
export HUBOT_ALIAS='!'
export ST2_API="${ST2_API:-https://${ST2_HOSTNAME}/api}"
export ST2_AUTH_URL="${ST2_AUTH_URL:-https://${ST2_HOSTNAME}/auth}"
export ST2_API_KEY="${ST2_API_KEY}"
export ST2_AUTH_USERNAME="${ST2_AUTH_USERNAME:-st2admin}"
export ST2_AUTH_PASSWORD="${ST2_AUTH_PASSWORD:-testp}"
export ST2_WEBUI_URL=https://${ST2_HOSTNAME}
export HUBOT_ADAPTER=slack
export HUBOT_SLACK_TOKEN=xoxb-1...Z
#4

Looks like you only ran the first part of my grep command, not the bit after the |?

(also, you can use Markdown in this forum, or use the editing options in the text entry box, to make blocks of text fixed-width, with syntax highlighting, etc)

#5

Your logs also don’t look right. Here’s my logs from a working system:

[Tue Sep 04 2018 06:38:39 GMT-0700 (PDT)] DEBUG Loading adapter slack
[Tue Sep 04 2018 06:38:40 GMT-0700 (PDT)] INFO Logged in as st2demo of StackStorm Demo
[Tue Sep 04 2018 06:38:41 GMT-0700 (PDT)] INFO Slack client now connected
[Tue Sep 04 2018 06:38:41 GMT-0700 (PDT)] DEBUG Loading scripts from /opt/stackstorm/chatops/scripts
[Tue Sep 04 2018 06:38:41 GMT-0700 (PDT)] DEBUG Loading scripts from /opt/stackstorm/chatops/src/scripts
[Tue Sep 04 2018 06:38:41 GMT-0700 (PDT)] DEBUG Loading external-scripts from npm packages
[Tue Sep 04 2018 06:38:41 GMT-0700 (PDT)] INFO Using ST2_API_KEY as authentication. Expiry will lead to bot exit.
[Tue Sep 04 2018 06:38:41 GMT-0700 (PDT)] DEBUG Using slack post data handler.
[Tue Sep 04 2018 06:38:41 GMT-0700 (PDT)] DEBUG Parsing help for /opt/stackstorm/chatops/node_modules/hubot-stackstorm/scripts/stackstorm.js
[Tue Sep 04 2018 06:38:41 GMT-0700 (PDT)] INFO Loading commands....
[Tue Sep 04 2018 06:38:41 GMT-0700 (PDT)] DEBUG Parsing help for /opt/stackstorm/chatops/node_modules/hubot-help/src/help.coffee
[Tue Sep 04 2018 06:38:41 GMT-0700 (PDT)] DEBUG Added command: test - Jonny 5 thinks he's alive

Note the paths there for “Loading scripts from…”

#6

Check this part of the README about enabling st2chatops: st2-docker/README.md at master · StackStorm/st2-docker · GitHub

Did you follow those directions?

#7

I did not. What is tje proper way to bind-mount /opt/stackstorm/chatops/st2chatops.env?

The docker-compose.yml file contains:

      - stackstorm-packs-volume:/opt/stackstorm/packs
      - stackstorm-virtualenvs-volume:/opt/stackstorm/virtualenvs
      - stackstorm-configs-volume:/opt/stackstorm/configs
      - stackstorm-log-volume:/var/log
      - ./packs.dev:/opt/stackstorm/packs.dev
      - ./runtime/entrypoint.d:/st2-docker/entrypoint.d
      - ./runtime/st2.d:/st2-docker/st2.d
      - ./conf/stackstorm.env:/st2-docker/env

I don’t understand why there isn’t a volume for /opt/stackstorm/chatops. What is the common place to park this file on the host?

#8

You don’t the entire /opt/stackstorm/chatops volume. That contains all the ChatOps-related packages, not just the environment.

Either add those environment variables the same way you pass other env vars, or bind-mount the file.

#9

I could not get the bind mount to work. So I passed in the variables through the stackstorm.env file instead:

ST2_USER=st2admin
ST2_PASSWORD=<masked>

######################################################################
# Chat service adapter settings
# Slack settings (https://github.com/slackhq/hubot-slack):
#

HUBOT_LOG_LEVEL=debug
HUBOT_NAME=friday
HUBOT_ADAPTER=slack
HUBOT_ALIAS='!'
HUBOT_SLACK_TOKEN=xoxb-<masked>
# Uncomment the following line to force hubot to exit if disconnected from slack.
#HUBOT_SLACK_EXIT_ON_DISCONNECT=1

# I had to add these because the /opt/stackstorm/chatops/st2chatops.env file does not 
# seem to get called
ST2_HOSTNAME="${ST2_HOSTNAME:-localhost}"
ST2_API="${ST2_API:-https://${ST2_HOSTNAME}/api}"
ST2_AUTH_URL="${ST2_AUTH_URL:-https://${ST2_HOSTNAME}/auth}"
ST2_API_KEY="${ST2_API_KEY}"
ST2_AUTH_USERNAME="${ST2_AUTH_USERNAME:-$(ST2_USER)}"
ST2_AUTH_PASSWORD="${ST2_AUTH_PASSWORD:-$(ST2_PASSWORD)}"
ST2_WEBUI_URL=https://${ST2_HOSTNAME}

Sadly, I get the same result when I debug:

[1] 569
root@7b4126265032:/opt/stackstorm/chatops/node_modules# [Fri Sep 07 2018 20:33:29 GMT+0000 (UTC)] DEBUG Loading adapter slack
[Fri Sep 07 2018 20:33:30 GMT+0000 (UTC)] INFO Logged in as friday of Opus Ops
[Fri Sep 07 2018 20:33:30 GMT+0000 (UTC)] INFO Slack client now connected
[Fri Sep 07 2018 20:33:30 GMT+0000 (UTC)] DEBUG Loading scripts from /opt/stackstorm/chatops/node_modules/scripts
[Fri Sep 07 2018 20:33:30 GMT+0000 (UTC)] DEBUG Loading scripts from /opt/stackstorm/chatops/node_modules/src/scripts
[Fri Sep 07 2018 20:33:43 GMT+0000 (UTC)] DEBUG Received message: '!' in channel: chatops, from: MysteryAchievement
[Fri Sep 07 2018 20:33:43 GMT+0000 (UTC)] DEBUG No listeners executed; falling back to catch-all
[Fri Sep 07 2018 20:34:00 GMT+0000 (UTC)] DEBUG Received message: '@friday help' in channel: chatops, from: MysteryAchievement
[Fri Sep 07 2018 20:34:00 GMT+0000 (UTC)] DEBUG No listeners executed; falling back to catch-all

The “loading scripts” messages look better than in my previous post but I’m still not getting responses.

Looking at /var/log/st2/st2chatops.log shows the following:

[Thu Sep 06 2018 17:53:41 GMT+0000 (UTC)] INFO Logged in as friday of Opus Ops
[Thu Sep 06 2018 17:53:41 GMT+0000 (UTC)] INFO Slack client now connected
[Thu Sep 06 2018 17:53:41 GMT+0000 (UTC)] DEBUG Loading scripts from /opt/stackstorm/chatops/scripts
[Thu Sep 06 2018 17:53:41 GMT+0000 (UTC)] DEBUG Loading scripts from /opt/stackstorm/chatops/src/scripts
[Thu Sep 06 2018 17:53:41 GMT+0000 (UTC)] DEBUG Loading external-scripts from npm packages
[Thu Sep 06 2018 17:53:41 GMT+0000 (UTC)] INFO Requesting a token...
[Thu Sep 06 2018 17:53:41 GMT+0000 (UTC)] DEBUG Using slack post data handler.
[Thu Sep 06 2018 17:53:41 GMT+0000 (UTC)] DEBUG Parsing help for /opt/stackstorm/chatops/node_modules/hubot-stackstorm/scripts/stackstorm.js
[Thu Sep 06 2018 17:53:41 GMT+0000 (UTC)] DEBUG Parsing help for /opt/stackstorm/chatops/node_modules/hubot-help/src/help.coffee
[Thu Sep 06 2018 17:53:41 GMT+0000 (UTC)] ERROR Failed to authenticate: Invalid or missing credentials
(node:1818) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): [object Object]
[Thu Sep 06 2018 18:05:08 GMT+0000 (UTC)] DEBUG Loading adapter slack
[Thu Sep 06 2018 18:05:09 GMT+0000 (UTC)] ERROR Error: listen EADDRINUSE 0.0.0.0:8081
    at Object.exports._errnoException (util.js:1020:11)
    at exports._exceptionWithHostPort (util.js:1043:20)
    at Server._listen2 (net.js:1262:14)
    at listen (net.js:1298:10)
    at net.js:1408:9
    at _combinedTickCallback (internal/process/next_tick.js:83:11)
    at process._tickCallback (internal/process/next_tick.js:104:9)
/
#10

You should fix this: ERROR Failed to authenticate: Invalid or missing credentials

Check your authentication. I strongly recommend you do not use ST2_AUTH_USERNAME/ST2_AUTH_PASSWORD, but instead use ST2_API_KEY

#11

I tested what you are saying with this:

ERROR: 401 Client Error: Unauthorized
MESSAGE: Unauthorized - ApiKey with key_hash=e5c264829e154b7ba056561826b3d2d0719b715c5dd00a6426cda763d9f0a9f24024d09a9cb1182829de1b7459a688bfd5daeebb507577e940790fc852d12747 not found. for url: http://127.0.0.1:9101/v1/actions/core.local

but when I try to generate an APIKey I get this:

root:/# st2 auth st2admin -p <password>-t
/opt/stackstorm/st2/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:852:
 InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly
 advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)

That leads to https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings and https://urllib3.readthedocs.io/en/latest/user-guide.html#ssl but neither of those things make sense to me.

#12

That ^^ is not generating an API key. That is an authentication token. See doc.stackstorm.com/authentication.html.

Generate an API key with st2 apikey create -k -m '{"used_by": "ChatOps"}'

Take the resulting key, and set the ST2_API_KEY environment variable.

#13

Lindsay - I apologize for all of the trouble and I really do appreciate your help. So thank you.

st2 apikey create, st2 apikey list and st2 pack list all generate the same error:

MESSAGE: Unauthorized - ApiKey with key_hash=e5<...lots of numbers and letters...>47
not found.
#14

Let’s go right back to the start. Can you run any st2 actions? It sounds like your system is in a bad state.

Tell me exactly how you set up your system, what steps you ran, what you changed from defaults.

#15

I did the following in my /opt directory:

git clone https://github.com/stackstorm/st2-docker
cd st2-docker
make env

I add this to docker-compose.yml in the end of the “env_file:” section:

      - conf/st2chatops.env

That file contains this (I removed comments for brevity):

ST2_HOSTNAME=localhost
NODE_TLS_REJECT_UNAUTHORIZED=0
EXPRESS_PORT=8081
HUBOT_LOG_LEVEL=debug
HUBOT_NAME=friday
HUBOT_ALIAS='!'
ST2_API=https://localhost/api
ST2_AUTH_URL=https://localhost/auth
ST2_STREAM_URL=https://localhost/stream
ST2_API_KEY="${ST2_API_KEY}"
ST2_AUTH_USERNAME=st2admin
ST2_AUTH_PASSWORD=<make env generated password>
ST2_WEBUI_URL=https://localhost
HUBOT_ADAPTER=slack
HUBOT_SLACK_TOKEN=xoxb-<buncha numbers and letters>

Then I copy the file:

cp /opt/my-st2-docker/conf/st2chatops.env /opt/st2-docker/conf/st2chatops.env

I also copy some .sh files:

cp /opt/my-st2-docker/runtime/entrypoint.d/*.sh /opt/st2-docker/runtime/entrypoint.d/

there are three - the first is add-nano.sh which contains:

#!/bin/bash

sudo apt-get -y install nano

the second is enable-chatops.sh which contains:

#!/bin/bash

sudo rm /etc/init/st2chatops.override
#16

I started over from scratch using the steps described above. I was able to generate an API key.

st2 apikey list
+--------------------------+----------+--------------------------+
| id                       | user     | metadata                 |
+--------------------------+----------+--------------------------+
| 5b95e55532c9d600f6dcd9d2 | st2admin | {u'used_by': u'ChatOps'} |
+--------------------------+----------+--------------------------+

Now, if I run

grep -v ^# /opt/stackstorm/chatops/st2chatops.env|grep -v ^$

I get:

export ST2_HOSTNAME="${ST2_HOSTNAME:-localhost}"
export NODE_TLS_REJECT_UNAUTHORIZED=0
export EXPRESS_PORT=8081
export HUBOT_LOG_LEVEL=debug
export HUBOT_NAME=hubot
export HUBOT_ALIAS='!'
export ST2_API="${ST2_API:-https://${ST2_HOSTNAME}/api}"
export ST2_AUTH_URL="${ST2_AUTH_URL:-https://${ST2_HOSTNAME}/auth}"
export ST2_API_KEY="${ST2_API_KEY}"
export ST2_AUTH_USERNAME="${ST2_AUTH_USERNAME:-st2admin}"
export ST2_AUTH_PASSWORD="${ST2_AUTH_PASSWORD:-testp}"
export ST2_WEBUI_URL=https://${ST2_HOSTNAME}

However, my st2chapops.env loads and sets the same variables. So when I run

printenv|grep "ST2_"

I get:

ST2_AUTH_URL=https://localhost/auth
ST2_API=https://localhost/api
ST2_USER=st2admin
ST2_WEBUI_URL=https://localhost
ST2_PASSWORD=<masked>
ST2_API_KEY=<masked>
ST2_STREAM_URL=https://localhost/stream
ST2_HOSTNAME=localhost

and this:

printenv|grep "HUB"

yields:

HUBOT_SLACK_TOKEN=xoxb-<masked>
HUBOT_LOG_LEVEL=debug
HUBOT_ALIAS='!'
HUBOT_NAME=friday
HUBOT_ADAPTER=slack

So I ran chatops-check.sh and received:

Starting the Hubot Self-Check Program
===============================================

Step 1 failed: Hubot is not running on this machine.

    Try restarting it with:

    service st2chatops restart

and I tried to restart and got:

st2chatops start/running, process 876
initctl: Job failed to start
.
.
.
st2chatops is not running.

So I tried running:

/opt/stackstorm/chatops/bin/hubot &

and got the following when I entered “!”, “@friday help” and “st2 --version” into slack:

[1] 1345
root@67e9952a8b7f:/st2-docker/bin# [Mon Sep 10 2018 16:02:17 GMT+0000 (UTC)] DEBUG Loading adapter slack
[Mon Sep 10 2018 16:02:18 GMT+0000 (UTC)] INFO Logged in as friday of Opus Ops
[Mon Sep 10 2018 16:02:18 GMT+0000 (UTC)] INFO Slack client now connected
[Mon Sep 10 2018 16:02:18 GMT+0000 (UTC)] DEBUG Loading scripts from /st2-docker/bin/scripts
[Mon Sep 10 2018 16:02:18 GMT+0000 (UTC)] DEBUG Loading scripts from /st2-docker/bin/src/scripts
[Mon Sep 10 2018 16:02:35 GMT+0000 (UTC)] DEBUG Received message: '!' in channel: chatops, from: MysteryAchievement
[Mon Sep 10 2018 16:02:35 GMT+0000 (UTC)] DEBUG No listeners executed; falling back to catch-all
[Mon Sep 10 2018 16:03:00 GMT+0000 (UTC)] DEBUG Received message: '@friday help' in channel: chatops, from: MysteryAchievement
[Mon Sep 10 2018 16:03:00 GMT+0000 (UTC)] DEBUG No listeners executed; falling back to catch-all
[Mon Sep 10 2018 16:03:18 GMT+0000 (UTC)] DEBUG Received message: 'st2 --version' in channel: chatops, from: MysteryAchievement
[Mon Sep 10 2018 16:03:18 GMT+0000 (UTC)] DEBUG No listeners executed; falling back to catch-all
#17

Change your HUBOT_NAME back to hubot.

You’ve still got some problem going on with your setup, because it does not seem to ever login to StackStorm, it only logs in to Slack. I don’t know what else is misconfigured - this is usually a pretty simple setup once the API key & Slack token is set.

#18

OK, I did that. I even generated a new apikey and removed the old one as well as the removed the friday bot from the channel. I added a new Hubot from scratch and generated a new UBOT_SLACK_TOKEN and updated my st2chatops.env file with the token and new apikey.

st2 apikey list
+--------------------------+----------+--------------------------+
| id                       | user     | metadata                 |
+--------------------------+----------+--------------------------+
| 5b95e55532c9d600f6dcd9d2 | st2admin | {u'used_by': u'ChatOps'} |
| 5b96dc8b619f6101249be99c | st2admin | {u'used_by': u'ChatOps'} |
+--------------------------+----------+--------------------------+

I removed the old APIKey:

st2 apikey delete 5b95e55532c9d600f6dcd9d2
Resource with id "5b95e55532c9d600f6dcd9d2" has been successfully deleted.
root@b7911abb832e:/# st2 apikey list
ERROR: 401 Client Error: Unauthorized
MESSAGE: Unauthorized - ApiKey with key_hash=e229440b72a51118063dbaf02ab4b14e9e35e82abe761a96b6643c6179f4e655871fb95fa3eebcf29661777340ab0aba4a46a92f4ea29714a942cedb7ce615ec not found. for url: http://127.0.0.1:9101/v1/apikeys

I took the error as I good sign that I was authorized by the old APIKey but no longer.

After a quick restart, we show the new ID:

 st2 apikey list
+--------------------------+----------+--------------------------+
| id                       | user     | metadata                 |
+--------------------------+----------+--------------------------+
| 5b96dc8b619f6101249be99c | st2admin | {u'used_by': u'ChatOps'} |
+--------------------------+----------+--------------------------+

but chatops won’t start:

st2ctl status|grep "chat"
st2chatops is not running.

/var/log/st2/st2chatops.log is no help - it just has a bunch of these:

Must provide HUBOT_ADAPTER ENV variable to start.
See README.md for more information

but the adapter appears to be set:

printenv|grep "HUBOT_ADAPTER"
HUBOT_ADAPTER=slack

Some questions:

  • How can I debug the Stackstorm login issue?
  • is inclusion of the st2chatops.env file as part of the docker-compose.yml a proper way to initialize the env variables?
  • does the existence of the /opt/stackstorm/chatops/st2chatops.env files interfering in some way with the variables that I set?
#19

Using printenv doesn’t validate what st2chatops has for its environment. It only tells you what your environment has.

#20

You also need to figure out why you’re getting 401 errors from your system when doing things like st2 key list

Something more fundamental is going on here.