ChatOps with Hubot and Slack

docker

#1

I tried to set up Chatops with Slack and invited the bot to a channel successfully. It is successfully logging and connecting in as you can see below but Hubot (aka friday in my case) does not seem to respond to examples that I have tried from examples I see online. I’ve tried “@friday: help”, “!help”, “st2 list actions” and “!st2 list actions”

When debugging, I see the following:

root@b31433eae1069:~# [Wed Sep 05 2018 13:29:47 GMT+0000 (UTC)] DEBUG Loading adapter slack
[Wed Sep 05 2018 13:29:48 GMT+0000 (UTC)] INFO Logged in as friday of Opus Ops
[Wed Sep 05 2018 13:29:48 GMT+0000 (UTC)] INFO Slack client now connected
[Wed Sep 05 2018 13:29:48 GMT+0000 (UTC)] DEBUG Loading scripts from /root/scripts
[Wed Sep 05 2018 13:29:48 GMT+0000 (UTC)] DEBUG Loading scripts from /root/src/scripts
[Wed Sep 05 2018 13:57:32 GMT+0000 (UTC)] DEBUG Received message: 'friday help' in channel: chatops, from:  <myname>
[Wed Sep 05 2018 13:57:32 GMT+0000 (UTC)] DEBUG No listeners executed; falling back to catch-all
[Wed Sep 05 2018 13:58:08 GMT+0000 (UTC)] DEBUG Received message: '@friday: help' in channel: chatops, from:  <myname>
[Wed Sep 05 2018 13:58:08 GMT+0000 (UTC)] DEBUG No listeners executed; falling back to catch-all
warn: Reconnecting, on attempt 1
[Wed Sep 05 2018 14:39:51 GMT+0000 (UTC)] INFO Logged in as friday of <companyname>
[Wed Sep 05 2018 14:39:51 GMT+0000 (UTC)] INFO Slack client now connected
[Wed Sep 05 2018 17:29:14 GMT+0000 (UTC)] DEBUG Received message: '!help' in channel: chatops, from:  <myname>
[Wed Sep 05 2018 17:29:14 GMT+0000 (UTC)] DEBUG No listeners executed; falling back to catch-all
[Wed Sep 05 2018 17:29:53 GMT+0000 (UTC)] DEBUG Received message: '@friday!help' in channel: chatops, from: <myname>
[Wed Sep 05 2018 17:29:53 GMT+0000 (UTC)] DEBUG No listeners executed; falling back to catch-all
[Wed Sep 05 2018 17:32:50 GMT+0000 (UTC)] DEBUG Received message: 'st2 list actions' in channel: chatops, from:  <myname>
[Wed Sep 05 2018 17:32:50 GMT+0000 (UTC)] DEBUG No listeners executed; falling back to catch-all
[Wed Sep 05 2018 17:32:54 GMT+0000 (UTC)] DEBUG Received message: '!st2 list actions' in channel: chatops, from:  <myname>
[Wed Sep 05 2018 17:32:54 GMT+0000 (UTC)] DEBUG No listeners executed; falling back to catch-all

So beside the fact that I don’t know what I am doing, there are a number of possibilities including I missed a critical setup step and I misconfigured something.

Can anyone point me in the right direction to get this working?


(Lindsay Hill) #2

How have you configured your system? Did you install StackStorm using the install script, or is this a manual setup?

What have you configured in st2chatops.env? What does grep -v ^# /opt/stackstorm/chatops/st2chatops.env|grep -v ^$ give you? (Mask out any tokens)


#3

Doh! That’s the most important thing, isn’t it. It’s the standard Docker image (Docker — StackStorm 2.8.1 documentation). The only changes that I’ve made are running through the examples.

here’s the grep:

export NODE_TLS_REJECT_UNAUTHORIZED=0
export EXPRESS_PORT=8081
export HUBOT_LOG_LEVEL=debug
export HUBOT_NAME=friday
export HUBOT_ALIAS='!'
export ST2_API="${ST2_API:-https://${ST2_HOSTNAME}/api}"
export ST2_AUTH_URL="${ST2_AUTH_URL:-https://${ST2_HOSTNAME}/auth}"
export ST2_API_KEY="${ST2_API_KEY}"
export ST2_AUTH_USERNAME="${ST2_AUTH_USERNAME:-st2admin}"
export ST2_AUTH_PASSWORD="${ST2_AUTH_PASSWORD:-testp}"
export ST2_WEBUI_URL=https://${ST2_HOSTNAME}
export HUBOT_ADAPTER=slack
export HUBOT_SLACK_TOKEN=xoxb-1...Z

(Lindsay Hill) #4

Looks like you only ran the first part of my grep command, not the bit after the |?

(also, you can use Markdown in this forum, or use the editing options in the text entry box, to make blocks of text fixed-width, with syntax highlighting, etc)


(Lindsay Hill) #5

Your logs also don’t look right. Here’s my logs from a working system:

[Tue Sep 04 2018 06:38:39 GMT-0700 (PDT)] DEBUG Loading adapter slack
[Tue Sep 04 2018 06:38:40 GMT-0700 (PDT)] INFO Logged in as st2demo of StackStorm Demo
[Tue Sep 04 2018 06:38:41 GMT-0700 (PDT)] INFO Slack client now connected
[Tue Sep 04 2018 06:38:41 GMT-0700 (PDT)] DEBUG Loading scripts from /opt/stackstorm/chatops/scripts
[Tue Sep 04 2018 06:38:41 GMT-0700 (PDT)] DEBUG Loading scripts from /opt/stackstorm/chatops/src/scripts
[Tue Sep 04 2018 06:38:41 GMT-0700 (PDT)] DEBUG Loading external-scripts from npm packages
[Tue Sep 04 2018 06:38:41 GMT-0700 (PDT)] INFO Using ST2_API_KEY as authentication. Expiry will lead to bot exit.
[Tue Sep 04 2018 06:38:41 GMT-0700 (PDT)] DEBUG Using slack post data handler.
[Tue Sep 04 2018 06:38:41 GMT-0700 (PDT)] DEBUG Parsing help for /opt/stackstorm/chatops/node_modules/hubot-stackstorm/scripts/stackstorm.js
[Tue Sep 04 2018 06:38:41 GMT-0700 (PDT)] INFO Loading commands....
[Tue Sep 04 2018 06:38:41 GMT-0700 (PDT)] DEBUG Parsing help for /opt/stackstorm/chatops/node_modules/hubot-help/src/help.coffee
[Tue Sep 04 2018 06:38:41 GMT-0700 (PDT)] DEBUG Added command: test - Jonny 5 thinks he's alive

Note the paths there for “Loading scripts from…”


(Lindsay Hill) #6

Check this part of the README about enabling st2chatops: st2-docker/README.md at master · StackStorm/st2-docker · GitHub

Did you follow those directions?


#7

I did not. What is tje proper way to bind-mount /opt/stackstorm/chatops/st2chatops.env?

The docker-compose.yml file contains:

      - stackstorm-packs-volume:/opt/stackstorm/packs
      - stackstorm-virtualenvs-volume:/opt/stackstorm/virtualenvs
      - stackstorm-configs-volume:/opt/stackstorm/configs
      - stackstorm-log-volume:/var/log
      - ./packs.dev:/opt/stackstorm/packs.dev
      - ./runtime/entrypoint.d:/st2-docker/entrypoint.d
      - ./runtime/st2.d:/st2-docker/st2.d
      - ./conf/stackstorm.env:/st2-docker/env

I don’t understand why there isn’t a volume for /opt/stackstorm/chatops. What is the common place to park this file on the host?


(Lindsay Hill) #8

You don’t the entire /opt/stackstorm/chatops volume. That contains all the ChatOps-related packages, not just the environment.

Either add those environment variables the same way you pass other env vars, or bind-mount the file.


#9

I could not get the bind mount to work. So I passed in the variables through the stackstorm.env file instead:

ST2_USER=st2admin
ST2_PASSWORD=<masked>

######################################################################
# Chat service adapter settings
# Slack settings (https://github.com/slackhq/hubot-slack):
#

HUBOT_LOG_LEVEL=debug
HUBOT_NAME=friday
HUBOT_ADAPTER=slack
HUBOT_ALIAS='!'
HUBOT_SLACK_TOKEN=xoxb-<masked>
# Uncomment the following line to force hubot to exit if disconnected from slack.
#HUBOT_SLACK_EXIT_ON_DISCONNECT=1

# I had to add these because the /opt/stackstorm/chatops/st2chatops.env file does not 
# seem to get called
ST2_HOSTNAME="${ST2_HOSTNAME:-localhost}"
ST2_API="${ST2_API:-https://${ST2_HOSTNAME}/api}"
ST2_AUTH_URL="${ST2_AUTH_URL:-https://${ST2_HOSTNAME}/auth}"
ST2_API_KEY="${ST2_API_KEY}"
ST2_AUTH_USERNAME="${ST2_AUTH_USERNAME:-$(ST2_USER)}"
ST2_AUTH_PASSWORD="${ST2_AUTH_PASSWORD:-$(ST2_PASSWORD)}"
ST2_WEBUI_URL=https://${ST2_HOSTNAME}

Sadly, I get the same result when I debug:

[1] 569
root@7b4126265032:/opt/stackstorm/chatops/node_modules# [Fri Sep 07 2018 20:33:29 GMT+0000 (UTC)] DEBUG Loading adapter slack
[Fri Sep 07 2018 20:33:30 GMT+0000 (UTC)] INFO Logged in as friday of Opus Ops
[Fri Sep 07 2018 20:33:30 GMT+0000 (UTC)] INFO Slack client now connected
[Fri Sep 07 2018 20:33:30 GMT+0000 (UTC)] DEBUG Loading scripts from /opt/stackstorm/chatops/node_modules/scripts
[Fri Sep 07 2018 20:33:30 GMT+0000 (UTC)] DEBUG Loading scripts from /opt/stackstorm/chatops/node_modules/src/scripts
[Fri Sep 07 2018 20:33:43 GMT+0000 (UTC)] DEBUG Received message: '!' in channel: chatops, from: MysteryAchievement
[Fri Sep 07 2018 20:33:43 GMT+0000 (UTC)] DEBUG No listeners executed; falling back to catch-all
[Fri Sep 07 2018 20:34:00 GMT+0000 (UTC)] DEBUG Received message: '@friday help' in channel: chatops, from: MysteryAchievement
[Fri Sep 07 2018 20:34:00 GMT+0000 (UTC)] DEBUG No listeners executed; falling back to catch-all

The “loading scripts” messages look better than in my previous post but I’m still not getting responses.

Looking at /var/log/st2/st2chatops.log shows the following:

[Thu Sep 06 2018 17:53:41 GMT+0000 (UTC)] INFO Logged in as friday of Opus Ops
[Thu Sep 06 2018 17:53:41 GMT+0000 (UTC)] INFO Slack client now connected
[Thu Sep 06 2018 17:53:41 GMT+0000 (UTC)] DEBUG Loading scripts from /opt/stackstorm/chatops/scripts
[Thu Sep 06 2018 17:53:41 GMT+0000 (UTC)] DEBUG Loading scripts from /opt/stackstorm/chatops/src/scripts
[Thu Sep 06 2018 17:53:41 GMT+0000 (UTC)] DEBUG Loading external-scripts from npm packages
[Thu Sep 06 2018 17:53:41 GMT+0000 (UTC)] INFO Requesting a token...
[Thu Sep 06 2018 17:53:41 GMT+0000 (UTC)] DEBUG Using slack post data handler.
[Thu Sep 06 2018 17:53:41 GMT+0000 (UTC)] DEBUG Parsing help for /opt/stackstorm/chatops/node_modules/hubot-stackstorm/scripts/stackstorm.js
[Thu Sep 06 2018 17:53:41 GMT+0000 (UTC)] DEBUG Parsing help for /opt/stackstorm/chatops/node_modules/hubot-help/src/help.coffee
[Thu Sep 06 2018 17:53:41 GMT+0000 (UTC)] ERROR Failed to authenticate: Invalid or missing credentials
(node:1818) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): [object Object]
[Thu Sep 06 2018 18:05:08 GMT+0000 (UTC)] DEBUG Loading adapter slack
[Thu Sep 06 2018 18:05:09 GMT+0000 (UTC)] ERROR Error: listen EADDRINUSE 0.0.0.0:8081
    at Object.exports._errnoException (util.js:1020:11)
    at exports._exceptionWithHostPort (util.js:1043:20)
    at Server._listen2 (net.js:1262:14)
    at listen (net.js:1298:10)
    at net.js:1408:9
    at _combinedTickCallback (internal/process/next_tick.js:83:11)
    at process._tickCallback (internal/process/next_tick.js:104:9)
/

(Lindsay Hill) #10

You should fix this: ERROR Failed to authenticate: Invalid or missing credentials

Check your authentication. I strongly recommend you do not use ST2_AUTH_USERNAME/ST2_AUTH_PASSWORD, but instead use ST2_API_KEY


#11

I tested what you are saying with this:

ERROR: 401 Client Error: Unauthorized
MESSAGE: Unauthorized - ApiKey with key_hash=e5c264829e154b7ba056561826b3d2d0719b715c5dd00a6426cda763d9f0a9f24024d09a9cb1182829de1b7459a688bfd5daeebb507577e940790fc852d12747 not found. for url: http://127.0.0.1:9101/v1/actions/core.local

but when I try to generate an APIKey I get this:

root:/# st2 auth st2admin -p <password>-t
/opt/stackstorm/st2/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:852:
 InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly
 advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)

That leads to https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings and https://urllib3.readthedocs.io/en/latest/user-guide.html#ssl but neither of those things make sense to me.


(Lindsay Hill) #12

That ^^ is not generating an API key. That is an authentication token. See doc.stackstorm.com/authentication.html.

Generate an API key with st2 apikey create -k -m '{"used_by": "ChatOps"}'

Take the resulting key, and set the ST2_API_KEY environment variable.


#13

Lindsay - I apologize for all of the trouble and I really do appreciate your help. So thank you.

st2 apikey create, st2 apikey list and st2 pack list all generate the same error:

MESSAGE: Unauthorized - ApiKey with key_hash=e5<...lots of numbers and letters...>47 
not found. 

(Lindsay Hill) #14

Let’s go right back to the start. Can you run any st2 actions? It sounds like your system is in a bad state.

Tell me exactly how you set up your system, what steps you ran, what you changed from defaults.


#15

I did the following in my /opt directory:

git clone https://github.com/stackstorm/st2-docker
cd st2-docker
make env

I add this to docker-compose.yml in the end of the “env_file:” section:

      - conf/st2chatops.env

That file contains this (I removed comments for brevity):

ST2_HOSTNAME=localhost
NODE_TLS_REJECT_UNAUTHORIZED=0
EXPRESS_PORT=8081
HUBOT_LOG_LEVEL=debug
HUBOT_NAME=friday
HUBOT_ALIAS='!'
ST2_API=https://localhost/api
ST2_AUTH_URL=https://localhost/auth
ST2_STREAM_URL=https://localhost/stream
ST2_API_KEY="${ST2_API_KEY}"
ST2_AUTH_USERNAME=st2admin
ST2_AUTH_PASSWORD=<make env generated password>
ST2_WEBUI_URL=https://localhost
HUBOT_ADAPTER=slack
HUBOT_SLACK_TOKEN=xoxb-<buncha numbers and letters>

Then I copy the file:

cp /opt/my-st2-docker/conf/st2chatops.env /opt/st2-docker/conf/st2chatops.env

I also copy some .sh files:

cp /opt/my-st2-docker/runtime/entrypoint.d/*.sh /opt/st2-docker/runtime/entrypoint.d/

there are three - the first is add-nano.sh which contains:

#!/bin/bash

sudo apt-get -y install nano

the second is enable-chatops.sh which contains:

#!/bin/bash

sudo rm /etc/init/st2chatops.override

#16

I started over from scratch using the steps described above. I was able to generate an API key.

st2 apikey list
+--------------------------+----------+--------------------------+
| id                       | user     | metadata                 |
+--------------------------+----------+--------------------------+
| 5b95e55532c9d600f6dcd9d2 | st2admin | {u'used_by': u'ChatOps'} |
+--------------------------+----------+--------------------------+

Now, if I run

grep -v ^# /opt/stackstorm/chatops/st2chatops.env|grep -v ^$

I get:

export ST2_HOSTNAME="${ST2_HOSTNAME:-localhost}"
export NODE_TLS_REJECT_UNAUTHORIZED=0
export EXPRESS_PORT=8081
export HUBOT_LOG_LEVEL=debug
export HUBOT_NAME=hubot
export HUBOT_ALIAS='!'
export ST2_API="${ST2_API:-https://${ST2_HOSTNAME}/api}"
export ST2_AUTH_URL="${ST2_AUTH_URL:-https://${ST2_HOSTNAME}/auth}"
export ST2_API_KEY="${ST2_API_KEY}"
export ST2_AUTH_USERNAME="${ST2_AUTH_USERNAME:-st2admin}"
export ST2_AUTH_PASSWORD="${ST2_AUTH_PASSWORD:-testp}"
export ST2_WEBUI_URL=https://${ST2_HOSTNAME}

However, my st2chapops.env loads and sets the same variables. So when I run

printenv|grep "ST2_"

I get:

ST2_AUTH_URL=https://localhost/auth
ST2_API=https://localhost/api
ST2_USER=st2admin
ST2_WEBUI_URL=https://localhost
ST2_PASSWORD=<masked>
ST2_API_KEY=<masked>
ST2_STREAM_URL=https://localhost/stream
ST2_HOSTNAME=localhost

and this:

printenv|grep "HUB"

yields:

HUBOT_SLACK_TOKEN=xoxb-<masked>
HUBOT_LOG_LEVEL=debug
HUBOT_ALIAS='!'
HUBOT_NAME=friday
HUBOT_ADAPTER=slack

So I ran chatops-check.sh and received:

Starting the Hubot Self-Check Program
===============================================

Step 1 failed: Hubot is not running on this machine.

    Try restarting it with:

    service st2chatops restart

and I tried to restart and got:

st2chatops start/running, process 876
initctl: Job failed to start
.
.
.
st2chatops is not running.

So I tried running:

/opt/stackstorm/chatops/bin/hubot &

and got the following when I entered “!”, “@friday help” and “st2 --version” into slack:

[1] 1345
root@67e9952a8b7f:/st2-docker/bin# [Mon Sep 10 2018 16:02:17 GMT+0000 (UTC)] DEBUG Loading adapter slack
[Mon Sep 10 2018 16:02:18 GMT+0000 (UTC)] INFO Logged in as friday of Opus Ops
[Mon Sep 10 2018 16:02:18 GMT+0000 (UTC)] INFO Slack client now connected
[Mon Sep 10 2018 16:02:18 GMT+0000 (UTC)] DEBUG Loading scripts from /st2-docker/bin/scripts
[Mon Sep 10 2018 16:02:18 GMT+0000 (UTC)] DEBUG Loading scripts from /st2-docker/bin/src/scripts
[Mon Sep 10 2018 16:02:35 GMT+0000 (UTC)] DEBUG Received message: '!' in channel: chatops, from: MysteryAchievement
[Mon Sep 10 2018 16:02:35 GMT+0000 (UTC)] DEBUG No listeners executed; falling back to catch-all
[Mon Sep 10 2018 16:03:00 GMT+0000 (UTC)] DEBUG Received message: '@friday help' in channel: chatops, from: MysteryAchievement
[Mon Sep 10 2018 16:03:00 GMT+0000 (UTC)] DEBUG No listeners executed; falling back to catch-all
[Mon Sep 10 2018 16:03:18 GMT+0000 (UTC)] DEBUG Received message: 'st2 --version' in channel: chatops, from: MysteryAchievement
[Mon Sep 10 2018 16:03:18 GMT+0000 (UTC)] DEBUG No listeners executed; falling back to catch-all

(Lindsay Hill) #17

Change your HUBOT_NAME back to hubot.

You’ve still got some problem going on with your setup, because it does not seem to ever login to StackStorm, it only logs in to Slack. I don’t know what else is misconfigured - this is usually a pretty simple setup once the API key & Slack token is set.


#18

OK, I did that. I even generated a new apikey and removed the old one as well as the removed the friday bot from the channel. I added a new Hubot from scratch and generated a new UBOT_SLACK_TOKEN and updated my st2chatops.env file with the token and new apikey.

st2 apikey list
+--------------------------+----------+--------------------------+
| id                       | user     | metadata                 |
+--------------------------+----------+--------------------------+
| 5b95e55532c9d600f6dcd9d2 | st2admin | {u'used_by': u'ChatOps'} |
| 5b96dc8b619f6101249be99c | st2admin | {u'used_by': u'ChatOps'} |
+--------------------------+----------+--------------------------+

I removed the old APIKey:

st2 apikey delete 5b95e55532c9d600f6dcd9d2
Resource with id "5b95e55532c9d600f6dcd9d2" has been successfully deleted.
root@b7911abb832e:/# st2 apikey list
ERROR: 401 Client Error: Unauthorized
MESSAGE: Unauthorized - ApiKey with key_hash=e229440b72a51118063dbaf02ab4b14e9e35e82abe761a96b6643c6179f4e655871fb95fa3eebcf29661777340ab0aba4a46a92f4ea29714a942cedb7ce615ec not found. for url: http://127.0.0.1:9101/v1/apikeys

I took the error as I good sign that I was authorized by the old APIKey but no longer.

After a quick restart, we show the new ID:

 st2 apikey list
+--------------------------+----------+--------------------------+
| id                       | user     | metadata                 |
+--------------------------+----------+--------------------------+
| 5b96dc8b619f6101249be99c | st2admin | {u'used_by': u'ChatOps'} |
+--------------------------+----------+--------------------------+

but chatops won’t start:

st2ctl status|grep "chat"
st2chatops is not running.

/var/log/st2/st2chatops.log is no help - it just has a bunch of these:

Must provide HUBOT_ADAPTER ENV variable to start.
See README.md for more information

but the adapter appears to be set:

printenv|grep "HUBOT_ADAPTER"
HUBOT_ADAPTER=slack

Some questions:

  • How can I debug the Stackstorm login issue?
  • is inclusion of the st2chatops.env file as part of the docker-compose.yml a proper way to initialize the env variables?
  • does the existence of the /opt/stackstorm/chatops/st2chatops.env files interfering in some way with the variables that I set?

(Lindsay Hill) #19

Using printenv doesn’t validate what st2chatops has for its environment. It only tells you what your environment has.


(Lindsay Hill) #20

You also need to figure out why you’re getting 401 errors from your system when doing things like st2 key list

Something more fundamental is going on here.