2018/07/09 21:46:14 [error] 132#132: *85 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.255.0.2, server: , request: "POST /auth/tokens HTTP/1.1", upstream: "http://127.0.0.1:9100/tokens", host

docker
swarm

(uday) #1

2018/07/09 21:46:14 [error] 132#132: *85 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.255.0.2, server: , request: “POST /auth/tokens HTTP/1.1”, upstream: “http://127.0.0.1:9100/tokens”, host


(Lindsay Hill) #2

Perhaps you could post just a tiny bit of context about what you’re doing here?

For example, the actions you are taking that lead to that message, information about your system configuration, if this has ever worked, what you last changed before it stopped working, etc.

A random log message on its own is not very helpful


(uday) #3

I have set up swarm cluster and am able to open console, but when i pass username and password am seeing above error.

I have 2 nodes, one is master and another is worker. from both nodes, am seeing same issue.

have modified time outs from 90 to 600 also , still no luck.


(Lindsay Hill) #4

I have set up swarm cluster and am able to open console,

Which console - the web UI? What happens if you try to authenticate via CLI?

What services are running on each of those nodes? Have you checked that all services are running? Since you’re seeing a timeout connecting to http://127.0.0.1:9100/tokens, have you checked that the st2auth service is listening on that IP + port?

I have modified time outs from 90 to 600 also , still no luck.

Which timeout, for which service, did you modify? Extending a timeout from 90s to 600s is probably not going to help, unless your systems have exceptionally high latency.


(uday) #5

Yes —SIT2auth is running
tcp 0 0 127.0.0.1:9100 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:55442 127.0.0.1:9100 ESTABLISHED -

Changed below time outs in si2.conf

location /api
proxy_read_timeout 3600;
proxy_connect_timeout 3600;

location /auth
proxy_read_timeout 3600;
proxy_connect_timeout 3600;


(Warren) #6

Thanks for sharing that st2auth is running. However, it’s still unclear which host it’s running on, and whether 10.255.0.2 can access 127.0.0.1:9100. Perhaps st2auth is not running on that host?

I’m with Lindsay… increasing the timeout from 90s to 600s probably won’t solve anything here.

How did you spin up your swarm cluster?


(uday) #7

its is running on maste node, where stacstorm image and all other images there…

Created docker-compose.yml file and added all images and ports and env configs and ran using docker stack command


(uday) #8