Houston not responding?

glenn1 · May 20, 2024, 4:17pm

I have noticed these errors come up randomly and last for a few hours or until I restart the container. Any ideas?

2024-05-20 16:10:21 quantrocket.countdown: ERROR Traceback (most recent call last):
2024-05-20 16:10:21 quantrocket.countdown: ERROR File "/opt/conda/lib/python3.9/site-packages/urllib3/connectionpool.py", line 449, in _make_request
2024-05-20 16:10:21 quantrocket.countdown: ERROR six.raise_from(e, None)
2024-05-20 16:10:21 quantrocket.countdown: ERROR File "", line 3, in raise_from
2024-05-20 16:10:21 quantrocket.countdown: ERROR File "/opt/conda/lib/python3.9/site-packages/urllib3/connectionpool.py", line 444, in _make_request
2024-05-20 16:10:21 quantrocket.countdown: ERROR httplib_response = conn.getresponse()
2024-05-20 16:10:21 quantrocket.countdown: ERROR File "/opt/conda/lib/python3.9/http/client.py", line 1377, in getresponse
2024-05-20 16:10:21 quantrocket.countdown: ERROR response.begin()
2024-05-20 16:10:21 quantrocket.countdown: ERROR File "/opt/conda/lib/python3.9/http/client.py", line 320, in begin
2024-05-20 16:10:21 quantrocket.countdown: ERROR version, status, reason = self._read_status()
2024-05-20 16:10:21 quantrocket.countdown: ERROR File "/opt/conda/lib/python3.9/http/client.py", line 281, in _read_status
2024-05-20 16:10:21 quantrocket.countdown: ERROR line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
2024-05-20 16:10:21 quantrocket.countdown: ERROR File "/opt/conda/lib/python3.9/socket.py", line 704, in readinto
2024-05-20 16:10:21 quantrocket.countdown: ERROR return self._sock.recv_into(b)
2024-05-20 16:10:21 quantrocket.countdown: ERROR socket.timeout: timed out
2024-05-20 16:10:21 quantrocket.countdown: ERROR
2024-05-20 16:10:21 quantrocket.countdown: ERROR During handling of the above exception, another exception occurred:
2024-05-20 16:10:21 quantrocket.countdown: ERROR
2024-05-20 16:10:21 quantrocket.countdown: ERROR Traceback (most recent call last):
2024-05-20 16:10:21 quantrocket.countdown: ERROR File "/opt/conda/lib/python3.9/site-packages/requests/adapters.py", line 440, in send
2024-05-20 16:10:21 quantrocket.countdown: ERROR resp = conn.urlopen(
2024-05-20 16:10:21 quantrocket.countdown: ERROR File "/opt/conda/lib/python3.9/site-packages/urllib3/connectionpool.py", line 785, in urlopen
2024-05-20 16:10:21 quantrocket.countdown: ERROR retries = retries.increment(
2024-05-20 16:10:21 quantrocket.countdown: ERROR File "/opt/conda/lib/python3.9/site-packages/urllib3/util/retry.py", line 550, in increment
2024-05-20 16:10:21 quantrocket.countdown: ERROR raise six.reraise(type(error), error, _stacktrace)
2024-05-20 16:10:21 quantrocket.countdown: ERROR File "/opt/conda/lib/python3.9/site-packages/urllib3/packages/six.py", line 770, in reraise
2024-05-20 16:10:21 quantrocket.countdown: ERROR raise value
2024-05-20 16:10:21 quantrocket.countdown: ERROR File "/opt/conda/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
2024-05-20 16:10:21 quantrocket.countdown: ERROR httplib_response = self._make_request(
2024-05-20 16:10:21 quantrocket.countdown: ERROR File "/opt/conda/lib/python3.9/site-packages/urllib3/connectionpool.py", line 451, in _make_request
2024-05-20 16:10:21 quantrocket.countdown: ERROR self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
2024-05-20 16:10:21 quantrocket.countdown: ERROR File "/opt/conda/lib/python3.9/site-packages/urllib3/connectionpool.py", line 340, in _raise_timeout
2024-05-20 16:10:21 quantrocket.countdown: ERROR raise ReadTimeoutError(
2024-05-20 16:10:21 quantrocket.countdown: ERROR urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='houston', port=80): Read timed out. (read timeout=1800)
2024-05-20 16:10:22 quantrocket.countdown: ERROR
2024-05-20 16:10:22 quantrocket.countdown: ERROR During handling of the above exception, another exception occurred:
2024-05-20 16:10:22 quantrocket.countdown: ERROR
2024-05-20 16:10:22 quantrocket.countdown: ERROR Traceback (most recent call last):
2024-05-20 16:10:22 quantrocket.countdown: ERROR File "/opt/conda/lib/python3.9/site-packages/quantrocket/_cli/commands.py", line 97, in main
2024-05-20 16:10:22 quantrocket.countdown: ERROR result, exit_code = func(**args)
2024-05-20 16:10:22 quantrocket.countdown: ERROR File "/opt/conda/lib/python3.9/site-packages/quantrocket/moonshot.py", line 738, in _cli_trade
2024-05-20 16:10:22 quantrocket.countdown: ERROR return json_to_cli(trade, *args, **kwargs)
2024-05-20 16:10:22 quantrocket.countdown: ERROR File "/opt/conda/lib/python3.9/site-packages/quantrocket/_cli/utils/output.py", line 37, in json_to_cli
2024-05-20 16:10:22 quantrocket.countdown: ERROR json_response = func(args, **kwargs)
2024-05-20 16:10:22 quantrocket.countdown: ERROR File "/opt/conda/lib/python3.9/site-packages/quantrocket/moonshot.py", line 726, in trade
2024-05-20 16:10:22 quantrocket.countdown: ERROR response = houston.get("/moonshot/orders.{0}".format(output), params=params, timeout=6030)
2024-05-20 16:10:22 quantrocket.countdown: ERROR File "/opt/conda/lib/python3.9/site-packages/requests/sessions.py", line 542, in get
2024-05-20 16:10:22 quantrocket.countdown: ERROR return self.request('GET', url, **kwargs)
2024-05-20 16:10:22 quantrocket.countdown: ERROR File "/opt/conda/lib/python3.9/site-packages/quantrocket/houston.py", line 183, in request
2024-05-20 16:10:22 quantrocket.countdown: ERROR return super(Houston, self).request(
2024-05-20 16:10:22 quantrocket.countdown: ERROR File "/opt/conda/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
2024-05-20 16:10:22 quantrocket.countdown: ERROR resp = self.send(prep, **send_kwargs)
2024-05-20 16:10:22 quantrocket.countdown: ERROR File "/opt/conda/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
2024-05-20 16:10:22 quantrocket.countdown: ERROR r = adapter.send(request, **kwargs)
2024-05-20 16:10:22 quantrocket.countdown: ERROR File "/opt/conda/lib/python3.9/site-packages/requests/adapters.py", line 532, in send
2024-05-20 16:10:22 quantrocket.countdown: ERROR raise ReadTimeout(e, request=request)
2024-05-20 16:10:22 quantrocket.countdown: ERROR requests.exceptions.ReadTimeout: HTTPConnectionPool(host='houston', port=80): Read timed out. (read timeout=1800)

Brian · May 21, 2024, 1:03pm

This error means that you ran the quantrocket moonshot trade command and it didn’t complete and return a response within 30 minutes, which is the timeout length for that command. This doesn’t tell you the underlying reason why the command didn’t return in a timely fashion. Possible examples of why you might see a timeout include: an overtaxed system that becomes slow; a backlog of trade requests that results in no workers being available to handle the new trade request; a remote API call that hangs; etc. Restarting the container clears out any stuck/slow/backlogged requests, which is why you would then see the error go away. I would try to run docker stats or check your system resources while this is happening to see if the server is under heavier load, and go from there.