Hey @Brian, the realtime collector has been solid for a few days, but ran into a crash this morning during the first collection of the day/pre-market session at 4:01AM EST.
I've included the logs below, but I think the issue MIGHT be due to another cron job I have that runs right at 4AM too. This other cron job queries the usstock-minute bundle and realtime aggregates database in order to generate a universe of stocks to trade for the day. From the logs, it almost looks as though PostgreSQL crash/disconnected because it was being overloaded by the query and the realtime collector? This is just a hunch.
The realtime collector got stuck at this point, and attempting to cancel the active collections via quantrocket realtime cancel --all and restart via quantrocket realtime collect usstock-realtime-polygon didn't work. I saw the Canceling market data request for ... messages in the log, but it never got past this point in order to re-start the collection process. Eventually, restarting the docker container and then restarting the collection process got things running again.
Note that I'm not looking for any specific resolution/response. My only hope in posting this issue, is to provide you as much detail as possible in case this helps with making the realtime collector more resilient. In the meantime, I'm working on a monitor script that will automatically restart the realtime container on any crash alerts from flightlog.
Let me know if you need any additional information.
2021-08-04 04:00:03 EDT| <14>1 2021-08-04T08:00:03Z 3a4b21e6bb8c quantrocket_houston_1 2860 - - 172.18.0.17 - - [04/Aug/2021:08:00:03 +0000] "GET /realtime/databases HTTP/1.1" 200 115 "-" "-"
2021-08-04 04:00:17 EDT| <14>1 2021-08-04T08:00:17Z 3a4b21e6bb8c quantrocket_houston_1 2860 - - 172.18.0.17 - - [04/Aug/2021:08:00:17 +0000] "GET /realtime/databases HTTP/1.1" 200 115 "-" "-"
2021-08-04 04:00:17 EDT| <14>1 2021-08-04T08:00:17Z 3a4b21e6bb8c quantrocket_houston_1 2860 - - 172.18.0.17 - - [04/Aug/2021:08:00:17 +0000] "GET /realtime/databases/usstock-minute-realtime-polygon HTTP/1.1" 308 381 "-" "-"
2021-08-04 04:00:17 EDT| <14>1 2021-08-04T08:00:17Z 3a4b21e6bb8c quantrocket_houston_1 2860 - - 172.18.0.17 - - [04/Aug/2021:08:00:17 +0000] "GET /realtime/databases/usstock-realtime-polygon/aggregates/usstock-minute-realtime-polygon HTTP/1.1" 200 187 "-" "-"
2021-08-04 04:00:17 EDT| <14>1 2021-08-04T08:00:17Z 3a4b21e6bb8c quantrocket_houston_1 2860 - - 172.18.0.17 - - [04/Aug/2021:08:00:17 +0000] "GET /realtime/usstock-minute-realtime-polygon.csv?start_date=2021-08-04&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteVolumeClose&fields=MinuteLowLow&fields=MinuteCloseClose HTTP/1.1" 400 74 "-" "-"
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z 3ccf9da518ab quantrocket_realtime_1 3554 - - Exception in thread notify_worker:
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z 3ccf9da518ab quantrocket_realtime_1 3554 - - Traceback (most recent call last):
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z 3ccf9da518ab quantrocket_realtime_1 3554 - - File "/opt/conda/lib/python3.8/threading.py", line 932, in _bootstrap_inner
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z d5635dcde2a3 quantrocket_flightlog_1 4748 - - 2021-08-04 04:01:02 quantrocket.realtime: ERROR Traceback (most recent call last):
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z 3ccf9da518ab quantrocket_realtime_1 3554 - - self.run()
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z 3ccf9da518ab quantrocket_realtime_1 3554 - - File "/opt/conda/lib/python3.8/threading.py", line 870, in run
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z 3ccf9da518ab quantrocket_realtime_1 3554 - - self._target(*self._args, **self._kwargs)
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z 3ccf9da518ab quantrocket_realtime_1 3554 - - File "sym://qrocket_realtime_collect_polygon_workers_py", line 16, in wrapped
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z 3ccf9da518ab quantrocket_realtime_1 3554 - - File "sym://qrocket_realtime_collect_notify_py", line 55, in run
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z 3ccf9da518ab quantrocket_realtime_1 3554 - - File "sym://qrocket_realtime_collect_notify_py", line 33, in are_listeners
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z 3ccf9da518ab quantrocket_realtime_1 3554 - - psycopg2.OperationalError: server closed the connection unexpectedly
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z 3ccf9da518ab quantrocket_realtime_1 3554 - - This probably means the server terminated abnormally
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z 3ccf9da518ab quantrocket_realtime_1 3554 - - before or while processing the request.
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z 3ccf9da518ab quantrocket_realtime_1 3554 - -
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z d5635dcde2a3 quantrocket_flightlog_1 4748 - - 2021-08-04 04:01:02 quantrocket.realtime: ERROR File "sym://qrocket_realtime_collect_polygon_workers_py", line 16, in wrapped
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z d5635dcde2a3 quantrocket_flightlog_1 4748 - - 2021-08-04 04:01:02 quantrocket.realtime: ERROR File "sym://qrocket_realtime_collect_notify_py", line 55, in run
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z d5635dcde2a3 quantrocket_flightlog_1 4748 - - 2021-08-04 04:01:02 quantrocket.realtime: ERROR File "sym://qrocket_realtime_collect_notify_py", line 33, in are_listeners
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z d5635dcde2a3 quantrocket_flightlog_1 4748 - - 2021-08-04 04:01:02 quantrocket.realtime: ERROR psycopg2.OperationalError: server closed the connection unexpectedly
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z d5635dcde2a3 quantrocket_flightlog_1 4748 - - 2021-08-04 04:01:02 quantrocket.realtime: ERROR This probably means the server terminated abnormally
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z d5635dcde2a3 quantrocket_flightlog_1 4748 - - 2021-08-04 04:01:02 quantrocket.realtime: ERROR before or while processing the request.
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z d5635dcde2a3 quantrocket_flightlog_1 4748 - - 2021-08-04 04:01:02 quantrocket.realtime: ERROR
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z d5635dcde2a3 quantrocket_flightlog_1 4748 - - 2021-08-04 04:01:02 quantrocket.realtime: ERROR
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z 3ccf9da518ab quantrocket_realtime_1 3554 - - ┌──────────────────────────────────────────────────┐
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z 3ccf9da518ab quantrocket_realtime_1 3554 - - │ Polygon market data received: │
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z 3ccf9da518ab quantrocket_realtime_1 3554 - - │ total_ticks unique_tickers │
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z 3ccf9da518ab quantrocket_realtime_1 3554 - - │ received at 08:00 UTC 206 103 │
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z 3ccf9da518ab quantrocket_realtime_1 3554 - - │ active collections 11052.0 │
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z 3ccf9da518ab quantrocket_realtime_1 3554 - - └──────────────────────────────────────────────────┘
2021-08-04 04:01:02 EDT| <11>1 2021-08-04T08:01:02Z 3ccf9da518ab quantrocket_realtime_1 3554 - - Worker exception detected, shutting down
2021-08-04 04:01:32 EDT| <11>1 2021-08-04T08:01:32Z d5635dcde2a3 quantrocket_flightlog_1 4748 - - 2021-08-04 04:01:32 quantrocket.realtime: INFO Exiting Polygon market data collection due to errors
2021-08-04 04:01:33 EDT| <11>1 2021-08-04T08:01:33Z 3ccf9da518ab quantrocket_realtime_1 3554 - - recycling mule after collecting market data