Realtime Collector Crash Recovery

charles · July 20, 2021, 2:12pm

When the realtime collector crashes, is there any sort of recovery process that occurs once it's restarted? For example, say the collector crashes and is out of service for 5 minutes; will it replay/sync back up those missed minutes and then resume realtime collection, or do those missed minutes need to be manually patched somehow?

On this topic, is there a good way to monitor the realtime collector and restart it automatically when it crashes? I've been experiences crashes similar (but not necessarily the same) to below about once a week.

Also, any idea why the specific crash below happened? The collection process has to be manually restarted to get things working again. Any help is appreciated; thanks!

# Realtime Collector Crash Below
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z d5635dcde2a3 quantrocket_flightlog_1 5693 - - 2021-07-20 09:36:32 quantrocket.realtime: ERROR Traceback (most recent call last):
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z 3889728f128e quantrocket_realtime_1 80571 - - Exception in thread polygon_websocket_worker:
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z 3889728f128e quantrocket_realtime_1 80571 - - Traceback (most recent call last):
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z 3889728f128e quantrocket_realtime_1 80571 - -   File "/opt/conda/lib/python3.8/threading.py", line 932, in _bootstrap_inner
2021-07-20 09:36:32 EDT| <14>1 2021-07-20T13:36:32Z 3a4b21e6bb8c quantrocket_houston_1 6700 - - 172.18.0.8 - - [20/Jul/2021:13:36:32 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z 3889728f128e quantrocket_realtime_1 80571 - -     self.run()
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z 3889728f128e quantrocket_realtime_1 80571 - -   File "/opt/conda/lib/python3.8/threading.py", line 870, in run
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z 3889728f128e quantrocket_realtime_1 80571 - -     self._target(*self._args, **self._kwargs)
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z 3889728f128e quantrocket_realtime_1 80571 - -   File "sym://qrocket_realtime_collect_polygon_workers_py", line 16, in wrapped
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z 3889728f128e quantrocket_realtime_1 80571 - -   File "sym://qrocket_realtime_collect_polygon_streamconn_py", line 206, in run
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z 3889728f128e quantrocket_realtime_1 80571 - -   File "/opt/conda/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z 3889728f128e quantrocket_realtime_1 80571 - -     return future.result()
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z 3889728f128e quantrocket_realtime_1 80571 - -   File "sym://qrocket_realtime_collect_polygon_streamconn_py", line 25, in wrapped
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z 3889728f128e quantrocket_realtime_1 80571 - -   File "sym://qrocket_realtime_collect_polygon_streamconn_py", line 184, in _run_until_no_subscriptions
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z 3889728f128e quantrocket_realtime_1 80571 - -   File "sym://qrocket_realtime_collect_polygon_streamconn_py", line 104, in _consume_msg
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z 3889728f128e quantrocket_realtime_1 80571 - -   File "sym://qrocket_realtime_collect_polygon_streamconn_py", line 231, in _dispatch
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z 3889728f128e quantrocket_realtime_1 80571 - -   File "sym://qrocket_realtime_collect_polygon_market_data_py", line 168, in _market_event_handler
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z 3889728f128e quantrocket_realtime_1 80571 - -   File "sym://qrocket_realtime_collect_polygon_tracker_py", line 47, in mark_tick_received
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z 3889728f128e quantrocket_realtime_1 80571 - - AttributeError: 'NoneType' object has no attribute 'security'
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z d5635dcde2a3 quantrocket_flightlog_1 5693 - - 2021-07-20 09:36:32 quantrocket.realtime: ERROR   File "sym://qrocket_realtime_collect_polygon_streamconn_py", line 25, in wrapped
2021-07-20 09:36:32 EDT| <14>1 2021-07-20T13:36:32Z 3a4b21e6bb8c quantrocket_houston_1 6700 - - 172.18.0.8 - - [20/Jul/2021:13:36:32 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z 3889728f128e quantrocket_realtime_1 80571 - - Worker exception detected, shutting down
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z d5635dcde2a3 quantrocket_flightlog_1 5693 - - 2021-07-20 09:36:32 quantrocket.realtime: ERROR   File "sym://qrocket_realtime_collect_polygon_streamconn_py", line 184, in _run_until_no_subscriptions
2021-07-20 09:36:32 EDT| <14>1 2021-07-20T13:36:32Z 3a4b21e6bb8c quantrocket_houston_1 6700 - - 172.18.0.8 - - [20/Jul/2021:13:36:32 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z d5635dcde2a3 quantrocket_flightlog_1 5693 - - 2021-07-20 09:36:32 quantrocket.realtime: ERROR   File "sym://qrocket_realtime_collect_polygon_streamconn_py", line 104, in _consume_msg
2021-07-20 09:36:32 EDT| <14>1 2021-07-20T13:36:32Z 3a4b21e6bb8c quantrocket_houston_1 6700 - - 172.18.0.8 - - [20/Jul/2021:13:36:32 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z d5635dcde2a3 quantrocket_flightlog_1 5693 - - 2021-07-20 09:36:32 quantrocket.realtime: ERROR   File "sym://qrocket_realtime_collect_polygon_streamconn_py", line 231, in _dispatch
2021-07-20 09:36:32 EDT| <14>1 2021-07-20T13:36:32Z 3a4b21e6bb8c quantrocket_houston_1 6700 - - 172.18.0.8 - - [20/Jul/2021:13:36:32 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z d5635dcde2a3 quantrocket_flightlog_1 5693 - - 2021-07-20 09:36:32 quantrocket.realtime: ERROR   File "sym://qrocket_realtime_collect_polygon_market_data_py", line 168, in _market_event_handler
2021-07-20 09:36:32 EDT| <14>1 2021-07-20T13:36:32Z 3a4b21e6bb8c quantrocket_houston_1 6700 - - 172.18.0.8 - - [20/Jul/2021:13:36:32 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z d5635dcde2a3 quantrocket_flightlog_1 5693 - - 2021-07-20 09:36:32 quantrocket.realtime: ERROR   File "sym://qrocket_realtime_collect_polygon_tracker_py", line 47, in mark_tick_received
2021-07-20 09:36:32 EDT| <14>1 2021-07-20T13:36:32Z 3a4b21e6bb8c quantrocket_houston_1 6700 - - 172.18.0.8 - - [20/Jul/2021:13:36:32 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z d5635dcde2a3 quantrocket_flightlog_1 5693 - - 2021-07-20 09:36:32 quantrocket.realtime: ERROR AttributeError: 'NoneType' object has no attribute 'security'
2021-07-20 09:36:32 EDT| <14>1 2021-07-20T13:36:32Z 3a4b21e6bb8c quantrocket_houston_1 6700 - - 172.18.0.8 - - [20/Jul/2021:13:36:32 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z d5635dcde2a3 quantrocket_flightlog_1 5693 - - 2021-07-20 09:36:32 quantrocket.realtime: ERROR 
2021-07-20 09:36:32 EDT| <14>1 2021-07-20T13:36:32Z 3a4b21e6bb8c quantrocket_houston_1 6700 - - 172.18.0.8 - - [20/Jul/2021:13:36:32 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z d5635dcde2a3 quantrocket_flightlog_1 5693 - - 2021-07-20 09:36:32 quantrocket.realtime: ERROR Traceback (most recent call last):
2021-07-20 09:36:32 EDT| <14>1 2021-07-20T13:36:32Z 3a4b21e6bb8c quantrocket_houston_1 6700 - - 172.18.0.8 - - [20/Jul/2021:13:36:32 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z 00bb8948199a quantrocket_account_1 4554 - - waiting until ECB's next expected 4PM CET update to collect exchange rates
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z d5635dcde2a3 quantrocket_flightlog_1 5693 - - 2021-07-20 09:36:32 quantrocket.realtime: ERROR   File "sym://qrocket_realtime_collect_polygon_workers_py", line 16, in wrapped
2021-07-20 09:36:32 EDT| <14>1 2021-07-20T13:36:32Z 3a4b21e6bb8c quantrocket_houston_1 6700 - - 172.18.0.8 - - [20/Jul/2021:13:36:32 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z d5635dcde2a3 quantrocket_flightlog_1 5693 - - 2021-07-20 09:36:32 quantrocket.realtime: ERROR   File "sym://qrocket_realtime_collect_polygon_streamconn_py", line 206, in run
2021-07-20 09:36:32 EDT| <14>1 2021-07-20T13:36:32Z 3a4b21e6bb8c quantrocket_houston_1 6700 - - 172.18.0.8 - - [20/Jul/2021:13:36:32 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z d5635dcde2a3 quantrocket_flightlog_1 5693 - - 2021-07-20 09:36:32 quantrocket.realtime: ERROR   File "/opt/conda/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
2021-07-20 09:36:32 EDT| <14>1 2021-07-20T13:36:32Z 3a4b21e6bb8c quantrocket_houston_1 6700 - - 172.18.0.8 - - [20/Jul/2021:13:36:32 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z d5635dcde2a3 quantrocket_flightlog_1 5693 - - 2021-07-20 09:36:32 quantrocket.realtime: ERROR     return future.result()
2021-07-20 09:36:32 EDT| <14>1 2021-07-20T13:36:32Z 3a4b21e6bb8c quantrocket_houston_1 6700 - - 172.18.0.8 - - [20/Jul/2021:13:36:32 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z d5635dcde2a3 quantrocket_flightlog_1 5693 - - 2021-07-20 09:36:32 quantrocket.realtime: ERROR   File "sym://qrocket_realtime_collect_polygon_streamconn_py", line 25, in wrapped
2021-07-20 09:36:32 EDT| <14>1 2021-07-20T13:36:32Z 3a4b21e6bb8c quantrocket_houston_1 6700 - - 172.18.0.8 - - [20/Jul/2021:13:36:32 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
2021-07-20 09:36:32 EDT| <14>1 2021-07-20T13:36:32Z 3a4b21e6bb8c quantrocket_houston_1 6700 - - 172.18.0.8 - - [20/Jul/2021:13:36:32 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z d5635dcde2a3 quantrocket_flightlog_1 5693 - - 2021-07-20 09:36:32 quantrocket.realtime: ERROR   File "sym://qrocket_realtime_collect_polygon_streamconn_py", line 184, in _run_until_no_subscriptions
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z d5635dcde2a3 quantrocket_flightlog_1 5693 - - 2021-07-20 09:36:32 quantrocket.realtime: ERROR   File "sym://qrocket_realtime_collect_polygon_streamconn_py", line 104, in _consume_msg
2021-07-20 09:36:32 EDT| <14>1 2021-07-20T13:36:32Z 3a4b21e6bb8c quantrocket_houston_1 6700 - - 172.18.0.8 - - [20/Jul/2021:13:36:32 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z d5635dcde2a3 quantrocket_flightlog_1 5693 - - 2021-07-20 09:36:32 quantrocket.realtime: ERROR   File "sym://qrocket_realtime_collect_polygon_streamconn_py", line 231, in _dispatch
2021-07-20 09:36:32 EDT| <14>1 2021-07-20T13:36:32Z 3a4b21e6bb8c quantrocket_houston_1 6700 - - 172.18.0.8 - - [20/Jul/2021:13:36:32 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z d5635dcde2a3 quantrocket_flightlog_1 5693 - - 2021-07-20 09:36:32 quantrocket.realtime: ERROR   File "sym://qrocket_realtime_collect_polygon_market_data_py", line 168, in _market_event_handler
2021-07-20 09:36:32 EDT| <14>1 2021-07-20T13:36:32Z 3a4b21e6bb8c quantrocket_houston_1 6700 - - 172.18.0.8 - - [20/Jul/2021:13:36:32 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z d5635dcde2a3 quantrocket_flightlog_1 5693 - - 2021-07-20 09:36:32 quantrocket.realtime: ERROR   File "sym://qrocket_realtime_collect_polygon_tracker_py", line 47, in mark_tick_received
2021-07-20 09:36:32 EDT| <14>1 2021-07-20T13:36:32Z 3a4b21e6bb8c quantrocket_houston_1 6700 - - 172.18.0.8 - - [20/Jul/2021:13:36:32 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z d5635dcde2a3 quantrocket_flightlog_1 5693 - - 2021-07-20 09:36:32 quantrocket.realtime: ERROR AttributeError: 'NoneType' object has no attribute 'security'
2021-07-20 09:36:32 EDT| <14>1 2021-07-20T13:36:32Z 3a4b21e6bb8c quantrocket_houston_1 6700 - - 172.18.0.8 - - [20/Jul/2021:13:36:32 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z d5635dcde2a3 quantrocket_flightlog_1 5693 - - 2021-07-20 09:36:32 quantrocket.realtime: ERROR 
2021-07-20 09:36:32 EDT| <14>1 2021-07-20T13:36:32Z 3a4b21e6bb8c quantrocket_houston_1 6700 - - 172.18.0.8 - - [20/Jul/2021:13:36:32 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
2021-07-20 09:36:32 EDT| <11>1 2021-07-20T13:36:32Z d5635dcde2a3 quantrocket_flightlog_1 5693 - - 2021-07-20 09:36:32 quantrocket.realtime: INFO Exiting Polygon market data collection due to errors

Brian · July 21, 2021, 12:04pm

The implication of the traceback is that a tick was received for a symbol that wasn’t requested (or perhaps a tick without a symbol was received). We probably just need to allow for that possibility by ignoring such ticks. If you have different tracebacks from this one, you should post those too.

(There’s no automatic recovery for an unhandled exception.)

charles · July 21, 2021, 3:45pm

@Brian I went back and looked at prior crashes in detail, and the only other unique traceback is the below which looks like a network connection issue, and unrelated to the received ticks issue.

In regards to automatic recovery, I was specifically looking for feedback on the following:

When the realtime collector is restarted, does it automatically fill in missed minutes since it last ran? I don't think so, but want to make sure I'm not missing something. I'm currently do this via a secondary history database that I manually fill with missed minutes, and then use both the realtime and history database in my strategy, but it would be amazing if the collector could be started with an optional switch to fill in missing minutes since last run.
In the event of a crash, what are your thoughts on automatic restarting of the service. Even if we eliminate every single bug, issues are still bound to happen, such as a dropped network connection, etc... In that case, would it maybe make sense to have a secondary script that reviews realtime logs every 5 minutes and automatically restarts the service if it details a crash in the logs? Have you seen any other clients do anything cool in regards to keeping the realtime collector up and running always?

Other unique traceback:

2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - - reconnecting to Polygon websocket after error: code = 1006 (connection closed abnormally [internal]), no reason
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - - Exception in thread polygon_websocket_worker:
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - - Traceback (most recent call last):
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -   File "/opt/conda/lib/python3.8/threading.py", line 932, in _bootstrap_inner
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -     self.run()
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -   File "/opt/conda/lib/python3.8/threading.py", line 870, in run
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -     self._target(*self._args, **self._kwargs)
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -   File "sym://qrocket_realtime_collect_polygon_workers_py", line 16, in wrapped
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -   File "sym://qrocket_realtime_collect_polygon_streamconn_py", line 206, in run
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -   File "/opt/conda/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -     return future.result()
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -   File "sym://qrocket_realtime_collect_polygon_streamconn_py", line 25, in wrapped
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -   File "sym://qrocket_realtime_collect_polygon_streamconn_py", line 179, in _run_until_no_subscriptions
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -   File "sym://qrocket_realtime_collect_polygon_streamconn_py", line 57, in _start_ws
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -   File "sym://qrocket_realtime_collect_polygon_streamconn_py", line 62, in connect
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -   File "/opt/conda/lib/python3.8/site-packages/websockets/client.py", line 535, in __await_impl__
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -     transport, protocol = await self._create_connection()
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -   File "/opt/conda/lib/python3.8/asyncio/base_events.py", line 986, in create_connection
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -     infos = await self._ensure_resolved(
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -   File "/opt/conda/lib/python3.8/asyncio/base_events.py", line 1365, in _ensure_resolved
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -     return await loop.getaddrinfo(host, port, family=family, type=type,
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -   File "/opt/conda/lib/python3.8/asyncio/base_events.py", line 825, in getaddrinfo
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -     return await self.run_in_executor(
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -   File "/opt/conda/lib/python3.8/concurrent/futures/thread.py", line 57, in run
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -     result = self.fn(*self.args, **self.kwargs)
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -   File "/opt/conda/lib/python3.8/socket.py", line 918, in getaddrinfo
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - -     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
2021-07-13 09:19:10 EDT| <11>1 2021-07-13T13:19:10Z 3889728f128e quantrocket_realtime_1 80571 - - socket.gaierror: [Errno -2] Name or service not known

Brian · July 28, 2021, 1:42pm

You can try pulling quantrocket/realtime:latest which will ignore ticks for unknown symbols.

charles · August 2, 2021, 2:22pm

Hey Brian, just wanted to let you know that this def appears to have fixed the issue I was having. This morning I was stuck in a loop where I couldn't get the realtime service to start... Basically, it would start up, collect a single minute of bars, and crash with the error log below.

Decided it was time to try this update you made for me, and after pulling the latest docker image and restarting, I was able to get the realtime service running correctly. To be clear, I had tried restarting the realtime service several times before resorting to the upgrade, so this fix def appears to have been the solution.

Just wanted to let you know for confirmation. Thanks brother.