Polygon realtime collection crash

Technical Help
#1

Error:

Exception in thread polygon_websocket_worker:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/opt/conda/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "sym://qrocket_realtime_collect_polygon_workers_py", line 16, in wrapped
  File "sym://qrocket_realtime_collect_polygon_streamconn_py", line 212, in run
AttributeError: type object '_asyncio.Task' has no attribute 'all_tasks'

context:

What we are trying to do

Consume real-time US equity trades from Polygon for a custom minute-bar builder.

  • Universe: 135 US stocks (custom pad-live universe, refreshed daily).
  • Tick DB: pad-live-trades, created via quantrocket realtime create-polygon-tick-db 'pad-live-trades' --universes 'pad-live' --fields 'LastPrice' 'LastSize' 'TradeConditions' (TradeId field omitted — see issue B below).
  • Collection: started daily by cron: quantrocket realtime collect 'pad-live-trades' --until '16:05:00 America/New_York'.
  • Client: a Python process running inside quantrocket-satellite-1 that (a) subscribes to ws://houston/realtime/stream to buffer live trades, and (b) calls quantrocket.realtime.download_market_data_file(...) at startup to backfill a ~1 minute gap.

What is happening

  • Collection starts cleanly and trades flow into the tick DB. quantrocket realtime active shows 135 active collections, and the per-minute "Polygon market data received" summary box shows ~1,500–2,500 ticks/min across ~100 unique tickers.
  • Intermittently the Polygon WebSocket worker thread crashes. After the crash:
    • quantrocket realtime active still reports 135 active collections.
    • The per-minute "Polygon market data received" box stops printing.
    • No new trades land in the tick DB.
    • Recovery requires an OS-level docker restart quantrocket-realtime-1 (uWSGI's mule respawn is not sufficient).
  • This has happens at least every 10 minutes but not regularily

Surrounding log (15:22:06 – 15:22:22 UTC, quantrocket-realtime-1)

15:22:06  Waiting for notifications on channel 'md'
15:22:19  [spooler /var/tmp/uwsgi/spool pid: 31] managing request uwsgi_spoolfile_on_..._214802 ...
15:22:19  [spooler /var/tmp/uwsgi/spool pid: 31] done with task ... after 0 seconds
15:22:20  updating market data collections
15:22:20  Canceling market data request for ABAT STK (sid FIBBG004M1KJN5)
15:22:20  Canceling market data request for ABCL STK (sid FIBBG00LLW2MF2)
... (all 135 sids canceled in this burst) ...
15:22:20  Exception in thread polygon_websocket_worker:
          Traceback ...
          AttributeError: type object '_asyncio.Task' has no attribute 'all_tasks'
15:22:21  recycling mule after collecting market data
15:22:22  OOOPS mule 2 (pid: 40) crippled...trying respawn...
15:22:22  spawned uWSGI mule 2 (pid: 144)

Preceding client-side events we can correlate (from pad.live flightlog):

15:22:06  Monitor process started (new Python client inside quantrocket-satellite-1)
15:22:06  WSConsumer started (135 sids)      # opened connection to ws://houston/realtime/stream
15:22:13  Backfilled 31579 trades from 2026-04-14 15:21:00
           ^ this was quantrocket.realtime.download_market_data_file(
               'pad-live-trades',
               sids=[...135 sids...],
               fields=['LastPrice' | 'LastSize' | 'TradeConditions'],  # three separate calls
               start_date='2026-04-14 15:21:00')

note: we had this other issue with TradeID, not sure if our workaround creates an issue:

Related issue B (already worked around, reporting for completeness)

On the first creation of pad-live-trades with --fields LastPrice LastSize TradeConditions TradeId, inserts began failing with:

psycopg2.errors.NumericValueOutOfRange: value "71679637467727" is out of range for type integer

71679637467727 is a Polygon trade ID (14-digit int64). The tick DB column for TradeId appears to be declared as PostgreSQL integer (max 2,147,483,647) rather than bigint. Workaround: we recreated the DB without the TradeId field.

Environment

  • QuantRocket version: [fill in from quantrocket --version]
  • Container Python: 3.11
  • Polygon Developer subscription (stocks real-time trades)
  • Host OS: Linux 6.8.0-107-generic

question: am I doing something wrong ? (did I miss something from the doc ?)

#2

Investigating...

#3

I have reproduced in another environment to make sure I did not screw up quantrocket environment by installing other components (I may still have done that though...)

#4

This error is happening during the data collection shutdown sequence (which occurs once all tickers have been cancelled or expired). A patch release is now available that fixes that issue as well as the TradeId issue (by switching to a BIGINT field). To get the patched container, please edit your docker-compose.yml, changing:

  realtime:
    image: 'quantrocket/realtime:2.11.0'

to:

  realtime:
    image: 'quantrocket/realtime:2.11.0.patch20260415'

Then deploy the new realtime container:

docker compose up -d realtime
#5

looks good, thanks

#6

Brian, just to let you know: it did not happen during the data collection shutdown; it happens randomly but in general quite fast after registration (I could get between one and 4 seconds of updates).

#7

The line reference in your traceback comes from code that only runs during data collection shutdown, but if it's not an intended shutdown, it could be the result of a different error, in which case that error should produce some kind of log message or traceback. You can also double check that you don't have some other scripts calling quantrocket realtime cancel or initiating data collection with a short expiration value for --until. (If one command says to collect ABC stock until 2 hours from now and another command says to collect it until 20 seconds from now, the shorter one will always win, and data will be canceled in 20 seconds.)