Polygon realtime collection crash

Technical Help
#1

Error:

Exception in thread polygon_websocket_worker:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/opt/conda/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "sym://qrocket_realtime_collect_polygon_workers_py", line 16, in wrapped
  File "sym://qrocket_realtime_collect_polygon_streamconn_py", line 212, in run
AttributeError: type object '_asyncio.Task' has no attribute 'all_tasks'

context:

What we are trying to do

Consume real-time US equity trades from Polygon for a custom minute-bar builder.

  • Universe: 135 US stocks (custom pad-live universe, refreshed daily).
  • Tick DB: pad-live-trades, created via quantrocket realtime create-polygon-tick-db 'pad-live-trades' --universes 'pad-live' --fields 'LastPrice' 'LastSize' 'TradeConditions' (TradeId field omitted — see issue B below).
  • Collection: started daily by cron: quantrocket realtime collect 'pad-live-trades' --until '16:05:00 America/New_York'.
  • Client: a Python process running inside quantrocket-satellite-1 that (a) subscribes to ws://houston/realtime/stream to buffer live trades, and (b) calls quantrocket.realtime.download_market_data_file(...) at startup to backfill a ~1 minute gap.

What is happening

  • Collection starts cleanly and trades flow into the tick DB. quantrocket realtime active shows 135 active collections, and the per-minute "Polygon market data received" summary box shows ~1,500–2,500 ticks/min across ~100 unique tickers.
  • Intermittently the Polygon WebSocket worker thread crashes. After the crash:
    • quantrocket realtime active still reports 135 active collections.
    • The per-minute "Polygon market data received" box stops printing.
    • No new trades land in the tick DB.
    • Recovery requires an OS-level docker restart quantrocket-realtime-1 (uWSGI's mule respawn is not sufficient).
  • This has happens at least every 10 minutes but not regularily

Surrounding log (15:22:06 – 15:22:22 UTC, quantrocket-realtime-1)

15:22:06  Waiting for notifications on channel 'md'
15:22:19  [spooler /var/tmp/uwsgi/spool pid: 31] managing request uwsgi_spoolfile_on_..._214802 ...
15:22:19  [spooler /var/tmp/uwsgi/spool pid: 31] done with task ... after 0 seconds
15:22:20  updating market data collections
15:22:20  Canceling market data request for ABAT STK (sid FIBBG004M1KJN5)
15:22:20  Canceling market data request for ABCL STK (sid FIBBG00LLW2MF2)
... (all 135 sids canceled in this burst) ...
15:22:20  Exception in thread polygon_websocket_worker:
          Traceback ...
          AttributeError: type object '_asyncio.Task' has no attribute 'all_tasks'
15:22:21  recycling mule after collecting market data
15:22:22  OOOPS mule 2 (pid: 40) crippled...trying respawn...
15:22:22  spawned uWSGI mule 2 (pid: 144)

Preceding client-side events we can correlate (from pad.live flightlog):

15:22:06  Monitor process started (new Python client inside quantrocket-satellite-1)
15:22:06  WSConsumer started (135 sids)      # opened connection to ws://houston/realtime/stream
15:22:13  Backfilled 31579 trades from 2026-04-14 15:21:00
           ^ this was quantrocket.realtime.download_market_data_file(
               'pad-live-trades',
               sids=[...135 sids...],
               fields=['LastPrice' | 'LastSize' | 'TradeConditions'],  # three separate calls
               start_date='2026-04-14 15:21:00')

note: we had this other issue with TradeID, not sure if our workaround creates an issue:

Related issue B (already worked around, reporting for completeness)

On the first creation of pad-live-trades with --fields LastPrice LastSize TradeConditions TradeId, inserts began failing with:

psycopg2.errors.NumericValueOutOfRange: value "71679637467727" is out of range for type integer

71679637467727 is a Polygon trade ID (14-digit int64). The tick DB column for TradeId appears to be declared as PostgreSQL integer (max 2,147,483,647) rather than bigint. Workaround: we recreated the DB without the TradeId field.

Environment

  • QuantRocket version: [fill in from quantrocket --version]
  • Container Python: 3.11
  • Polygon Developer subscription (stocks real-time trades)
  • Host OS: Linux 6.8.0-107-generic

question: am I doing something wrong ? (did I miss something from the doc ?)

#2

Investigating...