Alpaca Realtime Service Intermittently Hangs and Doesn't Start/Resume Startup as Scheduled until Docker Restart

charles · May 26, 2021, 9:13am

Struggling to track down an intermittent issue with the realtime data collection service not starting on some days. I currently have my realtime service scheduled via countdown to start running every morning at 3:30AM EST (30min before pre-market opens). It seems to run as scheduled sometimes, but othertimes it doesn't start, and I get no error in the logs. In fact, the logs indicate that the service did start, but I don't have any data, and no data is coming through the stream. Any help or guidance is much appreciated; thanks guys!

I attached a full log download from the realtime service from the last couple days, and also included relevant details below. See full logs here.

Here's my countdown schedule to start the realtime collection service; I'm using Alpaca's paid realtime aggregate minute service:

# Start real-time data collection from 3:30AM to 8:30PM market time (if market is open today)
30 3 * * mon-fri quantrocket master isopen 'NASDAQ' --in '7h' && quantrocket realtime collect 'usstock-realtime' --universes 'usstock-active' --until '20:30:00 America/New_York'

Here's the log output on days where it hangs. Note that on days where it starts correctly, the log output includes the below but also includes details for each symbol/sid being requested from Alpaca

# Below you see the prior day's collection being successfully closed down
2021-05-24 20:30:20 EDT| <11>1 2021-05-25T00:30:20Z 3889728f128e quantrocket_realtime_1 4444 - - closing connection to wss://stream.data.alpaca.markets/v2/sip

# Next morning you see the realtime collection process being kicked off, and all looks well except that I don't see any market data requests being issued like I normally do.
2021-05-25 03:30:02 EDT| <11>1 2021-05-25T07:30:02Z 3889728f128e quantrocket_realtime_1 4444 - - [spooler] written 196844 bytes to file /var/tmp/uwsgi/spool/uwsgi_spoolfile_on_3889728f128e_27_2_1350762257_1621927802_986540
2021-05-25 20:30:00 EDT| <11>1 2021-05-26T00:30:00Z 3889728f128e quantrocket_realtime_1 4444 - - [spooler /var/tmp/uwsgi/spool pid: 26] managing request uwsgi_spoolfile_on_3889728f128e_27_2_1350762257_1621927802_986540 ...
2021-05-25 20:30:00 EDT| <11>1 2021-05-26T00:30:00Z 3889728f128e quantrocket_realtime_1 4444 - - [spooler /var/tmp/uwsgi/spool pid: 26] done with task uwsgi_spoolfile_on_3889728f128e_27_2_1350762257_1621927802_986540 after 0 seconds
2021-05-26 03:30:02 EDT| <11>1 2021-05-26T07:30:02Z 3889728f128e quantrocket_realtime_1 4444 - - [spooler] written 196754 bytes to file /var/tmp/uwsgi/spool/uwsgi_spoolfile_on_3889728f128e_27_3_729580950_1622014202_873937
2021-05-26 04:38:05 EDT| <11>1 2021-05-26T08:38:05Z 3889728f128e quantrocket_realtime_1 4444 - - Waiting for notifications on channel 'md'
2021-05-26 04:46:20 EDT| <11>1 2021-05-26T08:46:20Z 3889728f128e quantrocket_realtime_1 4444 - - announcing my loyalty to the Emperor...
2021-05-26 04:46:20 EDT| <11>1 2021-05-26T08:46:20Z 3889728f128e quantrocket_realtime_1 4444 - - Wed May 26 08:46:20 2021 - [emperor] vassal /etc/uwsgi/uwsgi_websockets.ini is now loyal

Here's an example of a successful shutdown, and then subsequent startup the next morning:

# Realtime shutdown at end of day
2021-05-19 00:31:01 UTC| <11>1 2021-05-19T00:31:01Z 3889728f128e quantrocket_realtime_1 3506 - - closing connection to wss://stream.data.alpaca.markets/v2/sip
2021-05-19 00:31:12 UTC| <11>1 2021-05-19T00:31:12Z 3889728f128e quantrocket_realtime_1 3506 - - recycling mule after collecting market data
2021-05-19 00:31:12 UTC| <11>1 2021-05-19T00:31:12Z 3889728f128e quantrocket_realtime_1 3506 - - OOOPS mule 3 (pid: 155) crippled...trying respawn...
2021-05-19 00:31:12 UTC| <11>1 2021-05-19T00:31:12Z 3889728f128e quantrocket_realtime_1 3506 - - spawned uWSGI mule 3 (pid: 162)

# Realtime startup next morning
2021-05-19 07:30:03 UTC| <11>1 2021-05-19T07:30:03Z 3889728f128e quantrocket_realtime_1 3506 - - [spooler] written 170654 bytes to file /var/tmp/uwsgi/spool/uwsgi_spoolfile_on_3889728f128e_29_4_759377620_1621409403_596886
2021-05-19 07:30:08 UTC| <11>1 2021-05-19T07:30:08Z 3889728f128e quantrocket_realtime_1 3506 - - updating market data collections
2021-05-19 07:30:11 UTC| <11>1 2021-05-19T07:30:11Z 3889728f128e quantrocket_realtime_1 3506 - - Issuing market data request for A STK (sid FIBBG000C2V3D6)
2021-05-19 07:30:11 UTC| <11>1 2021-05-19T07:30:11Z 3889728f128e quantrocket_realtime_1 3506 - - connected to: wss://stream.data.alpaca.markets/v2/sip
2021-05-19 07:30:11 UTC| <11>1 2021-05-19T07:30:11Z 3889728f128e quantrocket_realtime_1 3506 - - Issuing market data request for AA STK (sid FIBBG00B3T3HD3)
2021-05-19 07:30:11 UTC| <11>1 2021-05-19T07:30:11Z 3889728f128e quantrocket_realtime_1 3506 - - Issuing market data request for AAA STK (sid FIBBG00X5FSP48)
2021-05-19 07:30:11 UTC| <11>1 2021-05-19T07:30:11Z 3889728f128e quantrocket_realtime_1 3506 - - Issuing market data request for AAAU STK (sid FIBBG00LPXX872)

Here you can see the output of realtime list

jupyter:/codeload $ quantrocket realtime list
usstock-realtime:
- usstock-realtime-minute

Lastly, I see this cron message which appears to indicate that everything started ok:

quantrocket_countdown_1|From root@6ebbec62f1f1 Wed May 26 03:30:02 2021
quantrocket_countdown_1|Return-path: <root@6ebbec62f1f1>
quantrocket_countdown_1|Envelope-to: root@6ebbec62f1f1
quantrocket_countdown_1|Delivery-date: Wed, 26 May 2021 03:30:02 -0400
quantrocket_countdown_1|Received: from root by 9525b2f600c2 with local (Exim 4.92)
quantrocket_countdown_1|  (envelope-from <root@6ebbec62f1f1>)
quantrocket_countdown_1|  id 1llnzO-0000ML-T8
quantrocket_countdown_1|  for root@6ebbec62f1f1; Wed, 26 May 2021 03:30:02 -0400
quantrocket_countdown_1|From: root@6ebbec62f1f1 (Cron Daemon)
quantrocket_countdown_1|To: root@6ebbec62f1f1
quantrocket_countdown_1|Subject: Cron <root@9525b2f600c2> quantrocket master isopen 'NASDAQ' --in '7h' && quantrocket realtime collect 'usstock-realtime' --universes 'usstock-active' --until '20:30:00 America/New_York'
quantrocket_countdown_1|MIME-Version: 1.0
quantrocket_countdown_1|Content-Type: text/plain; charset=UTF-8
quantrocket_countdown_1|Content-Transfer-Encoding: 8bit
quantrocket_countdown_1|X-Cron-Env: <SHELL=/bin/sh>
quantrocket_countdown_1|X-Cron-Env: <HOME=/root>
quantrocket_countdown_1|X-Cron-Env: <PATH=/usr/bin:/bin>
quantrocket_countdown_1|X-Cron-Env: <LOGNAME=root>
quantrocket_countdown_1|Message-Id: <E1llnzO-0000ML-T8@9525b2f600c2>
quantrocket_countdown_1|Date: Wed, 26 May 2021 03:30:02 -0400
quantrocket_countdown_1|
quantrocket_countdown_1|status: the market data will be collected until 20:30:00 America/New_York

charles · May 26, 2021, 10:32am

After more digging I realized that the issue started the prior day, when the realtime feed stalled during a reconnection and just seems to have never recovered. See logs below.

@Brian while fixing the root cause of this issue is ideal, in the meantime is there a workaround for ensuring the realtime service is restarted each day? Cancelling and restarting the collection process alone doesn't seem to clear the issue; I have to actually restart the docker container to get things working again; included logs from service restart below in case any of that's helpful.

Logs showing prior day reconnection stall


# Everything is running fine at this point; realtime service started at 4am EST and was collecting data normally
2021-05-24 04:10:46 EDT| <11>1 2021-05-24T08:10:46Z 3889728f128e quantrocket_realtime_1 4444 - - ┌──────────────────────────────────────────────────┐
2021-05-24 04:10:46 EDT| <11>1 2021-05-24T08:10:46Z 3889728f128e quantrocket_realtime_1 4444 - - │ Alpaca market data received:                     │
2021-05-24 04:10:46 EDT| <11>1 2021-05-24T08:10:46Z 3889728f128e quantrocket_realtime_1 4444 - - │                       total_ticks unique_tickers │
2021-05-24 04:10:46 EDT| <11>1 2021-05-24T08:10:46Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:00 UTC         195            195 │
2021-05-24 04:10:46 EDT| <11>1 2021-05-24T08:10:46Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:01 UTC          92             92 │
2021-05-24 04:10:46 EDT| <11>1 2021-05-24T08:10:46Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:02 UTC         102            102 │
2021-05-24 04:10:46 EDT| <11>1 2021-05-24T08:10:46Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:03 UTC         108            108 │
2021-05-24 04:10:46 EDT| <11>1 2021-05-24T08:10:46Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:04 UTC          72             72 │
2021-05-24 04:10:46 EDT| <11>1 2021-05-24T08:10:46Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:05 UTC          76             76 │
2021-05-24 04:10:46 EDT| <11>1 2021-05-24T08:10:46Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:06 UTC          69             69 │
2021-05-24 04:10:46 EDT| <11>1 2021-05-24T08:10:46Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:07 UTC          67             67 │
2021-05-24 04:10:46 EDT| <11>1 2021-05-24T08:10:46Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:08 UTC          64             64 │
2021-05-24 04:10:46 EDT| <11>1 2021-05-24T08:10:46Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:09 UTC          85             85 │
2021-05-24 04:10:46 EDT| <11>1 2021-05-24T08:10:46Z 3889728f128e quantrocket_realtime_1 4444 - - │ active collections                       10633.0 │
2021-05-24 04:10:46 EDT| <11>1 2021-05-24T08:10:46Z 3889728f128e quantrocket_realtime_1 4444 - - └──────────────────────────────────────────────────┘


# Noticed that after this reconnection below, the Alpaca market data received panel/count never received any new data. Not sure what happened here, but the realtime service seems to never recovered from this point, and stayed stuck until the next day when I manually restarted everything because I noticed no new data had been received for over a day.
2021-05-24 04:11:35 EDT| <11>1 2021-05-24T08:11:35Z 3889728f128e quantrocket_realtime_1 4444 - - reconnecting to Alpaca websocket after error: code = 1006 (connection closed abnormally [internal]), no reason
2021-05-24 04:11:35 EDT| <11>1 2021-05-24T08:11:35Z 3889728f128e quantrocket_realtime_1 4444 - - closing connection to wss://stream.data.alpaca.markets/v2/sip
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - connected to: wss://stream.data.alpaca.markets/v2/sip
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - re-subscribing to bars for 10633 symbols after reconnect
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - Alpaca status message: {'T': 'error', 'code': 400, 'msg': 'invalid syntax'}
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - Alpaca status message: {'T': 'error', 'code': 400, 'msg': 'invalid syntax'}
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - Alpaca status message: {'T': 'error', 'code': 400, 'msg': 'invalid syntax'}
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - Alpaca status message: {'T': 'error', 'code': 400, 'msg': 'invalid syntax'}
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - Alpaca status message: {'T': 'error', 'code': 400, 'msg': 'invalid syntax'}
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - Alpaca status message: {'T': 'error', 'code': 400, 'msg': 'invalid syntax'}
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - Alpaca status message: {'T': 'error', 'code': 400, 'msg': 'invalid syntax'}
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - Alpaca status message: {'T': 'error', 'code': 400, 'msg': 'invalid syntax'}
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - Alpaca status message: {'T': 'error', 'code': 400, 'msg': 'invalid syntax'}
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - Alpaca status message: {'T': 'error', 'code': 400, 'msg': 'invalid syntax'}
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - Alpaca status message: {'T': 'error', 'code': 400, 'msg': 'invalid syntax'}
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - Alpaca status message: {'T': 'error', 'code': 400, 'msg': 'invalid syntax'}
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - Alpaca status message: {'T': 'error', 'code': 400, 'msg': 'invalid syntax'}
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - Alpaca status message: {'T': 'error', 'code': 400, 'msg': 'invalid syntax'}
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - Alpaca status message: {'T': 'error', 'code': 400, 'msg': 'invalid syntax'}
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - Alpaca status message: {'T': 'error', 'code': 400, 'msg': 'invalid syntax'}
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - Alpaca status message: {'T': 'error', 'code': 400, 'msg': 'invalid syntax'}
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - Alpaca status message: {'T': 'error', 'code': 400, 'msg': 'invalid syntax'}
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - Alpaca status message: {'T': 'error', 'code': 400, 'msg': 'invalid syntax'}
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - Alpaca status message: {'T': 'error', 'code': 400, 'msg': 'invalid syntax'}
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - Alpaca status message: {'T': 'error', 'code': 400, 'msg': 'invalid syntax'}
2021-05-24 04:11:36 EDT| <11>1 2021-05-24T08:11:36Z 3889728f128e quantrocket_realtime_1 4444 - - Alpaca status message: {'T': 'error', 'code': 400, 'msg': 'invalid syntax'}
2021-05-24 04:11:51 EDT| <11>1 2021-05-24T08:11:51Z 3889728f128e quantrocket_realtime_1 4444 - - ┌──────────────────────────────────────────────────┐
2021-05-24 04:11:51 EDT| <11>1 2021-05-24T08:11:51Z 3889728f128e quantrocket_realtime_1 4444 - - │ Alpaca market data received:                     │
2021-05-24 04:11:51 EDT| <11>1 2021-05-24T08:11:51Z 3889728f128e quantrocket_realtime_1 4444 - - │                       total_ticks unique_tickers │
2021-05-24 04:11:51 EDT| <11>1 2021-05-24T08:11:51Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:01 UTC          92             92 │
2021-05-24 04:11:51 EDT| <11>1 2021-05-24T08:11:51Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:02 UTC         102            102 │
2021-05-24 04:11:51 EDT| <11>1 2021-05-24T08:11:51Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:03 UTC         108            108 │
2021-05-24 04:11:51 EDT| <11>1 2021-05-24T08:11:51Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:04 UTC          72             72 │
2021-05-24 04:11:51 EDT| <11>1 2021-05-24T08:11:51Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:05 UTC          76             76 │
2021-05-24 04:11:51 EDT| <11>1 2021-05-24T08:11:51Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:06 UTC          69             69 │
2021-05-24 04:11:51 EDT| <11>1 2021-05-24T08:11:51Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:07 UTC          67             67 │
2021-05-24 04:11:51 EDT| <11>1 2021-05-24T08:11:51Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:08 UTC          64             64 │
2021-05-24 04:11:51 EDT| <11>1 2021-05-24T08:11:51Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:09 UTC          85             85 │
2021-05-24 04:11:51 EDT| <11>1 2021-05-24T08:11:51Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:10 UTC          71             71 │
2021-05-24 04:11:51 EDT| <11>1 2021-05-24T08:11:51Z 3889728f128e quantrocket_realtime_1 4444 - - │ active collections                       10633.0 │
2021-05-24 04:11:51 EDT| <11>1 2021-05-24T08:11:51Z 3889728f128e quantrocket_realtime_1 4444 - - └──────────────────────────────────────────────────┘
2021-05-24 04:12:56 EDT| <11>1 2021-05-24T08:12:56Z 3889728f128e quantrocket_realtime_1 4444 - - ┌──────────────────────────────────────────────────┐
2021-05-24 04:12:56 EDT| <11>1 2021-05-24T08:12:56Z 3889728f128e quantrocket_realtime_1 4444 - - │ Alpaca market data received:                     │
2021-05-24 04:12:56 EDT| <11>1 2021-05-24T08:12:56Z 3889728f128e quantrocket_realtime_1 4444 - - │                       total_ticks unique_tickers │
2021-05-24 04:12:56 EDT| <11>1 2021-05-24T08:12:56Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:01 UTC          92             92 │
2021-05-24 04:12:56 EDT| <11>1 2021-05-24T08:12:56Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:02 UTC         102            102 │
2021-05-24 04:12:56 EDT| <11>1 2021-05-24T08:12:56Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:03 UTC         108            108 │
2021-05-24 04:12:56 EDT| <11>1 2021-05-24T08:12:56Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:04 UTC          72             72 │
2021-05-24 04:12:56 EDT| <11>1 2021-05-24T08:12:56Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:05 UTC          76             76 │
2021-05-24 04:12:56 EDT| <11>1 2021-05-24T08:12:56Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:06 UTC          69             69 │
2021-05-24 04:12:56 EDT| <11>1 2021-05-24T08:12:56Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:07 UTC          67             67 │
2021-05-24 04:12:56 EDT| <11>1 2021-05-24T08:12:56Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:08 UTC          64             64 │
2021-05-24 04:12:56 EDT| <11>1 2021-05-24T08:12:56Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:09 UTC          85             85 │
2021-05-24 04:12:56 EDT| <11>1 2021-05-24T08:12:56Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:10 UTC          71             71 │
2021-05-24 04:12:56 EDT| <11>1 2021-05-24T08:12:56Z 3889728f128e quantrocket_realtime_1 4444 - - │ active collections                       10633.0 │
2021-05-24 04:12:56 EDT| <11>1 2021-05-24T08:12:56Z 3889728f128e quantrocket_realtime_1 4444 - - └──────────────────────────────────────────────────┘
2021-05-24 04:14:02 EDT| <11>1 2021-05-24T08:14:02Z 3889728f128e quantrocket_realtime_1 4444 - - ┌──────────────────────────────────────────────────┐
2021-05-24 04:14:02 EDT| <11>1 2021-05-24T08:14:02Z 3889728f128e quantrocket_realtime_1 4444 - - │ Alpaca market data received:                     │
2021-05-24 04:14:02 EDT| <11>1 2021-05-24T08:14:02Z 3889728f128e quantrocket_realtime_1 4444 - - │                       total_ticks unique_tickers │
2021-05-24 04:14:02 EDT| <11>1 2021-05-24T08:14:02Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:01 UTC          92             92 │
2021-05-24 04:14:02 EDT| <11>1 2021-05-24T08:14:02Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:02 UTC         102            102 │
2021-05-24 04:14:02 EDT| <11>1 2021-05-24T08:14:02Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:03 UTC         108            108 │
2021-05-24 04:14:02 EDT| <11>1 2021-05-24T08:14:02Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:04 UTC          72             72 │
2021-05-24 04:14:02 EDT| <11>1 2021-05-24T08:14:02Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:05 UTC          76             76 │
2021-05-24 04:14:02 EDT| <11>1 2021-05-24T08:14:02Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:06 UTC          69             69 │
2021-05-24 04:14:02 EDT| <11>1 2021-05-24T08:14:02Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:07 UTC          67             67 │
2021-05-24 04:14:02 EDT| <11>1 2021-05-24T08:14:02Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:08 UTC          64             64 │
2021-05-24 04:14:02 EDT| <11>1 2021-05-24T08:14:02Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:09 UTC          85             85 │
2021-05-24 04:14:02 EDT| <11>1 2021-05-24T08:14:02Z 3889728f128e quantrocket_realtime_1 4444 - - │ received at 08:10 UTC          71             71 │
2021-05-24 04:14:02 EDT| <11>1 2021-05-24T08:14:02Z 3889728f128e quantrocket_realtime_1 4444 - - │ active collections                       10633.0 │
2021-05-24 04:14:02 EDT| <11>1 2021-05-24T08:14:02Z 3889728f128e quantrocket_realtime_1 4444 - - └──────────────────────────────────────────────────┘

Realtime Service Docker Restart Logs

quantrocket_realtime_1|[emperor] *** RAGNAROK EVOKED ***
quantrocket_realtime_1|Wed May 26 10:56:45 2021 - [emperor] stop the uwsgi instance /etc/uwsgi/uwsgi.ini
quantrocket_realtime_1|Wed May 26 10:56:45 2021 - [emperor] stop the uwsgi instance /etc/uwsgi/uwsgi_websockets.ini
quantrocket_realtime_1|Wed May 26 10:56:45 2021 - received message 0 from emperor
quantrocket_realtime_1|Wed May 26 10:56:45 2021 - received message 0 from emperor
quantrocket_realtime_1|SIGINT/SIGQUIT received...killing workers...
quantrocket_realtime_1|SIGINT/SIGQUIT received...killing workers...
quantrocket_realtime_1|killing the spooler with pid 26
quantrocket_realtime_1|[deadlock-detector] a process holding a robust mutex died. recovering...
quantrocket_realtime_1|worker 1 buried after 1 seconds
quantrocket_realtime_1|worker 2 buried after 1 seconds
quantrocket_realtime_1|worker 3 buried after 1 seconds
quantrocket_realtime_1|goodbye to uWSGI.
quantrocket_realtime_1|spooler (pid: 26) annihilated
quantrocket_realtime_1|worker 1 buried after 1 seconds
quantrocket_realtime_1|worker 2 buried after 1 seconds
quantrocket_realtime_1|worker 3 buried after 1 seconds
quantrocket_realtime_1|mule 1 (pid: 37) annihilated
quantrocket_realtime_1|mule 2 (pid: 40) annihilated
quantrocket_realtime_1|mule 4 (pid: 44) annihilated
quantrocket_realtime_1|Wed May 26 10:56:47 2021 - [emperor] removed uwsgi instance /etc/uwsgi/uwsgi_websockets.ini
quantrocket_houston_1|172.18.0.3 - - [26/May/2021:10:56:58 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.24.0"
quantrocket_houston_1|172.18.0.21 - - [26/May/2021:10:56:58 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 9 "-" "-"
quantrocket_realtime_1|*** Starting uWSGI 2.0.19.1 (64bit) on [Wed May 26 10:56:58 2021] ***
quantrocket_realtime_1|compiled with version: 7.3.0 on 10 March 2021 14:51:06
quantrocket_realtime_1|os: Linux-5.10.25-linuxkit #1 SMP Tue Mar 23 09:27:39 UTC 2021
quantrocket_realtime_1|nodename: 3889728f128e
quantrocket_realtime_1|machine: x86_64
quantrocket_realtime_1|clock source: unix
quantrocket_realtime_1|pcre jit disabled
quantrocket_realtime_1|detected number of CPU cores: 10
quantrocket_realtime_1|current working directory: /
quantrocket_realtime_1|detected binary path: /opt/conda/bin/uwsgi-quantrocket
quantrocket_realtime_1|uWSGI running as root, you can use --uid/--gid/--chroot options
quantrocket_realtime_1|*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
quantrocket_realtime_1|*** WARNING: you are running uWSGI without its master process manager ***
quantrocket_realtime_1|your memory page size is 4096 bytes
quantrocket_realtime_1|detected max file descriptor number: 1048576
quantrocket_realtime_1|*** starting uWSGI Emperor ***
quantrocket_realtime_1|*** has_emperor mode detected (fd: 6) ***
quantrocket_realtime_1|*** has_emperor mode detected (fd: 5) ***
quantrocket_realtime_1|[uWSGI] getting INI configuration from /etc/uwsgi/uwsgi_websockets.ini
quantrocket_realtime_1|[uWSGI] getting INI configuration from /etc/uwsgi/uwsgi.ini
quantrocket_realtime_1|[uWSGI] getting INI configuration from /etc/uwsgi/base.ini
quantrocket_realtime_1|[uWSGI] getting INI configuration from /etc/uwsgi/default.ini
quantrocket_realtime_1|[uWSGI] getting INI configuration from /etc/uwsgi/base.ini
quantrocket_realtime_1|*** Starting uWSGI 2.0.19.1 (64bit) on [Wed May 26 10:56:59 2021] ***
quantrocket_realtime_1|compiled with version: 7.3.0 on 10 March 2021 14:51:06
quantrocket_realtime_1|os: Linux-5.10.25-linuxkit #1 SMP Tue Mar 23 09:27:39 UTC 2021
quantrocket_realtime_1|nodename: 3889728f128e
quantrocket_realtime_1|machine: x86_64
quantrocket_realtime_1|clock source: unix
quantrocket_realtime_1|pcre jit disabled
quantrocket_realtime_1|detected number of CPU cores: 10
quantrocket_realtime_1|current working directory: /
quantrocket_realtime_1|detected binary path: /opt/conda/bin/uwsgi-quantrocket
quantrocket_realtime_1|uWSGI running as root, you can use --uid/--gid/--chroot options
quantrocket_realtime_1|*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
quantrocket_realtime_1|your memory page size is 4096 bytes
quantrocket_realtime_1|detected max file descriptor number: 1048576
quantrocket_realtime_1|lock engine: pthread robust mutexes
quantrocket_realtime_1|thunder lock: enabled
quantrocket_realtime_1|*** Starting uWSGI 2.0.19.1 (64bit) on [Wed May 26 10:56:59 2021] ***
quantrocket_realtime_1|compiled with version: 7.3.0 on 10 March 2021 14:51:06
quantrocket_realtime_1|os: Linux-5.10.25-linuxkit #1 SMP Tue Mar 23 09:27:39 UTC 2021
quantrocket_realtime_1|nodename: 3889728f128e
quantrocket_realtime_1|machine: x86_64
quantrocket_realtime_1|clock source: unix
quantrocket_realtime_1|pcre jit disabled
quantrocket_realtime_1|detected number of CPU cores: 10
quantrocket_realtime_1|current working directory: /
quantrocket_realtime_1|detected binary path: /opt/conda/bin/uwsgi-quantrocket
quantrocket_realtime_1|uWSGI running as root, you can use --uid/--gid/--chroot options
quantrocket_realtime_1|*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
quantrocket_realtime_1|your memory page size is 4096 bytes
quantrocket_realtime_1|detected max file descriptor number: 1048576
quantrocket_realtime_1|lock engine: pthread robust mutexes
quantrocket_realtime_1|uwsgi socket 0 bound to TCP address 0.0.0.0:81 fd 3
quantrocket_realtime_1|uWSGI running as root, you can use --uid/--gid/--chroot options
quantrocket_realtime_1|*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
quantrocket_realtime_1|Python version: 3.8.3 (default, Jul  2 2020, 16:21:59)  [GCC 7.3.0]
quantrocket_realtime_1|thunder lock: enabled
quantrocket_realtime_1|*** Cache "realtimecache" initialized: 6MB (key: 2136 bytes, keys: 25632 bytes, data: 6291456 bytes, bitmap: 0 bytes) preallocated ***
quantrocket_realtime_1|uwsgi socket 0 bound to TCP address 0.0.0.0:80 fd 4
quantrocket_realtime_1|uWSGI running as root, you can use --uid/--gid/--chroot options
quantrocket_realtime_1|*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
quantrocket_realtime_1|Python version: 3.8.3 (default, Jul  2 2020, 16:21:59)  [GCC 7.3.0]
quantrocket_realtime_1|Python main interpreter initialized at 0x55583c60cbf0
quantrocket_realtime_1|uWSGI running as root, you can use --uid/--gid/--chroot options
quantrocket_realtime_1|*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
quantrocket_realtime_1|python threads support enabled
quantrocket_realtime_1|your server socket listen backlog is limited to 100 connections
quantrocket_realtime_1|your mercy for graceful operations on workers is 60 seconds
quantrocket_realtime_1|Python main interpreter initialized at 0x5585d0f17110
quantrocket_realtime_1|uWSGI running as root, you can use --uid/--gid/--chroot options
quantrocket_realtime_1|*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
quantrocket_realtime_1|python threads support enabled
quantrocket_realtime_1|your server socket listen backlog is limited to 100 connections
quantrocket_realtime_1|your mercy for graceful operations on workers is 60 seconds
quantrocket_realtime_1|mapped 825016 bytes (805 KB) for 6 cores
quantrocket_realtime_1|mapped 825016 bytes (805 KB) for 6 cores
quantrocket_realtime_1|*** Operational MODE: preforking+threaded ***
quantrocket_realtime_1|*** Operational MODE: preforking+threaded ***
quantrocket_realtime_1|i am the bootstrap for uwsgi.SymbolsImporter
quantrocket_realtime_1|i am the bootstrap for uwsgi.SymbolsImporter
quantrocket_realtime_1|i am the bootstrap for uwsgi.SymbolsImporter
quantrocket_realtime_1|i am the bootstrap for uwsgi.SymbolsImporter
quantrocket_realtime_1|WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x5585d0f17110 pid: 9 (default app)
quantrocket_realtime_1|uWSGI running as root, you can use --uid/--gid/--chroot options
quantrocket_realtime_1|*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
quantrocket_realtime_1|spawned uWSGI master process (pid: 9)
quantrocket_realtime_1|Wed May 26 10:56:59 2021 - [emperor] vassal /etc/uwsgi/uwsgi_websockets.ini has been spawned
quantrocket_realtime_1|spawned uWSGI worker 1 (pid: 10, cores: 2)
quantrocket_realtime_1|spawned uWSGI worker 2 (pid: 11, cores: 2)
quantrocket_realtime_1|spawned uWSGI worker 3 (pid: 12, cores: 2)
quantrocket_realtime_1|Wed May 26 10:56:59 2021 - [emperor] vassal /etc/uwsgi/uwsgi_websockets.ini is ready to accept requests
quantrocket_realtime_1|[uwsgi-signal] signum 0 registered (wid: 0 modifier1: 0 target: mule)
quantrocket_realtime_1|[spooler] written 103 bytes to file /var/tmp/uwsgi/spool/uwsgi_spoolfile_on_3889728f128e_8_1_742503513_1622026619_923975
quantrocket_realtime_1|WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x55583c60cbf0 pid: 8 (default app)
quantrocket_realtime_1|uWSGI running as root, you can use --uid/--gid/--chroot options
quantrocket_realtime_1|*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
quantrocket_realtime_1|spawned uWSGI master process (pid: 8)
quantrocket_realtime_1|spawned the uWSGI spooler on dir /var/tmp/uwsgi/spool with pid 26
quantrocket_realtime_1|Wed May 26 10:56:59 2021 - [emperor] vassal /etc/uwsgi/uwsgi.ini has been spawned
quantrocket_realtime_1|spawned uWSGI worker 1 (pid: 28, cores: 2)
quantrocket_realtime_1|[spooler /var/tmp/uwsgi/spool pid: 26] managing request uwsgi_spoolfile_on_3889728f128e_8_1_742503513_1622026619_923975 ...
quantrocket_realtime_1|spawned uWSGI worker 2 (pid: 29, cores: 2)
quantrocket_realtime_1|Wed May 26 10:56:59 2021 - [emperor] vassal /etc/uwsgi/uwsgi.ini is ready to accept requests
quantrocket_realtime_1|spawned uWSGI worker 3 (pid: 33, cores: 2)
quantrocket_realtime_1|cache sweeper thread enabled
quantrocket_realtime_1|spawned uWSGI mule 1 (pid: 37)
quantrocket_realtime_1|spawned uWSGI mule 2 (pid: 40)
quantrocket_realtime_1|spawned uWSGI mule 3 (pid: 42)
quantrocket_realtime_1|spawned uWSGI mule 4 (pid: 44)
quantrocket_realtime_1|NOTICE:  version "2.1.0" of extension "timescaledb" is already installed
quantrocket_realtime_1|NOTICE:  version "2.1.0" of extension "timescaledb" is already installed
quantrocket_realtime_1|NOTICE:  version "2.1.0" of extension "timescaledb" is already installed
quantrocket_realtime_1|NOTICE:  version "2.1.0" of extension "timescaledb" is already installed
quantrocket_realtime_1|[spooler /var/tmp/uwsgi/spool pid: 26] done with task uwsgi_spoolfile_on_3889728f128e_8_1_742503513_1622026619_923975 after 1 seconds

charles · May 27, 2021, 4:04am

While waiting on support/feedback for this item I've migrated to Polygon for all realtime data to see if that resolves the issue; I know the Alpaca sip realtime feed is somewhat new, so maybe there's a bug with that code. Will report back whether or not this resolved the issue. That said, it would be ideal to use Alpaca given it's 1/4 the price for realtime data. Thanks!

Brian · May 27, 2021, 12:56pm

The "invalid syntax" messages seem problematic. Will attempt to reproduce.

charles · May 28, 2021, 12:20am

Yea I thought that was odd; sounds good. Let me know!

In the meantime, Polygon realtime streaming seems to be solid.

charles · June 11, 2021, 6:53am

Hey @Brian any updates on this? Just noticed today that Alpaca officially announced their pricing for realtime data and it's only $9/mo! as opposed to the original $47/mo mentioned.

Unfortunately because of this issue I'm currently spending $200/mo with Polygon, and would REALLY like to switch back to Alpaca if possible

Let me know; thanks mate!

P.S. I've been running on Polygon for two weeks now without any issues, so the issue appears to be exclusive to the Alpaca realtime integration.

Brian · June 11, 2021, 6:50pm

The upcoming 2.6 release will have an update to the reconnect logic that will ensure data gets flowing again. That said, the disconnects (which QuantRocket can't do anything about) still result in a temporary interruption of data. But I would not be surprised if that improves over time on Alpaca's end.

charles · June 12, 2021, 12:48am

That's great news! Thanks for the quick turnaround on this.

The good news is that because I'm using a system like QuantRocket, I have options. I love that I can easily plug in another data provider if one provider is having issues for whatever reason. That's so damn powerful.

Thanks again, and will be looking forward to that update.

Brian · June 21, 2021, 8:38pm

Better reconnect logic is included in version 2.6.0.

charles · June 24, 2021, 8:22pm

You da man! Thanks brother.