Alpaca realtime data produces zipline minutes with no price data

bjsun · July 2, 2021, 7:16pm

I've been testing the Alpaca v2 Data API as an alternative to IBKR. I set up the collection of minute aggregates in my Zipline live strategy as described in the docs (very simple to sub in Alpaca in place of IBKR btw, thanks to the universal database design). For the test day, I requested prices for just two assets, one stable (SPY) and one more volatile (MRNA).

Alpaca was working great for a while, until it started producing this error every few minutes at random:

2021-07-02 10:48:05 quantrocket.zipline: WARNING [alpacaTest] No current-day data after 13:47:00 is available in real-time database alpaca-tick-1min
2021-07-02 10:58:05 quantrocket.zipline: WARNING [alpacaTest] No current-day data after 13:57:00 is available in real-time database alpaca-tick-1min
2021-07-02 11:06:05 quantrocket.zipline: WARNING [alpacaTest] No current-day data after 14:05:00 is available in real-time database alpaca-tick-1min
2021-07-02 11:16:05 quantrocket.zipline: WARNING [alpacaTest] No current-day data after 14:15:00 is available in real-time database alpaca-tick-1min
2021-07-02 11:19:05 quantrocket.zipline: WARNING [alpacaTest] No current-day data after 14:18:00 is available in real-time database alpaca-tick-1min
2021-07-02 11:23:05 quantrocket.zipline: WARNING [alpacaTest] No current-day data after 14:22:00 is available in real-time database alpaca-tick-1min
2021-07-02 11:35:05 quantrocket.zipline: WARNING [alpacaTest] No current-day data after 14:34:00 is available in real-time database alpaca-tick-1min
2021-07-02 11:40:05 quantrocket.zipline: WARNING [alpacaTest] No current-day data after 14:39:00 is available in real-time database alpaca-tick-1min
2021-07-02 11:43:05 quantrocket.zipline: WARNING [alpacaTest] No current-day data after 14:42:00 is available in real-time database alpaca-tick-1min

Full traceback suggests that Zipline is querying the realtime database repeatedly for 5 seconds and then giving up (?) and feeding NaN prices to the strategy.

quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:01 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:01 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:01 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:01 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:02 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:02 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:02 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:02 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:02 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:02 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:02 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:02 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:02 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:03 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:03 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:03 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:03 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:03 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:04 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:04 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:04 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:04 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:04 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:04 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:04 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:04 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:04 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_houston_1|172.18.0.19 - - [02/Jul/2021:18:35:05 +0000] "GET /realtime/alpaca-tick-1min.csv?start_date=2021-07-02T18%3A34%3A00%2B00%3A00&end_date=2021-07-02T18%3A34%3A00%2B00%3A00&fields=MinuteCloseClose&fields=MinuteOpenOpen&fields=MinuteHighHigh&fields=MinuteLowLow&fields=MinuteVolumeSum HTTP/1.1" 400 74 "-" "-"
quantrocket_zipline_1|feeding 2021-07-02 14:35:00-04:00 BAR event to alpacaTest strategy
quantrocket_flightlog_1|2021-07-02 11:35:05 quantrocket.zipline: WARNING [alpacaTest] No current-day data after 14:34:00 is available in real-time database alpaca-tick-1min

There is no possibility that SPY has empty prices for any minute. Am I right in guessing that the Alpaca ticks are not arriving until very late in the minute and there's a timeout (5 seconds?) before Zipline gives up for the current minute? Is there a way to extend or eliminate the timeout?

Alternatively, is there any reason the collecting of the ticks would be delayed on the QuantRocket side, or must this be an Alpaca problem? It ran for 4 trading hours without any skipped minutes, and then it started happening fairly frequently in the afternoon. I'm also checking the Alpaca forums, but no mention so far.

Brian · July 7, 2021, 12:34pm

Zipline expects at least some data to start flowing within 5 seconds after the minute ends (and that’s not currently a configurable setting). I wasn’t able to reproduce this and before fiddling with the setting I would probably want to see more evidence that this is a somewhat consistent issue as opposed to a one-off day of extra latency on Alpaca’s end.

bjsun · July 8, 2021, 12:39am

I've been streaming prices for 500+ actively traded stocks from Alpaca this week. No null-data minutes for SPY on Tuesday, but several today, similar to last Friday when I reported this. If you stream just SPY all day for a few days, you might be able to reproduce the error. I also found multiple null-data gaps for most stocks in the S&P 500.

Regardless, this is more of an Alpaca issue. An inconsistent 5+ second latency for minute data is untradeable. I'll try reporting on their forums. It's a new API, so not surprised if they need to fix some kinks.

Here's my Alpaca forum post: https://forum.alpaca.markets/t/missing-minutes-in-real-time-aggregate-collection/6163