Zipline live doesn't provide partial daily data

Hi Brian,

As I further debug my live zipline strategy, I’m finding another inconsistency between live and backtest historical data, perhaps related to the previous issue that was just patched.

In backtest, when daily price history is retrieved using the ‘data’ object, zipline will fill in the data of the current trading day with partial data, even though the day is not complete. This partial day data is updated as of the last complete minute. For example:

hist = data.history(asset, ['open', 'high', 'low', 'close', 'price', 'volume'], 5, '1d')

today = hist.iloc[-1]

# This bool is always true in backtest
close_test = today['close'] == data.current(asset, 'close')

# These bools are true if the last complete minute set a new high or low for the day
new_high = today['high'] == data.current(asset, 'high')
new_low = today['low'] == data.current(asset, 'low')

In the current live implementation, the same calls will return NaN for all fields of the current day (the variable ‘today’ above), with the exception of ‘price’, which returns a value that does not correspond to the latest minute price.

Any chance this could be fixed to match backtest?

ps Happy New Year!

It sounds like we should be rolling up the real-time minute bars to produce the partial day bar. We can address that in the next release. Thanks for sharing.

1 Like

This is fixed in version 2.4.0:

After updating to v2.4.0, I’m getting an error when calling data.history() in a live strategy (doesn’t throw error in backtest). This is code that worked fine in v2.3.0. Here’s the traceback:

2021-01-15 11:29:42 quantrocket.zipline: ERROR Traceback (most recent call last):
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "sym://qrocket_log_py", line 34, in wrapped
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_trade_trade_py", line 63, in mule_trade_algo
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_trade_trade_py", line 187, in trade_algo
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/algorithm.py", line 669, in run
2021-01-15 11:29:42 quantrocket.zipline: ERROR     for perf in self.get_generator():
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/gens/tradesimulation.py", line 205, in transform
2021-01-15 11:29:42 quantrocket.zipline: ERROR     for capital_change_packet in every_bar(dt):
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/gens/tradesimulation.py", line 133, in every_bar
2021-01-15 11:29:42 quantrocket.zipline: ERROR     handle_data(algo, current_data, dt_to_use)
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/utils/events.py", line 218, in handle_data
2021-01-15 11:29:42 quantrocket.zipline: ERROR     dt,
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/utils/events.py", line 237, in handle_data
2021-01-15 11:29:42 quantrocket.zipline: ERROR     self.callback(context, data)
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_trade_algorithm_py", line 161, in handle_data
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/algorithm.py", line 479, in handle_data
2021-01-15 11:29:42 quantrocket.zipline: ERROR     self._handle_data(self, data)
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "divLive", line 407, in handle_data
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "zipline/_protocol.pyx", line 121, in zipline._protocol.check_parameters.__call__.assert_keywords_and_call (zipline/_protocol.c:3824)
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "zipline/_protocol.pyx", line 773, in zipline._protocol.BarData.history (zipline/_protocol.c:10268)
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_trade_data_portal_py", line 39, in get_history_window
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/data/data_portal.py", line 964, in get_history_window
2021-01-15 11:29:42 quantrocket.zipline: ERROR     "close", data_frequency)
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_trade_data_portal_py", line 105, in _get_history_daily_window
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/trading_calendars/trading_calendar.py", line 1060, in minute_index_to_session_labels
2021-01-15 11:29:42 quantrocket.zipline: ERROR     next_open=next_open, next_close=next_close)
2021-01-15 11:29:42 quantrocket.zipline: ERROR ValueError: 41 non-market minutes in minute_index_to_session_labels:
2021-01-15 11:29:42 quantrocket.zipline: ERROR First Bad Minute: 2021-01-15 13:50:00+00:00
2021-01-15 11:29:42 quantrocket.zipline: ERROR Previous Session: 2021-01-14 14:31:00 -> 2021-01-14 21:00:00
2021-01-15 11:29:42 quantrocket.zipline: ERROR Next Session: 2021-01-15 14:31:00 -> 2021-01-15 21:00:00
2021-01-15 11:29:42 quantrocket.zipline: ERROR 

The line of code from my strategy ‘divLive’ referenced in the traceback (line 407) is:

context.daily_hist = data.history(context.assets, ["price", "open", "high", "low", "close", "volume"], 150, "1d")

I reproduced this bug in a clean test strategy file with a different set of assets, so I don’t think it’s my code. I’ve also tried pulling the history of just a single asset and get the same error.

This topic was automatically closed after 2 days. New replies are no longer allowed.