Zipline live doesn't provide partial daily data

[bjsun]

Hi Brian,

As I further debug my live zipline strategy, I'm finding another inconsistency between live and backtest historical data, perhaps related to the previous issue that was just patched.

In backtest, when daily price history is retrieved using the 'data' object, zipline will fill in the data of the current trading day with partial data, even though the day is not complete. This partial day data is updated as of the last complete minute. For example:

hist = data.history(asset, ['open', 'high', 'low', 'close', 'price', 'volume'], 5, '1d')

today = hist.iloc[-1]

# This bool is always true in backtest
close_test = today['close'] == data.current(asset, 'close')

# These bools are true if the last complete minute set a new high or low for the day
new_high = today['high'] == data.current(asset, 'high')
new_low = today['low'] == data.current(asset, 'low')

In the current live implementation, the same calls will return NaN for all fields of the current day (the variable 'today' above), with the exception of 'price', which returns a value that does not correspond to the latest minute price.

Any chance this could be fixed to match backtest?

ps Happy New Year!

[Brian]

It sounds like we should be rolling up the real-time minute bars to produce the partial day bar. We can address that in the next release. Thanks for sharing.

[Brian]

This is fixed in version 2.4.0:

QuantRocket 2.4.0 Release Notes

QuantRocket 2.4.0 is now available. This release includes a number of enhancements related to real-time data, the US Stock dataset, and feature parity with Quantopian. All version 2 users are encouraged to update to the latest version. See how to update. Highlights There are two new additions to the Code Library: Quant Finance Lectures. This is the Quantopian Lecture Series updated for QuantRocket. Pipeline tutorial. This tutorial provides an in-depth walkthrough of the Pipeline API. It …

[bjsun]

After updating to v2.4.0, I'm getting an error when calling data.history() in a live strategy (doesn't throw error in backtest). This is code that worked fine in v2.3.0. Here's the traceback:

2021-01-15 11:29:42 quantrocket.zipline: ERROR Traceback (most recent call last):
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "sym://qrocket_log_py", line 34, in wrapped
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_trade_trade_py", line 63, in mule_trade_algo
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_trade_trade_py", line 187, in trade_algo
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/algorithm.py", line 669, in run
2021-01-15 11:29:42 quantrocket.zipline: ERROR     for perf in self.get_generator():
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/gens/tradesimulation.py", line 205, in transform
2021-01-15 11:29:42 quantrocket.zipline: ERROR     for capital_change_packet in every_bar(dt):
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/gens/tradesimulation.py", line 133, in every_bar
2021-01-15 11:29:42 quantrocket.zipline: ERROR     handle_data(algo, current_data, dt_to_use)
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/utils/events.py", line 218, in handle_data
2021-01-15 11:29:42 quantrocket.zipline: ERROR     dt,
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/utils/events.py", line 237, in handle_data
2021-01-15 11:29:42 quantrocket.zipline: ERROR     self.callback(context, data)
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_trade_algorithm_py", line 161, in handle_data
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/algorithm.py", line 479, in handle_data
2021-01-15 11:29:42 quantrocket.zipline: ERROR     self._handle_data(self, data)
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "divLive", line 407, in handle_data
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "zipline/_protocol.pyx", line 121, in zipline._protocol.check_parameters.__call__.assert_keywords_and_call (zipline/_protocol.c:3824)
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "zipline/_protocol.pyx", line 773, in zipline._protocol.BarData.history (zipline/_protocol.c:10268)
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_trade_data_portal_py", line 39, in get_history_window
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/data/data_portal.py", line 964, in get_history_window
2021-01-15 11:29:42 quantrocket.zipline: ERROR     "close", data_frequency)
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_trade_data_portal_py", line 105, in _get_history_daily_window
2021-01-15 11:29:42 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/trading_calendars/trading_calendar.py", line 1060, in minute_index_to_session_labels
2021-01-15 11:29:42 quantrocket.zipline: ERROR     next_open=next_open, next_close=next_close)
2021-01-15 11:29:42 quantrocket.zipline: ERROR ValueError: 41 non-market minutes in minute_index_to_session_labels:
2021-01-15 11:29:42 quantrocket.zipline: ERROR First Bad Minute: 2021-01-15 13:50:00+00:00
2021-01-15 11:29:42 quantrocket.zipline: ERROR Previous Session: 2021-01-14 14:31:00 -> 2021-01-14 21:00:00
2021-01-15 11:29:42 quantrocket.zipline: ERROR Next Session: 2021-01-15 14:31:00 -> 2021-01-15 21:00:00
2021-01-15 11:29:42 quantrocket.zipline: ERROR 

The line of code from my strategy 'divLive' referenced in the traceback (line 407) is:

context.daily_hist = data.history(context.assets, ["price", "open", "high", "low", "close", "volume"], 150, "1d")

I reproduced this bug in a clean test strategy file with a different set of assets, so I don't think it's my code. I've also tried pulling the history of just a single asset and get the same error.

[Brian]

It means you started collecting real-time data before the market opened and our test cases unfortunately did not consider that as they run during market hours. Zipline wants all the real-time data to be from minutes it knows about. For now the workaround would be to not start data collection until the market opens.

[bjsun]

Does that mean moving the collect_market_data() and set_realtime_db() calls to the first minute of handle_data() instead of before_trading_start()?

Will the first minute of data (opening minute at 9:30am) still be collected and rolled up properly?

[Brian]

We've applied a patch to the existing 2.4.0 image. To get the fix:

docker-compose pull zipline && docker-compose up -d zipline

[bjsun]

Just tried running this patch, I suppose I won't know if the intended fix works until I start the data collection before market opens tomorrow.

In the meantime, I'm getting a new error when calling minutes_in_range() from quantrocket_trading_calendars. I'm guessing it's related to the fix, since it appears to have something to do with minute prices, and this code worked fine before this patch. Here's the traceback from my strategy file:

2021-01-19 11:32:01 quantrocket.zipline: ERROR Traceback (most recent call last):
2021-01-19 11:32:01 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/trading_calendars/utils/memoize.py", line 47, in __get__
2021-01-19 11:32:01 quantrocket.zipline: ERROR     return self._cache[instance]
2021-01-19 11:32:01 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/weakref.py", line 394, in __getitem__
2021-01-19 11:32:01 quantrocket.zipline: ERROR     return self.data[ref(key)]
2021-01-19 11:32:01 quantrocket.zipline: ERROR KeyError: <weakref at 0x7f55ec42bbd8; to 'NYSEExchangeCalendar' at 0x7f55f1820278>
2021-01-19 11:32:01 quantrocket.zipline: ERROR 
2021-01-19 11:32:01 quantrocket.zipline: ERROR During handling of the above exception, another exception occurred:
2021-01-19 11:32:01 quantrocket.zipline: ERROR 
2021-01-19 11:32:01 quantrocket.zipline: ERROR Traceback (most recent call last):
2021-01-19 11:32:01 quantrocket.zipline: ERROR   File "sym://qrocket_log_py", line 34, in wrapped
2021-01-19 11:32:01 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_trade_trade_py", line 63, in mule_trade_algo
2021-01-19 11:32:01 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_trade_trade_py", line 187, in trade_algo
2021-01-19 11:32:01 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/algorithm.py", line 669, in run
2021-01-19 11:32:01 quantrocket.zipline: ERROR     for perf in self.get_generator():
2021-01-19 11:32:01 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/gens/tradesimulation.py", line 205, in transform
2021-01-19 11:32:01 quantrocket.zipline: ERROR     for capital_change_packet in every_bar(dt):
2021-01-19 11:32:01 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/gens/tradesimulation.py", line 133, in every_bar
2021-01-19 11:32:01 quantrocket.zipline: ERROR     handle_data(algo, current_data, dt_to_use)
2021-01-19 11:32:01 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/utils/events.py", line 218, in handle_data
2021-01-19 11:32:01 quantrocket.zipline: ERROR     dt,
2021-01-19 11:32:01 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/utils/events.py", line 237, in handle_data
2021-01-19 11:32:01 quantrocket.zipline: ERROR     self.callback(context, data)
2021-01-19 11:32:01 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_trade_algorithm_py", line 161, in handle_data
2021-01-19 11:32:01 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/algorithm.py", line 479, in handle_data
2021-01-19 11:32:01 quantrocket.zipline: ERROR     self._handle_data(self, data)
2021-01-19 11:32:01 quantrocket.zipline: ERROR   File "divLive", line 495, in handle_data
2021-01-19 11:32:01 quantrocket.zipline: ERROR   File "./codeload/local/custom/algo.py", line 385, in handle_hist
2021-01-19 11:32:01 quantrocket.zipline: ERROR     'rMinutes': len(context.cal.minutes_in_range(start_idx, current_time)),
2021-01-19 11:32:01 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/trading_calendars/trading_calendar.py", line 804, in minutes_in_range
2021-01-19 11:32:01 quantrocket.zipline: ERROR     return self.all_minutes[start_idx:end_idx]
2021-01-19 11:32:01 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/trading_calendars/utils/memoize.py", line 49, in __get__
2021-01-19 11:32:01 quantrocket.zipline: ERROR     self._cache[instance] = val = self._get(instance)
2021-01-19 11:32:01 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/trading_calendars/trading_calendar.py", line 944, in all_minutes
2021-01-19 11:32:01 quantrocket.zipline: ERROR     self.market_break_starts_nanos,
2021-01-19 11:32:01 quantrocket.zipline: ERROR AttributeError: 'NYSEExchangeCalendar' object has no attribute 'market_break_starts_nanos'
2021-01-19 11:32:01 quantrocket.zipline: ERROR

[Brian]

The problem is that you stored the calendar in your context which has resulted in loading an old calendar into a new version of trading_calendars which expects new attributes that yours doesn’t have. It’s a common pitfall with pickling. To avoid this, you should only store strings, integers, etc in your context and rebuild the Python objects as needed. A calendar doesn’t need to be stored in context at all since you can easily instantiate it. We are probably going to switch to storing the context as JSON to enforce not saving complex Python objects.

[bjsun]

Ah thanks for catching that. 'Context' is essentially the only global variable, so I tend to store anything used repeatedly in it. Easy enough to reinstantiate in before_trading_start().