Zipline backtests fail with timestamp key error

Technical Help
#1

Hi,
Couple of days ago all zipline backtests started to fail with following error:

HTTPError: ('500 Server Error: INTERNAL SERVER ERROR for url: http://houston/zipline/backtests/vrp_zl?data_frequency=daily&bundle=usstock-1min&start_date=2011-10-05&end_date=2021-07-23&progress=M', {'status': 'error', 'msg': "Timestamp('2022-08-19 00:00:00+0000', tz='UTC')"})

Detailed logs:
Traceback (most recent call last):
quantrocket_zipline_1| File "sym://qrocket_app_py", line 795, in post
quantrocket_zipline_1| File "sym://qrocket_qrzipline_backtest_py", line 115, in backtest_algo
quantrocket_zipline_1| File "/opt/conda/lib/python3.8/site-packages/zipline/data/data_portal.py", line 181, in init
quantrocket_zipline_1| last_minutes = [
quantrocket_zipline_1| File "/opt/conda/lib/python3.8/site-packages/zipline/data/data_portal.py", line 182, in
quantrocket_zipline_1| reader.last_available_dt
quantrocket_zipline_1| File "/opt/conda/lib/python3.8/site-packages/trading_calendars/utils/memoize.py", line 49, in get
quantrocket_zipline_1| self._cache[instance] = val = self._get(instance)
quantrocket_zipline_1| File "/opt/conda/lib/python3.8/site-packages/zipline/data/minute_bars.py", line 956, in last_available_dt
quantrocket_zipline_1| _, close = self.calendar.open_and_close_for_session(self._end_session)
quantrocket_zipline_1| File "/opt/conda/lib/python3.8/site-packages/trading_calendars/trading_calendar.py", line 848, in open_and_close_for_session
quantrocket_zipline_1| self.session_open(session_label),
quantrocket_zipline_1| File "/opt/conda/lib/python3.8/site-packages/trading_calendars/trading_calendar.py", line 873, in session_open
quantrocket_zipline_1| return self.schedule.at[session_label, "market_open"].tz_localize(UTC)
quantrocket_zipline_1| File "/opt/conda/lib/python3.8/site-packages/pandas/core/indexing.py", line 2156, in getitem
quantrocket_zipline_1| return super().getitem(key)
quantrocket_zipline_1| File "/opt/conda/lib/python3.8/site-packages/pandas/core/indexing.py", line 2103, in getitem
quantrocket_zipline_1| return self.obj._get_value(*key, takeable=self._takeable)
quantrocket_zipline_1| File "/opt/conda/lib/python3.8/site-packages/pandas/core/frame.py", line 3144, in _get_value
quantrocket_zipline_1| index = self.index.get_loc(index)
quantrocket_zipline_1| File "/opt/conda/lib/python3.8/site-packages/pandas/core/indexes/datetimes.py", line 688, in get_loc
quantrocket_zipline_1| raise KeyError(orig_key) from err
quantrocket_zipline_1|KeyError: Timestamp('2022-08-19 00:00:00+0000', tz='UTC')

Is seems that zipline comes across wrong date in the future ('2022-08-19') and fails to retrieve trading calendar data for this date. I tried to use different bundles, run backtests with different dates, recreated usstock bundles, .... without success.
BTW all zipline template strategies (sell_gap, winners) also fail to run with the same error.

Zipline "might be a bug" error
#2

I get a similar error sometimes when I try to use a bundle to create a zipline data object in jupyter, after the bundle has been updated with an ingest. It goes away if I restart the jupyter kernel - have you tried restarting the zipline container?

#3

@bjsun, thanx for advice! jupyter restart didn't help. But zipline container did.

#4

@bjsun big thanks for posting this solution, was banging my head against the wall all morning trying to figure out why this happened out of nowhere. I literally made no changes to my implementation, just suddenly woke up to this error after last night's ingestion.

@Brian this seems to be a creeping bug that happens intermittently after Zipline service has been running for a while. Any advice on preventing this issue from occurring in the future? Had I not been watching the markets today, I would have missed two really good trades due to my algo crashing without notice. At this point, I'm considering a scheduled restart of all services each weekend. Thoughts?

#5

Zipline workers recycle themselves every 24 hours, in part to avoid this issue, but based on the evidence of this ticket, it doesn't seem to be fully resolved...

#6

I still get this type of error now and then, seems to happen when a zipline bundle has previously been used to pull prices and is used again by the same code after bundle ingestion.

Is there something about the bundle config that stays in container memory after first use? E.g. there seems to be something about memoize/cache and last_available_dt in the traceback above.

1 Like
#7

For the record I'm on version 2.5, so not sure the 24 restart is included. That said, I've setup a scheduled job on my host to restart Zipline each weekend; may be redundant if it's already setup to restart nightly.