"No minute data for sid" error on attempt to run Zipline backtest

Hello everyone,

I am new to QuantRocket, so it is likely that I am doing something wrong, but I haven't managed to solve the issue by myself.

When I try to run a backtest, I get the following error:
quantrocket-zipline-1|Traceback (most recent call last):
quantrocket-zipline-1| File "/opt/conda/lib/python3.11/site-packages/zipline/data/minute_bars.py", line 1072, in _open_minute_file
quantrocket-zipline-1| carray = self._carrays[field][sid]
quantrocket-zipline-1| ~~~~~~~~~~~~~~~~~~~~^^^^^
quantrocket-zipline-1|KeyError: 20990
quantrocket-zipline-1|
quantrocket-zipline-1|During handling of the above exception, another exception occurred:
quantrocket-zipline-1|
quantrocket-zipline-1|Traceback (most recent call last):
quantrocket-zipline-1| File "/opt/conda/lib/python3.11/site-packages/zipline/data/minute_bars.py", line 1075, in _open_minute_file
quantrocket-zipline-1| carray = self._carrays[field][sid] = bcolz.carray(
quantrocket-zipline-1| ^^^^^^^^^^^^^
quantrocket-zipline-1| File "bcolz/carray_ext.pyx", line 1067, in bcolz.carray_ext.carray.cinit
quantrocket-zipline-1| File "bcolz/carray_ext.pyx", line 1369, in bcolz.carray_ext.carray._read_meta
quantrocket-zipline-1|FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/quantrocket/zipline/data/usstock-1min/2020-01-01T00;00;00/minute_equities.bcolz/02/09/020990.bcolz/open/meta/sizes'
quantrocket-zipline-1|
quantrocket-zipline-1|During handling of the above exception, another exception occurred:
quantrocket-zipline-1|
quantrocket-zipline-1|Traceback (most recent call last):
quantrocket-zipline-1| File "sym://qrocket_qrzipline_backtest_py", line 74, in mule_backtest_algo
quantrocket-zipline-1| File "sym://qrocket_qrzipline_backtest_py", line 253, in backtest_algo
quantrocket-zipline-1| File "/opt/conda/lib/python3.11/site-packages/zipline/algorithm.py", line 694, in run
quantrocket-zipline-1| for perf in self.get_generator():
quantrocket-zipline-1| File "/opt/conda/lib/python3.11/site-packages/zipline/gens/tradesimulation.py", line 182, in transform
quantrocket-zipline-1| for capital_change_packet in every_bar(dt):
quantrocket-zipline-1| File "/opt/conda/lib/python3.11/site-packages/zipline/gens/tradesimulation.py", line 117, in every_bar
quantrocket-zipline-1| handle_data(algo, current_data, dt_to_use)
quantrocket-zipline-1| File "/opt/conda/lib/python3.11/site-packages/zipline/utils/events.py", line 216, in handle_data
quantrocket-zipline-1| event.handle_data(
quantrocket-zipline-1| File "/opt/conda/lib/python3.11/site-packages/zipline/utils/events.py", line 238, in handle_data
quantrocket-zipline-1| self.callback(context, data)
quantrocket-zipline-1| File "/codeload/zipline/lugus.py", line 95, in find_gappers
quantrocket-zipline-1| today_opens = data.current(context.candidates.index, 'open')
quantrocket-zipline-1| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
quantrocket-zipline-1| File "zipline/_protocol.pyx", line 110, in zipline._protocol.check_parameters.call.assert_keywords_and_call
quantrocket-zipline-1| File "zipline/_protocol.pyx", line 336, in zipline._protocol.BarData.current
quantrocket-zipline-1| File "/opt/conda/lib/python3.11/site-packages/zipline/data/data_portal.py", line 401, in get_spot_value
quantrocket-zipline-1| return self._get_single_asset_value(
quantrocket-zipline-1| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
quantrocket-zipline-1| File "/opt/conda/lib/python3.11/site-packages/zipline/data/data_portal.py", line 355, in _get_single_asset_value
quantrocket-zipline-1| return self._get_minute_spot_value(asset, field, dt)
quantrocket-zipline-1| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
quantrocket-zipline-1| File "/opt/conda/lib/python3.11/site-packages/zipline/data/data_portal.py", line 570, in _get_minute_spot_value
quantrocket-zipline-1| return reader.get_value(asset.sid, dt, column)
quantrocket-zipline-1| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
quantrocket-zipline-1| File "/opt/conda/lib/python3.11/site-packages/zipline/data/dispatch_bar_reader.py", line 99, in get_value
quantrocket-zipline-1| return r.get_value(asset, dt, field)
quantrocket-zipline-1| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
quantrocket-zipline-1| File "/opt/conda/lib/python3.11/site-packages/zipline/data/minute_bars.py", line 1137, in get_value
quantrocket-zipline-1| value = self._open_minute_file(field, sid)[minute_pos]
quantrocket-zipline-1| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
quantrocket-zipline-1| File "/opt/conda/lib/python3.11/site-packages/zipline/data/minute_bars.py", line 1080, in _open_minute_file
quantrocket-zipline-1| raise NoDataForSid('No minute data for sid {}.'.format(sid))
quantrocket-zipline-1|zipline.data.bar_reader.NoDataForSid: No minute data for sid 20990.
quantrocket-zipline-1|

It sounds similar to this issue which was fixed a long time ago.

The backtest uses 1m usstock bundle

QuantRocket version: 2.10.2

UPD: It seems like the issue is reproduced when I try to run the backtest with very old data (starting from 2008-01-01), and it seems to work fine with later start date

Will be grateful for your help

I am able to reproduce this error. Investigating...

(This particular stock delisted in July 2009 which is why you only see the error with earlier start dates.)

1 Like

A fix for this will be included in the next release, but you can go ahead and pull in the fix now by running the following commands from your host machine:

docker compose exec zipline pip install git+https://github.com/quantrocket-llc/[email protected]
docker compose restart zipline

After that, you should be able to run the backtest without getting the error.

1 Like

Thank you. I can confirm that the issue is not reproduced with the patch