Minute bundle update crash under 2.6

kamran.sokhanvari · June 28, 2021, 11:51pm

I am getting the below error log messages on trying to update the minute bundle. The process starts fine and somewhere afterwards it fails.

Any ideas why this is happening. It's a new issue as I have had no problems in the past updating this bundle.

Thanks

2021-06-28 23:33:07 quantrocket.zipline: INFO [usstock-1min] Ingesting minute bars for 10728 securities in usstock-1min bundle
2021-06-28 23:39:42 quantrocket.zipline: ERROR Traceback (most recent call last):
2021-06-28 23:39:42 quantrocket.zipline: ERROR File "sym://qrocket_log_py", line 34, in wrapped
2021-06-28 23:39:42 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_ingest_py", line 77, in mule_ingest_bundle
2021-06-28 23:39:42 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_ingest_py", line 116, in _call_ingest_func
2021-06-28 23:39:42 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 98, in ingest
2021-06-28 23:39:42 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 650, in ingest
2021-06-28 23:39:42 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 684, in _wait_for_minute_workers
2021-06-28 23:39:42 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.8/threading.py", line 932, in _bootstrap_inner
2021-06-28 23:39:42 quantrocket.zipline: ERROR self.run()
2021-06-28 23:39:42 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.8/threading.py", line 870, in run
2021-06-28 23:39:42 quantrocket.zipline: ERROR self._target(*self._args, **self._kwargs)
2021-06-28 23:39:42 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 504, in wrapper
2021-06-28 23:39:42 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 801, in _ingest_minute_prices_for_sid
2021-06-28 23:39:42 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.8/site-packages/zipline/data/minute_bars.py", line 696, in write_sid
2021-06-28 23:39:42 quantrocket.zipline: ERROR self._write_cols(sid, dts, cols, invalid_data_behavior)
2021-06-28 23:39:42 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.8/site-packages/zipline/data/minute_bars.py", line 747, in _write_cols
2021-06-28 23:39:42 quantrocket.zipline: ERROR table = self._ensure_ctable(sid)
2021-06-28 23:39:42 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.8/site-packages/zipline/data/minute_bars.py", line 579, in _ensure_ctable
2021-06-28 23:39:42 quantrocket.zipline: ERROR return bcolz.ctable(rootdir=sidpath, mode='a')
2021-06-28 23:39:42 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.8/site-packages/bcolz/ctable.py", line 268, in init
2021-06-28 23:39:42 quantrocket.zipline: ERROR self._open_ctable()
2021-06-28 23:39:42 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.8/site-packages/bcolz/ctable.py", line 366, in _open_ctable
2021-06-28 23:39:42 quantrocket.zipline: ERROR self.cols.read_meta_and_open()
2021-06-28 23:39:42 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.8/site-packages/bcolz/ctable.py", line 65, in read_meta_and_open
2021-06-28 23:39:42 quantrocket.zipline: ERROR with open(rootsfile, 'rb') as rfile:
2021-06-28 23:39:42 quantrocket.zipline: ERROR FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/quantrocket/zipline/data/usstock-1min/2020-01-01T00;00;00/minute_equities.bcolz/02/05/020562.bcolz/rootdirs'
2021-06-28 23:39:42 quantrocket.zipline: ERROR>

detail log

quantrocket_zipline_1|Exception in thread zipline_minute_ingester_1:
quantrocket_zipline_1|Traceback (most recent call last):
quantrocket_zipline_1| File "/opt/conda/lib/python3.8/threading.py", line 932, in _bootstrap_inner
quantrocket_zipline_1| self.run()
quantrocket_zipline_1| File "/opt/conda/lib/python3.8/threading.py", line 870, in run
quantrocket_zipline_1| self._target(*self._args, **self._kwargs)
quantrocket_zipline_1| File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 504, in wrapper
quantrocket_zipline_1|exiting due to exception in another worker
quantrocket_zipline_1| File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 801, in _ingest_minute_prices_for_sid
quantrocket_zipline_1| File "/opt/conda/lib/python3.8/site-packages/zipline/data/minute_bars.py", line 696, in write_sid
quantrocket_zipline_1| self._write_cols(sid, dts, cols, invalid_data_behavior)
quantrocket_zipline_1| File "/opt/conda/lib/python3.8/site-packages/zipline/data/minute_bars.py", line 747, in _write_cols
quantrocket_zipline_1| table = self._ensure_ctable(sid)
quantrocket_zipline_1| File "/opt/conda/lib/python3.8/site-packages/zipline/data/minute_bars.py", line 579, in _ensure_ctable
quantrocket_zipline_1| return bcolz.ctable(rootdir=sidpath, mode='a')
quantrocket_zipline_1| File "/opt/conda/lib/python3.8/site-packages/bcolz/ctable.py", line 268, in init
quantrocket_zipline_1| self._open_ctable()
quantrocket_zipline_1| File "/opt/conda/lib/python3.8/site-packages/bcolz/ctable.py", line 366, in _open_ctable
quantrocket_zipline_1| self.cols.read_meta_and_open()
quantrocket_zipline_1| File "/opt/conda/lib/python3.8/site-packages/bcolz/ctable.py", line 65, in read_meta_and_open
quantrocket_zipline_1| with open(rootsfile, 'rb') as rfile:
quantrocket_zipline_1|FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/quantrocket/zipline/data/usstock-1min/2020-01-01T00;00;00/minute_equities.bcolz/02/05/020562.bcolz/rootdirs'
quantrocket_zipline_1|exiting due to exception in another worker
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR Traceback (most recent call last):
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR File "sym://qrocket_log_py", line 34, in wrapped
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_ingest_py", line 77, in mule_ingest_bundle
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_ingest_py", line 116, in _call_ingest_func
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 98, in ingest
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 650, in ingest
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 684, in _wait_for_minute_workers
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.8/threading.py", line 932, in _bootstrap_inner
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR self.run()
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.8/threading.py", line 870, in run
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR self._target(*self._args, **self._kwargs)
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 504, in wrapper
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 801, in _ingest_minute_prices_for_sid
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.8/site-packages/zipline/data/minute_bars.py", line 696, in write_sid
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR self._write_cols(sid, dts, cols, invalid_data_behavior)
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.8/site-packages/zipline/data/minute_bars.py", line 747, in _write_cols
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR table = self._ensure_ctable(sid)
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.8/site-packages/zipline/data/minute_bars.py", line 579, in _ensure_ctable
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR return bcolz.ctable(rootdir=sidpath, mode='a')
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.8/site-packages/bcolz/ctable.py", line 268, in init
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR self._open_ctable()
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.8/site-packages/bcolz/ctable.py", line 366, in _open_ctable
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR self.cols.read_meta_and_open()
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.8/site-packages/bcolz/ctable.py", line 65, in read_meta_and_open
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR with open(rootsfile, 'rb') as rfile:
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/quantrocket/zipline/data/usstock-1min/2020-01-01T00;00;00/minute_equities.bcolz/02/05/020562.bcolz/rootdirs'
quantrocket_houston_1|172.18.0.13 - - [28/Jun/2021:23:39:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_flightlog_1|2021-06-28 23:39:42 quantrocket.zipline: ERROR

Brian · June 29, 2021, 12:35pm

Try force ingesting that particular sid, which I think should force the missing directory to be synced:

curl -X POST 'http://houston/zipline/ingestions/usstock-1min?sids=FIBBG011F5ZCT2&force=true'

kamran.sokhanvari · June 29, 2021, 7:05pm

That worked. Thanks, Brian

bjsun · October 7, 2021, 1:56pm

I got this same bug this morning while trying to ingest the minute bundle. It came out of nowhere, nothing changed from how I ingest via cron every morning. The problem was similarly fixed by the suggested solution (with a different sid), but I'm concerned that this will happen again. Any idea what's causing it?

Error Traceback:

2021-10-07 04:48:51 quantrocket.zipline: ERROR Traceback (most recent call last):
2021-10-07 04:48:51 quantrocket.zipline: ERROR   File "sym://qrocket_log_py", line 34, in wrapped
2021-10-07 04:48:51 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_bundles_ingest_py", line 77, in mule_ingest_bundle
2021-10-07 04:48:51 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_bundles_ingest_py", line 116, in _call_ingest_func
2021-10-07 04:48:51 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 98, in ingest
2021-10-07 04:48:51 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 662, in ingest
2021-10-07 04:48:51 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 696, in _wait_for_minute_workers
2021-10-07 04:48:51 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.8/threading.py", line 932, in _bootstrap_inner
2021-10-07 04:48:51 quantrocket.zipline: ERROR     self.run()
2021-10-07 04:48:51 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.8/threading.py", line 870, in run
2021-10-07 04:48:51 quantrocket.zipline: ERROR     self._target(*self._args, **self._kwargs)
2021-10-07 04:48:51 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 504, in wrapper
2021-10-07 04:48:51 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 813, in _ingest_minute_prices_for_sid
2021-10-07 04:48:51 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.8/site-packages/zipline/data/minute_bars.py", line 696, in write_sid
2021-10-07 04:48:51 quantrocket.zipline: ERROR     self._write_cols(sid, dts, cols, invalid_data_behavior)
2021-10-07 04:48:51 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.8/site-packages/zipline/data/minute_bars.py", line 747, in _write_cols
2021-10-07 04:48:51 quantrocket.zipline: ERROR     table = self._ensure_ctable(sid)
2021-10-07 04:48:51 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.8/site-packages/zipline/data/minute_bars.py", line 579, in _ensure_ctable
2021-10-07 04:48:51 quantrocket.zipline: ERROR     return bcolz.ctable(rootdir=sidpath, mode='a')
2021-10-07 04:48:51 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.8/site-packages/bcolz/ctable.py", line 268, in __init__
2021-10-07 04:48:51 quantrocket.zipline: ERROR     self._open_ctable()
2021-10-07 04:48:51 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.8/site-packages/bcolz/ctable.py", line 366, in _open_ctable
2021-10-07 04:48:51 quantrocket.zipline: ERROR     self.cols.read_meta_and_open()
2021-10-07 04:48:51 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.8/site-packages/bcolz/ctable.py", line 65, in read_meta_and_open
2021-10-07 04:48:51 quantrocket.zipline: ERROR     with open(rootsfile, 'rb') as rfile:
2021-10-07 04:48:51 quantrocket.zipline: ERROR FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/quantrocket/zipline/data/usstock-minute/2020-01-01T00;00;00/minute_equities.bcolz/02/12/021262.bcolz/__rootdirs__'
2021-10-07 04:48:51 quantrocket.zipline: ERROR

Fixed with:

curl -X POST 'http://houston/zipline/ingestions/usstock-minute?sids=FIBBG012JCMG16&force=true'

charles · October 7, 2021, 8:04pm

Ran into the same issue during normal ingestion. Thanks for the fix @bjsun

Brian · October 8, 2021, 1:06pm

There is a fix for this in the zipline master branch. It will be in the next release but you are welcome to pull the code into your zipline container in the meantime:

pip install git+https://github.com/quantrocket-llc/zipline.git@master

After which it’s a good idea to restart (but not recreate) the zipline container to make sure the new code gets loaded.

charles · October 13, 2021, 7:09pm

Thanks @Brian, I've installed the update (after running into same error again today), and will let you know if I run into any issues.

P.S. Anyone wanting to try this patch, make sure you're on version 2.6, otherwise you'll break your zipline. And for the record, the issue mentioned in this post appears to impact zipline 2.5 as well.

@Brian is there any issue with running zipline service 2.6 with all other services still on 2.5? Not quite ready to upgrade everything, but wanted to fix this Zipline issue. Let me know!

Brian · December 16, 2021, 3:06pm

This issue is fixed in release 2.7.0:

Brian · December 23, 2021, 3:06pm

This topic was automatically closed after 7 days. New replies are no longer allowed.