Failure when trying to ingest new zipline minute bundle

I tried re-ingesting the entire minute bundle this weekend, per the recent QuantRocket email. I’m running v2.5.0.

First I tried to delete the existing zipline bundle, which resulted in the following error message:

drop_bundle("usstock-minute", confirm_by_typing_bundle_code_again="usstock-minute")

---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
<ipython-input-14-d0f909364bbd> in <module>
      1 from quantrocket.zipline import drop_bundle
----> 2 drop_bundle("usstock-minute", confirm_by_typing_bundle_code_again="usstock-minute")

/opt/conda/lib/python3.8/site-packages/quantrocket/zipline.py in drop_bundle(code, confirm_by_typing_bundle_code_again)
    313     response = houston.delete("/zipline/bundles/{}".format(code), params=params, timeout=6*60*60)
    314 
--> 315     houston.raise_for_status_with_json(response)
    316     return response.json()
    317 

/opt/conda/lib/python3.8/site-packages/quantrocket/houston.py in raise_for_status_with_json(response)
    204                 e.json_response = {}
    205                 e.args = e.args + ("please check the logs for more details",)
--> 206             raise e
    207 
    208 # Instantiate houston so that all callers can share a TCP connection (for

/opt/conda/lib/python3.8/site-packages/quantrocket/houston.py in raise_for_status_with_json(response)
    196         """
    197         try:
--> 198             response.raise_for_status()
    199         except requests.exceptions.HTTPError as e:
    200             try:

/opt/conda/lib/python3.8/site-packages/requests/models.py in raise_for_status(self)
    939 
    940         if http_error_msg:
--> 941             raise HTTPError(http_error_msg, response=self)
    942 
    943     def close(self):

HTTPError: ('500 Server Error: Internal Server Error for url: http://houston/zipline/bundles/usstock-minute?confirm_by_typing_bundle_code_again=usstock-minute', {'status': 'error', 'msg': 'an unhandled exception occurred, please check flightlog for the traceback and notify support if the problem persists as this might be a bug', 'error': "[Errno 74] Bad message: '/var/lib/quantrocket/zipline/data/usstock-minute/2020-01-01T00;00;00/minute_equities.bcolz/01/31/013128.bcolz'"})

The traceback from flightlog:

2021-04-05 23:53:19 quantrocket.zipline: ERROR Traceback (most recent call last):
2021-04-05 23:53:19 urllib3.connectionpool: DEBUG Resetting dropped connection: houston
2021-04-05 23:53:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR   File "sym://qrocket_wsgi_py", line 41, in app
2021-04-05 23:53:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1994, in __call__
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR     return self.wsgi_app(environ, start_response)
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1985, in wsgi_app
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR     response = self.handle_exception(e)
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask_restful/__init__.py", line 271, in error_router
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR     return original_handler(e)
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1540, in handle_exception
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR     reraise(exc_type, exc_value, tb)
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask/_compat.py", line 32, in reraise
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR     raise value.with_traceback(tb)
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR     response = self.full_dispatch_request()
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR     rv = self.handle_user_exception(e)
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask_restful/__init__.py", line 271, in error_router
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR     return original_handler(e)
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR     reraise(exc_type, exc_value, tb)
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask/_compat.py", line 32, in reraise
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR     raise value.with_traceback(tb)
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR     rv = self.dispatch_request()
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR     return self.view_functions[rule.endpoint](**req.view_args)
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask_restful/__init__.py", line 477, in wrapper
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR     resp = resource(*args, **kwargs)
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask/views.py", line 84, in view
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR     return self.dispatch_request(*args, **kwargs)
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask_restful/__init__.py", line 587, in dispatch_request
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR     resp = meth(*args, **kwargs)
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/webargs/core.py", line 441, in wrapper
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR     return func(*args, **kwargs)
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR   File "sym://qrocket_app_py", line 369, in delete
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/shutil.py", line 480, in rmtree
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR     _rmtree_safe_fd(fd, path, onerror)
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/shutil.py", line 418, in _rmtree_safe_fd
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR     _rmtree_safe_fd(dirfd, fullname, onerror)
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/shutil.py", line 402, in _rmtree_safe_fd
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR     onerror(os.listdir, path, sys.exc_info())
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/shutil.py", line 399, in _rmtree_safe_fd
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR     names = os.listdir(topfd)
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR OSError: [Errno 74] Bad message: '/var/lib/quantrocket/zipline/data/usstock-minute/2020-01-01T00;00;00/minute_equities.bcolz/01/31/013128.bcolz'
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR 
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20

After this error, the bundle no longer appeared to be registered (which I checked with list_bundles()). So I re-created the bundled and tried to ingest it. I got this error mid-way through:

quantrocket_zipline_1|Ingesting minute bars for FIBBG000BWJYL8 (4689 of 19936)
quantrocket_zipline_1|Ingesting minute bars for FIBBG000BWKB81 (4690 of 19936)
quantrocket_zipline_1|Ingesting minute bars for FIBBG000BWKQX0 (4691 of 19936)
quantrocket_zipline_1|Ingesting minute bars for FIBBG000BWKX65 (4692 of 19936)
quantrocket_zipline_1|Ingesting minute bars for FIBBG000BWKXD7 (4693 of 19936)
quantrocket_zipline_1|Ingesting minute bars for FIBBG000BWKXN6 (4694 of 19936)
quantrocket_zipline_1|Ingesting minute bars for FIBBG000BWL1D6 (4695 of 19936)
quantrocket_zipline_1|Exception in thread zipline_minute_sync_1:
quantrocket_zipline_1|Traceback (most recent call last):
quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/threading.py", line 916, in _bootstrap_inner
quantrocket_zipline_1|    self.run()
quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/threading.py", line 864, in run
quantrocket_zipline_1|    self._target(*self._args, **self._kwargs)
quantrocket_zipline_1|  File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 519, in wrapper
quantrocket_zipline_1|  File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 618, in _sync_minute_bars_for_sid
quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/subprocess.py", line 291, in check_call
quantrocket_zipline_1|    raise CalledProcessError(retcode, cmd)
quantrocket_zipline_1|subprocess.CalledProcessError: Command '['aws', 's3', 'sync', '--quiet', '--exact-timestamps', '--delete', 's3://data.quantrocket.com/usstock/zipline/usstock-1min/minute_equities.bcolz/01/25/012544.bcolz', '/var/lib/quantrocket/zipline/data/usstock-minute/2020-01-01T00;00;00/minute_equities.bcolz/01/25/012544.bcolz']' returned non-zero exit status 1.
quantrocket_zipline_1|
quantrocket_zipline_1|exiting due to exception in another worker

I’ve tried recreating the zipline bundle, with no success.

Try manually deleting the bundle data (will take a while):

rm -r /var/lib/quantrocket/zipline/data/usstock-minute

This worked, thanks.