I tried re-ingesting the entire minute bundle this weekend, per the recent QuantRocket email. I'm running v2.5.0.
First I tried to delete the existing zipline bundle, which resulted in the following error message:
drop_bundle("usstock-minute", confirm_by_typing_bundle_code_again="usstock-minute")
---------------------------------------------------------------------------
HTTPError Traceback (most recent call last)
<ipython-input-14-d0f909364bbd> in <module>
1 from quantrocket.zipline import drop_bundle
----> 2 drop_bundle("usstock-minute", confirm_by_typing_bundle_code_again="usstock-minute")
/opt/conda/lib/python3.8/site-packages/quantrocket/zipline.py in drop_bundle(code, confirm_by_typing_bundle_code_again)
313 response = houston.delete("/zipline/bundles/{}".format(code), params=params, timeout=6*60*60)
314
--> 315 houston.raise_for_status_with_json(response)
316 return response.json()
317
/opt/conda/lib/python3.8/site-packages/quantrocket/houston.py in raise_for_status_with_json(response)
204 e.json_response = {}
205 e.args = e.args + ("please check the logs for more details",)
--> 206 raise e
207
208 # Instantiate houston so that all callers can share a TCP connection (for
/opt/conda/lib/python3.8/site-packages/quantrocket/houston.py in raise_for_status_with_json(response)
196 """
197 try:
--> 198 response.raise_for_status()
199 except requests.exceptions.HTTPError as e:
200 try:
/opt/conda/lib/python3.8/site-packages/requests/models.py in raise_for_status(self)
939
940 if http_error_msg:
--> 941 raise HTTPError(http_error_msg, response=self)
942
943 def close(self):
HTTPError: ('500 Server Error: Internal Server Error for url: http://houston/zipline/bundles/usstock-minute?confirm_by_typing_bundle_code_again=usstock-minute', {'status': 'error', 'msg': 'an unhandled exception occurred, please check flightlog for the traceback and notify support if the problem persists as this might be a bug', 'error': "[Errno 74] Bad message: '/var/lib/quantrocket/zipline/data/usstock-minute/2020-01-01T00;00;00/minute_equities.bcolz/01/31/013128.bcolz'"})
The traceback from flightlog:
2021-04-05 23:53:19 quantrocket.zipline: ERROR Traceback (most recent call last):
2021-04-05 23:53:19 urllib3.connectionpool: DEBUG Resetting dropped connection: houston
2021-04-05 23:53:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR File "sym://qrocket_wsgi_py", line 41, in app
2021-04-05 23:53:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1994, in __call__
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR return self.wsgi_app(environ, start_response)
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1985, in wsgi_app
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR response = self.handle_exception(e)
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.6/site-packages/flask_restful/__init__.py", line 271, in error_router
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR return original_handler(e)
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1540, in handle_exception
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR reraise(exc_type, exc_value, tb)
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.6/site-packages/flask/_compat.py", line 32, in reraise
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR raise value.with_traceback(tb)
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR response = self.full_dispatch_request()
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR rv = self.handle_user_exception(e)
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.6/site-packages/flask_restful/__init__.py", line 271, in error_router
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR return original_handler(e)
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR reraise(exc_type, exc_value, tb)
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.6/site-packages/flask/_compat.py", line 32, in reraise
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR raise value.with_traceback(tb)
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR rv = self.dispatch_request()
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR return self.view_functions[rule.endpoint](**req.view_args)
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.6/site-packages/flask_restful/__init__.py", line 477, in wrapper
2021-04-05 23:53:20 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR resp = resource(*args, **kwargs)
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.6/site-packages/flask/views.py", line 84, in view
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR return self.dispatch_request(*args, **kwargs)
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.6/site-packages/flask_restful/__init__.py", line 587, in dispatch_request
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR resp = meth(*args, **kwargs)
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.6/site-packages/webargs/core.py", line 441, in wrapper
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR return func(*args, **kwargs)
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR File "sym://qrocket_app_py", line 369, in delete
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.6/shutil.py", line 480, in rmtree
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR _rmtree_safe_fd(fd, path, onerror)
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.6/shutil.py", line 418, in _rmtree_safe_fd
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR _rmtree_safe_fd(dirfd, fullname, onerror)
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.6/shutil.py", line 402, in _rmtree_safe_fd
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR onerror(os.listdir, path, sys.exc_info())
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.6/shutil.py", line 399, in _rmtree_safe_fd
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR names = os.listdir(topfd)
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR OSError: [Errno 74] Bad message: '/var/lib/quantrocket/zipline/data/usstock-minute/2020-01-01T00;00;00/minute_equities.bcolz/01/31/013128.bcolz'
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-05 23:53:19 quantrocket.zipline: ERROR
2021-04-05 23:53:21 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
After this error, the bundle no longer appeared to be registered (which I checked with list_bundles()). So I re-created the bundled and tried to ingest it. I got this error mid-way through:
quantrocket_zipline_1|Ingesting minute bars for FIBBG000BWJYL8 (4689 of 19936)
quantrocket_zipline_1|Ingesting minute bars for FIBBG000BWKB81 (4690 of 19936)
quantrocket_zipline_1|Ingesting minute bars for FIBBG000BWKQX0 (4691 of 19936)
quantrocket_zipline_1|Ingesting minute bars for FIBBG000BWKX65 (4692 of 19936)
quantrocket_zipline_1|Ingesting minute bars for FIBBG000BWKXD7 (4693 of 19936)
quantrocket_zipline_1|Ingesting minute bars for FIBBG000BWKXN6 (4694 of 19936)
quantrocket_zipline_1|Ingesting minute bars for FIBBG000BWL1D6 (4695 of 19936)
quantrocket_zipline_1|Exception in thread zipline_minute_sync_1:
quantrocket_zipline_1|Traceback (most recent call last):
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/threading.py", line 916, in _bootstrap_inner
quantrocket_zipline_1| self.run()
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/threading.py", line 864, in run
quantrocket_zipline_1| self._target(*self._args, **self._kwargs)
quantrocket_zipline_1| File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 519, in wrapper
quantrocket_zipline_1| File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 618, in _sync_minute_bars_for_sid
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/subprocess.py", line 291, in check_call
quantrocket_zipline_1| raise CalledProcessError(retcode, cmd)
quantrocket_zipline_1|subprocess.CalledProcessError: Command '['aws', 's3', 'sync', '--quiet', '--exact-timestamps', '--delete', 's3://data.quantrocket.com/usstock/zipline/usstock-1min/minute_equities.bcolz/01/25/012544.bcolz', '/var/lib/quantrocket/zipline/data/usstock-minute/2020-01-01T00;00;00/minute_equities.bcolz/01/25/012544.bcolz']' returned non-zero exit status 1.
quantrocket_zipline_1|
quantrocket_zipline_1|exiting due to exception in another worker
I've tried recreating the zipline bundle, with no success.