INGEST crashes at random

Technical Help
#1

Hi Brian,

I think I'm experiencing something that is a deeper bug than in my local system as I get it now local and in the cloud. Somehow when I start the Ingest it runs and then it crashes with the same error but at a different location. First one at item 86, then 100, then 158. Below the print of the detail log and under that the main log.  Any Idea on what to do?


quantrocket_houston_1|172.18.0.11 - - [21/Jun/2021:12:14:15 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 3 "-" "-"
        quantrocket_zipline_1|Ingesting minute bars for FIBBG000BB6G37 (156 of 10813)
        quantrocket_zipline_1|Ingesting minute bars for FIBBG000BB6H99 (157 of 10813)
        quantrocket_zipline_1|Exception in thread zipline_minute_sync_2:
        quantrocket_zipline_1|Traceback (most recent call last):
        quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/threading.py", line 916, in _bootstrap_inner
        quantrocket_zipline_1|    self.run()
        quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/threading.py", line 864, in run
        quantrocket_zipline_1|    self._target(*self._args, **self._kwargs)
        quantrocket_zipline_1|  File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 519, in wrapper
        quantrocket_zipline_1|  File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 618, in _sync_minute_bars_for_sid
        quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/subprocess.py", line 291, in check_call
        quantrocket_zipline_1|    raise CalledProcessError(retcode, cmd)
        quantrocket_zipline_1|subprocess.CalledProcessError: Command '['aws', 's3', 'sync', '--quiet', '--exact-timestamps', '--delete', 's3://data.quantrocket.com/usstock/zipline/usstock-1min/minute_equities.bcolz/00/02/000253.bcolz', '/var/lib/quantrocket/zipline/data/usstock-1min/2020-01-01T00;00;00/minute_equities.bcolz/00/02/000253.bcolz']' returned non-zero exit status 1.
        quantrocket_zipline_1|
        quantrocket_zipline_1|exiting due to exception in another worker

main log:

2021-06-21 12:14:21 quantrocket.zipline: ERROR Traceback (most recent call last):
2021-06-21 12:14:21 quantrocket.zipline: ERROR   File "sym://qrocket_log_py", line 34, in wrapped
2021-06-21 12:14:21 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_bundles_ingest_py", line 77, in mule_ingest_bundle
2021-06-21 12:14:21 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_bundles_ingest_py", line 126, in _call_ingest_func
2021-06-21 12:14:21 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 107, in ingest
2021-06-21 12:14:21 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 553, in ingest
2021-06-21 12:14:21 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 576, in _wait_for_minute_workers
2021-06-21 12:14:21 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/threading.py", line 916, in _bootstrap_inner
2021-06-21 12:14:21 quantrocket.zipline: ERROR     self.run()
2021-06-21 12:14:21 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/threading.py", line 864, in run
2021-06-21 12:14:21 quantrocket.zipline: ERROR     self._target(*self._args, **self._kwargs)
2021-06-21 12:14:21 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 519, in wrapper
2021-06-21 12:14:21 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 618, in _sync_minute_bars_for_sid
2021-06-21 12:14:21 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/subprocess.py", line 291, in check_call
2021-06-21 12:14:21 quantrocket.zipline: ERROR     raise CalledProcessError(retcode, cmd)
2021-06-21 12:14:21 quantrocket.zipline: ERROR subprocess.CalledProcessError: Command '['aws', 's3', 'sync', '--quiet', '--exact-timestamps', '--delete', 's3://data.quantrocket.com/usstock/zipline/usstock-1min/minute_equities.bcolz/00/02/000253.bcolz', '/var/lib/quantrocket/zipline/data/usstock-1min/2020-01-01T00;00;00/minute_equities.bcolz/00/02/000253.bcolz']' returned non-zero exit status 1.
2021-06-21 12:14:21 quantrocket.zipline: ERROR 
 
2021-06-21 12:27:22 quantrocket.zipline: INFO [usstock-1min] Ingesting minute bars for 10813 securities in usstock-1min bundle
#2

It’s likely a network issue, since it happens on different sids and happens in the call to S3. Network issues can cluster in time so you might expect to have trouble for a period of time and then see the problem clear up on its own.