Timestamp error while ingesting zipline bundle from IBKR database

Hi Brian,

I've gotten this error several times when ingesting a zipline bundle from the minute IBKR database that it was created from. It looks like the ingester is trying to access price data in the database for a timestamp on 7-15-2021, but the bundle was created with a 7-16-2021 start date (same as the database).

The problem goes away if I delete and re-create the bundle (with the exact same start date), then ingest. However, it pops the same error on the next ingestion, after the database collects new data from IBKR. Any idea what's happening?

quantrocket_zipline_1|Ingesting 43 rolled-up daily bars for AHLpD STK (sid FIBBG00DSQ3278)
quantrocket_zipline_1|Ingesting 780 minute bars for AHPI STK (sid FIBBG000C2RD58)
quantrocket_zipline_1|Ingesting 43 rolled-up daily bars for AHLpE STK (sid FIBBG00PXGMSB4)
quantrocket_zipline_1|Ingesting 390 minute bars for AHT STK (sid FIBBG000PRZVD1)
quantrocket_zipline_1|Exception in thread zipline_minute_ingester:
quantrocket_zipline_1|Traceback (most recent call last):
quantrocket_zipline_1|  File "pandas/_libs/index.pyx", line 444, in pandas._libs.index.DatetimeEngine.get_loc
quantrocket_zipline_1|  File "pandas/_libs/hashtable_class_helper.pxi", line 1625, in pandas._libs.hashtable.Int64HashTable.get_item
quantrocket_zipline_1|  File "pandas/_libs/hashtable_class_helper.pxi", line 1632, in pandas._libs.hashtable.Int64HashTable.get_item
quantrocket_zipline_1|KeyError: 1626307200000000000
quantrocket_zipline_1|
quantrocket_zipline_1|During handling of the above exception, another exception occurred:
quantrocket_zipline_1|
quantrocket_zipline_1|Traceback (most recent call last):
quantrocket_zipline_1|  File "/opt/conda/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3080, in get_loc
quantrocket_zipline_1|    return self._engine.get_loc(casted_key)
quantrocket_zipline_1|  File "pandas/_libs/index.pyx", line 413, in pandas._libs.index.DatetimeEngine.get_loc
quantrocket_zipline_1|  File "pandas/_libs/index.pyx", line 446, in pandas._libs.index.DatetimeEngine.get_loc
quantrocket_zipline_1|KeyError: Timestamp('2021-07-15 00:00:00+0000', tz='UTC')
quantrocket_zipline_1|
quantrocket_zipline_1|The above exception was the direct cause of the following exception:
quantrocket_zipline_1|
quantrocket_zipline_1|Traceback (most recent call last):
quantrocket_zipline_1|  File "/opt/conda/lib/python3.8/site-packages/pandas/core/indexes/datetimes.py", line 686, in get_loc
quantrocket_zipline_1|    return Index.get_loc(self, key, method, tolerance)
quantrocket_zipline_1|  File "/opt/conda/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3082, in get_loc
quantrocket_zipline_1|    raise KeyError(key) from err
quantrocket_zipline_1|KeyError: Timestamp('2021-07-15 00:00:00+0000', tz='UTC')
quantrocket_zipline_1|
quantrocket_zipline_1|The above exception was the direct cause of the following exception:
quantrocket_zipline_1|
quantrocket_zipline_1|Traceback (most recent call last):
quantrocket_zipline_1|  File "/opt/conda/lib/python3.8/threading.py", line 932, in _bootstrap_inner
quantrocket_zipline_1|    self.run()
quantrocket_zipline_1|  File "/opt/conda/lib/python3.8/threading.py", line 870, in run
quantrocket_zipline_1|    self._target(*self._args, **self._kwargs)
quantrocket_zipline_1|  File "sym://qrocket_qrzipline_bundles_fromdb_minute_py", line 270, in _consume_prices
quantrocket_zipline_1|  File "/opt/conda/lib/python3.8/site-packages/zipline/data/minute_bars.py", line 819, in data_len_for_day
quantrocket_zipline_1|    day_ix = self._session_labels.get_loc(day)
quantrocket_zipline_1|  File "/opt/conda/lib/python3.8/site-packages/pandas/core/indexes/datetimes.py", line 688, in get_loc
quantrocket_zipline_1|    raise KeyError(orig_key) from err
quantrocket_zipline_1|KeyError: Timestamp('2021-07-15 00:00:00+0000', tz='UTC')
quantrocket_zipline_1|Ingesting 43 rolled-up daily bars for AHPI STK (sid FIBBG000C2RD58)
quantrocket_zipline_1|No price data to ingest for AHTpF STK (sid FIBBG00D8Y9RD5)
quantrocket_zipline_1|No price data to ingest for AHTpG STK (sid FIBBG00F0GJH93)

Can you provide simplified code to reproduce the error, with just a few sids? I am not able to reproduce it using the sids in your output.

This error happens consistently when ingesting after database update, but with a different sid each time. The only way to fix it is to delete the bundle, recreate it, and ingest from scratch. For now, I've scheduled this delete/recreate/ingest process on the daily cron as a workaround.

I'll try to isolate a situation to reproduce, but the instigating sid seems to depend on the day.