Error ingesting zipline bundle from IBKR database

bjsun · November 23, 2021, 1:51am

Hi Brian,

Since about two weeks ago, I've been getting this error when trying to ingest my IBKR minute database into a zipline bundle:

quantrocket_houston_1|172.18.0.2 - - [23/Nov/2021:01:42:28 +0000] "GET /zipline/bundles HTTP/1.1" 200 52 "-" "-"
quantrocket_houston_1|172.18.0.2 - - [23/Nov/2021:01:42:28 +0000] "GET /zipline/bundles HTTP/1.1" 200 52 "-" "-"
quantrocket_houston_1|172.18.0.2 - - [23/Nov/2021:01:42:29 +0000] "PUT /zipline/bundles/ibkr-minute-bundle?ingest_type=from_db&from_db=ibkr-minute-db&calendar=NYSE&start_date=09-28-2021 HTTP/1.1" 200 79 "-" "-"
quantrocket_houston_1|172.18.0.2 - - [23/Nov/2021:01:42:29 +0000] "POST /zipline/ingestions/ibkr-minute-bundle HTTP/1.1" 202 55 "-" "-"
quantrocket_flightlog_1|2021-11-22 17:42:29 quantrocket.zipline: INFO [ibkr-minute-bundle] Ingesting ibkr-minute-bundle bundle
quantrocket_zipline_1|Ingesting 13260 minute bars for A STK (sid FIBBG000C2V3D6)
quantrocket_zipline_1|error ingesting A STK (sid FIBBG000C2V3D6)
quantrocket_zipline_1|Traceback (most recent call last):
quantrocket_zipline_1|  File "sym://qrocket_qrzipline_bundles_fromdb_minute_py", line 279, in _consume_prices
quantrocket_zipline_1|  File "/opt/conda/lib/python3.8/site-packages/zipline/data/resample.py", line 65, in minute_frame_to_session_frame
quantrocket_zipline_1|    labels = calendar.minute_index_to_session_labels(minute_frame.index)
quantrocket_zipline_1|  File "/opt/conda/lib/python3.8/site-packages/trading_calendars/trading_calendar.py", line 1053, in minute_index_to_session_labels
quantrocket_zipline_1|    self._opens.values.searchsorted(index.values, side="right") - 1
quantrocket_zipline_1|TypeError: '<' not supported between instances of 'int' and 'Timestamp'
quantrocket_zipline_1|
quantrocket_flightlog_1|2021-11-22 17:42:49 quantrocket.zipline: ERROR error ingesting A STK (sid FIBBG000C2V3D6), see detailed logs for traceback, continuing with next security
quantrocket_zipline_1|Ingesting 13260 minute bars for AA STK (sid FIBBG00B3T3HD3)
quantrocket_zipline_1|error ingesting AA STK (sid FIBBG00B3T3HD3)
quantrocket_zipline_1|Traceback (most recent call last):
quantrocket_zipline_1|  File "sym://qrocket_qrzipline_bundles_fromdb_minute_py", line 279, in _consume_prices
quantrocket_zipline_1|  File "/opt/conda/lib/python3.8/site-packages/zipline/data/resample.py", line 65, in minute_frame_to_session_frame
quantrocket_zipline_1|    labels = calendar.minute_index_to_session_labels(minute_frame.index)
quantrocket_zipline_1|  File "/opt/conda/lib/python3.8/site-packages/trading_calendars/trading_calendar.py", line 1053, in minute_index_to_session_labels
quantrocket_zipline_1|    self._opens.values.searchsorted(index.values, side="right") - 1
quantrocket_zipline_1|TypeError: '<' not supported between instances of 'int' and 'Timestamp'
quantrocket_zipline_1|
quantrocket_flightlog_1|2021-11-22 17:43:04 quantrocket.zipline: ERROR error ingesting AA STK (sid FIBBG00B3T3HD3), see detailed logs for traceback, continuing with next security
quantrocket_zipline_1|No price data to ingest for AAAU STK (sid FIBBG00LPXX872)
quantrocket_zipline_1|Ingesting 8238 minute bars for AAA STK (sid FIBBG00X5FSP48)
quantrocket_zipline_1|error ingesting AAA STK (sid FIBBG00X5FSP48)
quantrocket_zipline_1|Traceback (most recent call last):
quantrocket_zipline_1|  File "sym://qrocket_qrzipline_bundles_fromdb_minute_py", line 279, in _consume_prices
quantrocket_zipline_1|  File "/opt/conda/lib/python3.8/site-packages/zipline/data/resample.py", line 65, in minute_frame_to_session_frame
quantrocket_zipline_1|    labels = calendar.minute_index_to_session_labels(minute_frame.index)
quantrocket_zipline_1|  File "/opt/conda/lib/python3.8/site-packages/trading_calendars/trading_calendar.py", line 1053, in minute_index_to_session_labels
quantrocket_zipline_1|    self._opens.values.searchsorted(index.values, side="right") - 1
quantrocket_zipline_1|TypeError: '<' not supported between instances of 'int' and 'Timestamp'
quantrocket_zipline_1|
quantrocket_flightlog_1|2021-11-22 17:43:19 quantrocket.zipline: ERROR error ingesting AAA STK (sid FIBBG00X5FSP48), see detailed logs for traceback, continuing with next security
quantrocket_zipline_1|Ingesting 13260 minute bars for AAAU STK (sid FIBBG00QFQ8FL3)
quantrocket_zipline_1|error ingesting AAAU STK (sid FIBBG00QFQ8FL3)
quantrocket_zipline_1|Traceback (most recent call last):
quantrocket_zipline_1|  File "sym://qrocket_qrzipline_bundles_fromdb_minute_py", line 279, in _consume_prices
quantrocket_zipline_1|  File "/opt/conda/lib/python3.8/site-packages/zipline/data/resample.py", line 65, in minute_frame_to_session_frame
quantrocket_zipline_1|    labels = calendar.minute_index_to_session_labels(minute_frame.index)
quantrocket_zipline_1|  File "/opt/conda/lib/python3.8/site-packages/trading_calendars/trading_calendar.py", line 1053, in minute_index_to_session_labels
quantrocket_zipline_1|    self._opens.values.searchsorted(index.values, side="right") - 1
quantrocket_zipline_1|TypeError: '<' not supported between instances of 'int' and 'Timestamp'
quantrocket_zipline_1|

As you can see, the same error happens for every security. If I restore my IBKR database from before this started happening, the ingestion is successful. This suggests that the ingestion system is intact and that the error is caused by the newly collected IBKR data. Any idea what's happening?

I'm unable to maintain a backup bundle using IBKR data while this error persists, so any thoughts would be much appreciated.

Brian · November 24, 2021, 1:49pm

Have you looked at the newly collected IBKR data to see if anything looks different? The traceback shows the IBKR minute data datetime index being used to search the calendar, and it fails because it's trying to compare ints and Timestamps. I would assume the problem is with the data not the calendar, so maybe there are ints where there should be dates? As always, a small toy example that reproduces the issue would be helpful.

bjsun · November 29, 2021, 6:28am

The following should recreate the issue for a single sid:

from quantrocket import get_prices
from quantrocket.history import create_ibkr_db, collect_history
from quantrocket.zipline import create_bundle_from_db, ingest_bundle

# Create a test database
create_ibkr_db('test-ibkr-db', sids='FIBBG00B3T3HD3', bar_size="1 min", shard='sid')

# Collect historical data for single sid
collect_history('test-ibkr-db', start_date='11-01-2021')

# Verify that data was collected - this should return normal data
get_prices('test-ibkr-db', sids='FIBBG00B3T3HD3', start_date='2021-11-01', end_date='2021-11-02', fields=['Open', 'High', 'Low', 'Close', 'Volume'], data_frequency='minute')

# Create a zipline bundle based on the test database
create_bundle_from_db('test-bundle', 
                      from_db='test-ibkr-db', 
                      calendar="NYSE", 
                      sids='FIBBG00B3T3HD3',
                      start_date="2021-11-01")

# This raises the exception in the zipline container
ingest_bundle('test-bundle')

And here's the logged exception again:

quantrocket_houston_1|172.18.0.2 - - [29/Nov/2021:06:16:44 +0000] "PUT /zipline/bundles/test-bundle?ingest_type=from_db&from_db=test-ibkr-db&calendar=NYSE&start_date=2021-11-01&sids=FIBBG00B3T3HD3 HTTP/1.1" 200 72 "-" "-"
quantrocket_houston_1|172.18.0.2 - - [29/Nov/2021:06:16:45 +0000] "POST /zipline/ingestions/test-bundle HTTP/1.1" 202 55 "-" "-"
quantrocket_flightlog_1|2021-11-28 22:16:46 quantrocket.zipline: INFO [test-bundle] Ingesting test-bundle bundle
quantrocket_zipline_1|Ingesting 7230 minute bars for AA STK (sid FIBBG00B3T3HD3)
quantrocket_zipline_1|error ingesting AA STK (sid FIBBG00B3T3HD3)
quantrocket_zipline_1|Traceback (most recent call last):
quantrocket_zipline_1|  File "sym://qrocket_qrzipline_bundles_fromdb_minute_py", line 279, in _consume_prices
quantrocket_zipline_1|  File "/opt/conda/lib/python3.8/site-packages/zipline/data/resample.py", line 65, in minute_frame_to_session_frame
quantrocket_zipline_1|    labels = calendar.minute_index_to_session_labels(minute_frame.index)
quantrocket_zipline_1|  File "/opt/conda/lib/python3.8/site-packages/trading_calendars/trading_calendar.py", line 1053, in minute_index_to_session_labels
quantrocket_zipline_1|    self._opens.values.searchsorted(index.values, side="right") - 1
quantrocket_zipline_1|TypeError: '<' not supported between instances of 'int' and 'Timestamp'
quantrocket_zipline_1|
quantrocket_flightlog_1|2021-11-28 22:17:00 quantrocket.zipline: ERROR error ingesting AA STK (sid FIBBG00B3T3HD3), see detailed logs for traceback, continuing with next security
quantrocket_flightlog_1|2021-11-28 22:17:00 quantrocket.zipline: ERROR skipped 1 securities with errors
quantrocket_flightlog_1|2021-11-28 22:17:00 quantrocket.zipline: ERROR test-bundle bundle ingestion failed: No data found in test-ibkr-db matching the ingestion parameters

Brian · November 30, 2021, 1:11pm

Thank you for the sample code, I was able to reproduce the error. Pandas is not properly casting the minute index to datetimes when the query crosses over the threshold of daylight savings time, which occurred on November 7. This regression probably arose with the pandas update in version 2.6.0. A fix will be included in the next release.

Brian · December 16, 2021, 3:03pm

This issue is fixed in release 2.7.0:

Brian · December 20, 2021, 1:00pm

This topic was automatically closed after 3 days. New replies are no longer allowed.