Calspread demo collect_history problem

Technical Help
#1

I am walking through the calspread demo, and am running into trouble with the collect_history("cl-1min-bbo") step. Everything up to this step seems to go well. I've tried to stop/start the IB gateway, and recreating containers. I've also tried running during off hours and RTH hours.

I see two types of warnings when trying to collect the history. This seems like missing historical data:

2024-02-23 15:41:28 quantrocket.history: WARNING [cl-1min-bbo] IBKR reports CLZ7 FUT (sid QF000000023490) cannot be found in their system, this is commonly due to a stock delisting or switching exchanges or a derivative contract expiring, see http://qrok.it/h/err/200 for more help (error code 200: No security definition has been found for the request)
2024-02-23 15:41:28 quantrocket.history: WARNING [cl-1min-bbo] IBKR reports CLZ8 FUT (sid QF000000023508) cannot be found in their system, this is commonly due to a stock delisting or switching exchanges or a derivative contract expiring, see http://qrok.it/h/err/200 for more help (error code 200: No security definition has been found for the request)
2024-02-23 15:41:28 quantrocket.history: WARNING [cl-1min-bbo] IBKR reports CLZ9 FUT (sid QF000000023536) cannot be found in their system, this is commonly due to a stock delisting or switching exchanges or a derivative contract expiring, see http://qrok.it/h/err/200 for more help (error code 200: No security definition has been found for the request)

And this seems like an issue downloading data that seems to exist:

2024-02-23 15:42:58 quantrocket.history: WARNING [cl-1min-bbo] Increasing IBKR historical data timeout from 60 to 70 seconds after response time of 54 seconds for historical data request for 7 D of 1 min BID_ASK for CLF4 FUT (sid QF000000024686) ending 20230308 04:59:59 UTC
2024-02-23 15:43:58 quantrocket.history: WARNING [cl-1min-bbo] Increasing IBKR historical data timeout from 70 to 80 seconds after IBKR reported they were still processing historical data request for 7 D of 1 min BID_ASK for CLF4 FUT (sid QF000000024686) ending 20230315 04:59:59 UTC
2024-02-23 15:45:08 quantrocket.history: WARNING [cl-1min-bbo] Increasing IBKR historical data timeout from 80 to 90 seconds after IBKR reported they were still processing historical data request for 7 D of 1 min BID_ASK for CLF4 FUT (sid QF000000024686) ending 20230322 04:59:59 UTC
2024-02-23 15:47:40 quantrocket.history: WARNING [cl-1min-bbo] Increasing IBKR historical data timeout from 90 to 162 seconds after response time of 152 seconds for historical data request for 7 D of 1 min BID_ASK for CLF4 FUT (sid QF000000024686) ending 20230315 04:59:59 UTC

I've tried reducing the duration of history collection to 1 year from 2.5 in the original code to see if I might fly below some throttling limit, but still get this error. Any suggestions?

#2

IBKR only provides historical data for contracts that expired less than 2 years ago. The first set of warnings reflects that and is normal. (Those contracts are from 2017-2019 so IBKR doesn't provide historical data for them anymore.)

The second set of warnings indicates that the IBKR API is taking longer than expected to respond, but QuantRocket should be adapting to that, so it's not necessarily problematic on its own. Minute bid ask futures data is not always the easiest to collect from IBKR (arguably not the best choice for a free tutorial), but so far this looks okay. If you were to see messages that the IBKR API wasn't responding and QuantRocket was retrying lots of requests, I would recommend canceling and restarting the data collection, but these log messages don't reflect that.