Doubts about collecting IBKR db historical data

I have a doubt regarding the process of history collection for ibkr databases:

First I update the ibkr listings.
Then, I create a ibkr db using:

securities = get_securities(vendors='ibkr', sec_types = ['STK', 'ETF']) germany_securities = securities[(securities['Country']=="DE") & (securities.Delisted==False)]

germany_securities = germany_securities.reset_index()

germany_sids = germany_securities[‘Sid’].to_list()

create_ibkr_db(“ibkr-germany-1d”, sids= germany_sids , bar_size=“1 day”, outside_rth=False, primary_exchange=False)

Finally, I collect the history using collect_history(“ibkr-germany-1d”)

sometime after the db creation, I recollect the listings and delist some securities using the quantrocket master diff-ibkr command as shown in the documentation.

After that I run the history collection again using:

securities = get_securities(vendors=‘ibkr’, sec_types = [‘STK’, ‘ETF’])
germany_securities = securities[(securities[‘Country’]==“DE”) & (securities.Delisted==False)]
germany_securities = germany_securities.reset_index()
germany_sids = germany_securities[‘Sid’].to_list()
collect_history(“ibkr-germany-1d”, sids=country_sids)

In this second history collection, the sids passed to the “collect_history” function are different from the ones used when the db was created.

My doubt is if the sids contained in the db can be expanded after the db creation or if it is necessary to create a new db each time the listings are collected in order to maintain a db with all the active german stocks.

I’m currently having some problems collecting history. The collection process is much slower than it was when the db was created, and I get a lot of errors like this:

2021-05-11 06:51:16 UTC| <14>1 2021-05-11T06:51:16Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT 2021-05-11 06:52:11 UTC| <14>1 2021-05-11T06:52:11Z 1c3039bfc9fc quantrocket_history_1 1769 - - Retrying historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after no response received from IBKR after 55 seconds (retry 1 of 10) 2021-05-11 06:53:06 UTC| <14>1 2021-05-11T06:53:06Z 1c3039bfc9fc quantrocket_history_1 1769 - - Retrying historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after no response received from IBKR after 55 seconds (retry 1 of 10) 2021-05-11 06:53:06 UTC| <14>1 2021-05-11T06:53:06Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT 2021-05-11 06:54:01 UTC| <14>1 2021-05-11T06:54:01Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT 2021-05-11 06:54:56 UTC| <14>1 2021-05-11T06:54:56Z 1c3039bfc9fc quantrocket_history_1 1769 - - Retrying historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after no response received from IBKR after 55 seconds (retry 3 of 10) 2021-05-11 06:55:52 UTC| <14>1 2021-05-11T06:55:52Z 1c3039bfc9fc quantrocket_history_1 1769 - - Retrying historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after no response received from IBKR after 55 seconds (retry 3 of 10) 2021-05-11 06:55:52 UTC| <14>1 2021-05-11T06:55:52Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT 2021-05-11 06:56:46 UTC| <14>1 2021-05-11T06:56:46Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT 2021-05-11 06:57:41 UTC| <14>1 2021-05-11T06:57:41Z 1c3039bfc9fc quantrocket_history_1 1769 - - Retrying historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after no response received from IBKR after 55 seconds (retry 5 of 10) 2021-05-11 06:58:36 UTC| <14>1 2021-05-11T06:58:36Z 1c3039bfc9fc quantrocket_history_1 1769 - - Retrying historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after no response received from IBKR after 55 seconds (retry 5 of 10) 2021-05-11 06:58:36 UTC| <14>1 2021-05-11T06:58:36Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT 2021-05-11 06:59:31 UTC| <14>1 2021-05-11T06:59:31Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT 2021-05-11 07:00:26 UTC| <14>1 2021-05-11T07:00:26Z 1c3039bfc9fc quantrocket_history_1 1769 - - Retrying historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after no response received from IBKR after 55 seconds (retry 7 of 10) 2021-05-11 07:09:36 UTC| <14>1 2021-05-11T07:09:36Z 1c3039bfc9fc quantrocket_history_1 1769 - - Retrying historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after no response received from IBKR after 55 seconds (retry 7 of 10) 2021-05-11 07:09:36 UTC| <14>1 2021-05-11T07:09:36Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT 2021-05-11 07:10:31 UTC| <14>1 2021-05-11T07:10:31Z 1c3039bfc9fc quantrocket_history_1 1769 - - Retrying historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after no response received from IBKR after 55 seconds (retry 8 of 10) 2021-05-11 07:10:31 UTC| <14>1 2021-05-11T07:10:31Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT 2021-05-11 07:11:26 UTC| <14>1 2021-05-11T07:11:26Z 1c3039bfc9fc quantrocket_history_1 1769 - - Retrying historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after no response received from IBKR after 55 seconds (retry 9 of 10) 2021-05-11 07:11:26 UTC| <14>1 2021-05-11T07:11:26Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT 2021-05-11 07:12:21 UTC| <14>1 2021-05-11T07:12:21Z 1c3039bfc9fc quantrocket_history_1 1769 - - Retrying historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after no response received from IBKR after 55 seconds (retry 10 of 10) 2021-05-11 07:12:21 UTC| <14>1 2021-05-11T07:12:21Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT 2021-05-11 07:13:16 UTC| <14>1 2021-05-11T07:13:16Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT 2021-05-11 07:13:16 UTC| <11>1 2021-05-11T07:13:16Z fb565ac9d539 quantrocket_flightlog_1 2644 - - 2021-05-11 07:13:16 quantrocket.history: WARNING [ibkr-germany-1d] No response received from IBKR for historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after 11 attempts, won't request any more data for this security at present to avoid data gaps, will retry this security next time ibkr-germany-1d is collected

You don’t need to recreate the database. The sids you pass each time will override the sids you passed originally.

Another approach is to create a named universe such as “germany-stk” and specify that universe when you create the database. Then you can update the universe definition over time and don’t have to pass the sids each time you collect history.

I suggest restarting IB Gateway to try to deal with the timeouts, or just wait for it to resolve on its own.