I have a doubt regarding the process of history collection for ibkr databases:
First I update the ibkr listings.
Then, I create a ibkr db using:
securities = get_securities(vendors='ibkr', sec_types = ['STK', 'ETF'])
germany_securities = securities[(securities['Country']=="DE") & (securities.Delisted==False)]
germany_securities = germany_securities.reset_index()
germany_sids = germany_securities['Sid'].to_list()
create_ibkr_db("ibkr-germany-1d", sids= germany_sids , bar_size="1 day", outside_rth=False, primary_exchange=False)
Finally, I collect the history using collect_history("ibkr-germany-1d")
sometime after the db creation, I recollect the listings and delist some securities using the quantrocket master diff-ibkr command as shown in the documentation.
After that I run the history collection again using:
securities = get_securities(vendors='ibkr', sec_types = ['STK', 'ETF'])
germany_securities = securities[(securities['Country']=="DE") & (securities.Delisted==False)]
germany_securities = germany_securities.reset_index()
germany_sids = germany_securities['Sid'].to_list()
collect_history("ibkr-germany-1d", sids=country_sids)
In this second history collection, the sids passed to the "collect_history" function are different from the ones used when the db was created.
My doubt is if the sids contained in the db can be expanded after the db creation or if it is necessary to create a new db each time the listings are collected in order to maintain a db with all the active german stocks.
I'm currently having some problems collecting history. The collection process is much slower than it was when the db was created, and I get a lot of errors like this:
2021-05-11 06:51:16 UTC| <14>1 2021-05-11T06:51:16Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT
2021-05-11 06:52:11 UTC| <14>1 2021-05-11T06:52:11Z 1c3039bfc9fc quantrocket_history_1 1769 - - Retrying historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after no response received from IBKR after 55 seconds (retry 1 of 10)
2021-05-11 06:53:06 UTC| <14>1 2021-05-11T06:53:06Z 1c3039bfc9fc quantrocket_history_1 1769 - - Retrying historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after no response received from IBKR after 55 seconds (retry 1 of 10)
2021-05-11 06:53:06 UTC| <14>1 2021-05-11T06:53:06Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT
2021-05-11 06:54:01 UTC| <14>1 2021-05-11T06:54:01Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT
2021-05-11 06:54:56 UTC| <14>1 2021-05-11T06:54:56Z 1c3039bfc9fc quantrocket_history_1 1769 - - Retrying historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after no response received from IBKR after 55 seconds (retry 3 of 10)
2021-05-11 06:55:52 UTC| <14>1 2021-05-11T06:55:52Z 1c3039bfc9fc quantrocket_history_1 1769 - - Retrying historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after no response received from IBKR after 55 seconds (retry 3 of 10)
2021-05-11 06:55:52 UTC| <14>1 2021-05-11T06:55:52Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT
2021-05-11 06:56:46 UTC| <14>1 2021-05-11T06:56:46Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT
2021-05-11 06:57:41 UTC| <14>1 2021-05-11T06:57:41Z 1c3039bfc9fc quantrocket_history_1 1769 - - Retrying historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after no response received from IBKR after 55 seconds (retry 5 of 10)
2021-05-11 06:58:36 UTC| <14>1 2021-05-11T06:58:36Z 1c3039bfc9fc quantrocket_history_1 1769 - - Retrying historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after no response received from IBKR after 55 seconds (retry 5 of 10)
2021-05-11 06:58:36 UTC| <14>1 2021-05-11T06:58:36Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT
2021-05-11 06:59:31 UTC| <14>1 2021-05-11T06:59:31Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT
2021-05-11 07:00:26 UTC| <14>1 2021-05-11T07:00:26Z 1c3039bfc9fc quantrocket_history_1 1769 - - Retrying historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after no response received from IBKR after 55 seconds (retry 7 of 10)
2021-05-11 07:09:36 UTC| <14>1 2021-05-11T07:09:36Z 1c3039bfc9fc quantrocket_history_1 1769 - - Retrying historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after no response received from IBKR after 55 seconds (retry 7 of 10)
2021-05-11 07:09:36 UTC| <14>1 2021-05-11T07:09:36Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT
2021-05-11 07:10:31 UTC| <14>1 2021-05-11T07:10:31Z 1c3039bfc9fc quantrocket_history_1 1769 - - Retrying historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after no response received from IBKR after 55 seconds (retry 8 of 10)
2021-05-11 07:10:31 UTC| <14>1 2021-05-11T07:10:31Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT
2021-05-11 07:11:26 UTC| <14>1 2021-05-11T07:11:26Z 1c3039bfc9fc quantrocket_history_1 1769 - - Retrying historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after no response received from IBKR after 55 seconds (retry 9 of 10)
2021-05-11 07:11:26 UTC| <14>1 2021-05-11T07:11:26Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT
2021-05-11 07:12:21 UTC| <14>1 2021-05-11T07:12:21Z 1c3039bfc9fc quantrocket_history_1 1769 - - Retrying historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after no response received from IBKR after 55 seconds (retry 10 of 10)
2021-05-11 07:12:21 UTC| <14>1 2021-05-11T07:12:21Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT
2021-05-11 07:13:16 UTC| <14>1 2021-05-11T07:13:16Z 1c3039bfc9fc quantrocket_history_1 1769 - - Issuing to ibg1 historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT
2021-05-11 07:13:16 UTC| <11>1 2021-05-11T07:13:16Z fb565ac9d539 quantrocket_flightlog_1 2644 - - 2021-05-11 07:13:16 quantrocket.history: WARNING [ibkr-germany-1d] No response received from IBKR for historical data request for 2 Y of 1 day TRADES for VTLN STK (sid FIBBG000BSPJ15) ending 20210511 21:59:59 GMT after 11 attempts, won't request any more data for this security at present to avoid data gaps, will retry this security next time ibkr-germany-1d is collected