Unable to collect ibkr historical fx data

Technical Help
#1

Hi I am trying to collect historical fx data for some sids from ibkr .

My db config is

{'universes': ['idealpro-fx'],
 'vendor': 'ibkr',
 'bar_size': '1 day',
 'fields': ['Open', 'High', 'Low', 'Close', 'Volume', 'Wap', 'TradeCount']}

I am collecting data using the following code

from quantrocket.history import collect_history
collect_history(codes="idealpro-fx-1d-history" , start_date= '2022-01-01', sids= 'FXAUDUSD')

But for some reason it is collecting data for only certain days and giving following logs.

 quantrocket-history-1|ibg1 client 4590 got IBKR message code 2104: Market data farm connection is OK:cashfarm
  quantrocket-history-1|ibg1 client 4590 got IBKR message code 2104: Market data farm connection is OK:usfarm
  quantrocket-history-1|ibg1 client 4590 got IBKR message code 2106: HMDS data farm connection is OK:cashhmds
  quantrocket-history-1|ibg1 client 4590 got IBKR message code 2107: HMDS data farm connection is inactive but should be available upon demand.ushmds
  quantrocket-history-1|Issuing to ibg1 historical data request for 4 D of 1 day MIDPOINT for AUD.USD CASH (sid FXAUDUSD) ending 20240308 04:59:59 UTC
  quantrocket-history-1|Processing 3 raw bars from IBKR for AUD.USD CASH (sid FXAUDUSD)
  quantrocket-history-1|Saving 3 historical bars for 1 securities to quantrocket.v2.history.idealpro-fx-1d-history.sqlite

I tried for some other sids getting similar kind of response

#2

The log output is what you would expect to see when collecting an update for that sid. In other words, the log message implies to me that you previously collected AUD.USD data, and now you're collecting an update, so QuantRocket only queries the last few days to get the latest updates. Did you query the database to see if the data you are expecting is already in there?

#3

Yes i queried the database it has some rows for some dates but almost all rows have prices empty. I tried doing it other SIDS same kind of behaviour it was showing .

from quantrocket import get_prices
prices = get_prices("idealpro-fx-1d-history")
prices.loc['Close']['FXAUDUSD'].to_csv('prices.csv')

I queired prices using above code it gave me almost rows empty and only for very few dates.

#4

I just checked and I am able to collect data for AUD.USD without a problem. Can you create a fresh database for AUD.USD and try again and see if the same issue occurs? Please monitor the detailed logs for any messages. Just collecting one currency will make it easier to monitor.

#5

Okay got it. I recreated the database it worked. Thanks

#6

Hi Brian I am still facing this issue again though i have created the database again from scracth following is the log

using the following command:

from quantrocket.history import collect_history
collect_history(codes="ibkr-fx-eurusd-1min-history", start_date= '2017-01-01', sids = 'FXEURUSD')

quantrocket-houston-1|172.18.0.11 - - [22/Mar/2024:17:41:56 +0000] "POST /history/queue?codes=ibkr-fx-eurusd-1min-history&sids=FXEURUSD&start_date=2017-01-01 HTTP/1.1" 202 67 "-" "python-urllib3/2.1.0"
quantrocket-houston-1|172.18.0.14 - - [22/Mar/2024:17:41:56 +0000] "GET /master/securities.csv?sids=FXEURUSD&exclude_delisted=True&fields=ibkr_ConId&fields=ibkr_LocalSymbol&fields=ibkr_SecType&fields=ibkr_PrimaryExchange&fields=Timezone&fields=ibkr_LastTradeDate&fields=ibkr_ValidExchanges HTTP/1.1" 200 183 "-" "python-urllib3/1.26.8"
quantrocket-houston-1|172.18.0.14 - - [22/Mar/2024:17:41:57 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket-flightlog-1|2024-03-22 17:41:57 quantrocket.history: INFO [ibkr-fx-eurusd-1min-history] Collecting history from IBKR for 1 securities in ibkr-fx-eurusd-1min-history
  quantrocket-history-1|Setting IBKR historical data timeout to initial value of 45 seconds
  quantrocket-houston-1|172.18.0.14 - - [22/Mar/2024:17:41:57 +0000] "GET /ibgrouter/config HTTP/1.1" 204 0 "-" "python-urllib3/1.26.8"
  quantrocket-houston-1|172.18.0.16 - - [22/Mar/2024:17:41:57 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.27.1"
  quantrocket-houston-1|172.18.0.14 - - [22/Mar/2024:17:41:57 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 9 "-" "python-urllib3/1.26.8"
  quantrocket-history-1|ibg1 client 7253 got IBKR message code 2104: Market data farm connection is OK:cashfarm
  quantrocket-history-1|ibg1 client 7253 got IBKR message code 2104: Market data farm connection is OK:usfarm
  quantrocket-history-1|ibg1 client 7253 got IBKR message code 2106: HMDS data farm connection is OK:cashhmds
  quantrocket-history-1|ibg1 client 7253 got IBKR message code 2107: HMDS data farm connection is inactive but should be available upon demand.ushmds
  quantrocket-history-1|Issuing to ibg1 historical data request for 7 D of 1 min MIDPOINT for EUR.USD CASH (sid FXEURUSD) ending 20170120 17:42:00 UTC
  quantrocket-houston-1|172.18.0.16 - - [22/Mar/2024:17:42:32 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.27.1"
  quantrocket-houston-1|172.18.0.5 - - [22/Mar/2024:17:42:32 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 9 "-" "python-urllib3/1.26.8"
  quantrocket-houston-1|172.18.0.4 - - [22/Mar/2024:17:42:32 +0000] "GET /master/securities.csv?vendors=ibkr&fields=ibkr_ConId HTTP/1.1" 200 77497 "-" "python-urllib3/1.26.8"
  quantrocket-houston-1|172.18.0.7 - - [22/Mar/2024:17:42:33 +0000] "GET /account/balances.csv?latest=True&fields=NetLiquidation&fields=Paper HTTP/1.1" 200 146 "-" "python-urllib3/1.26.8"
  quantrocket-houston-1|172.18.0.4 - - [22/Mar/2024:17:42:33 +0000] "GET /master/securities.csv?vendors=alpaca&fields=alpaca_AssetId HTTP/1.1" 400 139 "-" "python-urllib3/1.26.8"
  quantrocket-houston-1|172.18.0.16 - - [22/Mar/2024:17:42:33 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.27.1"
  quantrocket-houston-1|172.18.0.4 - - [22/Mar/2024:17:42:33 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 9 "-" "python-urllib3/1.26.8"
  quantrocket-houston-1|172.18.0.16 - - [22/Mar/2024:17:42:39 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.27.1"
  quantrocket-houston-1|172.18.0.4 - - [22/Mar/2024:17:42:39 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 9 "-" "python-urllib3/1.26.8"
  quantrocket-history-1|Retrying historical data request for 7 D of 1 min MIDPOINT for EUR.USD CASH (sid FXEURUSD) ending 20170120 17:42:00 UTC after no response received from IBKR after 45 seconds (retry 1 of 10)
  quantrocket-history-1|Issuing to ibg1 historical data request for 7 D of 1 min MIDPOINT for EUR.USD CASH (sid FXEURUSD) ending 20170127 17:42:00 UTC
  quantrocket-history-1|Retrying historical data request for 7 D of 1 min MIDPOINT for EUR.USD CASH (sid FXEURUSD) ending 20170120 17:42:00 UTC after no response received from IBKR after 45 seconds (retry 1 of 10)
  quantrocket-history-1|Issuing to ibg1 historical data request for 7 D of 1 min MIDPOINT for EUR.USD CASH (sid FXEURUSD) ending 20170120 17:42:00 UTC
  quantrocket-history-1|[ibkr-fx-eurusd-1min-history] ibg1 got IBKR message code 162 for historical data request for 7 D of 1 min MIDPOINT for EUR.USD CASH (sid FXEURUSD) ending 20170120 17:42:00 UTC: Historical Market Data Service error message:API historical data query cancelled: 0
  quantrocket-houston-1|172.18.0.14 - - [22/Mar/2024:17:42:42 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket-flightlog-1|2024-03-22 17:42:42 quantrocket.history: WARNING [ibkr-fx-eurusd-1min-history] Increasing IBKR historical data timeout from 45 to 55 seconds after IBKR reported they were still processing historical data request for 7 D of 1 min MIDPOINT for EUR.USD CASH (sid FXEURUSD) ending 20170120 17:42:00 UTC
  quantrocket-history-1|ibg1 client 7253 got IBKR message code 366: No historical data query found for ticker id:2

Even after the data collection (checked via history queue command)stops it has data only for few days.like one or two weeks max

#7

It seems like for some reason in your case IBKR is taking longer than usual to return the data. QuantRocket is timing out after 45 seconds and retrying, but that doesn't appear to be a good solution here. It may be that IBKR is still working on the request and doesn't like getting a retry request.

There are two environment variables that can you play with to adjust the settings. In your docker-compose.override.yml, play with the following settings (the settings shown below are the defaults):

version: '2.4' # this is the Compose file version, not the QuantRocket version, and should always be 2.4
services:
  history:
    environment:
      HISTORICAL_DATA_REQUEST_TIMEOUT: 45
      IB_DURATION_1_MIN: '7 D'

HISTORICAL_DATA_REQUEST_TIMEOUT sets the initial timeout value. You could set it higher to tell QuantRocket to give IBKR more time to send the response back. IB_DURATION_1_MIN controls how much data QuantRocket asks for at a time when collecting 1-minute bars. If you reduce the amount from '7 D' to, say, '4 D', each request is for a smaller amount of data which should lead to a faster response from IBKR per request. I would think tinkering with one or the other would be sufficient. After editing the file:

docker compose up -d history
#8

i tried the above solution did the following

HISTORICAL_DATA_REQUEST_TIMEOUT: 45
IB_DURATION_1_MIN: '4 D'

 quantrocket-history-1|Retrying historical data request for 4 D of 1 min MIDPOINT for EUR.USD CASH (sid FXEURUSD) ending 20170103 04:59:59 UTC after no response received from IBKR after 130 seconds (retry 4 of 10)
  quantrocket-history-1|Issuing to ibg1 historical data request for 4 D of 1 min MIDPOINT for EUR.USD CASH (sid FXEURUSD) ending 20170103 04:59:59 UTC
  quantrocket-history-1|ibg1 client 3049 got IBKR message code 366: No historical data query found for ticker id:4
  quantrocket-houston-1|172.18.0.16 - - [23/Mar/2024:18:27:59 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.27.1"
  quantrocket-houston-1|172.18.0.14 - - [23/Mar/2024:18:27:59 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 9 "-" "python-urllib3/1.26.8"
  quantrocket-history-1|┌────────────────────────────────────────────────┐
  quantrocket-history-1|│ ibkr-fx-eurusd-1min-history processing stages: │
  quantrocket-history-1|│ head timestamps          0                     │
  quantrocket-history-1|│ historical data          0                     │
  quantrocket-history-1|│ waiting for responses    0                     │
  quantrocket-history-1|│ bar processing           0                     │
  quantrocket-history-1|│ database storage         0                     │
  quantrocket-history-1|│ completed                0                     │

It just keeps on retrying again and again .

this time i am just trying to collect data for two days.

from quantrocket.history import collect_history
collect_history(codes="ibkr-fx-eurusd-1min-history", start_date= '2017-01-02',end_date = '2017-01-02', sids = 'FXEURUSD')

it ends like the following

 quantrocket-history-1|ibg1 client 3049 got IBKR message code 366: No historical data query found for ticker id:9
  quantrocket-history-1|Retrying historical data request for 4 D of 1 min MIDPOINT for EUR.USD CASH (sid FXEURUSD) ending 20170103 04:59:59 UTC after no response received from IBKR after 130 seconds (retry 10 of 10)
  quantrocket-history-1|Issuing to ibg1 historical data request for 4 D of 1 min MIDPOINT for EUR.USD CASH (sid FXEURUSD) ending 20170103 04:59:59 UTC
  quantrocket-houston-1|172.18.0.16 - - [23/Mar/2024:18:40:46 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.27.1"
  quantrocket-houston-1|172.18.0.14 - - [23/Mar/2024:18:40:46 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 9 "-" "python-urllib3/1.26.8"
  quantrocket-houston-1|172.18.0.14 - - [23/Mar/2024:18:40:46 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
  quantrocket-history-1|Issuing to ibg1 historical data request for 4 D of 1 min MIDPOINT for EUR.USD CASH (sid FXEURUSD) ending 20170103 04:59:59 UTC
  quantrocket-history-1|Processing 29520 raw bars from IBKR for EUR.USD CASH (sid FXEURUSD)
quantrocket-flightlog-1|2024-03-23 18:40:46 quantrocket.history: WARNING [ibkr-fx-eurusd-1min-history] No response received from IBKR for historical data request for 4 D of 1 min MIDPOINT for EUR.USD CASH (sid FXEURUSD) ending 20170103 04:59:59 UTC after 11 attempts, won't request any more data for this security at present to avoid data gaps, will retry this security next time ibkr-fx-eurusd-1min-history is collected
  quantrocket-history-1|Saving 900 historical bars for 1 securities to quantrocket.v2.history.ibkr-fx-eurusd-1min-history.sqlite
  quantrocket-history-1|ibg1 client 3049 got IBKR message code 366: No historical data query found for ticker id:10
  quantrocket-history-1|ibg1 client 3049 got IBKR message code 366: No historical data query found for ticker id:11
quantrocket-flightlog-1|2024-03-23 18:40:49 quantrocket.history: INFO [ibkr-fx-eurusd-1min-history] Saved 900 total records for 1 total securities to quantrocket.v2.history.ibkr-fx-eurusd-1min-history.sqlite
  quantrocket-houston-1|172.18.0.14 - - [23/Mar/2024:18:40:49 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
  quantrocket-history-1|recycling mule after collecting history
  quantrocket-history-1|OOOPS mule 1 (pid: 49) crippled...trying respawn...
  quantrocket-history-1|spawned uWSGI mule 1 (pid: 61)
#9

For 1 min data HISTORICAL_DATA_REQUEST_TIMEOUT: 360 and IB_DURATION_1_MIN: '2 D' did the trick.