Unable to query realtime aggregate database

Technical Help
#1

Created a tick db:

from quantrocket.realtime import create_ibkr_tick_db
create_ibkr_tick_db("ibkr-fx-eurusd-1min-tick", universes=["ibkr-fx-eurusd-1min"],
                        fields=["LastPrice", "Volume", "BidPrice",
                                "AskPrice", "BidSize", "AskSize"])

Created an realtime aggregate db:

from quantrocket.realtime import create_agg_db
create_agg_db("ibkr-fx-eurusd-1min",
              tick_db_code="ibkr-fx-eurusd-1min-tick",
              bar_size="5m",
              fields={"LastPrice":["Open","High","Low","Close"], "Volume": ["Close"]})

List dbs:

from quantrocket.realtime import list_databases
list_databases()

{'ibkr-fx-eurusd-1min-tick': ['ibkr-fx-eurusd-1min']}

Check db config for aggregate db:

from quantrocket.realtime import get_db_config
get_db_config("ibkr-fx-eurusd-1min")

{'tick_db_code': 'ibkr-fx-eurusd-1min-tick', 'bar_size': '5m', 'fields': ['LastPriceClose', 'LastPriceHigh', 'LastPriceLow', 'LastPriceOpen', 'VolumeClose']}

Fails to query realtime aggregate db:

download_market_data_file("ibkr-fx-eurusd-1min",
                          start_date="2023-09-29",
                          sids=["FXEURUSD"],
                          fields=['LastPriceClose', 
                                  'LastPriceHigh', 
                                  'LastPriceLow', 
                                  'LastPriceOpen', 
                                  'VolumeClose'],
                              filepath_or_buffer="ibkr-fx-eurusd-1min.csv"
                         )

NoRealtimeData: ('400 Client Error: BAD REQUEST for url: http://houston/realtime/ibkr-fx-eurusd-1min.csv?start_date=2023-09-29&sids=FXEURUSD', {'status': 'error', 'msg': 'no market data matches the query parameters'})

Can you help me investigate this?

#2

You left out one critical step before querying the data, which is to collect the data.

#3

Oh, yeah, I missed adding that in the post. I collect data using:

from quantrocket.realtime import collect_market_data
collect_market_data(codes="ibkr-fx-eurusd-1min-tick", 
                    sids=["FXEURUSD"], 
                    universes=["ibkr-fx-eurusd-1min"])

And still see the same issue!

#4

If you’re getting no data on a Saturday, try on Sunday or during the week. IBKR does system maintenance on Saturday and the API becomes unreliable.

If you’re getting no data during the week, what do the detailed logs say?

You can also log into Trader Workstation and make sure real-time data for EUR.USD is flowing there; if it’s not, it won’t come to QuantRocket either.

#5

I must've mentioned this; it's strange that I am not able to query real time aggregate database. I am able to query real time tick database though. I will share more details when market opens.

#6

I tried to query real time aggregate data while real time tick data collection was happening.

Collect real time tick data

from quantrocket.realtime import collect_market_data
collect_market_data(codes="ibkr-fx-eurusd-1min-tick", 
                    sids=["FXEURUSD"], 
                    universes=["ibkr-fx-eurusd-1min"])

Check if collection is happening

from quantrocket.realtime import get_active_collections
get_active_collections()

Output
{'ibkr': {'ibkr-fx-eurusd-1min-tick': 1}}

When I query the real time tick db, it takes around 15-20 seconds and outputs a dataframe:

df
df1920×698 92.6 KB

When I switch to real time aggregate data:

import pandas as pd
from quantrocket.realtime import download_market_data_file
download_market_data_file("ibkr-fx-eurusd-1min",
                          start_date="2023-09-29",
                          sids=["FXEURUSD"],
                          filepath_or_buffer="ibkr-fx-eurusd-1min.csv"
                         )
ticks = pd.read_csv("ibkr-fx-eurusd-1min.csv", parse_dates=["Date"])
ticks.tail()

Error:

NoRealtimeData: ('400 Client Error: BAD REQUEST for url: http://houston/realtime/ibkr-fx-eurusd-1min.csv?start_date=2023-09-29&sids=FXEURUSD', {'status': 'error', 'msg': 'no market data matches the query parameters'})
#7

Your aggregate database is aggregating LastPrice and Volume, but those fields aren't available for FX because FX is an OTC market with no central exchange to report trade prices and sizes. All you can get with FX is bid and ask, from which you can then calculate the midpoint on your own. IBKR reports the midpoint when you collect historical FX data, but for real-time data you need to calculate it yourself using the bid price and ask price. Try creating your aggregate database using bid price and ask price and you should get the expected results.