IB collection errors after 2.2.0 upgrade

Technical Help
#1

Hi there,

I'm getting multiple errors when trying to collect data from IB after upgrading to 2.2.0.

Here are some example messages:

quantrocket_flightlog_1|2020-09-07 16:37:43 quantrocket.history: WARNING ibg2 client 3824 got IBKR error code 320: Error reading request.Message id 6.  Unable to parse data. java.lang.NumberFormatException: For input string: "7341.0"
    quantrocket_houston_1|172.18.0.5 - - [07/Sep/2020:20:37:43 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
  quantrocket_flightlog_1|2020-09-07 16:37:43 quantrocket.history: WARNING ibg1 client 3824 got IBKR error code 320: Error reading request.Message id 5.  Unable to parse data. java.lang.NumberFormatException: For input string: "132900672.0"
    quantrocket_houston_1|172.18.0.5 - - [07/Sep/2020:20:37:43 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
  quantrocket_flightlog_1|2020-09-07 16:37:43 quantrocket.history: WARNING ibg4 client 3824 got IBKR error code 320: Error reading request.Message id 4.  Unable to parse data. java.lang.NumberFormatException: For input string: "76227954.0"

Data collection retries for each symbol, then eventually times out.

Please advise.

David

Broken Pipe Error in Market Data Collection
#2

The error message suggests that IBKR conids are being sent by QuantRocket as strings instead of integers, and the IBKR API is complaining about that. I'm unable to reproduce a situation where this would happen, however.

Please try running the following code, first replacing SIDS and UNIVERSES with the appropriate values for your database. This code will print any records in your universe that have non-integer ibkr_ConId s. Please share the output you get, if any.

import io
import pandas as pd

SIDS = None
UNIVERSES = None

f = io.StringIO()
download_master_file(
    f, sids=SIDS, universes=UNIVERSES,
    fields=["ibkr_ConId", "ibkr_LocalSymbol", "ibkr_SecType",
            "ibkr_PrimaryExchange", "Timezone",
            "ibkr_LastTradeDate",
            "ibkr_ValidExchanges", "ibkr_MdSizeMultiplier"],
    exclude_delisted=True
)

securities = pd.read_csv(f)
securities = securities[securities.ibkr_ConId.notnull()].copy()

securities["ibkr_ConId_int"] = pd.to_numeric(securities.ibkr_ConId, errors="coerce")
print(securities[securities.ibkr_ConId_int.isnull()].to_string())
#3

Here are the SIDS and UNIVERSES I used with the above code:

SIDS = ["FIBBG000QH56C1","FIBBG000F7RCJ1"]
UNIVERSES = ["sp500members","sp500current","spyetf"]

And here is the result:

Empty DataFrame
Columns: [Sid, Timezone, ibkr_ConId, ibkr_LastTradeDate, ibkr_LocalSymbol, ibkr_MdSizeMultiplier, ibkr_PrimaryExchange, ibkr_SecType, ibkr_ValidExchanges, ibkr_ConId_int]
Index: []

Those SIDS were specifically generating errors.

#4

No output means the conids are integers, as they should be. Fortunately or unfortunately, I can’t reproduce the error message you're seeing with those sids or any others. That makes it hard to address, but we can try forcing strings to integers in an extra place. First, please try collecting the data again to verify that the problem is still happening. If it is, then please edit your docker-compose.yml by replacing the line:

image: 'quantrocket/history:2.2.0'

with

image: 'quantrocket/history:patch20200909'

and then run

docker-compose up -d history

and then try collecting the data again. Let me know if using the patched image fixes the problem.

QuantRocket 2.3.0 Release Notes
#5

That worked! Thanks!

#6

This issue is fixed in version 2.3.0.