IB connection error when fetching NASDAQ historical eod data

Technical Help
#1

Dear Q,
I'm following the define universe tutorial and trying to fetch historical data for all NASDAQ stocks. At first, everything runs well but then there are some errors:

20180212225900
201802122259001907×791 42.1 KB

201802122304271
2018021223042711603×958 18.8 KB

It seems many stocks' historical eod data in nasdaq list are not avaliable in IB and I lost the connection with IB when fetching data. I tried to test the connection with IB using "quantrocket account balance --fields 'NetLiquidation' --latest --force-refresh --pretty" and it shows me the connection is good.
I have tried to fetch the data at least three times and this connection error always happen.

#2

There are several things to mention based on your log output:

  1. IB's reliability on Saturdays can be spotty sometimes. You may want to retry on another day.
  2. When IB isn't responding as expected it's often helpful to restart IB gateway: quantrocket launchpad stop --wait && quantrocket launchpad start
  3. A large universe such as all Nasdaq listings will likely include many illiquid securities, so not getting a certain amount of data back may be expected. Unfortunately IB is not always consistent with the error codes they return for unavailable securities.
  4. IB's API mirrors TWS, so you can spot check individual stocks by seeing if data is available in TWS.
  5. Re-run the data fetching and see if the same securities are affected each time. If they are, there's a good chance they're unavailable. On the other hand, if it's a transient error then you will probably get them the second time. The "Request Timed Out" errors are usually transient.

Hope this helps.

#3

Thank you mach8, I'm re-running the data fetching and will compare the securities. There are two more questions based on your answer:

  1. If I want to fetch the data that with transient error at the second time, is it I just need to use the same code (quantrocket history fetch 'nasdaq-eod') again to fetch those data for the same database? If I understand it correctly, it will only try to fetch those securities that failed at the first time and ignore those that fetched successfully before.
  2. I opened the IB GUI to check my connection, On API client, most time it is disconnect. Just very short time it will show connected as showed in the pic. Is it related to my slow and many faild data fetching? If so, how should I fix this?

20180214095423
20180214095423707×575 5.01 KB

Thank you!

#4

Correct. Re-running will bring the history up-to-date, taking account of what's already in your database. So it will try to get the stocks that were missed, as well as collecting any new price history for the stocks you already have.

If API Client is usually disconnected, that's fine. It just means that at the moment, none of the QuantRocket containers (the API clients) are connected to IB Gateway and asking for data. If Interactive Brokers API Server is usually disconnected, that indicates a bad internet connection.