Data download time

#1

my download times for data seem excessively slow. Almost to the point of useless. Here is an example of 1hr bars for one symbole.

2019-02-20 15:35:46 quantrocket.history: INFO [fx-1h] Collecting history from IB for 1 securities in fiber-1h
2019-02-20 15:45:47 quantrocket.history: INFO [fx-1h] Saved 85647 total records for 1 total securities to quantrocket.history.fx-1h.sqlite

it took 10 minutes to download 1 hour bars for 1 symbol. This is something i could do in IQFeed in <10 seconds.

is this normal?

#2

That's fairly typical for IB data. IB rate limits their historical data; QuantRocket pulls the data as fast as IB provides it. You can reduce the runtime by running multiple IB Gateways.

The best practice with initial data collection is to start working with one dataset and while doing so to kick off data collection for the datasets you anticipate wanting next and let it run in the background. Once you've synced a library of IB data to your local databases it's much faster to collect the updates.

IB data is better for its breadth of coverage than its speed of download. The initial data collection can be annoying but it's a one-time cost.

More on initial data collection here.