Huge amount of time to get just two historical 5secs bars

Technical Help
#1

We are trying to get the 5 seconds historical bars.
Just for the last two days and just the opening bar and just one ticker. This is a total of 2 bars.
We are using a code similar to the following:

from timeit import default_timer as timer
create_db(my_db_name, conids=[416904], bar_size='5 secs', shard='off', bar_type='BID_ASK', times=['09:30:00'], start_date='20200601', end_date='20200603')
.......
start = timer()
collect_history(my_db_name, start_date='20200601', end_date='20200603')
wait_for_collections(my_db_name)
end = timer()

It is taking 1623 seconds. Is this normal?

#2

Maybe paste quantrocket flightlog stream --detail while you're running that? It will show you data about what it is fetching and how long requests take. Might help narrow down your problem.

Only other thing which stands out is you wrote dates like 20200601 but the docs say the format is 2020-06-01. Maybe those are being ignored and you're collecting a ton of history.

1 Like
#3

There are two reasons it is so slow. The IBKR API imposes more onerous rate limits for bar sizes smaller than 1 minute: only 6 requests per minute. On top of that, requesting BID_ASK counts as two requests which results in a further doubling of the response time.

Depending on your use case, I would suggest collecting 1-minute BID_ASK bars to avoid the extra rate limiting. Although 1-minute might seem like a large bar size for examining spreads, IBKR provides the time-averaged bid and ask price which ends up being a pretty useful gauge of average spread.

1 Like
#4

Thanks for both responses.

Does any kind of restriction exist for requesting data from the actual day? We are not being able to get that data with any history service configuration.

For example I can request for 1 minute opening bar from yesterday without any problem. And it is taking less than 1 second:

create_db(db_name, conids=[conid], bar_size="1 min", start_date="2020-06-17", times=["09:30:00"], shard='off')

2020-06-18 15:46:48 UTC| <14>1 2020-06-18T15:46:48Z a0a9d11caad2 quantrocket_history_1 14483 - - Issuing to ibg1 historical data request for 2 D of 1 min TRADES for SPY STK (conid 756733) ending 20200618 15:47:00 GMT
2020-06-18 15:46:48 UTC| <14>1 2020-06-18T15:46:48Z a0a9d11caad2 quantrocket_history_1 14483 - - ibg1 client 3668 got IB error code 2104: Market data farm connection is OK:usfarm
2020-06-18 15:46:48 UTC| <14>1 2020-06-18T15:46:48Z a0a9d11caad2 quantrocket_history_1 14483 - - ibg1 client 3668 got IB error code 2107: HMDS data farm connection is inactive but should be available upon demand.ushmds.nj
2020-06-18 15:46:48 UTC| <14>1 2020-06-18T15:46:48Z a0a9d11caad2 quantrocket_history_1 14483 - - ibg1 client 3668 got IB error code 2107: HMDS data farm connection is inactive but should be available upon demand.ushmds
2020-06-18 15:46:48 UTC| <14>1 2020-06-18T15:46:48Z a0a9d11caad2 quantrocket_history_1 14483 - - ibg1 client 3668 got IB error code 2106: HMDS data farm connection is OK:ushmds
2020-06-18 15:46:48 UTC| <14>1 2020-06-18T15:46:48Z a0a9d11caad2 quantrocket_history_1 14483 - - Processing 527 raw bars from IB for SPY STK (conid 756733)
2020-06-18 15:46:49 UTC| <14>1 2020-06-18T15:46:49Z a0a9d11caad2 quantrocket_history_1 14483 - - Saving 1 historical bars for 1 securities to quantrocket.history.spy1mindb.sqlite

But when I try to request the same 1 minute bar data but from the actual day, the request is taking huge amount of time (More than 1 hour in some cases). Here the first 3 minutes of the log:

create_db(db_name, conids=[conid], bar_size="1 min", start_date="2020-06-18", times=["09:30:00"], shard='off')

2020-06-18 15:47:05 UTC| <14>1 2020-06-18T15:47:05Z a0a9d11caad2 quantrocket_history_1 14483 - - Issuing to ibg1 historical data request for 1 D of 1 min TRADES for SPY STK (conid 756733) ending 20200618 15:48:00 GMT
2020-06-18 15:47:05 UTC| <14>1 2020-06-18T15:47:05Z a0a9d11caad2 quantrocket_history_1 14483 - - ibg1 client 3681 got IB error code 2104: Market data farm connection is OK:usfarm
2020-06-18 15:47:05 UTC| <14>1 2020-06-18T15:47:05Z a0a9d11caad2 quantrocket_history_1 14483 - - ibg1 client 3681 got IB error code 2107: HMDS data farm connection is inactive but should be available upon demand.ushmds.nj
2020-06-18 15:47:05 UTC| <14>1 2020-06-18T15:47:05Z a0a9d11caad2 quantrocket_history_1 14483 - - ibg1 client 3681 got IB error code 2106: HMDS data farm connection is OK:ushmds
2020-06-18 15:47:06 UTC| <14>1 2020-06-18T15:47:06Z a0a9d11caad2 quantrocket_history_1 14483 - - Processing 138 raw bars from IB for SPY STK (conid 756733)
2020-06-18 15:47:08 UTC| <14>1 2020-06-18T15:47:08Z a0a9d11caad2 quantrocket_history_1 14483 - - ┌──────────────────────────────┐
2020-06-18 15:47:08 UTC| <14>1 2020-06-18T15:47:08Z a0a9d11caad2 quantrocket_history_1 14483 - - │ spy1mindb IB response times: │
2020-06-18 15:47:08 UTC| <14>1 2020-06-18T15:47:08Z a0a9d11caad2 quantrocket_history_1 14483 - - │        ibg1                  │
2020-06-18 15:47:08 UTC| <14>1 2020-06-18T15:47:08Z a0a9d11caad2 quantrocket_history_1 14483 - - │ count   1.0                  │
2020-06-18 15:47:08 UTC| <14>1 2020-06-18T15:47:08Z a0a9d11caad2 quantrocket_history_1 14483 - - │ mean    0.2                  │
2020-06-18 15:47:08 UTC| <14>1 2020-06-18T15:47:08Z a0a9d11caad2 quantrocket_history_1 14483 - - │ std     NaN                  │
2020-06-18 15:47:08 UTC| <14>1 2020-06-18T15:47:08Z a0a9d11caad2 quantrocket_history_1 14483 - - │ min     0.2                  │
2020-06-18 15:47:08 UTC| <14>1 2020-06-18T15:47:08Z a0a9d11caad2 quantrocket_history_1 14483 - - │ 25%     0.2                  │
2020-06-18 15:47:08 UTC| <14>1 2020-06-18T15:47:08Z a0a9d11caad2 quantrocket_history_1 14483 - - │ 50%     0.2                  │
2020-06-18 15:47:08 UTC| <14>1 2020-06-18T15:47:08Z a0a9d11caad2 quantrocket_history_1 14483 - - │ 75%     0.2                  │
2020-06-18 15:47:08 UTC| <14>1 2020-06-18T15:47:08Z a0a9d11caad2 quantrocket_history_1 14483 - - │ max     0.2                  │
2020-06-18 15:47:08 UTC| <14>1 2020-06-18T15:47:08Z a0a9d11caad2 quantrocket_history_1 14483 - - └──────────────────────────────┘
2020-06-18 15:48:05 UTC| <14>1 2020-06-18T15:48:05Z a0a9d11caad2 quantrocket_history_1 14483 - - ┌──────────────────────────────┐
2020-06-18 15:48:05 UTC| <14>1 2020-06-18T15:48:05Z a0a9d11caad2 quantrocket_history_1 14483 - - │ spy1mindb processing stages: │
2020-06-18 15:48:05 UTC| <14>1 2020-06-18T15:48:05Z a0a9d11caad2 quantrocket_history_1 14483 - - │ head timestamps          0   │
2020-06-18 15:48:05 UTC| <14>1 2020-06-18T15:48:05Z a0a9d11caad2 quantrocket_history_1 14483 - - │ historical data          0   │
2020-06-18 15:48:05 UTC| <14>1 2020-06-18T15:48:05Z a0a9d11caad2 quantrocket_history_1 14483 - - │ waiting for responses    0   │
2020-06-18 15:48:05 UTC| <14>1 2020-06-18T15:48:05Z a0a9d11caad2 quantrocket_history_1 14483 - - │ bar processing           0   │
2020-06-18 15:48:05 UTC| <14>1 2020-06-18T15:48:05Z a0a9d11caad2 quantrocket_history_1 14483 - - │ database storage         1   │
2020-06-18 15:48:05 UTC| <14>1 2020-06-18T15:48:05Z a0a9d11caad2 quantrocket_history_1 14483 - - │ completed                0   │
2020-06-18 15:48:05 UTC| <14>1 2020-06-18T15:48:05Z a0a9d11caad2 quantrocket_history_1 14483 - - └──────────────────────────────┘
2020-06-18 15:49:05 UTC| <14>1 2020-06-18T15:49:05Z a0a9d11caad2 quantrocket_history_1 14483 - - ┌──────────────────────────────┐
2020-06-18 15:49:05 UTC| <14>1 2020-06-18T15:49:05Z a0a9d11caad2 quantrocket_history_1 14483 - - │ spy1mindb processing stages: │
2020-06-18 15:49:05 UTC| <14>1 2020-06-18T15:49:05Z a0a9d11caad2 quantrocket_history_1 14483 - - │ head timestamps          0   │
2020-06-18 15:49:05 UTC| <14>1 2020-06-18T15:49:05Z a0a9d11caad2 quantrocket_history_1 14483 - - │ historical data          0   │
2020-06-18 15:49:05 UTC| <14>1 2020-06-18T15:49:05Z a0a9d11caad2 quantrocket_history_1 14483 - - │ waiting for responses    0   │
2020-06-18 15:49:05 UTC| <14>1 2020-06-18T15:49:05Z a0a9d11caad2 quantrocket_history_1 14483 - - │ bar processing           0   │
2020-06-18 15:49:05 UTC| <14>1 2020-06-18T15:49:05Z a0a9d11caad2 quantrocket_history_1 14483 - - │ database storage         1   │
2020-06-18 15:49:05 UTC| <14>1 2020-06-18T15:49:05Z a0a9d11caad2 quantrocket_history_1 14483 - - │ completed                0   │
2020-06-18 15:49:05 UTC| <14>1 2020-06-18T15:49:05Z a0a9d11caad2 quantrocket_history_1 14483 - - └──────────────────────────────┘
#5

I can recreate this issue but would consider it an edge case. Making a historical database that starts today is unusual; it's not a scenario for which we have test coverage. The intended design pattern is to collect a larger date range then query the subset of data you need. If you only want today's data you can specify that at query time. If for some reason you want separate databases per day, you can try using shard="day" and look at the individual shards.