1 gateway vs 3 gateways: Same amount of time to get data

Technical Help
#1

We have been measuring the time that our daily snapshots are taking with just 1 gateway compared with using 3 gateways. In both cases, they are taking the same amount of time. Here the logs:

2020-07-06 11:35:01 quantrocket.ibg3: INFO stopping ibg3
2020-07-06 11:35:01 quantrocket.ibg2: INFO stopping ibg2
2020-07-06 11:35:05 quantrocket.ibg1: INFO stopping ibg1
2020-07-06 11:35:18 quantrocket.ibg1: INFO starting ibg1
#We executed list_gateway_statuses here to double check that just ibg1 is running

2020-07-06 11:37:00 quantrocket.realtime: INFO Collecting market data snapshot for 29 securities in etfuniverse20200706t113632db
2020-07-06 11:37:14 quantrocket.realtime: INFO Finished collecting IB market data
2020-07-06 11:37:15 logger: INFO Snapshot took 15.080867748707533 seconds


2020-07-06 11:37:15 logger: INFO ------------------------------------


2020-07-06 11:37:27 quantrocket.ibg2: INFO starting ibg2
2020-07-06 11:37:46 quantrocket.ibg3: INFO starting ibg3
#We executed list_gateway_statuses here to double check that the 3 gateways are running

2020-07-06 11:39:00 logger: INFO Starting to collet snapshot market data to database etfuniverse20200706t113836db
2020-07-06 11:39:00 quantrocket.realtime: INFO Collecting market data snapshot for 29 securities in etfuniverse20200706t113836db
2020-07-06 11:39:15 quantrocket.realtime: INFO Finished collecting IB market data
2020-07-06 11:39:16 logger: INFO Snapshot took 16.10690214857459 seconds

All the 3 gateways have the required market data subscription and the Market data permission file is correctly created.

Is it not supposed to take much less time when running 3 gateways than with just 1 (spreading requests among the running gateways)?

#2

For real-time data, multiple gateways is not really about speeding up the requests but overcoming the concurrent ticker limit. Since you are only requesting 29 tickers and the basic ticker limit is 100, that's not applicable to your situation. Also, for snapshot requests, IBKR imposes an 11-second delay per 100 tickers (or whatever your ticker limit is) so you are likely running into that in both cases. You will only see a speedup if you collect at least hundreds of tickers.

If you only have 29 tickers and you care about speed, the better approach is not to use snapshots but streaming data. Start collecting streaming data a minute or two before you need the quotes then query the tick database (or aggregate database) at the time you need them. The main use case for --snapshot is when you need quotes for more tickers than you are allowed with IBKR. Otherwise streaming will generally be preferable.

1 Like