Collecting 15 Min Bar ALL US stock taking insane amout of time

[raj.priyesh25]

Hi,

I started collecting ALL US Stock from IB with 15 Min Bar 5 days back. However, till now, only 5 GB of 150 GB data has been collected. When going through logs, I found one statement very prominent.

TWS Session is connected from Different IP Address. Waiting for Session to End.

How can we speed up the collection? We are using 2 gateways with different users.

Thanks

[Brian]

I've added a help guide for this error message:

Error code 162: TWS session is connected from a different IP address

When collecting historical data from IB, you might see this error message in the logs: quantrocket.history: WARNING [demo-stk-1d] ibg1 got IB error code 162 for historical data request for 1 M of 15 mins TRADES for EDC STK (conid 225312853) ending 20181122 21:00:00 GMT: Historical Market Data Service error message:Trading TWS session is connected from a different IP address quantrocket.history: WARNING [demo-stk-1d] Waiting 60 seconds before collecting demo-stk-1d history to see if other TWS se…

[raj.priyesh25]

Brian, Thanks for the help.

We are trying to figure out what is happening at our end. However, the accounts are as you suggested, one for QuantRocket and one for another purpose. So, I guess this issue should not come.

However, the process of collecting data is still very slow. It's running since I opened this question, and it's going on as I write here and the DB size is 32 GB only. Can someone or you tell me why the data collection is very slow? What are the parameters that hamper the collection speed?

Please note that we have 2 gateways configured and everything runs on cloud instance.

[Brian]

Here are some things to check:

• Check the IB response times which are printed in a table in the detailed logs every few minutes. I would expect a mean response time of maybe 2 seconds, but this can vary. The estimates in the usage guide are based on a 2 second response time, I believe.
• If the whole system is slow (not just data collection), that would probably indicate a lack of memory. Run docker stats to see memory usage by container.
• monitor the detailed logs to see if the history collection is pausing a lot for some reason, as when the duplicate session was detected. Filtering the logs is useful for this purpose.