Increase request timeout

Technical Help
#1

Trying to quantrocket master exchanges --regions 'north_america' --sec-types 'STK' returns a connection timeout while working from Australia:

Traceback (most recent call last):
File "/opt/conda/bin/quantrocket", line 11, in
sys.exit(main())
File "/opt/conda/lib/python3.6/site-packages/quantrocket/cli/commands.py", line 97, in main
result, exit_code = func(**args)
File "/opt/conda/lib/python3.6/site-packages/quantrocket/master.py", line 50, in _cli_list_exchanges
return json_to_cli(list_exchanges, *args, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/quantrocket/cli/utils/output.py", line 37, in json_to_cli
json_response = func(*args, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/quantrocket/master.py", line 45, in list_exchanges
response = houston.get("/master/exchanges", params=params)
File "/opt/conda/lib/python3.6/site-packages/requests/sessions.py", line 521, in get
return self.request('GET', url, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/quantrocket/houston.py", line 135, in request
return super(Houston, self).request(method, url, *args, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/opt/conda/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/requests/adapters.py", line 521, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='houston', port=80): Read timed out. (read timeout=30)

It is not my connection speed (which is broadband) but rather responsiveness of IB web site from Australia (which redirects to www.interactivebrokers.com.au for Australian clients, which, I suppose is a proxy on itself and all this contribute to response time significantly). Anyway, there needs to be a way to increase the timeout as it is not possible to complete even simple operations with the IB web site.

Thanks!

#2

When I go to www.interactivebrokers.com.au I see it is very slow; are you saying this is regular occurrence for Australian clients accessing the IB site or is it just slow right now?

The particular traceback indicates a timeout on the client side, waiting for the master service to return a response. You might be able to work around it by using an HTTP request instead:

curl -X GET 'http://houston/master/exchanges?sec_types=STK&regions=north_america' | json2yml

However, if the IB site is so slow there may be timeouts in the master service itself, in which case this workaround wouldn't help. In that case another workaround is to simply get the exchange codes directly from browsing the IB website. This command is a convenience but not so essential.

The only other command that uses the IB website is quantrocket master collect, which uses the website to get a list of all symbols for an exchange. Timeouts would be a bigger problem if they effect that command since it's more important and might call for a configurable timeout as you suggest. QuantRocket has customers in Australia and I haven't heard about this before, so I assume it must not have effected those customers. Maybe the slowness is temporary?

#3

Hi Brian,

The following command hangs:

while flightlog displays:

2019-03-05 01:47:26 quantrocket.master: INFO Pausing 10 seconds before retrying after timeout: Exchange Listings - North America | Interactive Brokers LLC
2019-03-05 01:47:53 quantrocket.master: INFO Pausing 20 seconds before retrying after timeout: Exchange Listings - North America | Interactive Brokers LLC
2019-03-05 01:48:29 quantrocket.master: INFO Pausing 40 seconds before retrying after timeout: Exchange Listings - North America | Interactive Brokers LLC
2019-03-05 01:49:26 quantrocket.master: INFO Pausing 80 seconds before retrying after timeout: Exchange Listings - North America | Interactive Brokers LLC

What is interesting is that it keeps showing "Home | Interactive Brokers LLC" as the target. Please make sure your system can handle redirects because IB web site will redirect for Australian clients to https://www.interactivebrokers.com.au
It does not seem to be a temporary slowness BTW.

I could have collected exchange codes manually, unfortunately, quantrocket master collect command does not work either and it is much harder to work with universes without it.

Thanks again!

#4

PS: If you willing to investigate it further, I'll be happy to assist you.

#5

It does not look like a redirect problem now as quantrocket master exchanges --regions 'asia' --sec-types 'STK' works (because it returns response quicker).

#6

The quantrocket master collect --exchanges 'SEHK' --sec-types 'STK' also seem to be running. After data initially collected from IB web site there seem to be no issues collecting data from IB Gateway. The issue relates to the IB web site only. Clearly, request timeouts need to be increased.

IBG1 fails / does not fetch historical data
#7

OK, quantrocket master collect --exchanges 'NYSE' --sec-types 'STK' started collecting now. But IB web site response times are still super slow, and they are typically like that, so it might be a good idea to increase timeouts (or make them configurable) in future releases.
Thanks!

#8

Now I am getting errors:

2019-03-05 10:31:41 quantrocket.history: WARNING [nasdaq-stk-eod] IB reports no market data subscription for AAPL STK (conid 265598), please enable subscription in IB Account Management for the username associated with ibg1, see Error code 162: Historical Market Data Service error message:No market data permissions - Help Guides - QuantRocket Support Forum for more help (error code 162: Historical Market Data Service error message:No market data permissions for ISLAND STK)

The detailed log indicates no data is actually getting downloaded. I have followed the link provided in the error message. I can clearly see data from TWS and request historical data as well. Also, I have purchased US Securities Snapshot and Futures Value Bundle (NP,L1) IB data subscription, although I was able to see data in TWS without it.
I am logging in ibg1 with a paper trading account. I have enabled "share market data with paper trading" and I can see data in TWS when I am logged in with the same credentials in paper trading mode.
Could you please suggest what am I doing wrong here?
Thanks!

#9

Does the problem persist after stopping and restarting ibg1?

#10

Now it is not collecting history at all.

If I call quantrocket history collect ... I can see history queue grows, but no log activity or any other responses related to history collection. Canceling requests simply removes them from the queue.
I emptied the queue, rebooted the server, checked that ibg1 is running, confirmed that quantrocket account balance --force-refresh returns valid data (although it now returns two accounts - paper and live, and paper account has more than allowed 1M assets), tried to run quantrocket history collect 'nasdaq-stk-eod' -s '2018-01-01' which simply added it to queue but no activity in the flightlog and nothing interesting with --detail options (no history collection or any confirmation), simply silence.

Is there a way to troubleshoot it without re-installing from scratch? (I know I'm being a pain)
Thanks!

PS: IB now has two-factor authentication with their mobile app which is kind of neat because it does not require UI access to IBGateway. If I try to quantrocket launchpad credentials 'ibg1' --live it reports error that 2FA is enabled, then mobile app bumps in, I confirm connection and quantrocket launchpad status reports running. Also, I can make a request to quantrocket account balance --force-refresh but no history (as described above). Problem with history persists even after switching to paper account which does not require 2FA.
I'd like to keep 2FA on as you are storing IB live account name/password in plain text, so at least I'd like to keep 2FA to myself. With the new IB mobile app this should be easy so I suggest you update documentation and reactions of quantrocket. Probably not related to my history problem anyway.

#11

Two things I would try:

First, download the detailed log file, filtering to quantrocket_history. See what the last log line was. Silence implies it might have gotten stuck on something earlier?

Second, try restarting or re-creating the history container (although rebooting the server should have accomplished the same thing). If there is a stalled collection for any reason this will clear it out.

The 2FA or paper account balance >1M shouldn't be relevant. One way or another there should be activity in the logs.

Thanks for the tip on using the mobile app for 2FA, that is certainly easier than opening the IB Gateway GUI!

#12

After I deleted and re-created databases the collection seems to be going now.
The problem might be me putting the machine to sleep mode yesterday without stopping collection. This may have corrupted databases somehow. Will try to remember not to put Ubuntu machines to sleep from now on :slight_smile:
Thank you!

#13

One more question, Brian.
After initially collecting of approximately 2400 securities from nasdaq it is now "retrying" securities it wasn't able to collect. Actually more than 500 of them, 10 attempts for each, synchronously. The log is now filling with repeating records like:

    quantrocket_license-service_1|fetching license profile for Z29vZ........wODM=
    quantrocket_houston_1|172.18.0.14 - - [06/Mar/2019:23:17:25 +0000] "GET /account/balances.csv?latest=True&fields=NetLiquidation HTTP/1.1" 200 138 "-" "python-requests/2.18.4"
    quantrocket_houston_1|172.18.0.13 - - [06/Mar/2019:23:17:25 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.18.4"
    quantrocket_houston_1|172.18.0.6 - - [06/Mar/2019:23:17:25 +0000] "GET /launchpad/gateways?status=running HTTP/1.1" 200 9 "-" "python-requests/2.18.4"
    quantrocket_account_1|waiting until ECB's next expected 4PM CET update to collect exchange rates
    quantrocket_blotter_1|[spooler /var/tmp/uwsgi/spool pid: 423] managing request uwsgi_spoolfile_on_d154a11cd06a_19_2_35000897_1551826049_475887 ...
    quantrocket_houston_1|172.18.0.11 - - [06/Mar/2019:23:17:28 +0000] "POST /launchpad/registration/ibg1 HTTP/1.1" 200 5 "-" "python-requests/2.18.4"
    quantrocket_history_1|Retrying historical data request for 4 Y of 1 day TRADES for IAC STK (conid 53145883) ending 20190306 04:59:59 GMT after no response received from IB after 50 seconds (retry 6 of 10)
    quantrocket_history_1|ibg1 client 3030 got IB error code 366: No historical data query found for ticker id:7669
    quantrocket_history_1|Issuing to ibg1 historical data request for 4 Y of 1 day TRADES for IAC STK (conid 53145883) ending 20190306 04:59:59 GMT
    quantrocket_houston_1|172.18.0.13 - - [06/Mar/2019:23:17:28 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.18.4"
    quantrocket_houston_1|172.18.0.19 - - [06/Mar/2019:23:17:28 +0000] "GET /launchpad/gateways?status=running HTTP/1.1" 200 9 "-" "python-requests/2.18.4"
    quantrocket_houston_1|172.18.0.13 - - [06/Mar/2019:23:17:29 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.18.4"
    quantrocket_houston_1|172.18.0.2 - - [06/Mar/2019:23:17:29 +0000] "GET /launchpad/gateways?status=running HTTP/1.1" 200 9 "-" "python-requests/2.18.4"
       quantrocket_ibg1_1|[spooler /var/tmp/uwsgi/spool/monitor pid: 12] managing request uwsgi_spoolfile_on_5419b1c55ee4_11_2_1005688585_1550136170_758789 ...
    quantrocket_blotter_1|[spooler /var/tmp/uwsgi/spool pid: 423] managing request uwsgi_spoolfile_on_d154a11cd06a_19_2_35000897_1551826049_475887 ...
    quantrocket_blotter_1|[spooler /var/tmp/uwsgi/spool pid: 423] managing request uwsgi_spoolfile_on_d154a11cd06a_19_2_35000897_1551826049_475887 ...
    quantrocket_history_1|┌───────────────────────────────────┐
    quantrocket_history_1|│ nasdaq-stk-eod processing stages: │
    quantrocket_history_1|│ head timestamps             0     │
    quantrocket_history_1|│ historical data           427     │
    quantrocket_history_1|│ waiting for responses       0     │
    quantrocket_history_1|│ bar processing              0     │
    quantrocket_history_1|│ database storage            0     │
    quantrocket_history_1|│ completed                2448     │
    quantrocket_history_1|└───────────────────────────────────┘
    quantrocket_history_1|┌────────────────────────────────────────────────┐
    quantrocket_history_1|│ nasdaq-stk-eod completion report:              │
    quantrocket_history_1|│ successful completions                    2101 │
    quantrocket_history_1|│ data unavailable                           287 │
    quantrocket_history_1|│ partial completions before no response     128 │
    quantrocket_history_1|│ no permission                              106 │
    quantrocket_history_1|│ security not found (error 200)               4 │
    quantrocket_history_1|│ head timestamp errors/timeouts              55 │
    quantrocket_history_1|└────────────────────────────────────────────────┘
    quantrocket_blotter_1|[spooler /var/tmp/uwsgi/spool pid: 423] managing request uwsgi_spoolfile_on_d154a11cd06a_19_2_35000897_1551826049_475887 ...
    quantrocket_history_1|┌───────────────────────────────────┐
    quantrocket_history_1|│ nasdaq-stk-eod IB response times: │
    quantrocket_history_1|│          ibg1                     │
    quantrocket_history_1|│ count  2216.0                     │
    quantrocket_history_1|│ mean      4.3                     │
    quantrocket_history_1|│ std       4.7                     │
    quantrocket_history_1|│ min       0.4                     │
    quantrocket_history_1|│ 25%       0.9                     │
    quantrocket_history_1|│ 50%       1.8                     │
    quantrocket_history_1|│ 75%       5.5                     │
    quantrocket_history_1|│ max      30.0                     │

At this speed it will take days to finish the collection, actually just to confirm it cannot collect remaining 500 securities. I am only trying to follow documentation to define universes: Usage Guide
Am I missing some subscriptions? It is totally unclear from IB web site what data subscriptions do I need to simply collect historical data for nasdaq tickers. Can you please advise what am I doing wrong here as I don't really think collecting data to define universes should take days to complete.
Thank you!

#14

To follow the documentation you linked you would want to collect fundamentals rather than price history. You could then collect history for the universe you create based on the fundamentals.

Regarding the retries, sometimes the IB API can be spotty but it should usually sort itself out. You could also cancel and re-queue the collection and/or restart IB Gateway and that should help get the data flowing again.