Master collect-ibkr timeouts

Issues
#1

Hi,
I ran the command last night to create a security master from IBKR data. I looked in the flight log and it was downloading things. This morning the data browser has nothing in it. I ran the collect-ibkr command again for US stocks and ETFs. I get these timeouts. The IB GUI shows everything green. When I run an account command from your console it shows me correct recently updated information which implies that at some level the IB api connection i working.
upyter:/codeload $ quantrocket master collect-ibkr --countries 'US' --sec-types 'STK' 'ETF'
Traceback (most recent call last):
File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 462, in _make_request
httplib_response = conn.getresponse()
^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/http/client.py", line 1378, in getresponse
response.begin()
File "/opt/conda/lib/python3.11/http/client.py", line 318, in begin
version, status, reason = self._read_status()
^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/socket.py", line 706, in readinto
return self._sock.recv_into(b)
^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 799, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/urllib3/packages/six.py", line 770, in reraise
raise value
File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 469, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 358, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='houston', port=80): Read timed out. (read timeout=120)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/bin/quantrocket", line 8, in
sys.exit(main())
^^^^^^
File "/opt/conda/lib/python3.11/site-packages/quantrocket/_cli/commands.py", line 97, in main
result, exit_code = func(**args)
^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/quantrocket/master.py", line 388, in _cli_collect_ibkr_listings
return json_to_cli(collect_ibkr_listings, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/quantrocket/_cli/utils/output.py", line 37, in json_to_cli
json_response = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/quantrocket/master.py", line 383, in collect_ibkr_listings
response = houston.post("/master/securities/ibkr", params=params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/quantrocket/houston.py", line 183, in request
return super(Houston, self).request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/requests/adapters.py", line 532, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='houston', port=80): Read timed out. (read timeout=120)
jupyter:/codeload $

#2

There is an issue with collect-ibkr as of last night. This command first collects symbols from the IBKR website (because symbol lists are not available by API) then queries listing details from the IB API. IBKR made a change to their website last night that broke the pagination of the symbol lists, which is causing QuantRocket to retrieve pages endlessly. It's likely that this has consumed too much memory or otherwise clogged your system and is causing the timeouts you're seeing.

First, please restart the master service to clear out the endless data collection:

docker compose restart master

You can run

docker stats

before and after to make sure the memory usage looks okay afterward.

A fix for the root issue will be forthcoming.

#3

Brian,
Thanks for your prompt reply. I am using IB as my broker but I have been trying to move beyond them for data. I subscribe to a couple of historical and real time data providers to that end. However it seems that IB would be a logical source for the security master since I need them to trade index options, ETFs, and futures. What is the actual need for the security master in Quantrocket? In the past I have used Factset and it is good but a bit expensive.

#4

You will need to collect IBKR listings for the products you want to trade before you can trade them, the main reason being that QuantRocket submits IBKR orders using IBKR's unique contract ID (conid), which is one of the fields collected. You will also need to collect IBKR listings if you want to collect real-time or historical data from IBKR. However, you don't need to collect IBKR listings to use other historical data providers such as the built-in US Stock dataset or the Sharadar or EDI datasets, etc.

The pagination issue can be seen on IBKR's website, where if you browse products and click through the pages, the first page of results is re-displayed over and over again on every subsequent page. I have reported this issue to IBKR. Hopefully it should get resolved quickly since it affects all of their users, but we will have to wait for IBKR to resolve it.

Depending on your goals, the easiest course would be to start working with the included US Stock dataset or one of the other data providers while giving this a day or two to hopefully get resolved promptly. If this is blocking you from your next steps, direct message me and I can probably get the securities master records to you in a different way.

#5

ok I will turn my attention to porting my algorithms to zipline, I dont think moonshot supports my use cases.

#6

Interactive Brokers has acknowledged this issue and escalated to the appropriate team. Continuing to monitor...

#7

Interactive Brokers has resolved the pagination issue on their end, and QuantRocket is able to collect listings again. Please re-run the original command that was timing out.

#8

This topic was automatically closed after 10 days. New replies are no longer allowed.