Listings collection fails silently

Technical Help
#1

Hello, we have recently upgraded from the evaluation Plan to Supersonic to access all exchanges.
This has necessitated collecting new listings, in the process of which we've run into an issue: No listings whatsoever are being downloaded.
In fact we can not even list the exchanges we should have available to us, executing "quantrocket master exchanges" returns nothing. Flightlog (in debug mode) doesn't emit any error messages either, as far as I can tell.

What I have noticed however is that when triggering the listings collection, requests are being made to "interactivebrokers.com". Opening the URL printed in flightlog however I get redirected to the homepage, leading me to believe that the URL QuantRocket is requesting listing information from no longer exists.
If I replace ".com" with ".co.uk" the list of listings comes up. I'm not quite sure whether this is an issue on QuantRocket's or IB's side or whether we have simply overlooked something, but for the moment it appears as though we are unable to interact with listings in any way.

Screen%20Shot
Screen%20Shot2124×600 182 KB

Existing listings continue to work, for example we had NYSE listings downloaded and exporting those as well as data collection for databases based on NYSE stocks created before this happened work. However, new listings can not be obtained.

To replicate this (at least on our end) it suffices to execute one of the following commands:
quantrocket master exchanges
quantrocket master collect -e ARCA (or any other exchange)

QuantRocket 1.6.0 Release Notes
#2

Sorry for the trouble. I believe that IB redirects from interactivebrokers.com based on your location in order to show information for the correct IB subsidiary. QuantRocket uses Python requests, which fill follow these redirects. But it sounds like IB has temporarily or permanently changed the redirect such that from your location you get redirected to the homepage, which would explain the warning message in your log capture that no listings were found (since there are none on the home page).

Both of the following URLs work fine from my US location:

This might be a temporary problem that will resolve itself on IB's end, but in case it's permanent, we will add an environment variable toggle in the pending release so you can change your base url to .co.uk.

The new release should be out before end of week.

#3

Did this clear up on its own or is it still failing?

If failing, the new release allows you to override the IB URL using the IB_WEB environment variable on quantrocket/master:

#4

Hey Brian, first off, thanks for the quick turnaround on this!

Unfortunately the issue has not yet been resolved.
The listings collection fails after a simple update (with no change to the IB_WEB variable), however it also fails when IB_WEB is set to

https://www.interactivebrokers.co.uk

as instructed in the version notes.

Here is the combined config output of "docker-compose config": hastebin
The environment variable appears to get picked up correctly, so that part of the process appears to work.

However, once we actually look at the logs we see a lot of exceptions. Additionally, it appears as though the listings collection does not honor the IB_WEB variable and still connects to the default

interactivebrokers. com

instead.

#5

Here is an excerpt of the log that was captured today, after an upgrade to the latest version, setting the environment variable and doing a full reboot of everything: MEGA

I'm posting this separately because the forum will not allow me to add more than two links per reply.

#6

You have two sets of containers running, one set like quantrocket_master_1 etc and another set like quantrocket_docker_master_1. Your use of the listing command is still hitting the old deployment. This could happen if you put your compose file in a new directory called quantrocket_docker instead of overwriting the old compose file in the quantrocket directory.

The container prefix is called the project name and defaults to the directory of the compose file, which by convention should be quantrocket. You can override it with the -p option if you're using a different directory: docker-compose -p quantrocket up -d.

You should terminate both sets of containers:

docker-compose -p quantrocket down
docker-compose -p quantrocket_docker down

Then bring up the deployment using the new compose file, either after moving it to a directory called quantrocket or always remembering to use the -p quantrocket option with docker-compose.

Can not see my files, only QuickStart folder
#7

Ah, a very stupid mistake, I do apologise.
After properly bringing up the new deployment the listings collection is indeed working as expected.

The only thing that still does not work is listing the exchanges via

quantrocket master exchanges

or any of its more specific variants, i.e. with a region specified as it always comes up empty.
There are also no mentions of failure in the log, it simply sends a successful request to Houston.

Screen%20Shot
Screen%20Shot1438×124 12.1 KB

#8

Alas we failed to apply IB_WEB to that command. For now you can get the exchange codes from the quantrocket.com account page or from the IB website.

In the next patch release we're actually going to replace IB_WEB with auto-detection and handling of the redirect (and will apply it to this command too). IB's redirecting of European customers to the .co.uk homepage is probably motivated by the cookie notice displayed there and may be here to stay. It's a poor user experience for new users to have to set IB_WEB just to collect listings, so QuantRocket will handle this automatically.

#9

That's great news, thanks so much!
For now this works perfectly fine for us as well.

#10

A new patch release addressing this issue is now available.