Error setting credentials for multiple gateways

Technical Help
#1

I was successfully running 4 gateways and am now attempting to add a 5th gateway. I am getting the following error when calling set_credentials:

HTTPError: ('500 Server Error: Internal Server Error for url: http://localhost:1969/ibg5/credentials', {'status': 'error', 'msg': 'an unhandled exception occurred, please check flightlog for the traceback and notify support if the problem persists as this might be a bug', 'error': "Destination path '/root/Jts/ebpefcdobokjkclmjkdkfbachdldnonaodinmdlo' already exists"})

The first 4 calls to set_credentials work fine, it is the 5th one that keeps failing. I have tried to recreate all the gateways, even removing them all and re-adding them (by removing the docker-compose-override.yml and rebuilding the docker containers, removing the orphans)

I also received this exact same error when first moving from a single gateway to 4 gateways, but I just rebuilt the services a couple of times and finally it decided to work. This time it fails every time.

I tried to revert back to 4 gateways but somehow it remembers that I tried to add a 5th gateway when I try to stop/start gateways (even though I removed the ibg5 from override.yml and rebuilt the services) and now I can't seem to revert back to what was working before. I am stuck.

Any suggestions?

#2

I have seen that error message a couple times lately and it seems to be a red herring. If setting the credentials fails, QuantRocket tries to roll back the changes and restore the previous settings. This error message is happening when trying to roll back. What seems to be happening is that the login succeeds (i.e. valid credentials), at which point IB Gateway creates the settings directory (/root/Jts/ebpefcdobokjkclmjkdkfbachdldnonaodinmdlo), but then IB Gateway has a problem of some sort which QuantRocket detects and tries to roll back by moving the old settings directory back but fails because it encounters the new settings directory. (With a normal login failure IB Gateway doesn't create the settings directory so the roll back works.)

Tips:

  • It sounds like you are calling set_credentials in succession on all gateways. If so, try just doing one at a time, manually. Once you have set credentials for a gateway, you don't need to do it again unless you redeploy to a new server. So you can just focus on getting that 5th one working. The problem might come from trying to set so many credentials at once, since starting up seems to cause the IB Gateways to do a lot of work and maybe it's too much at one time.
  • Have the IB Gateway GUI open while setting credentials to see if that reveals anything.
  • The ibgrouter service will forget the removed gateway in a couple minutes. You can also restart the ibgrouter service to make it find the gateways anew.
QuantRocket 2.1.0 Release Notes
#3

FYI - I finally tracked this down to an issue with the ib account setup - it was configured as an "ib lite" account which does not support API access, which was causing repeated failures for this specific account.