How to release the license from a server to move it to a new server

Technical Help
#1

Hi guys,

Does any of you have experience in migrating to a new server? I backed up databases and the codeload folder .Started a new server and transferred the files over. Unfortunately I ran both servers at the sametime, getting a licencing error: "account limit". Now both the old and the new server won't connect to IB due to the not existing/running license.

Any idea how to get out of this deadlock situation?

#2

There's nothing in place that would shut down the new deployment because of the old one using the license key. The terms of use specifically allow you to run the same license on a dev and production server (and enforcement of that limit is handled asynchronously, not by the deployment itself). So whatever is causing the issue, it's not that. Have you checked your detailed logs for error messages?

#3

Yes I have. Please see log messages:
2018-07-17T09:41:37-07:00 IP quantrocket.license-service ERROR: failure checking license:
2018-07-17T09:41:37-07:00 IP quantrocket.license-service ERROR: ValueError('The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()',)

This message is repeated again and again on both servers at the moment.

I have tried to restart the license container, and done the quantrocket license get --force-refresh still the same.

#4

I think there may be something unexpected about your account balances database. Can you reproduce the ValueError using this code?

import io
import pandas as pd
from quantrocket.account import download_account_balances    
f = io.StringIO()
download_account_balances(f, latest=True, fields=["NetLiquidation"])
account_balances = pd.read_csv(f)
account_balances = account_balances.loc[~account_balances.Account.str.startswith("D")]
bool((account_balances.Currency != "USD").any())
#5

That's all I'm getting out of running that code.. :frowning: Other quantrocket.master modules are loading fine.

ImportError Traceback (most recent call last)
in ()
1 import io
2 import pandas as pd
----> 3 from quantrocket.master import download_account_balances
4
5 f = io.StringIO()

ImportError: cannot import name 'download_account_balances'

#6

Sorry should be

from quantrocket.account import download_account_balances
#7

Very interesting, the old server returns with the account balance for 2 of our IB accounts. The new server returns True

What's the expected output?

#8

Additional information: I didn't notice the error in the log when running the above code. But I still get the warnings when I'm trying to start one of the gateways.

Jul 16 22:30:34 167.99.87.189 quantrocket.ibg1:  INFO Updated INI settings 
Jul 16 22:30:34 167.99.87.189 quantrocket.ibg1:  INFO Starting and stopping gateway to generate API settings file 
Jul 16 22:30:55 167.99.87.189 quantrocket.ibg1:  INFO Updated API settings 
Jul 16 22:31:05 167.99.87.189 quantrocket.ibg1:  ERROR Can't start gateway without active license (run `quantrocket license get` to view license details if this is unexpected)
#9 
bool((account_balances.Currency != "USD").any())

should return True or False. If it returns a Series (such as 2 account balances), that would cause the ValueError (which presumably you're seeing on your old server). Try print(account_balances) and see if something looks screwy?

Also, if you're having issues with two deployments, treat them separately. It's not clear which error messages you've posted go with which deployment. I wouldn't assume the two deployments are having the same issue.

#10

Thanks a lot for the quick replies here late in the evening, it's much appreciated.

You are right, I'm getting the balance for 2 accounts. And you are also right I should keep the issues separate, even I got confused about it.

with that said, they are actually giving the same output. True. And when you put the print in at the end I'm getting the same table and values for both servers. Account number and balance has been changed in the post to keep some privacy.

     Account Currency  NetLiquidation          LastUpdated
  2  U23xxx     USD        0.00              2018-07-16 22:59:09
  3  U25xxx     EUR        0.00              2018-07-16 22:59:09

Any log messages or other debug prints I can assist with please let me know.

#11

Please update your deployment to quantrocket/license-service:debug20180718 and reproduce the error. This should print the full traceback to your logs after "failure checking license".

#12

Hi Brian,

Thanks for taking the time to create the debug image, but had the issue resolved before we got to try it out. The issue was the 2 different currencies EUR/USD.

The reason we experienced the same issue on both servers was down to our backup of the db's to S3. When starting the new server all backed up db's were downloaded including the account.balance db which included the error.

The solution was to delete the account.balance db on S3, start up the new server from scratch, change the currency for all IB accounts to USD, then start the IB gateways one by one and do a force-refresh on the account.balance db. We then pushed this new db to S3 so it's all backed up again.

#13

I was able to reproduce this bug when there are multiple base currencies. Multiple base currencies are supposed to be supported. The issue is fixed in quantrocket/license-service:1.2.1.

#14

Thanks, I upgraded to the new image.