Lot of license issues leading to crashing backtests

Technical Help
#1

For a few days I get errors like this on 2.6, docker on mac, 64GB:

2021-12-14 07:08:44 quantrocket.blotter: ERROR Traceback (most recent call last):
2021-12-14 07:08:44 quantrocket.blotter: ERROR   File "sym://qrocket_log_py", line 34, in wrapped
2021-12-14 07:08:44 quantrocket.blotter: ERROR   File "sym://qrocket_spooler_py", line 75, in spooler_start_order_monitor
2021-12-14 07:08:44 quantrocket.blotter: ERROR   File "sym://qrocket_blotter_ibkr_sid_py", line 66, in refresh_cache
2021-12-14 07:08:44 quantrocket.blotter: ERROR   File "/opt/conda/lib/python3.8/site-packages/quantrocket/master.py", line 545, in download_master_file
2021-12-14 07:08:44 quantrocket.blotter: ERROR     houston.raise_for_status_with_json(response)
2021-12-14 07:08:44 quantrocket.blotter: ERROR   File "/opt/conda/lib/python3.8/site-packages/quantrocket/houston.py", line 206, in raise_for_status_with_json
2021-12-14 07:08:44 quantrocket.blotter: ERROR     raise e
2021-12-14 07:08:44 quantrocket.blotter: ERROR   File "/opt/conda/lib/python3.8/site-packages/quantrocket/houston.py", line 198, in raise_for_status_with_json
2021-12-14 07:08:44 quantrocket.blotter: ERROR     response.raise_for_status()
2021-12-14 07:08:44 quantrocket.blotter: ERROR   File "/opt/conda/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
2021-12-14 07:08:44 quantrocket.blotter: ERROR     raise HTTPError(http_error_msg, response=self)
2021-12-14 07:08:44 quantrocket.blotter: ERROR requests.exceptions.HTTPError: ('403 Client Error: FORBIDDEN for url: http://houston/master/securities.csv?vendors=ibkr&fields=ibkr_ConId', {'status': 'error', 'error': 'A license key is required (view li

If I force a refresh of the licence it works for a bit and then it starts again to spit these errors. My backtests are sometimes running for 24hrs+ and they start to crash now because of this error. Did something change? Is there an issue with the licence server? How can I improve this unworkable situation?

cheers, P

#2

There is no backend issue going on, moreover if there were, the license service will continue to use the cached license profile so that you’re not locked out. So I think it must be something local. Have you tried restarting all the containers? Particularly the master and license-service as they seem to be the two involved. Is there anything in the license service logs? Try a restart.

#3

FWIW, I've also had long-running backtests interrupted twice with these random license errors in the last couple months. Same exception that's posted above. They happened randomly, and I wasn't able to reproduce (didn't occur again with same backtest after a docker restart).

#4

Looking a little closer, the above error wouldn’t actually cause a backtest to crash. That error is happening in the blotter’s order monitor, and the blotter will just try again in a little bit. A zipline backtest doesn’t interact with the blotter. Not saying you couldn’t have a backtest fail because of a similar error, but it wouldn’t be due to that particular traceback.

1 Like