Backtest randomly stops with error

Issues
#1

Hi Brian,

Lately and often I get the below error after starting a backtest run. It usually runs fine. However, randomly after the backtest has been running for over 20 minutes or more it suddenly throws the below error. I am always connected to the internet and have no connectivity issues either. Any idea why this keeps happening?

Thanks

Here is the log output:

HTTPError Traceback (most recent call last)
File :10, in

File /opt/conda/lib/python3.9/site-packages/quantrocket/zipline.py:636, in backtest(strategy, data_frequency, capital_base, bundle, start_date, end_date, progress, params, filepath_or_buffer)
632 _params["progress"] = progress
634 response = houston.post("/zipline/backtests/{0}".format(strategy), params=_params, timeout=606096)
--> 636 houston.raise_for_status_with_json(response)
638 filepath_or_buffer = filepath_or_buffer or sys.stdout
639 write_response_to_filepath_or_buffer(filepath_or_buffer, response)

File /opt/conda/lib/python3.9/site-packages/quantrocket/houston.py:212, in Houston.raise_for_status_with_json(response)
210 e.json_response = {}
211 e.args = e.args + ("please check the logs for more details",)
--> 212 raise e

File /opt/conda/lib/python3.9/site-packages/quantrocket/houston.py:204, in Houston.raise_for_status_with_json(response)
199 """
200 Raises 400/500 error codes, attaching a json response to the
201 exception, if possible.
202 """
203 try:
--> 204 response.raise_for_status()
205 except requests.exceptions.HTTPError as e:
206 try:

File /opt/conda/lib/python3.9/site-packages/requests/models.py:960, in Response.raise_for_status(self)
957 http_error_msg = u'%s Server Error: %s for url: %s' % (self.status_code, reason, self.url)
959 if http_error_msg:
--> 960 raise HTTPError(http_error_msg, response=self)

HTTPError: ('400 Client Error: BAD REQUEST for url: http://houston/zipline/backtests/LongShortStratProdRefe?data_frequency=daily&capital_base=5000000&bundle=usstock-1d-bundle&start_date=2021-01-01&end_date=2023-03-21&progress=D', {'status': 'error', 'msg': "('403 Client Error: FORBIDDEN for url: http://houston/history/refe-fundamentals.csv?start_date=2019-09-01&end_date=2021-07-14&fields=ForwardEnterpriseValueToOperatingCashFlow&fields=LongTermGrowth_Mean&fields=EnterpriseValueToEBITDA_&fields=CombinedAlpha&fields=CompanyMarketCap', {'status': 'error', 'error': 'A license key is required (view license details if this is unexpected, see Usage Guide for help)'}) (see detailed logs for full traceback)"})

QuantRocket 2.9.0 Release Notes
#2

I apologize, that's very annoying for a license service error to interrupt a long-running backtest. I have seen that error occasionally (though not often). Sporadic errors are by their nature hard to pin down, but I suspect it has nothing to do with your outside internet but is some sort of transient cache miss when one container is talking to another. In the upcoming 2.9 release there is actually a new retry around this part of this code that will hopefully reduce or eliminate the error.

For the time being, my only tentative advice would be to restart the license-service container and perhaps restart any containers that are holding a lot of memory (as revealed by docker stats). I don't know that those suggestions are applicable here, but experience tells me that weird transient errors like this tend to be more common when systems are under greater load, so it's worth a try.

#3

Hi Brian,

I have tried the container restart on many occasions but as you mention it's not really a fix as it does still keep happing. I will upgrade once 2.9 is available.

Thanks for the info.

#4

Version 2.9.0 is now available and includes a retry on license service errors. If you still see the errors after upgrading, please let me know.

#5

Thanks very much.

#6

BTW has the Theia editor gone now?

#7

Hi Brian,

I am sending this in again as the license error seems to be back for some reason. The system has plenty of resources and memory and even after restarting QuantRocket the error occurs often. I am not also sure why the system needs to check the license status so often from a process perspective.

here is the error. Thanks for any further advice you may have.


File /opt/conda/lib/python3.9/site-packages/quantrocket/houston.py:225, in Houston.raise_for_status_with_json(response)
    223     e.json_response = {}
    224     e.args = e.args + ("please check the logs for more details",)
--> 225 raise e

File /opt/conda/lib/python3.9/site-packages/quantrocket/houston.py:217, in Houston.raise_for_status_with_json(response)
    212 """
    213 Raises 400/500 error codes, attaching a json response to the
    214 exception, if possible.
    215 """
    216 try:
--> 217     response.raise_for_status()
    218 except requests.exceptions.HTTPError as e:
    219     try:

File /opt/conda/lib/python3.9/site-packages/requests/models.py:1021, in Response.raise_for_status(self)
   1016     http_error_msg = (
   1017         f"{self.status_code} Server Error: {reason} for url: {self.url}"
   1018     )
   1020 if http_error_msg:
-> 1021     raise HTTPError(http_error_msg, response=self)

HTTPError: ('400 Client Error: BAD REQUEST for url: http://houston/zipline/backtests/LS-Strat-2sig?data_frequency=daily&capital_base=5000000&bundle=usstock-1d-bundle&start_date=2012-01-01&end_date=2023-06-07&progress=D', {'status': 'error', 'msg': "('403 Client Error: FORBIDDEN for url: http://houston/history/refe-fundamentals.csv?start_date=2017-12-09&end_date=2019-02-06&fields=CompanyCommonName&fields=GICSSectorName&fields=Symbol', {'status': 'error', 'error': 'A license key is required (view license details if this is unexpected, see http://qrok.it/h/lkey for help)'}) (see detailed logs for full traceback)"})
#8

I saw this error this week as well and was able to pin down the root cause. There will be a 2.9.1 patch release soon with a fix. The retry logic in 2.9.0 was a hopeful fix based on not being able to replicate the error but this will be a definitive fix. I apologize for the inconvenience.

#9

Ok, Thank you

#10

QuantRocket 2.9.1 with a fix for this issue is now available.