Zipline backtest with CustomFundamental times out

Technical Help
#1

I have been trying to use the Zipline research and algo environment by using CustomFundamentals.

I have successfully imported the CustomFundamnetal data set and can run a pipeline request successfully in research. Although the data does take some time to come through for just one day's worth.

pipe shape (2097, 38)
Time taken: 0:01:28.179415

However, as soon as I try to run a similar pipeline in a strategy code, Zipline times out with the below message. I have tested it with a pipe just with price and volume info and it works fine but as soon as I uncomment a CustomFundamental field it times out.

Thanks for any help on this as I have no idea where the issue lies.

--------------------------------------------------------------------------

HTTPError Traceback (most recent call last)
in ()
7 end_date= "2021-05-01",
8
----> 9 filepath_or_buffer="Long_Short_results.csv")

/opt/conda/envs/zipline/lib/python3.6/site-packages/quantrocket/zipline.py in backtest(strategy, data_frequency, capital_base, bundle, start_date, end_date, progress, filepath_or_buffer)
551 response = houston.post("/zipline/backtests/{0}".format(strategy), params=params, timeout=606096)
552
--> 553 houston.raise_for_status_with_json(response)
554
555 filepath_or_buffer = filepath_or_buffer or sys.stdout

/opt/conda/envs/zipline/lib/python3.6/site-packages/quantrocket/houston.py in raise_for_status_with_json(response)
204 e.json_response = {}
205 e.args = e.args + ("please check the logs for more details",)
--> 206 raise e
207
208 # Instantiate houston so that all callers can share a TCP connection (for

/opt/conda/envs/zipline/lib/python3.6/site-packages/quantrocket/houston.py in raise_for_status_with_json(response)
196 """
197 try:
--> 198 response.raise_for_status()
199 except requests.exceptions.HTTPError as e:
200 try:

/opt/conda/envs/zipline/lib/python3.6/site-packages/requests/models.py in raise_for_status(self)
941
942 if http_error_msg:
--> 943 raise HTTPError(http_error_msg, response=self)
944
945 def close(self):

HTTPError: ('500 Server Error: INTERNAL SERVER ERROR for url: http://houston/zipline/backtests/LongShortStrat?data_frequency=daily&bundle=usstock-1d-bundle&start_date=2020-05-01&end_date=2021-05-01&progress=M', {'status': 'error', 'msg': "HTTPConnectionPool(host='houston', port=80): Read timed out. (read timeout=120)"})

#2

Does the timeout persist if you restart zipline and run one backtest at a time?

docker-compose restart zipline

Also check the detailed logs as that should reveal what exactly is timing out.

#3

Hi Brian,

The issue seems to happen when I include a screen filter referring to or any type of Custom Fundamental data field in the pipeline definition. If I comment them out the error does not occur.

I seem to be able to use Custom Fundamentals in with exact definitions in research without such errors

e.g.

pipe = Pipeline(screen= Universe("us-nys"),
columns={
'name': CustomFundamentals.Symbol.latest,

'market_cap': CustomFundamentals.market_val.latest,

'entval': CustomFundamentals.entval_cmpt.latest,

'sector': CustomFundamentals.RBICSFocus_l2_name.latest,

                         'price' : USEquityPricing.close.latest,
                         'volume':  USEquityPricing.volume.latest,

Here is the log output relate to this.

Blockquote
quantrocket_houston_1|172.18.0.5 - - [14/May/2021:20:31:32 +0000] "GET /history/databases HTTP/1.1" 200 25 "-" "-"
quantrocket_houston_1|172.18.0.5 - - [14/May/2021:20:31:32 +0000] "GET /realtime/databases HTTP/1.1" 200 3 "-" "-"
quantrocket_houston_1|172.18.0.18 - - [14/May/2021:20:31:35 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.24.0"
quantrocket_houston_1|172.18.0.15 - - [14/May/2021:20:31:35 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 3 "-" "-"
quantrocket_blotter_1|recycling spooler after 90 tasks
quantrocket_blotter_1|OOOPS the spooler is no more...trying respawn...
quantrocket_blotter_1|spawned the uWSGI spooler on dir /var/tmp/uwsgi/spool with pid 249
quantrocket_houston_1|172.18.0.15 - - [14/May/2021:20:31:55 +0000] "GET /master/securities.csv?vendors=ibkr&fields=ibkr_ConId HTTP/1.1" 400 137 "-" "-"
quantrocket_houston_1|172.18.0.15 - - [14/May/2021:20:31:55 +0000] "GET /master/securities.csv?vendors=alpaca&fields=alpaca_AssetId HTTP/1.1" 400 139 "-" "-"
quantrocket_houston_1|172.18.0.15 - - [14/May/2021:20:31:56 +0000] "GET /account/balances.csv?latest=True&fields=Account&fields=Broker HTTP/1.1" 200 79 "-" "-"
quantrocket_houston_1|172.18.0.18 - - [14/May/2021:20:31:56 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.24.0"
quantrocket_houston_1|172.18.0.15 - - [14/May/2021:20:31:56 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 3 "-" "-"
quantrocket_jupyter_1|[I 20:32:16.571 NotebookApp] Saving file at /intro_zipline/Untitled4.ipynb
quantrocket_jupyter_1|[W 20:32:16.572 NotebookApp] Notebook intro_zipline/Untitled4.ipynb is not trusted
quantrocket_houston_1|172.18.0.16 - - [14/May/2021:20:32:21 +0000] "POST /ibgrouter/registration/ibg1 HTTP/1.1" 200 5 "-" "python-requests/2.24.0"
quantrocket_houston_1|172.18.0.18 - - [14/May/2021:20:32:21 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.24.0"
quantrocket_houston_1|172.18.0.7 - - [14/May/2021:20:32:21 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 3 "-" "-"
quantrocket_account_1|waiting until ECB's next expected 4PM CET update to collect exchange rates
quantrocket_account_1|recycling mule after 30 tasks
quantrocket_account_1|OOOPS mule 1 (pid: 162) crippled...trying respawn...
quantrocket_account_1|spawned uWSGI mule 1 (pid: 180)
quantrocket_houston_1|172.18.0.18 - - [14/May/2021:20:32:56 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.24.0"
quantrocket_houston_1|172.18.0.15 - - [14/May/2021:20:32:56 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 3 "-" "-"
quantrocket_blotter_1|recycling spooler after 90 tasks
quantrocket_blotter_1|OOOPS the spooler is no more...trying respawn...
quantrocket_blotter_1|spawned the uWSGI spooler on dir /var/tmp/uwsgi/spool with pid 251
quantrocket_houston_1|172.18.0.15 - - [14/May/2021:20:33:17 +0000] "GET /master/securities.csv?vendors=ibkr&fields=ibkr_ConId HTTP/1.1" 400 137 "-" "-"
quantrocket_houston_1|172.18.0.15 - - [14/May/2021:20:33:17 +0000] "GET /master/securities.csv?vendors=alpaca&fields=alpaca_AssetId HTTP/1.1" 400 139 "-" "-"
quantrocket_houston_1|172.18.0.15 - - [14/May/2021:20:33:17 +0000] "GET /account/balances.csv?latest=True&fields=Account&fields=Broker HTTP/1.1" 200 79 "-" "-"
quantrocket_houston_1|172.18.0.18 - - [14/May/2021:20:33:17 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.24.0"
quantrocket_houston_1|172.18.0.15 - - [14/May/2021:20:33:17 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 3 "-" "-"
quantrocket_houston_1|172.18.0.18 - - [14/May/2021:20:33:21 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.24.0"
quantrocket_houston_1|172.18.0.7 - - [14/May/2021:20:33:21 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 3 "-" "-"
quantrocket_license-service_1|fetching license profile for YXV0........NTcx
quantrocket_houston_1|172.18.0.22 - - [14/May/2021:20:33:24 +0000] "GET /account/balances.csv?latest=True&fields=NetLiquidation&fields=Paper HTTP/1.1" 200 101 "-" "-"
quantrocket_houston_1|172.18.0.5 - - [14/May/2021:20:33:32 +0000] "GET /zipline/bundles HTTP/1.1" 499 0 "-" "-"
quantrocket_zipline_1|Traceback (most recent call last):
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 426, in _make_request
quantrocket_zipline_1| six.raise_from(e, None)
quantrocket_zipline_1| File "", line 3, in raise_from
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request
quantrocket_zipline_1| httplib_response = conn.getresponse()
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/http/client.py", line 1331, in getresponse
quantrocket_zipline_1| response.begin()
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/http/client.py", line 297, in begin
quantrocket_zipline_1| version, status, reason = self._read_status()
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/http/client.py", line 258, in _read_status
quantrocket_zipline_1| line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/socket.py", line 586, in readinto
quantrocket_zipline_1| return self._sock.recv_into(b)
quantrocket_zipline_1|socket.timeout: timed out
quantrocket_zipline_1|
quantrocket_zipline_1|During handling of the above exception, another exception occurred:
quantrocket_zipline_1|
quantrocket_zipline_1|Traceback (most recent call last):
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
quantrocket_zipline_1| timeout=timeout
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
quantrocket_zipline_1| method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/urllib3/util/retry.py", line 410, in increment
quantrocket_zipline_1| raise six.reraise(type(error), error, _stacktrace)
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/urllib3/packages/six.py", line 735, in reraise
quantrocket_zipline_1| raise value
quantrocket_houston_1|172.18.0.17 - - [14/May/2021:20:33:32 +0000] "POST /zipline/backtests/LongShortStrat?data_frequency=daily&bundle=usstock-1d-bundle&start_date=2020-05-01&end_date=2021-05-01&progress=M HTTP/1.1" 500 110 "-" "python-urllib3/1.26.3"
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
quantrocket_zipline_1| chunked=chunked,
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 428, in _make_request
quantrocket_zipline_1| self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 336, in _raise_timeout
quantrocket_zipline_1| self, url, "Read timed out. (read timeout=%s)" % timeout_value
quantrocket_zipline_1|urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='houston', port=80): Read timed out. (read timeout=120)
quantrocket_zipline_1|
quantrocket_zipline_1|During handling of the above exception, another exception occurred:
quantrocket_zipline_1|
quantrocket_zipline_1|Traceback (most recent call last):
quantrocket_zipline_1| File "sym://qrocket_app_py", line 807, in post
quantrocket_zipline_1| File "sym://qrocket_qrzipline_backtest_py", line 167, in backtest_algo
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/zipline/algorithm.py", line 675, in run
quantrocket_zipline_1| for perf in self.get_generator():
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/zipline/gens/tradesimulation.py", line 223, in transform
quantrocket_zipline_1| algo.before_trading_start(self.current_data)
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/zipline/algorithm.py", line 470, in before_trading_start
quantrocket_zipline_1| self.compute_eager_pipelines()
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/zipline/algorithm.py", line 643, in compute_eager_pipelines
quantrocket_zipline_1| self.pipeline_output(name)
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/zipline/utils/api_support.py", line 103, in wrapped_method
quantrocket_zipline_1| return method(self, *args, **kwargs)
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/zipline/algorithm.py", line 2467, in pipeline_output
quantrocket_zipline_1| return self._pipeline_output(pipe, chunks, name)
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/zipline/algorithm.py", line 2479, in _pipeline_output
quantrocket_zipline_1| pipeline, today, next(chunks),
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/zipline/algorithm.py", line 2526, in run_pipeline
quantrocket_zipline_1| self.engine.run_pipeline(pipeline, start_session, end_session),
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/zipline/pipeline/engine.py", line 392, in run_pipeline
quantrocket_zipline_1| hooks,
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/zipline/pipeline/engine.py", line 442, in _run_pipeline_impl
quantrocket_zipline_1| hooks=hooks,
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/zipline/pipeline/engine.py", line 692, in compute_chunk
quantrocket_zipline_1| domain, to_load, mask_dates, sids, mask,
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/zipline/pipeline/loaders/db.py", line 68, in load_adjusted_array
quantrocket_zipline_1| data_frequency=dataset.DATA_FREQUENCY)
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/quantrocket/price.py", line 672, in get_prices_reindexed_like
quantrocket_zipline_1| data_frequency=data_frequency)
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/quantrocket/price.py", line 220, in get_prices
quantrocket_zipline_1| zipline_bundles = set(list_bundles())
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/quantrocket/zipline.py", line 255, in list_bundles
quantrocket_zipline_1| response = houston.get("/zipline/bundles")
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/requests/sessions.py", line 543, in get
quantrocket_zipline_1| return self.request('GET', url, **kwargs)
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/quantrocket/houston.py", line 165, in request
quantrocket_zipline_1| method, url, *args, headers=self.headers, **kwargs)
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
quantrocket_zipline_1| resp = self.send(prep, **send_kwargs)
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
quantrocket_zipline_1| r = adapter.send(request, **kwargs)
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/requests/adapters.py", line 529, in send
quantrocket_zipline_1| raise ReadTimeout(e, request=request)
quantrocket_zipline_1|requests.exceptions.ReadTimeout: HTTPConnectionPool(host='houston', port=80): Read timed out. (read timeout=120)

#4

The issue is that, for some reason, you don’t have enough zipline workers. When you include custom fundamentals, that results in a call to get_prices, which in turn queries zipline for a list of bundles to see if the database you specified is a zipline bundle. That call is timing out because there are no zipline workers available to handle the request. By default there are two zipline workers, and one of them is busy running your backtest, so there should be another available to handle the request to list the bundles but it seems not in your case. That’s why I suggested restarting zipline just in case there was a stuck worker, and making sure you’re only running one backtest and/or zipline query at a time.

As a workaround, you can increase the number of workers to 3 (or higher) by creating a docker-compose.override.yml that looks like this:

version: '2.4'
services:
  zipline:
    environment:
      UWSGI_WORKERS: 3

Then re-deploy zipline:

docker-compose up -d zipline

I think we might also bump up the default number of workers in the next release to allow a bit more cushion.

#5

Thanks, Brian, That fixed it.

I also took out a few redundant master.securities lookup functions.

Any specific tips on how to ensure the most efficient way to get zipline backtests to run? I am looking for any kind of best practices recommendations.

Thanks again for your response.