Backtest keep failing during simultaneous backtest runs

Technical Help
#1

Brian,

Lately when I run a couple of simultaneous backtests the entire backtest fails towards the end with Jupyter getting killed as well as lots of hanging processes. It also corrupts the database containers so badly that I have had to resort to recreating the containers as the only way to start clean.

I have no memory issues or disk issues as far as the space is concerned. Do you know what could be causing this? Here are the latest error messages from the log file. This has been somewhat prevalent since the upgrade to 2.10 version.

Thanks

quantrocket-zipline-1|Traceback (most recent call last):
        quantrocket-zipline-1|  File "/opt/conda/lib/python3.11/site-packages/zipline/utils/cache.py", line 151, in get
        quantrocket-zipline-1|    return self._cache[key].unwrap(dt)
        quantrocket-zipline-1|           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
        quantrocket-zipline-1|  File "/opt/conda/lib/python3.11/site-packages/zipline/utils/cache.py", line 81, in unwrap
        quantrocket-zipline-1|    raise Expired(self._expires)
        quantrocket-zipline-1|zipline.utils.cache.Expired: 2023-02-17 00:00:00
        quantrocket-zipline-1|
        quantrocket-zipline-1|During handling of the above exception, another exception occurred:
        quantrocket-zipline-1|
        quantrocket-zipline-1|Traceback (most recent call last):
        quantrocket-zipline-1|  File "/opt/conda/lib/python3.11/site-packages/zipline/algorithm.py", line 2348, in _pipeline_output
        quantrocket-zipline-1|    data = self._pipeline_cache.get(name, today)
        quantrocket-zipline-1|           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        quantrocket-zipline-1|  File "/opt/conda/lib/python3.11/site-packages/zipline/utils/cache.py", line 155, in get
        quantrocket-zipline-1|    raise KeyError(key)
        quantrocket-zipline-1|KeyError: 'my_pipeline'
        quantrocket-zipline-1|
        quantrocket-zipline-1|During handling of the above exception, another exception occurred:
        quantrocket-zipline-1|
        quantrocket-zipline-1|Traceback (most recent call last):
        quantrocket-zipline-1|  File "/opt/conda/lib/python3.11/site-packages/quantrocket/_cli/utils/files.py", line 38, in write_response_to_filepath_or_buffer
        quantrocket-zipline-1|    f.write(chunk)
        quantrocket-zipline-1|OSError: [Errno 28] No space left on device
        quantrocket-zipline-1|
        quantrocket-zipline-1|During handling of the above exception, another exception occurred:
        quantrocket-zipline-1|
        quantrocket-zipline-1|Traceback (most recent call last):
        quantrocket-zipline-1|  File "sym://qrocket_qrzipline_backtest_py", line 75, in mule_backtest_algo
        quantrocket-zipline-1|  File "sym://qrocket_qrzipline_backtest_py", line 258, in backtest_algo
        quantrocket-zipline-1|  File "/opt/conda/lib/python3.11/site-packages/zipline/algorithm.py", line 694, in run
        quantrocket-zipline-1|    for perf in self.get_generator():
        quantrocket-zipline-1|  File "/opt/conda/lib/python3.11/site-packages/zipline/gens/tradesimulation.py", line 200, in transform
        quantrocket-zipline-1|    algo.before_trading_start(self.current_data)
        quantrocket-zipline-1|  File "/opt/conda/lib/python3.11/site-packages/zipline/algorithm.py", line 484, in before_trading_start
        quantrocket-zipline-1|    self.compute_eager_pipelines()
        quantrocket-zipline-1|  File "/opt/conda/lib/python3.11/site-packages/zipline/algorithm.py", line 662, in compute_eager_pipelines
        quantrocket-zipline-1|    self.pipeline_output(name)
        quantrocket-zipline-1|  File "/opt/conda/lib/python3.11/site-packages/zipline/utils/api_support.py", line 103, in wrapped_method
        quantrocket-zipline-1|    return method(self, *args, **kwargs)
        quantrocket-zipline-1|           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        quantrocket-zipline-1|  File "/opt/conda/lib/python3.11/site-packages/zipline/algorithm.py", line 2320, in pipeline_output
        quantrocket-zipline-1|    return self._pipeline_output(pipe, chunks, name)
        quantrocket-zipline-1|           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        quantrocket-zipline-1|  File "/opt/conda/lib/python3.11/site-packages/zipline/algorithm.py", line 2351, in _pipeline_output
        quantrocket-zipline-1|    data, valid_until = self.run_pipeline(
        quantrocket-zipline-1|                        ^^^^^^^^^^^^^^^^^^
        quantrocket-zipline-1|  File "/opt/conda/lib/python3.11/site-packages/zipline/algorithm.py", line 2405, in run_pipeline
        quantrocket-zipline-1|    self.engine.run_pipeline(pipeline, start_session, end_session), \
        quantrocket-zipline-1|    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        quantrocket-zipline-1|  File "/opt/conda/lib/python3.11/site-packages/zipline/pipeline/engine.py", line 395, in run_pipeline
        quantrocket-zipline-1|    return self._run_pipeline_impl(
      quantrocket-flightlog-1|--- Logging error ---
        quantrocket-zipline-1|           ^^^^^^^^^^^^^^^^^^^^^^^^
        quantrocket-zipline-1|  File "/opt/conda/lib/python3.11/site-packages/zipline/pipeline/engine.py", line 442, in _run_pipeline_impl
        quantrocket-zipline-1|    results = self.compute_chunk(
        quantrocket-zipline-1|              ^^^^^^^^^^^^^^^^^^^
        quantrocket-zipline-1|  File "/opt/conda/lib/python3.11/site-packages/zipline/pipeline/engine.py", line 765, in compute_chunk
        quantrocket-zipline-1|    loaded = loader.load_adjusted_array(
        quantrocket-zipline-1|             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
        quantrocket-zipline-1|  File "/opt/conda/lib/python3.11/site-packages/zipline/pipeline/loaders/db.py", line 56, in load_adjusted_array
        quantrocket-zipline-1|    prices = get_prices_reindexed_like(
        quantrocket-zipline-1|             ^^^^^^^^^^^^^^^^^^^^^^^^^^
        quantrocket-zipline-1|  File "/opt/conda/lib/python3.11/site-packages/quantrocket/price.py", line 734, in get_prices_reindexed_like
        quantrocket-zipline-1|    prices = get_prices(
        quantrocket-zipline-1|             ^^^^^^^^^^^
        quantrocket-zipline-1|  File "/opt/conda/lib/python3.11/site-packages/quantrocket/price.py", line 366, in get_prices
        quantrocket-zipline-1|    download_history_file(db, tmp_filepath, **kwargs)
        quantrocket-zipline-1|  File "/opt/conda/lib/python3.11/site-packages/quantrocket/history.py", line 886, in download_history_file
        quantrocket-zipline-1|    write_response_to_filepath_or_buffer(filepath_or_buffer, response)
        quantrocket-zipline-1|  File "/opt/conda/lib/python3.11/site-packages/quantrocket/_cli/utils/files.py", line 35, in write_response_to_filepath_or_buffer
        quantrocket-zipline-1|    with open(filepath_or_buffer, "wb") as f:
        quantrocket-zipline-1|OSError: [Errno 28] No space left on device

 quantrocket-jupyter-1|[I 2024-05-20 18:45:20.734 ServerApp] Saving file at /intro_zipline/Untitled20.ipynb
        quantrocket-jupyter-1|[E 2024-05-20 18:45:20.735 ServerApp] Error while saving file: intro_zipline/Untitled20.ipynb database or disk is full
        quantrocket-jupyter-1|    Traceback (most recent call last):
        quantrocket-jupyter-1|      File "/opt/conda/lib/python3.11/site-packages/jupyter_server/services/contents/filemanager.py", line 959, in save
        quantrocket-jupyter-1|        self.check_and_sign(nb, path)
        quantrocket-jupyter-1|      File "/opt/conda/lib/python3.11/site-packages/jupyter_server/services/contents/manager.py", line 723, in check_and_sign
        quantrocket-jupyter-1|        self.notary.sign(nb)
        quantrocket-jupyter-1|      File "/opt/conda/lib/python3.11/site-packages/nbformat/sign.py", line 480, in sign
        quantrocket-jupyter-1|        self.store.store_signature(signature, self.algorithm)
        quantrocket-jupyter-1|      File "/opt/conda/lib/python3.11/site-packages/nbformat/sign.py", line 216, in store_signature
        quantrocket-jupyter-1|        self.db.execute(
        quantrocket-jupyter-1|    sqlite3.OperationalError: database or disk is full
        quantrocket-jupyter-1|[W 2024-05-20 18:45:20.744 ServerApp] 500 PUT /jupyter/api/contents/intro_zipline/Untitled20.ipynb?1716230720727 (172.22.0.10): Unexpected error while saving file: intro_zipline/Untitled20.ipynb database or disk is full
        quantrocket-jupyter-1|[W 2024-05-20 18:45:20.744 ServerApp] wrote error: 'Unexpected error while saving file: intro_zipline/Untitled20.ipynb database or disk is full'
        quantrocket-jupyter-1|    Traceback (most recent call last):
        quantrocket-jupyter-1|      File "/opt/conda/lib/python3.11/site-packages/jupyter_server/services/contents/filemanager.py", line 959, in save
        quantrocket-jupyter-1|        self.check_and_sign(nb, path)
        quantrocket-jupyter-1|      File "/opt/conda/lib/python3.11/site-packages/jupyter_server/services/contents/manager.py", line 723, in check_and_sign
        quantrocket-jupyter-1|        self.notary.sign(nb)
        quantrocket-jupyter-1|      File "/opt/conda/lib/python3.11/site-packages/nbformat/sign.py", line 480, in sign
        quantrocket-jupyter-1|        self.store.store_signature(signature, self.algorithm)
        quantrocket-jupyter-1|      File "/opt/conda/lib/python3.11/site-packages/nbformat/sign.py", line 216, in store_signature
        quantrocket-jupyter-1|        self.db.execute(
        quantrocket-jupyter-1|    sqlite3.OperationalError: database or disk is full
        quantrocket-jupyter-1|    
        quantrocket-jupyter-1|    The above exception was the direct cause of the following exception:
#2

The traceback indicates that you're out of disk space. This doesn't necessarily mean your computer is out of space but that Docker's virtual disk is out of space. You can run

df -h

from a JupyterLab terminal to check the space. Then you can open Docker Desktop > Resources to increase the virtual disk limit. You can also run

docker system prune --all

to prune any unused images that might be taking up space. If you didn't run that after updating to 2.10, that might be why you're bumping up against the disk limits since then.

#3

I had pruned all unused space. I will try to increase the vdisk size but I had 128 GB allocated and I can't see how that was close to being used up!

Thanks

#4

I think I found the culprit. The deleted files in Jupyter were being put in the .Trash folder. I remember this behavior had been the default and then suddenly it wasn't happening anymore after one of the updates.

And now it's back to doing it when I delete a file in Jupyter.

Strange!

1 Like