* Error in `uwsgi-quantrocket': malloc(): smallbin double linked list corrupted: 0x00007fb65aa95350 *

peterfabakker · January 23, 2021, 4:35am

I'm stuggling ith the following error: *** Error in `uwsgi-quantrocket': malloc(): smallbin double linked list corrupted: 0x00007fb65aa95350 ***

I use zipline within the last version of quantrocket 2.4.1

This started to happen after I upgraded from 2.3.x to 2.4.1.

Except for reinstalling and downloading all data again, which will take a while is there something else
I could do?

Brian · January 25, 2021, 1:26pm

What are you doing when that error occurs? And what operating system are you using? If you can simplify it to a reproducible error and post the code to reproduce it, that would help.

It's an error from Zipline's C extensions, and we have seen a few similar ones occasionally in backtests recently. If it’s happening in a backtest, the short-term workaround unfortunately is to try to pin down what part of the backtest triggers the error and avoid it. Or you can edit docker-compose.yml and switch back and forth between versions of the Zipline container, if it only happens in the newer version.

Big picture, Zipline’s Achilles heel is that it uses lots of C extensions and it is pinned to fairly old versions of various libraries. That’s a bit of a recipe for problems. An important item on our road map is to modernize the Zipline codebase by updating it to work with recent versions of Python/pandas/etc. While Zipline will never be as simple and predictable as something like Moonshot, modernizing the codebase is the best strategy for eliminating these kinds of issues.

peterfabakker · February 3, 2021, 7:15am

after endless testing I found it must be the database as it fails on the same day on different algo's. I threw away the database and installed again. Now the ingest fails at ticker 4851 out of 19229. I redoing the ingest (after waiting 6 hours and 30GB downloaded) and I'll post the error.

peterfabakker · February 3, 2021, 8:39am

2021-02-03 07:01:54 quantrocket.zipline: INFO [usstock] Ingesting minute bars for 19229 securities in usstock bundle
2021-02-03 07:58:16 quantrocket.zipline: INFO [usstock] Ingesting daily bars for usstock bundle
2021-02-03 07:58:17 quantrocket.zipline: ERROR Traceback (most recent call last):
2021-02-03 07:58:17 quantrocket.zipline: ERROR File "sym://qrocket_log_py", line 34, in wrapped
2021-02-03 07:58:17 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_ingest_py", line 77, in mule_ingest_bundle
2021-02-03 07:58:17 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_ingest_py", line 130, in _call_ingest_func
2021-02-03 07:58:17 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 109, in ingest
2021-02-03 07:58:17 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 495, in _sync_daily_bars
2021-02-03 07:58:17 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.6/subprocess.py", line 291, in check_call
2021-02-03 07:58:17 quantrocket.zipline: ERROR raise CalledProcessError(retcode, cmd)
2021-02-03 07:58:17 quantrocket.zipline: ERROR subprocess.CalledProcessError: Command '['aws', 's3', 'sync', '--quiet', '--exact-timestamps', 's3://data.quantrocket.com/usstock/zipline/usstock-1min/daily_equities.bcolz', '/var/lib/quantrocket/zipline/data/usstock/2020-01-01T00;00;00/daily_equities.bcolz']' returned non-zero exit status 1.
2021-02-03 07:58:17 quantrocket.zipline: ERROR

Brian · February 3, 2021, 6:27pm

According to your log output, the minute bars finished ingesting at 7:58 and then there was a failure syncing the daily data directory. A temporary network error could be a possible cause (the failed command is just syncing a directory from S3 to your machine, it's not getting into any of Zipline’s C extensions and not related to the original error), so the first thing I would try is just to run the ingest command again which will try to sync the daily directory again.

What is the date that fails in backtests?

peterfabakker · February 5, 2021, 8:14am

So I bought a new computer (MacbookPro 64GB just to be sure) installed everything from scratch....

Identical error pattern

I suspect it has to do with values in the database for SPY (maybe the volumes are too big for ints?)

I'm willing privately to share my algo file for bugfixing purposes.... I'm pretty desparate

the dates are consistent: ["20200508","20200506","20200507","20200507", "20200308"]

My universe is fixed:
context.SPY = algo.sid('FIBBG000BDTBL9')
context.GLD = algo.sid("FIBBG000CRF6Q8")#"SPDR GOLD SHARES"
context.VNQ= algo.sid("FIBBG000Q89NG6")#"VANGUARD REAL ESTATE ETF"
context.EEM= algo.sid("FIBBG000M0P5L2")#"ISHARES MSCI EMERGING MARKET"
context.RWX= algo.sid('FIBBG000Q8TD76')
context.IEF=algo.sid('FIBBG000BHR296')
context.TLT= algo.sid('FIBBG000BJKYW3')
context.EWJ=algo.sid('FIBBG000BK38F5')
context.VGK=algo.sid('FIBBG000HT8SM3')
context.SLV=algo.sid('FIBBG000NDCRW7')
context.DBB=algo.sid('FIBBG000QNK9K6')
context.UUP=algo.sid('FIBBG000Q7R318')
context.TLH=algo.sid('FIBBG000QN16D1')

context.XLC=algo.sid("FIBBG00L5F9F38") #""COMM SERV SELECT SECTOR SPDR"
context.XLY=algo.sid("FIBBG000BJ1MF9") #""CONSUMER DISCRETIONARY SELT"
context.XLP=algo.sid("FIBBG000BJ10N8") #""CONSUMER STAPLES SPDR"
context.XLE=algo.sid("FIBBG000BJ20S2") #""ENERGY SELECT SECTOR SPDR"
context.XLF=algo.sid("FIBBG000BJ29X7") #""FINANCIAL SELECT SECTOR SPDR"
context.XLV=algo.sid("FIBBG000BHVSG6") #""HEALTH CARE SELECT SECTOR"
context.XLI=algo.sid("FIBBG000BJ2RF7") #""INDUSTRIAL SELECT SECT SPDR"
context.XLB=algo.sid("FIBBG000BHNNF7") #""MATERIALS SELECT SECTOR SPDR"
context.XLRE=algo.sid("FIBBG00B597128") #"REAL ESTATE SELECT SECT SPDR"
context.XLK=algo.sid("FIBBG000BJ7007") #""TECHNOLOGY SELECT SECT SPDR"
context.XLU=algo.sid("FIBBG000BJ7G75") #""UTILITIES SELECT SECTOR SPDR"

Sometimes the error is :

quantrocket_zipline_1|Fatal Python error: GC object already tracked
quantrocket_zipline_1|
quantrocket_zipline_1|Thread 0x00007fbb0b3e7700 (most recent call first):
quantrocket_zipline_1|
quantrocket_zipline_1|Thread 0x00007fbb0abe6700 (most recent call first):
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/threading.py", line 295 in wait
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/queue.py", line 164 in get
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/logging/handlers.py", line 1401 in dequeue
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/logging/handlers.py", line 1452 in monitor
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/threading.py", line 864 in run
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/threading.py", line 916 in bootstrap_inner
quantrocket_zipline_1| File "/opt/c, line 884t (most recent call first):
quantrocket_zipline_1|"/opt/conda/lib/python3.6/site-packages/zipline/utils/eventst File "/optl/th3.s-agktp", line ps File "/opt/co/ry441rpe File /opt/conda/lib/pythkfksl_it.py", line 587 in dispatch_request
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/flask/views.py", line 84 in view
quantrocket_zipline_1| File "/opt/conda/lib/lib/python3.6/siteaksa/py", line 1598 in dispatch_request
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1612 in full_dispatch_request
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1982 in wsgi_app
quantrocket_zipline_1| File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1994 in call
quantrocket_zipline_1| File "sym://qrocket_wsgi_py", line 30 in app
quantrocket_houston_1|2021/02/05 07:46:35 [error] 12#12: *153 upstream prematurely closed connection while reading response header from upstream, client: 172.18.0.9, server: , request: "POST /zipline/backtests/riskpar?start_date=2020-01-01&end_date=2021-02-02&progress=W HTTP/1.1", upstream: "uwsgi://172.18.0.19:80", host: "houston"
quantrocket_houston_1|172.18.0.9 - - [05/Feb/2021:07:46:35 +0000] "POST /zipline/backtests/riskpar?start_date=2020-01-01&end_date=2021-02-02&progress=W HTTP/1.1" 502 0 "-" "python-urllib3/1.26.2"
quantrocket_zipline_1|DAMN ! worker 1 (pid: 37) died, killed by signal 6 trying respawn ...
quantrocket_zipline_1|Respawned uWSGI worker 1 (new pid: 49)
quantrocket_zipline_1|[deadlock-detector] a process holding a robust mutex died. recovering..

Sometimes it's

    quantrocket_houston_1|172.18.0.11 - - [05/Feb/2021:08:03:40 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
    quantrocket_zipline_1|Traceback (most recent call last):
    quantrocket_zipline_1|  File "sym://qrocket_app_py", line 798, in post
    quantrocket_zipline_1|  File "sym://qrocket_qrzipline_backtest_py", line 167, in backtest_algo
    quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/algorithm.py", line 669, in run
    quantrocket_zipline_1|    for perf in self.get_generator():
    quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/gens/tradesimulation.py", line 205, in transform
    quantrocket_zipline_1|    for capital_change_packet in every_bar(dt):
    quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/gens/tradesimulation.py", line 133, in every_bar
    quantrocket_zipline_1|    handle_data(algo, current_data, dt_to_use)
    quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/utils/events.py", line 218, in handle_data
    quantrocket_zipline_1|    dt,
    quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/utils/events.py", line 237, in handle_data
    quantrocket_zipline_1|    self.callback(context, data)
    quantrocket_zipline_1|  File "riskpar", line 672, in record_vars
    quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/algorithm.py", line 1578, in account
    quantrocket_zipline_1|    self._sync_last_sale_prices()
    quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/algorithm.py", line 1567, in _sync_last_sale_prices
    quantrocket_zipline_1|    self.data_portal,
    quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/finance/metrics/tracker.py", line 201, in sync_last_sale_prices
    quantrocket_zipline_1|    handle_non_market_minutes=handle_non_market_minutes,
    quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/finance/ledger.py", line 464, in sync_last_sale_prices
    quantrocket_zipline_1|    handle_non_market_minutes=handle_non_market_minutes,
    quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/finance/ledger.py", line 285, in sync_last_sale_prices
    quantrocket_zipline_1|    update_position_last_sale_prices(self.positions, get_price, dt)
    quantrocket_zipline_1|  File "zipline/finance/_finance_ext.pyx", line 14, in zipline.finance._finance_ext.update_position_last_sale_prices (zipline/finance/_finance_ext.c:2133)
    quantrocket_zipline_1|  File "zipline/finance/_finance_ext.pyx", line 32, in zipline.finance._finance_ext.update_position_last_sale_prices (zipline/finance/_finance_ext.c:1945)
    quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/data/data_portal.py", line 572, in get_scalar_asset_spot_value
    quantrocket_zipline_1|    data_frequency,
    quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/data/data_portal.py", line 468, in _get_single_asset_value
    quantrocket_zipline_1|    asset, "close", dt, ffill=True,
    quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/data/data_portal.py", line 718, in _get_minute_spot_value
    quantrocket_zipline_1|    query_dt = reader.get_last_traded_dt(asset, dt)
    quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/data/dispatch_bar_reader.py", line 101, in get_last_traded_dt
    quantrocket_zipline_1|    return r.get_last_traded_dt(asset, dt)
    quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/data/minute_bars.py", line 1153, in get_last_traded_dt
    quantrocket_zipline_1|    minute_pos = self._find_last_traded_position(asset, dt)
    quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/data/minute_bars.py", line 1179, in _find_last_traded_position
    quantrocket_zipline_1|    self._minutes_per_day,
    quantrocket_zipline_1|  File "zipline/data/_minute_bar_internal.pyx", line 130, in zipline.data._minute_bar_internal.find_last_traded_position_internal (zipline/data/_minute_bar_internal.c:2371)
    quantrocket_zipline_1|  File "zipline/data/_minute_bar_internal.pyx", line 87, in zipline.data._minute_bar_internal.find_position_of_minute (zipline/data/_minute_bar_internal.c:2094)
    quantrocket_zipline_1|OverflowError: value too large to convert to int

and somtimes its:
quantrocket_flightlog_1|2021-02-05 08:11:26 quantrocket.zipline: INFO [riskpar] ███------- 31% 2020-05-04 8% 3.01 -4% $79208
quantrocket_houston_1|172.18.0.4 - - [05/Feb/2021:08:11:26 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket_zipline_1|!!! uWSGI process 39 got Segmentation Fault !!!
quantrocket_zipline_1|*** backtrace of 39 ***
quantrocket_zipline_1|uwsgi-quantrocket(uwsgi_backtrace+0x30) [0x46ea00]
quantrocket_zipline_1|uwsgi-quantrocket(uwsgi_segfault+0x21) [0x46eda1]
quantrocket_zipline_1|/lib/x86_64-linux-gnu/libc.so.6(+0x350e0) [0x7fbe11ca10e0]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x1482e3) [0x7fbe123962e3]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x26fd) [0x7fbe1239915d]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x146a60) [0x7fbe12394a60]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x14848a) [0x7fbe1239648a]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x1e7d) [0x7fbe123988dd]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x145e74) [0x7fbe12393e74]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x1485e8) [0x7fbe123965e8]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x26fd) [0x7fbe1239915d]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x146a60) [0x7fbe12394a60]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyFunction_FastCallDict+0x10c) [0x7fbe12394cfc]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x166) [0x7fbe122b9ba6]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyObject_Call_Prepend+0xcc) [0x7fbe122b9dfc]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x9e) [0x7fbe122b9ade]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x1ba22f) [0x7fbe1240822f]
quantrocket_zipline_1|/opt/conda/lib/python3.6/site-packages/zipline/finance/_finance_ext.cpython-36m-x86_64-linux-gnu.so(+0x1050b) [0x7fbde75ca50b]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyCFunction_FastCallDict+0x102) [0x7fbe12310df2]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x1484ec) [0x7fbe123964ec]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x26fd) [0x7fbe1239915d]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x146a60) [0x7fbe12394a60]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x14848a) [0x7fbe1239648a]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x1e7d) [0x7fbe123988dd]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x146a60) [0x7fbe12394a60]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x14848a) [0x7fbe1239648a]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x1e7d) [0x7fbe123988dd]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x146a60) [0x7fbe12394a60]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x14848a) [0x7fbe1239648a]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x26fd) [0x7fbe1239915d]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x146a60) [0x7fbe12394a60]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x14848a) [0x7fbe1239648a]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x26fd) [0x7fbe1239915d]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x146a60) [0x7fbe12394a60]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(PyEval_EvalCodeEx+0x63) [0x7fbe12394ee3]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x9edc2) [0x7fbe122ecdc2]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(PyObject_Call+0x56) [0x7fbe122b9e96]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x8b63e) [0x7fbe122d963e]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyObject_GenericGetAttrWithDict+0x93) [0x7fbe12314843]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x4598) [0x7fbe1239aff8]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x145e74) [0x7fbe12393e74]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x1485e8) [0x7fbe123965e8]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x26fd) [0x7fbe1239915d]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x145e74) [0x7fbe12393e74]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x1485e8) [0x7fbe123965e8]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x26fd) [0x7fbe1239915d]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x145e74) [0x7fbe12393e74]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x1485e8) [0x7fbe123965e8]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x26fd) [0x7fbe1239915d]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x95e33) [0x7fbe122e3e33]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x248e) [0x7fbe12398eee]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x95e33) [0x7fbe122e3e33]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x248e) [0x7fbe12398eee]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x146a60) [0x7fbe12394a60]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x14848a) [0x7fbe1239648a]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x26fd) [0x7fbe1239915d]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x146a60) [0x7fbe12394a60]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x14848a) [0x7fbe1239648a]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x1e7d) [0x7fbe123988dd]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x146a60) [0x7fbe12394a60]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(PyEval_EvalCodeEx+0x63) [0x7fbe12394ee3]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x9eee1) [0x7fbe122ecee1]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(PyObject_Call+0x56) [0x7fbe122b9e96]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x37d6) [0x7fbe1239a236]
quantrocket_zipline_1|*** end of backtrace ***
quantrocket_houston_1|2021/02/05 08:11:26 [error] 12#12: *184 upstream prematurely closed connection while reading response header from upstream, client: 172.18.0.13, server: , request: "POST /zipline/backtests/riskpar?start_date=2020-01-01&end_date=2021-02-02&progress=W HTTP/1.1", upstream: "uwsgi://172.18.0.4:80", host: "houston"
quantrocket_houston_1|172.18.0.13 - - [05/Feb/2021:08:11:26 +0000] "POST /zipline/backtests/riskpar?start_date=2020-01-01&end_date=2021-02-02&progress=W HTTP/1.1" 502 0 "-" "python-urllib3/1.26.2"
quantrocket_zipline_1|DAMN ! worker 1 (pid: 39) died, killed by signal 11 trying respawn ...
quantrocket_zipline_1|Respawned uWSGI worker 1 (new pid: 51)
quantrocket_zipline_1|[deadlock-detector] a process holding a robust mutex died. recovering...

Brian · February 5, 2021, 9:58pm

I believe the triggering problem might be associated with 2020-05-08. Can you try ending your backtest before that date or starting it after that date and see if you can work around the error that way?

peterfabakker · February 6, 2021, 1:45am

That is correct.... when I end the backtest on the 7th of May 2020 or start it at the 9th of May 2020 it will continue. Is there something I can do about that as I need the full backtest to check things?

bjsun · February 8, 2021, 2:20am

I'm getting a similar error on two backtest runs this weekend on the exact same date (May 8 2020). The error was different on each run, both are below. Will I need to re-download the data bundles if I edit docker-compose.yml to use an older version of the zipline container?

Error from first backtest:

quantrocket_zipline_1|Traceback (most recent call last):
quantrocket_zipline_1|  File "sym://qrocket_app_py", line 798, in post
quantrocket_zipline_1|  File "sym://qrocket_qrzipline_backtest_py", line 167, in backtest_algo
quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/algorithm.py", line 669, in run
quantrocket_zipline_1|    for perf in self.get_generator():
quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/gens/tradesimulation.py", line 205, in transform
quantrocket_zipline_1|    for capital_change_packet in every_bar(dt):
quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/gens/tradesimulation.py", line 133, in every_bar
quantrocket_zipline_1|    handle_data(algo, current_data, dt_to_use)
quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/utils/events.py", line 218, in handle_data
quantrocket_zipline_1|    dt,
quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/utils/events.py", line 237, in handle_data
quantrocket_zipline_1|    self.callback(context, data)
quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/algorithm.py", line 479, in handle_data
quantrocket_zipline_1|    self._handle_data(self, data)
quantrocket_zipline_1|  File "backAlgo", line 416, in handle_data
quantrocket_zipline_1|  File "zipline/_protocol.pyx", line 121, in zipline._protocol.check_parameters.__call__.assert_keywords_and_call (zipline/_protocol.c:3824)
quantrocket_zipline_1|  File "zipline/_protocol.pyx", line 773, in zipline._protocol.BarData.history (zipline/_protocol.c:10268)
quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/data/data_portal.py", line 967, in get_history_window
quantrocket_zipline_1|    field, data_frequency)
quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/data/data_portal.py", line 806, in _get_history_daily_window
quantrocket_zipline_1|    assets, days_for_window, end_dt, field_to_use, data_frequency
quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/data/data_portal.py", line 836, in _get_history_daily_window_data
quantrocket_zipline_1|    days_for_window[0:-1]
quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/data/data_portal.py", line 1117, in _get_daily_window_data
quantrocket_zipline_1|    extra_slot)
quantrocket_zipline_1|  File "/opt/conda/lib/python3.6/site-packages/zipline/data/history_loader.py", line 554, in history
quantrocket_zipline_1|    axis=1,
quantrocket_zipline_1|  File "<__array_function__ internals>", line 6, in concatenate
quantrocket_zipline_1|ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 0, the array at index 0 has size 149 and the array at index 22 has size 597375
quantrocket_zipline_1|
quantrocket_zipline_1|!!! uWSGI process 27 got Segmentation Fault !!!
quantrocket_zipline_1|*** backtrace of 27 ***
quantrocket_zipline_1|uwsgi-quantrocket(uwsgi_backtrace+0x30) [0x46ea00]
quantrocket_zipline_1|uwsgi-quantrocket(uwsgi_segfault+0x21) [0x46eda1]
quantrocket_zipline_1|/lib/x86_64-linux-gnu/libc.so.6(+0x350e0) [0x7f11ccc850e0]
quantrocket_zipline_1|/lib/x86_64-linux-gnu/libc.so.6(cfree+0x14) [0x7f11ccccc614]
quantrocket_zipline_1|/opt/conda/lib/python3.6/site-packages/numpy/core/_multiarray_umath.cpython-36m-x86_64-linux-gnu.so(+0x7202e) [0x7f11c65a902e]
quantrocket_zipline_1|/opt/conda/lib/python3.6/site-packages/numpy/core/_multiarray_umath.cpython-36m-x86_64-linux-gnu.so(+0x2d7de) [0x7f11c65647de]
quantrocket_zipline_1|/opt/conda/lib/python3.6/site-packages/pandas/_libs/lib.cpython-36m-x86_64-linux-gnu.so(+0x15856) [0x7f11b897f856]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0xd750a) [0x7f11cd30950a]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0xa1635) [0x7f11cd2d3635]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0xd750a) [0x7f11cd30950a]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0xb27ef) [0x7f11cd2e47ef]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0xb8904) [0x7f11cd2ea904]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0xd7675) [0x7f11cd309675]
quantrocket_zipline_1|/opt/conda/lib/python3.6/site-packages/zipline/finance/_finance_ext.cpython-36m-x86_64-linux-gnu.so(+0x32ff) [0x7f11946e12ff]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0xb874d) [0x7f11cd2ea74d]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0xd7675) [0x7f11cd309675]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0xb874d) [0x7f11cd2ea74d]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0xd7675) [0x7f11cd309675]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0xb874d) [0x7f11cd2ea74d]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0xd7675) [0x7f11cd309675]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x8586d) [0x7f11cd2b786d]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x9d372) [0x7f11cd2cf372]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x184d87) [0x7f11cd3b6d87]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x184d97) [0x7f11cd3b6d97]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x184d97) [0x7f11cd3b6d97]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x184d97) [0x7f11cd3b6d97]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x981) [0x7f11cd37b3e1]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x146a60) [0x7f11cd378a60]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(PyEval_EvalCodeEx+0x63) [0x7f11cd378ee3]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x9eee1) [0x7f11cd2d0ee1]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(PyObject_Call+0x56) [0x7f11cd29de96]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x37d6) [0x7f11cd37e236]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x146a60) [0x7f11cd378a60]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyFunction_FastCallDict+0x10c) [0x7f11cd378cfc]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x166) [0x7f11cd29dba6]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyObject_Call_Prepend+0xcc) [0x7f11cd29ddfc]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(PyObject_Call+0x56) [0x7f11cd29de96]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x37d6) [0x7f11cd37e236]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x146a60) [0x7f11cd378a60]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyFunction_FastCallDict+0x10c) [0x7f11cd378cfc]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x166) [0x7f11cd29dba6]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyObject_Call_Prepend+0xcc) [0x7f11cd29ddfc]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(PyObject_Call+0x56) [0x7f11cd29de96]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x37d6) [0x7f11cd37e236]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x146a60) [0x7f11cd378a60]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(PyEval_EvalCodeEx+0x63) [0x7f11cd378ee3]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x9eee1) [0x7f11cd2d0ee1]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(PyObject_Call+0x56) [0x7f11cd29de96]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x37d6) [0x7f11cd37e236]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x146a60) [0x7f11cd378a60]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(PyEval_EvalCodeEx+0x63) [0x7f11cd378ee3]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x9eee1) [0x7f11cd2d0ee1]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(PyObject_Call+0x56) [0x7f11cd29de96]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x37d6) [0x7f11cd37e236]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x145e74) [0x7f11cd377e74]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x1485e8) [0x7f11cd37a5e8]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x26fd) [0x7f11cd37d15d]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x145e74) [0x7f11cd377e74]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x1485e8) [0x7f11cd37a5e8]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x26fd) [0x7f11cd37d15d]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x145e74) [0x7f11cd377e74]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x1485e8) [0x7f11cd37a5e8]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x26fd) [0x7f11cd37d15d]
quantrocket_zipline_1|/opt/conda/lib/libpython3.6m.so.1.0(+0x145e74) [0x7f11cd377e74]
quantrocket_zipline_1|*** end of backtrace ***

The line referenced from handle_data in my algo (line 416) is:

context.daily_hist = data.history(context.assets, ["price", "open", "high", "low", "close", "volume"], 150, "1d")

Error from second backtest (nothing out of the ordinary preceded this):

quantrocket_zipline_1|*** Error in `uwsgi-quantrocket': malloc(): smallbin double linked list corrupted: 0x000000000992ad70 ***
quantrocket_houston_1|2021/02/07 17:01:21 [error] 11#11: *62419 upstream prematurely closed connection while reading response header from upstream, client: 172.18.0.21, server: , request: "POST /zipline/backtests/backAlgo?data_frequency=minute&bundle=usstock-minute&start_date=2020-01-01&end_date=2020-12-31 HTTP/1.1", upstream: "uwsgi://172.18.0.19:80", host: "houston"
quantrocket_houston_1|172.18.0.21 - - [07/Feb/2021:17:01:21 +0000] "POST /zipline/backtests/backAlgo?data_frequency=minute&bundle=usstock-minute&start_date=2020-01-01&end_date=2020-12-31 HTTP/1.1" 502 0 "-" "-"
quantrocket_zipline_1|DAMN ! worker 1 (pid: 27) died, killed by signal 6 :( trying respawn ...
quantrocket_zipline_1|Respawned uWSGI worker 1 (new pid: 42)
quantrocket_zipline_1|[deadlock-detector] a process holding a robust mutex died. recovering...

bjsun · February 8, 2021, 9:03am

Btw you can easily reproduce this error in a backtest strategy file with the following code (run the backtest with a date range that includes May 8 2020). The error always pops at 14:51 (US/Eastern time) on that date. What's interesting is that the error message/backtrace is different each time, perhaps because it's a memory error of some sort (and different things are in memory with each run).

def handle_data(context, data):

    asset = sid('FIBBG000BDTBL9')  #SPY
    print('BACKTEST TIME:', data.current_dt.astimezone(eastern))
    daily_hist = data.history(asset, ["price", "open", "high", "low", "close", "volume"], 150, "1d")

Brian · February 8, 2021, 8:41pm

This is under investigation but in the meantime the only workaround is to avoid that date.

peterfabakker · February 10, 2021, 12:12pm

if peeps need to have the full test: here a script that stitches things together

#STITCHING MULTIPLE BACKTESTS
from quantrocket.zipline import backtest
import pyfolio as pf
from datetime import datetime as dt
import pandas as pd
from quantrocket.zipline import ZiplineBacktestResult

name_test = "dev"
algo= "XXXXX"
algofile="../zipline/{}.py".format(algo)
basename = "{}-tmp-results.csv" #base result filename 
tot_result = '{}-results_tot.csv'.format(algo) ## what will te consolidated file called
start_date1 ="2008-01-01"
end_date1 = "2020-05-07" # between 2020-05-07 and 2020-05-09 aka 2020-05-08, all backtest fail
start_date2 = "2020-05-09"
end_date2 = "2021-02-02"


BT_result1 = ''
BT_result2 = ''

now = dt.now()
BT_result1 = basename.format(now)
file_algo = "{}-{}-{}.py".format(now,algo,name_test)
shutil.copyfile(algofile, file_algo)
backtest(
    algo,
    capital_base = 1000000,
    start_date= start_date1,
    end_date=  end_date1, 
    progress="W",
    filepath_or_buffer=BT_result1)

backtest1= pd.read_csv(BT_result1)
perf1 = backtest1[backtest1.dataframe=='perf']
maxi=max(perf1['index'])
portfolio_value = float(perf1[perf1['index']==maxi][perf1[perf1['index']==maxi]['column']=='ending_value'].value)

now = dt.now()
BT_result2 = basename.format(now)

backtest(
    algo,
    capital_base = portfolio_value,
    start_date=start_date2,
    end_date=end_date2, 
    progress="W",
    filepath_or_buffer=BT_result2)



backtest2= pd.read_csv(BT_result2)

benchmark1 = backtest1[backtest1.dataframe=='benchmark']
maxindex1 = max(benchmark.index)
maxindex1 = maxindex1+1
benchmark2 = backtest2[backtest2.dataframe=='benchmark']
benchmark2["index"] = benchmark2["index"]+ maxindex1
benchmarktot = pd.concat([benchmark1,benchmark2])

perf2 = backtest2[backtest2.dataframe=='perf']
maxindexperf = max(perf1.index)
maxindexperf = maxindexperf+1

maxdd = float(perf1[perf1['index']==maxi][perf1[perf1['index']==maxi]['column']=='max_drawdown'].value)
maxlev = float(perf1[perf1['index']==maxi][perf1[perf1['index']==maxi]['column']=='max_leverage'].value)
def transform_row(r):
    if r['column']== 'max_drawdown':
        if float(r['value']) < maxdd:
            r['value'] = str(maxdd)
    if r['column']== 'max_leverage':
        if float(r['value']) < maxlev:
            r['value'] = str(maxlev)
    return r

perf2.apply(transform_row, axis=1)
perf2["index"] = perf2["index"]+ maxindexperf
perftot = pd.concat([perf1,perf2])

tran1 = backtest1[backtest1.dataframe=='transactions']
maxindextran= max(tran1.index)
maxindextran = maxindextran+1
tran2 = backtest2[backtest2.dataframe=='transactions']
tran2["index"] = tran2["index"]+ maxindextran
trantot = pd.concat([tran1,tran2])

returns1 = backtest1[backtest1.dataframe=='returns']
maxindexreturns= max(returns1.index)
maxindexreturns = maxindexreturns+1
returns2 = backtest2[backtest2.dataframe=='returns']
returns2["index"] = returns2["index"]+ maxindexreturns
returnstot = pd.concat([returns1,returns2])

positions1 = backtest1[backtest1.dataframe=='positions']
maxindexpositions= max(positions1.index)
maxindexpositions = maxindexpositions+1
positions2 = backtest2[backtest2.dataframe=='positions']
positions2["index"] = backtest2["index"]+ maxindexpositions
positionstot = pd.concat([positions1,positions2])

backtesttot = pd.concat([benchmarktot,perftot,trantot,returnstot,positionstot])

result = backtesttot.to_csv(tot_result)
pf.from_zipline_csv(tot_result, round_trips=True)

mantle · February 10, 2021, 10:57pm

I can confirm that I'm getting the same issue as reported above.

Brian · February 15, 2021, 6:04pm

We rebuilt the daily bundle a couple different ways over the weekend as most of the evidence seems to suggest the problem may be related to that. Unfortunately this didn’t resolve the issue so we are still investigating.

peterfabakker · February 25, 2021, 3:47am

Any progress? this really handicaps my research

Brian · February 26, 2021, 2:16pm

We’re still working on it. Sorry for the inconvenience. We’ve never had a bug open this long, but this is a slippery fish.

Brian · March 11, 2021, 7:14pm

This issue is fixed in version 2.5.0.

Brian · March 18, 2021, 4:00am

This topic was automatically closed after 6 days. New replies are no longer allowed.