Moonshot segment concat memory intensive

Technical Help
#1

Running a segment="A" backtest.

image
image732×181 28.8 KB

Logs show proper yearly segmentation. Is the final moonshot concatenation running out of memory? 12GB are allocated.

quantrocket_moonshot_1| Backtesting strategy
quantrocket_moonshot_1| Backtesting strategy from 2009-01-01 to 2009-12-30
quantrocket_moonshot_1| Backtesting strategy from 2009-12-31 to 2010-12-30
....
quantrocket_moonshot_1| Backtesting strategy from 2019-12-31 to 2020-12-31
...
# hangs 3-5 min then errors 

quantrocket_flightlog_1|2021-05-15 05:36:21 
quantrocket.moonshot: ERROR the system killed the worker handling the request, 
likely an Out Of Memory error; if you were backtesting, try a segmented backtest to reduce 
memory usage (for example `segment="A"`), or add more memory, see http://qrok.it/h/oom for more help

If this is the case then a segmented backtest isn't very helpful if the final concat fails to bring together those segmented years. Is there a work-around or other practice for this scenario?

#2

Using details=True with a segmented backtest will tend to negate some or all of the memory benefit of a segmented backtest. The intention is to use details=False with segmented backtests.