Why do we shift() to get future returns / fill positions?

Technical Help
#1

I've bee going through the sample code and the one thing I can't intuitively understand is the nature of the shift() function on the DataFrame when we are calculating gross returns or filling positions. I looked up what shift does and it looks like the equivalent of zero-padding in a convolutional neural network, just curious why its needed to fill positions and calculate returns specifically though. What if you didn't shift()?

#2

shift() moves the values down a row and is simply about properly aligning your DataFrames according to what you're trying to calculate. A simple example: if you want to know the size of an opening gap, you want opens - closes.shift(), which means the current period's open minus the prior period's close. In contrast, opens - closes is the current period's open minus the current period's close.

Whether to shift depends entirely on what you're trying to model. In the case of modeling fills, if you generate signals based on a closing price and enter the following day, you probably want to shift your signals to get your positions. If you enter the same session as the signal, you probably don't want to shift.

Try making a toy DataFrame of signals and derived positions and returns and see how it all lines up. Do it for a strategy that enters the next day and then for one that enters the same day.

#3

Awesome, great explanation. Thank you Brian!

#4

Hi @Brian , can you explain something:

In Usage Guide - > Moonshot - > How a Moonshot backtest works, we have this code:

from moonshot import Moonshot

class DualMovingAverageStrategy(Moonshot):

CODE = "dma-tech"
DB = "usstock-1d"
UNIVERSES = "tech-giants"
LMAVG_WINDOW = 300
SMAVG_WINDOW = 100

def prices_to_signals(self, prices):
    closes = prices.loc["Close"]

    # Compute long and short moving averages
    lmavgs = closes.rolling(self.LMAVG_WINDOW).mean()
    smavgs = closes.rolling(self.SMAVG_WINDOW).mean()

    # Go long when short moving average is above long moving average
    signals = smavgs.shift() > lmavgs.shift()

    return signals.astype(int)

def signals_to_target_weights(self, signals, prices):
    # spread our capital equally among our trades on any given day
    weights = self.allocate_equal_weights(signals) # provided by moonshot.mixins.WeightAllocationMixin
    return weights

def target_weights_to_positions(self, weights, prices):
    # we'll enter in the period after the signal
    positions = weights.shift()
    return positions

def positions_to_gross_returns(self, positions, prices):
    # Our return is the security's close-to-close return, multiplied by
    # the size of our position. We must shift the positions DataFrame because
    # we don't have a return until the period after we open the position
    closes = prices.loc["Close"]
    gross_returns = closes.pct_change() * positions.shift()
    return gross_returns

But "positions" have a shift as "positions = weights.shift()" in target_weights_to_positions() function, so isn't it worth to replace in last function

gross_returns = closes.pct_change() * positions.shift()

by this

gross_returns = closes.pct_change().shift() * positions

??

#5

How and whether to shift depends on the strategy. In this end-of-day strategy, if you get a signal on Tuesday, you enter on Wednesday. That’s one shift: positions = weights.shift(). You don’t get your return on that position until Thursday. That’s the second shift: positions.shift(). Thursday’s pct_change() is the Wednesday to Thursday return, so you want to leave that as is, not shift forward.

My best advice is to make toy dataframes and step through it to see what’s happening.

2 Likes