Nan values in us_stock db / daily (in zipline) bundle

The following code is triggering the following warnings / Errors / the concern is that in sometime in September, SPY IEF and other tickers got data gaps. / log is provided as well.

def hist(sec, prc, bars, freq):
sec_ts = data.history(sec, prc, bars, freq)
if np.isnan(np.sum(np.array(sec_ts))):
if is_live_trading(): log_info('ERROR: nan in data history time series!!' + str(sec))
sec_ts = sec_ts.fillna(method='ffill')
return sec_ts

2020-12-21 17:48:53 quantrocket.zipline: INFO [a13v_spyx3] Trading a13v_test3 in account DUXXX
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BDTBL9 [SPY])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BDTBL9 [SPY])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BDTBL9 [SPY])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BDTBL9 [SPY])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BDTBL9 [SPY])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BBV9N3 [LQD])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BDTBL9 [SPY])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BHR296 [IEF])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BBV9N3 [LQD])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BHR296 [IEF])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BDTBL9 [SPY])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BDTBL9 [SPY])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000CX0W11 [TMF])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000CX0W11 [TMF])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BBV9N3 [LQD])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BBV9N3 [LQD])

It sounds like you are encountering a problem that was fixed in the 2.3.2 release and should update to the latest version:

Thank you Brian for a quick response - we will make the upgrade, and report back,
that said - the release notes refer to:
//
In live trading, return real-time data from the previously completed minute, not from the current, partially completed minute.
//

which sounds different from the ticket above
In our case - nan values show up in the end-of-day price series in zipline. it seems that this issue started in early September - I can try to find and send the exact dates when it started - potentially / likely after the update to 2.3.1 version.

Also "nan" values show up in backtests as well as in live trading

thank you for your support and happy holidays!

Are you sure you’ve updated the bundle? It’s not self-updating.

https://www.quantrocket.com/docs/#historical-update-minute-bundle

the following scheduled ingestion db / bundle updates run every day. in cron. So yes... sure.
including the fact that we re-run the ingestion - just in case manually - and still keep getting nan values. let me trace it for you with exact dates - where I believe there are data gaps in daily database.

DATA COLLECTION

20 11 * * mon-fri quantrocket history collect --priority 'spyx-1d-db'
25 11 * * mon-fri quantrocket history collect --priority 'vixd-1d-db'
30 11 * * mon-fri quantrocket fundamental collect-sharadar-sp500 --country 'US'
35 11 * * mon-fri quantrocket fundamental collect-sharadar-fundamentals --country 'US'
40 11 * * mon-fri quantrocket zipline ingest 'usstock-1d-bundle'

You don’t provide enough detail to reproduce the problem, but if I had to guess from your naming conventions, I would guess you are ingesting from a history database and have not updated the underlying history database.