Nan values in us_stock db / daily (in zipline) bundle

Technical Help
#1

The following code is triggering the following warnings / Errors / the concern is that in sometime in September, SPY IEF and other tickers got data gaps. / log is provided as well.

def hist(sec, prc, bars, freq):
sec_ts = data.history(sec, prc, bars, freq)
if np.isnan(np.sum(np.array(sec_ts))):
if is_live_trading(): log_info('ERROR: nan in data history time series!!' + str(sec))
sec_ts = sec_ts.fillna(method='ffill')
return sec_ts

2020-12-21 17:48:53 quantrocket.zipline: INFO [a13v_spyx3] Trading a13v_test3 in account DUXXX
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BDTBL9 [SPY])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BDTBL9 [SPY])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BDTBL9 [SPY])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BDTBL9 [SPY])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BDTBL9 [SPY])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BBV9N3 [LQD])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BDTBL9 [SPY])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BHR296 [IEF])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BBV9N3 [LQD])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BHR296 [IEF])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BDTBL9 [SPY])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BDTBL9 [SPY])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000CX0W11 [TMF])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000CX0W11 [TMF])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BBV9N3 [LQD])
2020-12-21 17:48:53 algo_spyx3: INFO ERROR: nan in data history time series!!Equity(FIBBG000BBV9N3 [LQD])

#2

It sounds like you are encountering a problem that was fixed in the 2.3.2 release and should update to the latest version:

#3

Thank you Brian for a quick response - we will make the upgrade, and report back,
that said - the release notes refer to:
//
In live trading, return real-time data from the previously completed minute, not from the current, partially completed minute.
//

which sounds different from the ticket above
In our case - nan values show up in the end-of-day price series in zipline. it seems that this issue started in early September - I can try to find and send the exact dates when it started - potentially / likely after the update to 2.3.1 version.

Also "nan" values show up in backtests as well as in live trading

thank you for your support and happy holidays!

#4

Are you sure you’ve updated the bundle? It’s not self-updating.

https://www.quantrocket.com/docs/#historical-update-minute-bundle

#5

the following scheduled ingestion db / bundle updates run every day. in cron. So yes... sure.
including the fact that we re-run the ingestion - just in case manually - and still keep getting nan values. let me trace it for you with exact dates - where I believe there are data gaps in daily database.

DATA COLLECTION

20 11 * * mon-fri quantrocket history collect --priority 'spyx-1d-db'
25 11 * * mon-fri quantrocket history collect --priority 'vixd-1d-db'
30 11 * * mon-fri quantrocket fundamental collect-sharadar-sp500 --country 'US'
35 11 * * mon-fri quantrocket fundamental collect-sharadar-fundamentals --country 'US'
40 11 * * mon-fri quantrocket zipline ingest 'usstock-1d-bundle'

#6

You don’t provide enough detail to reproduce the problem, but if I had to guess from your naming conventions, I would guess you are ingesting from a history database and have not updated the underlying history database.