We’re working with ES futures data from IBKR. When we originally collected data, it looks like some days didn’t get data for the ESZ0 contract (Thursdays between 10/1 and 7/23, inclusive). Using some other threads in the forum, we put this workaround together:
- Identify missing data and contract SID.
- Drop last saved date for the contract:
jupyter:/var/lib/quantrocket/quantrocket.v2.history.es-fut-1min.sqlite $ sqlite3 quantrocket.v2.history.es-fut-1min.sqlite 'DELETE FROM PenultimateMaxDate WHERE Sid = "QF000000023069"'
- Re-collect history for missing contract:
quantrocket history collect 'es-fut-1min' --sids 'QF000000023069'
- (optional) Sync DBs in S3
$ quantrocket db s3push --services 'history' $ quantrocket db s3pull --services 'history' -f
- Drop, recreate and ingest bundle:
$ quantrocket zipline drop-bundle 'es-fut-1min-bundle' --confirm-by-typing-bundle-code-again 'es-fut-1min-bundle' $ quantrocket zipline create-bundle-from-db 'es-fut-1min-bundle' --from-db 'es-fut-1min' --calendar 'GLOBEX' $ quantrocket zipline ingest 'es-fut-1min-bundle'
Our specific feedback is that it should be possible to force re-collection of history without modifying the sqlite DB directly. Also, ingesting the bundle didn’t pick up the missing data, so we had to drop and recreate the bundle to fix that issue.