The usstock dataset (bundle and history database) is delayed due to a supplier delay. Prices for 2021-07-12 are not yet available. We are working with the supplier to get the issue resolved.
The supplier has restored data and we are now processing the update.
Thanks for being on top of this! It's mornings like this that I'm grateful for the get_prices function supporting multiple databases/datasource 
Data for 2021-07-12 is now available. We apologize for the delay.
Due to another supplier delay, usstock data for 2021-07-13 is not yet available. We will process the data as soon as we have it, but it will be after today's open. Due to back-to-back days of supplier delays, please have other data sources on standby until we see evidence that the supplier has worked out these issues.
I'm sure this is frustrating for you guys as well. Still planning to update the bundle today or skipping until tomorrow?
I think he meant to say, he wasn't going to process the bundle until after market closed.
Fortunately, after yesterday's miss I refactored my price getting logic to leverage a backup source and so I didn't have any hiccups today. Ping me if interested in the code to seamlessly fallback to a backup data source.
The supplier has moved to a new storage backend and we will need to access the raw data files from there going forward. We are expediting the switchover as fast as possible but cannot yet say whether it will be completed in time for tomorrow's session.
@charles Thanks for the offer, but my setup is very specific. I need a replacement bundle to feed zipline live and goes back at least 1.5 years daily and 1 month minutely. It's very difficult to reproduce a minute bundle that goes back 1.5 years for all US stocks. To workaround, I'm almost done implementing the use of two different bundles (one daily and one shorter minute bundle) in the same zipline instance. Just didn't have it done in time for these delays, hopefully will have tomorrow. If you know of an easily accessible multi-year minute bundle, do let me know!
@Brian Good luck, fingers crossed!
The usstock dataset is now updated with prices for 2021-07-13. We anticipate that the 2021-07-14 prices should be available on time (i.e. overnight).
1 Like
Hey @bjsun, I totally understand; my setup is very specific as well. In regards to bundles, one idea is using Polygon (or similar) to pull down prices into a history database, and then ingest that history database into a Zipline bundle. This can be a flexible way to generate whatever ad-hoc bundles you might need. Personally I use Moonshot, but should be just as good for Zipline.
Here's the docs on ingesting history databases into a Zipline bundle: Usage Guide
Polygon has been a super helpful backup provider for me; I use it to provided pre/post market data, as well as pull down missing data on-demand when needed. For example, if the realtime collector crashes for whatever reason, I have a "bring-prices-current.sh" script that downloads the missing prices for the day, week, etc into a history database, and then restarts the realtime collector so that I can resume live trading. Then, my strategies are setup to use both a realtime aggregate database, and the backup history databases so that together I have a full dataset.
I currently stream the full us stock market (~10k) for my realtime trading, and it's been a journey to implement a robust system that resilient to various data failures.