Hey guys noticed I was missing price for yesterday when running strategy scripts this morning, and I saw in my logs that when the zipline ingest ran it's saying no data ingested because it's already up to date. Tried again manually and getting the same thing.
Is it just delayed today or is something wrong?
The minute bundle is unfortunately delayed due to a processing issue overnight. The usstock history database is unaffected. The minute bundle will be available later this morning, and we will look into addressing the particular root cause. I apologize for the delay.
Thanks for the quick response Brian; was worried some of my recent changes messed something up.
For others, it's a good idea to keep some overlapping realtime data on hand; I keep a week's worth, and it's useful for situations like this.
Is there a contingency when this happens using a different database? I'm unable to generate a pipeline for live trading. This is potentially a money losing issue...
The minute data for May 18 is now available. This was a follow-on consequence of the recent bundle rebuild and the root cause has been addressed.
Regarding backup data sources, that would be ideal and has been considered but isn’t easy. It’s a long journey from the data vendor through processing and Zipline prep to your deployment, and a backup solution only works if it bypasses the particular area where an issue occurred. The particular area can vary, so theoretically you end up needing an entirely different start-to-finish pipeline that shares nothing in common with the main pipeline. This introduces considerable complexity and has its own problems.
The corresponding real-time database used for live trading is probably the best and most reliably available duplicate data source as you already have it locally so it bypasses the bulk of the data pipeline. Perhaps there could be a way to allow hooking Zipline into that more easily as a temporary backup.
The realtime database would only have ticks on the assets specifically collected from the prior day, no? So prob no more than a few hundred.
This wouldn't be enough to generate pipeline, but I suppose useful for continued trend tracking for open positions that you were collecting ticks for the day before.
Hi @Brian, is it safe to assume that collection issues with usstock-1min will be paralleled for usstock-1d as well?
Did this have any known bearing on the ibgs starting? I would suspect no, but today I got this error on at market close for an algo that collects & executes right at 4pm EST. It's ran for multiple months without error:
2021-05-19 16:00:01 quantrocket.history: WARNING [sp500-top20-1day] Waiting to collect sp500-top20-1day history because no ibg services are running, please start service(s)
This sits in my cron and haven't typically had issues. I saw in version 2.5 we don't typically need this anymore but left it in as a safety measure.
0 2 * * mon-fri quantrocket ibg start --gateways ibg1 ibg2
I manually started the ibgs and the orders on the blotter went through.
Let me tweak and generalize my earlier response.
The best backup solution, which is available now, is to maintain a secondary database that fits your needs. QuantRocket has lots of ways to get data, and the best choice for a secondary database will depend on your situation. For running pipelines, you could have a secondary bundle ingested from a US stock history db (which was unaffected by this delay), a Sharadar db, or an Interactive Brokers end of day db. If you also need minute data in your Zipline strategy, you could collect and ingest historical Interactive Brokers minute data, or collect real-time minute aggregates from Alpaca, and ingest them into a bundle. Then point your strategies to the secondary bundle if there is a delay with the US stock bundle. For Moonshot strategies it would work similarly, except that the secondary could be a history database and need not be a bundle.
It’s QuantRocket’s job to minimize delays and disruptions with the US stock bundle as much as possible (hopefully to zero incidents), but it’s each trader’s job to take advantage of QuantRocket’s flexibility to ensure that, if there is a delay, your live trading is not interrupted.
Assuming the zipline DB is our primary, ussstock-1d-bundle, how do convert to US stock history DB, usstock-1d, if a failure happens in the Cron (does Moonshot have this functionality)?
# Cron to pull both databases each morning 6:30am EST
30 6 * * mon-fri quantrocket zipline ingest 'usstock-1d-bundle'
30 6 * * mon-fri quantrocket zipline ingest 'usstock-1d'
# If this fails, switch from ussstock-1d-bundle to usstock-1d
25 9 * * mon-fri quantrocket moonshot trade 'MyStrategy' | quantrocket blotter order -f -
Moonshot:
class Strategy(Moonshot):
CODE = "MyStrategy"
DB = ["usstock-1d-bundle", "usstock-1d"] # use DB[0] primarily, but use DB[1] if failure in cronThere's no automatic failover process. You would need to manually change the Moonshot DB parameter or the zipline --bundle parameter, or set up your own automated process.
This topic was automatically closed after 5 days. New replies are no longer allowed.