No pipeline data for Zipline live trading

Technical Help
#1

Hi Brian,

I've been getting very erratic pipeline generation while paper trading a live Zipline strategy. The pipeline definition uses both standard price inputs (from USEquityPricing), as well as custom factors. The pipeline produces empty values for USEquityPricing.high and USEquityPricing.low, and 0.0 values for USEquityPricing.volume.

I've been running this strategy for a couple weeks, and I've seen the pipeline generation work properly on certain days before. But I can't predict whether it will or won't each day.

Things I've already tried:

  • I've recollected both the local minute and daily price databases. I usually collect updates for both on a daily basis before starting the strategy (all via countdown).

  • I've deleted the context joblib to start the strategy from scratch.

  • I've recreated the zipline container.

  • The pipeline generates as expected when I run it in a notebook in a zipline environment (via run_pipeline), using the same pipeline definition and specifying a date range that includes the last trading day before the current one.

Any ideas how to test why the live pipeline would not be passed last day prices during live trading?

Related - when a live strategy is started manually in the middle of a trading day (via zipline.trade), am I right in assuming that it will try to pull pipeline prices for the previous trading day (since there are no current day prices in the database)?

Thanks,
Paul.

#2

I apologize, we were running a QA procedure on the bundle overnight and unfortunately/ironically it caused the daily bars portion of the bundle to be delayed this morning. Shouldn’t happen again.

Regarding your second question, yes, Pipeline always loads the previous day.

#3

Gotcha, the pipeline data generated correctly today, thanks. However, in trying to verify that the data was consistent, I noticed that the run_pipeline method does not generate a pipeline for the current date when asked. I.e. when I use:

output = run_pipeline(pipe, 
                      start_date="2020-12-01", 
                      end_date="2020-12-03", 
                      bundle='usstock-minute')

the returned output has all empty values for today's date (output.loc['2020-12-03']). Is this by design?

#4

The US Stock dataset is updated overnight, so the data will not be available until the following morning. The update schedule varies by dataset and is outlined in the docs:

https://www.quantrocket.com/docs/#historical-us-stock-data-guide

But also, the design of Pipeline is to run (typically in before_trading_start) using yesterday's data.