Usstock-1min data missing

Technical Help
#1

I setup our historic usstock-1min database using the steps found in Usage Guide

We noticed that the data had signification gaps in close, high, low, volume, and open. about 1 in every 10 candles has a blank for stock sid FIBBG000FQRVM3 (ticker: WAT).

I ran the following code to get an export from the zipline database directly:

image
image818×382 15 KB

and here are examples of the data missing:

image
image407×550 7.5 KB

(I have the entire csv if that would be helpful)

We've noticed that most of the sids have gaps in the data.

We update the data every night:
image

I tend to operate under the motto "bad data in bad trades out."
Is there anyway to address this data quality issue?

#2

Not all stocks will have data every minute because not all stocks trade every minute. People commonly assume that every stock trades every minute, but that's not the case. It's true for larger, more liquid stocks like AAPL (which are the kinds of stocks most people are used to looking at), but for many, many stocks it's not the case. Your example, WAT, only trades a few hundred thousand shares a day - stocks in that daily volume range usually won't trade every minute.

Think about it this way: there are 390 minutes in a trading day. Suppose a stock trades 390,000 shares per day, which is about what WAT trades. If trading were completely uniform throughout the day, that would be only 1,000 shares traded per minute, which is not very much. In reality, some minutes a lot more than 1,000 shares will trade and some minutes no shares will trade.

If you have access to another source of minute data, such as a broker data feed, you can cross-reference and you will see that this is the case.