I am trying to collect recent ETF data from the EDI XSTO database, but for some reason the data stops at 2024-12-05 for most of the listed ETFs on XSTO:
Non-ETF XSTO data is updating just fine.
Any ideas why this is happening?
Investigating...
To add -- the discontinuation in the data on 2024-12-05 also seems to affect a small group of stocks on XSTO:
Appreciate the help.
Inquiring with EDI...
EDI is investigating this.
EDI has learned from the exchange that there is additional ETF file that must be processed. They are working to incorporate the file into their process. Once they go live with the new file (this weekend), they will backfill the file historically, at which point QuantRocket will backfill it as well. I will post another update when all of that has happened.
Thanks updating me on this. Will keep an eye out for your update next week.
The ETF counts have come back up on a forward basis but have not yet been backfilled. The backfill is in queue with EDI's IT team, and I'm working to get an estimated completion date.
Regarding the drop in stock counts, I don't believe there is any missing data. I suspect, instead, that the same reporting change that affected the ETF counts may have caused some inactive securities to disappear from the data that had been getting carried forward (with unchanged prices) prior to the reporting change. You can see this by plotting the count of closes that changed from the prior close, which doesn't show a drop on Dec 6 (the drop to 0 on Jan 6 is due to an exchange holiday as discussed in another ticket).
Interesting that the drop in counts on Dec 6 goes away when only looking at closes that changed from prior close, which suggests your explanation is valid.
Awaiting the backfilled ETF data.
Sorry for the continued delay, still working to get this moved along with EDI.
Unfortunately the exchange sent the files to EDI in a different format than EDI needs, so EDI had to request the files again.
@Brian any development with EDI? Any chance for them to expedite the data? As the need is somewhat time sensitive. Much appreciated
Sorry for the delay. I’ve requested another update and tried to inject some urgency. It’s better when the problem is in QuantRocket so the fix can be quicker.
EDI is still waiting on the upstream data source. I'm trying to pin down an ETA.
Updated files have been made available by EDI. Most of the missing ETF data is now present. However, I've identified 4 symbols that are still missing some dates in January and February. The ETFs still affected are:
- BITCOIN XBT
- BITCOIN XBTE
- ETHEREUM XBT
- ETHEREUM XBTE
If you would like to access the restored data while these ETFs are still being investigated, you can do so by dropping and re-collecting your XSTO database. Note the following as you look for low counts in the new data:
- The low count on January 6 is correct because it was an XSTO holiday, but there is data for a handful of symbols whose primary listing is on XSWX.
- DNBNOR OBXO is missing data on one date, but that is correct because it is illiquid and didn't trade that day.
I will let you know when I have a further update on the 4 symbols that are still incomplete.
That's great news, thanks for the update. I have dropped and recollected the XSTO database, and most of the previously missing ETFs have now appeared (except the crypto ones you mention). Noted on Jan 6 low counts. Please do update me on the remaining 4 symbols.
Data for the 4 remaining symbols is now available. Please drop and re-collect the XSTO database to access it.
Great. Thank you.