Maintain an Historical Database

Technical Help
#1

Hello everyone!

Sorry in advance for the long post, I hope, I will make my point clear.

My goal is to maintain an Historical Database with both eod-price and fundamentals data.
I set the cron-jobs below into countdown (Timezone America/New York).

# Update the 'nyse' universe, Monday-Friday evenings at 16:30
30 16 * * 1-5 quantrocket master get -e 'NYSE' | quantrocket master universe 'nyse' -a -f -

# Fetch historical price data for the 'nyse' universe, Monday-Friday evenings at 17:30
30 17 * * 1-5 quantrocket history fetch 'nyse-eod'

# Fetch fundamental data for the 'nyse' universe, Monday-Friday evenings at 18:30
30 18 * * 1-5 quantrocket fundamental fetch-financials --universes 'nyse'

The first step is important to update the list of active contracts traded on the NYSE (delisting, new entries, etc...)

Then after having created a database with the following command (needed one, therefore no cronjob):
quantrocket history create-db 'nyse-eod' --universes 'nyse' --bar-size '1 day'
I fetch price and fundamentals data daily.

I've noted that fetching price data uses the db ('nyse-eod'), while fetching fundamentals uses directly the universe ('nyse').
My question is: if there are changes in the universe (for example new listings), will they be taken into account automatically when downloading the price data or I need previously to update the database 'nyse-eod'?
And If a database update is a necessary step, how do I do that? (I've only found 'create-db' but no 'update-db').

Thank you very much!
Costantino

#2

The history db configuration (which is stored in a table inside the database itself) only maintains a reference to the universe ("nyse") and re-queries the master service each time to determine the universe constituents. So it will pick up the new constituents and ignore the delisted ones.

Note that quantrocket master get only queries what's already in your master db. So running the first cron job nightly won't accomplish anything unless you first fetched new listings and/or delisted old listings.

https://www.quantrocket.com/docs/#maintain-listings

#3

I'm not sure, I've understood.

I can simply delete the first cron job or I have to run previously this command?
quantrocket master listings -e 'NYSE'

#4

To add new listings, you would need to run:

quantrocket master listings -e 'NYSE'

Give that enough time to finish running, then later you can update your universe to include the new listings:

quantrocket master get -e 'NYSE' | quantrocket master universe 'nyse' -a -f -

To automatically delist securities which IB removes from their system, you could run:

quantrocket master diff -u 'nyse' --delist-missing

This doesn't require updating your universe afterwards; when you "delist" a security, it just gets marked as "delisted" in the master database. When your history db later queries the master service to determine the universe constituents, any delisted symbols will be excluded automatically.

Also note that if IB removes securities and you don't "delist" them, this just means that the history service will request data for them and IB will respond with error 200: "No security definition has been found for the request". This warning message will get logged to flightlog. At that time you could see it and manually delist the security using quantrocket master delist --conid XXXXX if you want to silence the warnings.