QuantRocket doesn't collect historical data for futures that previously had no data available and now have data available

Issues
#1

I have a US Futures universe and related daily price database. There's a contract, ConId 346233391, it's a 10 year treasury future expiring in september. I have an issue where the contract is in the master securities database and it is included in the US Futures universe, but it is not present in the database which uses that universe to get data.

I know it's in the universe because if I pull the master file using:

download_master_file(f, universes  = ['us-futures'], symbols = futures_symbols, exclude_delisted = False)

The conid is in the master file. However, when I get pricing data from the database the conid is not in the dataframe.

I created a test database that only has that specific contract to see if it was a data permission issue, and that test DB works fine. I can get pricing data no problem.

Any idea what might be going on and how to fix it?

Thanks!

Walter

#2

Did you watch the detailed logs while trying to collect data? Each request is logged there as well as any error messages. You can use the conids parameter to request just that conid. That's where I would start.

#3

If I only try to collect that ConId these are the detailed logs.

image

If I use the other db I created to test if it was working these are the logs.

image

I am having the same issue with another ZN contract, ConId: 358060603 which expires 19/12/2019. However ZN ConId: 333866892 which expires 19/06/2019 does work.

#4

No requests being issued suggests the software is calculating there is no date range to collect. The start date for requests is calculated as the penultimate max date of the already collected data for the security, if any, otherwise the start date from the db config (check quantrocket db config), if set, otherwise the head timestamp, which can be viewed using download_history_availability_file. The end date is calculated as the earliest of (1) the end date from the db config, if set, (2) the LastTradeDate for futures, or (3) now. If the resulting start date is later than the end date, no requests will be issued, which seems to be what is happening.

#5

When I download the head timestamps for my DB, I see that the contract in question has a date of 2200-01-01, there are a bunch of contracts with this date. For the test DB (which works) I see a date of 2019-03-19.

Is there a way to re-download the head timestamps?

#6

No doubt when you first collected data there was no data available for that contract (indicated by the far future head timestamp) but now there is. So the saved head timestamp is a problem. Probably QuantRocket should ignore far future head timestamps for futures and always re-request the head timestamp in case data has since become available.

In the meantime, you can delete the head timestamp in the database to force QuantRocket to re-request it:

sqlite3 /var/lib/quantrocket/quantrocket.history.availability.sqlite 'DELETE FROM HeadTimestamp WHERE ConId = 346233391'
Historical price data from IB is available further back than what's in my database
#7

Thanks a lot, this worked. I deleted all the HeadTimestamps which were greater than today's date and now am able to collect the missing data. Thanks!

QuantRocket 1.7.0 Release Notes
#8

This is fixed in version 1.7.0:

#9

This topic was automatically closed after 4 days. New replies are no longer allowed.