Manually include additional names in the `master` db?

Technical Help
#1

Hi Team,

Just wondering if there is a good way to include additional names in the master db? This would be helpful if I could include some crypto currencies manually.

Best regards,

Peter

#2

Adding custom securities to the securities master database isn't currently supported.

#3

Note that you can technically add records directly to the security master sqlite database; see /var/lib/quantrocket for a list of databases. You should see the security master; just open it with sqlite3 "database_name" and peek at the table schema and example records to get an understanding of the record structure. Like Brian mentioned, it's not supported, but it should work. Might be more work than it's worth, but I've found that almost anything is possible with quantrocket if you're willing to get under the hood. Just note that once you go outside of what's supported, you're basically on your own. Fortunately, if you mess something up you can simply rebuild the docker services as needed. Good luck!

#4

Are we talking about Bitcoin Futures or Bitcoin itself? IB has Bitcoin Futures here.
I don't think IB or Alpaca support the currencies directly. I've asked Alpaca if they have a roadmap for crypto support via Polygon.io's crypto api.
For others info, I pinged brian a week ago and he stated he thought it would be a few months for QR to put it into the roadmap (not putting words into his mouth if things have changed since).

#5

I have a different reason for wanting this functionality, so I thought I'd pile on.

IBKR provides only 2 years worth of history for futures, but it's not that hard to find futures data from another provider. I was hoping to download what's available via IBKR and backfill data manually (using IBKR to keep the data updated).

The SecurityIBKR table is straightforward to update manually except that I don't have reliable values for Sid or ConId. I'm not sure the latter is used anywhere so I think a placeholder would be OK, but I'm curious how QuantRocket generates values for Sid.

Thanks

#6

FYI, talked with alpaca a week or so ago, they told me late Q3/Q4 for Crypto.
Dan also confirmed in this thread.
I didn't get confirmation if it will parallel the way their equity model works w/ PFOF.

#7

The only rule for sids is that they must be globally unique. QuantRocket’s convention is to assign a 2-character prefix indicating the source of the identifier, followed by the identifier. For example, for options, IBKR numeric conids are used, combined with a prefix IB: IB123456789. FIGI-based sids are FI followed by the FIGI: FIBBG000B9XRY4. The prefixes ensure that, even if two sources were to use the same identifier for two different securities, the resulting sids would still be unique.

WIth an eye to possibly supporting loading custom securities in the future, we’ve reserved the prefix range U[A-Z] for user-defined securities. So custom sids could be as simple as UA1, UA2, …

It would probably work to put custom securities in SecurityIBKR, although we haven’t tested it and don't officially support it. Feel free to try.

#8

Thanks, that helps a lot. Can you elaborate where the numbers come from for futures (QF*)? I thought perhaps it was the conid but those don't match.

#9

It’s an internal numbering system. You should use your own prefix and numbering system (which could be conids if you like).