Create dbs for "global market profiles"

[sculd3]

Hi, I am trying to replicate the result of the "global-market-profiles" codes, which I can clone by running

$ quantrocket codeload clone 'global-market-profiles'

The code presumes the presence of the dbs

countries = (
    # North America
    ("usa-stk-1d", "United States"),
    ("canada-stk-1d", "Canada"),
    ("mexico-stk-1d", "Mexico"),
    
    # Asia
    ("australia-stk-1d", "Australia"),
    ("japan-stk-1d", "Japan"),
    ("hongkong-stk-1d", "Hong Kong"),
    ("singapore-stk-1d", "Singapore"),
    
    # Europe
    ("spain-stk-1d", "Spain"),
    ("netherlands-stk-1d", "Netherlands"),
    ("uk-stk-1d", "United Kingdom"),
    ("germany-stk-1d", "Germany"),
    ("france-stk-1d", "France"),
    ("belgium-stk-1d", "Belgium"),
    ("swiss-stk-1d", "Switzerland"),
    ("austria-stk-1d", "Austria"),
    ("sweden-stk-1d", "Sweden"),
    ("italy-stk-1d", "Italy"),
)

which the prerequisite describes "Data collection is not covered by this tutorial.".

I, however, would really appreciate some direction as to what functions to run to create the global non-us databases.

Thank you.

[Brian]

Global non-US data is available from EDI or IBKR (if you have an IBKR account). EDI data is generally preferable for historical research because (1) the data is survivorship-bias-free and (2) it is much easier to collect.

EDI data collection is straightforward: purchase the data then collect it as covered in the docs. For example, the canada db would be collected like this:

>>> from quantrocket.history import create_edi_db, collect_history
>>> create_edi_db("canada-stk-1d", exchanges="XTSE")
>>> collect_history("canada-stk-1d")

For the short sale notebook, you will need to collect IBKR shortable shares data (which does not require an IBKR account).

The primary vs consolidated notebook can't be recreated with EDI data as all of their data is primary exchange.

For the sector notebook, you would want to look at edi_Industry, instead of looking at ibkr_Sector as shown in the notebook. That notebook also expects universes (named 'canada-stk' for example) which can be made as shown in the docs.

The global-market-profiles repo predates the EDI data integration, so IBKR data was used originally. Due to the amount of effort required to collect that many exchanges from IBKR, data collection steps were omitted from the tutorial as it's not the greatest user experience. Collecting IBKR data requires the following steps:

• Subscribe to the relevant market data through your IBKR account
• Collect listings for the desired exchange
• Create a universe of the listings for an exchange, named, for example, 'canada-stk'
• Create a history database tied to the universe and collect the historical price data. If you want to recreate the primary vs consolidated notebook, you would need to create one database with the primary_exchange flag to limit to primary exchange prices (see the API reference) and another without the flag to collect consolidated prices. The naming convention used in the notebook is canada-stk-1d for consolidated prices and canada-stk-1d-p for primary exchange prices.
• When collecting IBKR data for so many exchanges, it will be common to encounter some permission errors from the IBKR API, which will require double-checking IBKR market data permissions to figure out if you're subscribed to all the correct market data subscriptions.

Collecting IBKR data is entirely doable but the level of patience required increases in proportion to the number of exchanges.