Can't collect listings for RUT (Russell 2k index)

Technical Help
#1

I'm trying to collect historical data for RUT. Using TWS, I found RUT is listed on the RUSSELL exchange: IB Products & Exchange | Interactive Brokers LLC

But QR doesn't show RUSSELL as an exchange in the US:

$ quantrocket master list-ibkr-exchanges --sec-types IND --regions north_america 
IND:
  Canada:
  - CDE
  - TSE
  United States:
  - CFE
  - ECBOT
  - CBOE
  - GLOBEX
  - ISEOPT
  - NASDAQ
  - NSX
  - NYMEX
  - NYSE
  - AMEX
  - PSE
  - NYSELIFFE
  - PHLX

And it throws an error when trying to collect listings:

$ quantrocket master collect-ibkr --exchanges RUSSELL --sec-types IND --symbols RUT

2021-04-11 00:18:51 quantrocket.master: INFO Collecting RUSSELL IND listings from IBKR website (RUT only)
2021-04-11 00:18:52 quantrocket.master: INFO Requesting details for 1 RUSSELL listings found on IBKR website
2021-04-11 00:18:52 quantrocket.master: ERROR The IBKRTimezone table in the securities master database doesn't contain the timezone for exchange RUSSELL, it should be added, please notify support
2021-04-11 00:19:00 quantrocket.master: INFO Saved 1 RUSSELL listings to securities master database

And even though it says it's saved the listings, I can't find it in the master database:

$ quantrocket master get --symbols RUT
msg: No securities match the query parameters
status: error

I filed a ticket with IBKR since they don't properly list RUSSELL here, but I'm assuming there might also be a QR bug for the error mentioned above.

Thanks!

#2

Since the exchange was not listed on IBKR's website, QuantRocket hadn’t assigned sids yet. If you try collecting it again, it should work now. Note that it still won’t show up in list-ibkr-exchanges unless/until IBKR adds it to the website.

#3

Thanks, that works now. What are the downsides for it not being available via list-ibkr-exchanges? I think data collection still works (it's running right now).

#4

I think there's still a problem with RUT. I can't create a bundle from a DB that contains this security:

2021-05-05 06:57:55 quantrocket.zipline: INFO [rut-1min-bundle] Ingesting rut-1min-bundle bundle
2021-05-05 06:58:59 quantrocket.zipline: ERROR Traceback (most recent call last):
2021-05-05 06:58:59 quantrocket.zipline: ERROR   File "sym://qrocket_log_py", line 34, in wrapped
2021-05-05 06:58:59 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_bundles_ingest_py", line 77, in mule_ingest_bundle
2021-05-05 06:58:59 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_bundles_ingest_py", line 126, in _call_ingest_func
2021-05-05 06:58:59 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_bundles_base_py", line 142, in ingest
2021-05-05 06:58:59 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_bundles_fromdb_base_py", line 252, in _write_assets
2021-05-05 06:58:59 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/assets/asset_writer.py", line 809, in write
2021-05-05 06:58:59 quantrocket.zipline: ERROR     chunk_size=chunk_size,
2021-05-05 06:58:59 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/assets/asset_writer.py", line 485, in _real_write
2021-05-05 06:58:59 quantrocket.zipline: ERROR     chunk_size,
2021-05-05 06:58:59 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/assets/asset_writer.py", line 824, in _write_df_to_table
2021-05-05 06:58:59 quantrocket.zipline: ERROR     chunksize=chunk_size,
2021-05-05 06:58:59 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/pandas/core/generic.py", line 1534, in to_sql
2021-05-05 06:58:59 quantrocket.zipline: ERROR     chunksize=chunksize, dtype=dtype)
2021-05-05 06:58:59 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/pandas/io/sql.py", line 473, in to_sql
2021-05-05 06:58:59 quantrocket.zipline: ERROR     chunksize=chunksize, dtype=dtype)
2021-05-05 06:58:59 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/pandas/io/sql.py", line 1512, in to_sql
2021-05-05 06:58:59 quantrocket.zipline: ERROR     table.insert(chunksize)
2021-05-05 06:58:59 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/pandas/io/sql.py", line 670, in insert
2021-05-05 06:58:59 quantrocket.zipline: ERROR     self._execute_insert(conn, keys, chunk_iter)
2021-05-05 06:58:59 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/pandas/io/sql.py", line 1299, in _execute_insert
2021-05-05 06:58:59 quantrocket.zipline: ERROR     conn.executemany(self.insert_statement(), data_list)
2021-05-05 06:58:59 quantrocket.zipline: ERROR sqlite3.IntegrityError: NOT NULL constraint failed: exchanges.country_code
2021-05-05 06:58:59 quantrocket.zipline: ERROR

Looking at the master db, in the Security table Country is NULL, and in the SecurityIBKR table Mic is NULL. I think this is causing bundle creation to fail.

#5

You’re right that the lack of Mic is causing the error. Although we could add a dummy Mic for RUSSELL, it wouldn’t help because Zipline doesn’t support ingesting indexes anyway, only stocks and futures. (There should be a better error message when trying to ingest sec types other than STK and FUT to avoid the unhandled exception.)

To use index data in Zipline, you can either create a custom Pipeline dataset that points to your database (but that's only for daily frequency) or use get_prices to query the database directly from your Zipline strategy.

#6

Yeah, we're using get_prices() to load details for SPX and COMP and that works OK. The problem still exists though because the RUT data is part of the same historical db where we're ingesting RTY and IWM, so we need to explore the workaround to fix the Mic details even if the RUT data won't be in the Zipline bundle.

EDIT: as a workaround, I'm using XCBO as the exchange since CBOE provides index options for RUT, so it seems like a reasonable placeholder.

#7

Just exclude RUT. See the --exclude-sids parameter to quantrocket zipline create-bundle-from-db.