Error with zipline lngest

Technical Help
#1

i get the following error with:

quantrocket zipline ingest --history-db 'h-nyse5y'

any help would be appreciated
thanks
darryl

------------error----------------------------------------------------------------------------
error: "Ambiguous ownership for 6 symbols, multiple assets held the following symbols:

CLNS:
intersections: (('NaT', 'NaT'),)
start_date end_date
ConId
261256894 NaT NaT
321328204 NaT NaT
CLNY PRB:
intersections: (('NaT', 'NaT'),)
start_date end_date
ConId
261990723 NaT NaT
323020895 NaT NaT
CLNY PRE:
intersections: (('NaT', 'NaT'),)
start_date end_date
ConId
261990729 NaT NaT
323020950 NaT NaT
CLNY PRG:
intersections: (('NaT', 'NaT'),)
start_date end_date
ConId
261992562 NaT NaT
323020885 NaT NaT
CLNY PRI:
intersections: (('NaT', 'NaT'),)
start_date end_date
ConId
277582148 NaT NaT
323020947 NaT NaT
CLNY PRJ:
intersections: (('NaT', 'NaT'),)
start_date end_date
ConId
289088606 NaT NaT
323020941 NaT NaT
msg: an unhandled exception occurred, please check flightlog for the traceback and
notify support as this might be a bug
status: error

--------------flightlog log-------------------------------------------------------------
2018-06-27 18:10:33 quantrocket.zipline: ERROR Traceback (most recent call last):
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "sym://qrocket_wsgi_py", line 30, in app
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.5/site-packages/flask/app.py", line 1994, in call
2018-06-27 18:10:33 quantrocket.zipline: ERROR return self.wsgi_app(environ, start_response)
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.5/site-packages/flask/app.py", line 1985, in wsgi_app
2018-06-27 18:10:33 quantrocket.zipline: ERROR response = self.handle_exception(e)
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.5/site-packages/flask_restful/init.py", line 271, in error_router
2018-06-27 18:10:33 quantrocket.zipline: ERROR return original_handler(e)
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.5/site-packages/flask/app.py", line 1540, in handle_exception
2018-06-27 18:10:33 quantrocket.zipline: ERROR reraise(exc_type, exc_value, tb)
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.5/site-packages/flask/_compat.py", line 32, in reraise
2018-06-27 18:10:33 quantrocket.zipline: ERROR raise value.with_traceback(tb)
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.5/site-packages/flask/app.py", line 1982, in wsgi_app
2018-06-27 18:10:33 quantrocket.zipline: ERROR response = self.full_dispatch_request()
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.5/site-packages/flask/app.py", line 1614, in full_dispatch_request
2018-06-27 18:10:33 quantrocket.zipline: ERROR rv = self.handle_user_exception(e)
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.5/site-packages/flask_restful/init.py", line 271, in error_router
2018-06-27 18:10:33 quantrocket.zipline: ERROR return original_handler(e)
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.5/site-packages/flask/app.py", line 1517, in handle_user_exception
2018-06-27 18:10:33 quantrocket.zipline: ERROR reraise(exc_type, exc_value, tb)
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.5/site-packages/flask/_compat.py", line 32, in reraise
2018-06-27 18:10:33 quantrocket.zipline: ERROR raise value.with_traceback(tb)
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.5/site-packages/flask/app.py", line 1612, in full_dispatch_request
2018-06-27 18:10:33 quantrocket.zipline: ERROR rv = self.dispatch_request()
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.5/site-packages/flask/app.py", line 1598, in dispatch_request
2018-06-27 18:10:33 quantrocket.zipline: ERROR return self.view_functionsrule.endpoint
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.5/site-packages/flask_restful/init.py", line 477, in wrapper
2018-06-27 18:10:33 quantrocket.zipline: ERROR resp = resource(*args, **kwargs)
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.5/site-packages/flask/views.py", line 84, in view
2018-06-27 18:10:33 quantrocket.zipline: ERROR return self.dispatch_request(*args, **kwargs)
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.5/site-packages/flask_restful/init.py", line 587, in dispatch_request
2018-06-27 18:10:33 quantrocket.zipline: ERROR resp = meth(*args, **kwargs)
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.5/site-packages/webargs/core.py", line 441, in wrapper
2018-06-27 18:10:33 quantrocket.zipline: ERROR return func(*args, **kwargs)
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "sym://qrocket_app_py", line 134, in post
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.5/site-packages/zipline/data/bundles/core.py", line 451, in ingest
2018-06-27 18:10:33 quantrocket.zipline: ERROR pth.data_path([name, timestr], environ=environ),
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.5/site-packages/zipline_extensions/data/bundles/history.py", line 89, in quantrocket_history_bundle
2018-06-27 18:10:33 quantrocket.zipline: ERROR self._write_assets(prices, asset_db_writer)
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.5/site-packages/zipline_extensions/data/bundles/history.py", line 177, in _write_assets
2018-06-27 18:10:33 quantrocket.zipline: ERROR root_symbols=root_symbols)
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.5/site-packages/zipline/assets/asset_writer.py", line 473, in write
2018-06-27 18:10:33 quantrocket.zipline: ERROR else pd.DataFrame()
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.5/site-packages/zipline/assets/asset_writer.py", line 711, in _load_data
2018-06-27 18:10:33 quantrocket.zipline: ERROR equities_output, equities_mappings = self._normalize_equities(equities)
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.5/site-packages/zipline/assets/asset_writer.py", line 658, in _normalize_equities
2018-06-27 18:10:33 quantrocket.zipline: ERROR return _split_symbol_mappings(equities_output)
2018-06-27 18:10:33 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.5/site-packages/zipline/assets/asset_writer.py", line 264, in _split_symbol_mappings
2018-06-27 18:10:33 quantrocket.zipline: ERROR key=first,
2018-06-27 18:10:33 quantrocket.zipline: ERROR ValueError: Ambiguous ownership for 6 symbols, multiple assets held the following symbols:
2018-06-27 18:10:33 quantrocket.zipline: ERROR CLNS:
2018-06-27 18:10:33 quantrocket.zipline: ERROR intersections: (('NaT', 'NaT'),)
2018-06-27 18:10:33 quantrocket.zipline: ERROR start_date end_date
2018-06-27 18:10:33 quantrocket.zipline: ERROR ConId
2018-06-27 18:10:33 quantrocket.zipline: ERROR 261256894 NaT NaT
2018-06-27 18:10:33 quantrocket.zipline: ERROR 321328204 NaT NaT
2018-06-27 18:10:33 quantrocket.zipline: ERROR CLNY PRB:
2018-06-27 18:10:33 quantrocket.zipline: ERROR intersections: (('NaT', 'NaT'),)
2018-06-27 18:10:33 quantrocket.zipline: ERROR start_date end_date
2018-06-27 18:10:33 quantrocket.zipline: ERROR ConId
2018-06-27 18:10:33 quantrocket.zipline: ERROR 261990723 NaT NaT
2018-06-27 18:10:33 quantrocket.zipline: ERROR 323020895 NaT NaT
2018-06-27 18:10:33 quantrocket.zipline: ERROR CLNY PRE:
2018-06-27 18:10:33 quantrocket.zipline: ERROR intersections: (('NaT', 'NaT'),)
2018-06-27 18:10:33 quantrocket.zipline: ERROR start_date end_date
2018-06-27 18:10:33 quantrocket.zipline: ERROR ConId
2018-06-27 18:10:33 quantrocket.zipline: ERROR 261990729 NaT NaT
2018-06-27 18:10:33 quantrocket.zipline: ERROR 323020950 NaT NaT
2018-06-27 18:10:33 quantrocket.zipline: ERROR CLNY PRG:
2018-06-27 18:10:33 quantrocket.zipline: ERROR intersections: (('NaT', 'NaT'),)
2018-06-27 18:10:33 quantrocket.zipline: ERROR start_date end_date
2018-06-27 18:10:33 quantrocket.zipline: ERROR ConId
2018-06-27 18:10:33 quantrocket.zipline: ERROR 261992562 NaT NaT
2018-06-27 18:10:33 quantrocket.zipline: ERROR 323020885 NaT NaT
2018-06-27 18:10:33 quantrocket.zipline: ERROR CLNY PRI:
2018-06-27 18:10:33 quantrocket.zipline: ERROR intersections: (('NaT', 'NaT'),)
2018-06-27 18:10:33 quantrocket.zipline: ERROR start_date end_date
2018-06-27 18:10:33 quantrocket.zipline: ERROR ConId
2018-06-27 18:10:33 quantrocket.zipline: ERROR 277582148 NaT NaT
2018-06-27 18:10:33 quantrocket.zipline: ERROR 323020947 NaT NaT
2018-06-27 18:10:33 quantrocket.zipline: ERROR CLNY PRJ:
2018-06-27 18:10:33 quantrocket.zipline: ERROR intersections: (('NaT', 'NaT'),)
2018-06-27 18:10:33 quantrocket.zipline: ERROR start_date end_date
2018-06-27 18:10:33 quantrocket.zipline: ERROR ConId
2018-06-27 18:10:33 quantrocket.zipline: ERROR 289088606 NaT NaT
2018-06-27 18:10:33 quantrocket.zipline: ERROR 323020941 NaT NaT
2018-06-27 18:10:33 quantrocket.zipline: ERROR

#2

What's your database config? (quantrocket history config <db>)

Off the top of my head, I wonder if those are illiquid symbols which are part of the universe but have no history in the history db, so zipline is showing NaT for the start and end dates and that is leading to these collisions. The error should be handled better but as a workaround maybe try removing them from the universe?

#3

my database config is:
bar_size: 1 day
shard: 'off'
universes:

  • u-nyse5y
    vendor: ib

i have tried creating a new unverse using the command:

quantrocket master get --exchanges NYSE --sec-type 'STK' --outfile 'u-nyse5y.csv'
then editing the csv file to removed the conIds before creating the univese from the file,
but the conIds listed in the error message do not exist in the csv file.
i wasnt expecting this,
am i doing something wrong or is this a bug?
any help would be appreciated
thanks
darryl

#4

If the conids are gone maybe the errors are gone too?

#5

no i still get the same errors, also with the same conIds

#6

Maybe the conids didn't show up in your query because they are delisted and need to be included with --delisted:

quantrocket master get --exchanges NYSE --sec-type ‘STK’ --outfile ‘u-nyse5y.csv’ --delisted

The latter is more like the query that would get run during Zipline ingestion.