Zipline w/ EDI JPX data, SQLite file .../adjustments.sqlite doesn't exist

dominic.cooney · June 27, 2020, 10:56am

I'm trying to run the Zipline "crossover" strategy from the docs:

$ quantrocket master get -e XJPX | quantrocket master universe xjpx -f -
$ quantrocket zipline create-bundle-from-db xjpx-eod-bundle --from-db xjpx-eod --calendar TSEJ --start-date 2007-01-01 --universe xjpx
$ quantrocket zipline ingest xjpx-eod-bundle # How do you tell when this is done?
$ quantrocket zipline default-bundle xjpx-eod-bundle
$ quantrocket zipline backtest crossover --bundle xjpx-eod-bundle -s 2007-01-01 -e 2020-01-01 -o dma_results.csv
msg: SQLite file '/var/lib/quantrocket/zipline/data/xjpx-eod-bundle/2020-01-01T00;00;00/adjustments.sqlite'
  doesn't exist.
status: error

Flightlog says:

2020-06-27 08:53:13 quantrocket.zipline: INFO [xjpx-eod-bundle] Ingesting xjpx-eod-bundle bundle

And nothing else that looks relevant... I left it alone for 24 hours.

If I connect to the zipline container, I see this... sizes seem suspiciously small:

root@...:/# ls -al /var/lib/quantrocket/zipline/data/xjpx-eod-bundle/2020-01-01T00\;00\;00/
total 232
drwxr-xr-x 4 root root   4096 Jun 26 23:46 .
drwxr-xr-x 4 root root   4096 Jun 26 23:46 ..
-rw-r--r-- 1 root root  98304 Jun 26 23:46 assets-8.sqlite
-rw-r--r-- 1 root root 114688 Jun 26 23:46 assets-8.sqlite.tmp
drwxr-xr-x 2 root root   4096 Jun 26 23:46 daily_equities.bcolz
drwxr-xr-x 2 root root   4096 Jun 26 23:46 minute_equities.bcolz

What am I doing wrong here?

Brian · June 29, 2020, 1:11pm

I can re-create this issue. It is caused by the volume of data. For daily databases, Zipline tries to load the entire database into memory, which is not a good idea with a large exchange like this.

This will be fixed in the next release. Daily database ingestion will be modified to ingest one security at a time (the same as currently happens for minute databases). For now, the workaround is to specify a smaller universe or shorter date range in the create-bundle-from-db command.

Depending on the date range and securities you ingest, there is also an issue where the ingestion may fail due to a handful of delisted XJPX securities that don’t have a Symbol in the securities master database. This will also be fixed in the next release. The workaround is to leave the securities without symbols out of your universe:

quantrocket master get -u xjpx | csvgrep -c Symbol -r '^$' --invert-match | quantrocket master universe xjpx -f - -r

Brian · July 23, 2020, 4:29pm

QuantRocket 2.1.0 is now available and includes a fix for these issues.

Brian · July 27, 2020, 10:00pm

This topic was automatically closed after 4 days. New replies are no longer allowed.