Summary
The location where QuantRocket stores your code and databases has changed. This change is in force for YAML files generated from the configuration wizard on or after March 18, 2018. This change is only applicable to local deployments (not Docker Cloud) and has several performance benefits. Existing users are encouraged to upgrade their deployments to use the new storage location.
What's changed
Previously, code and databases were stored under the user's home directory, in ~/quantrocket/codeload and ~/quantrocket/databases. Docker containers would read from and write to these directories. The section of the docker-compose.yml which defined this behavior looked similar to this:
services:
db:
image: 'quantrocket/db:1.1.0'
volumes:
- '/c/users/myuser/quantrocket/databases:/var/lib/quantrocket'
The volumes directive told Docker to use C:\Users\myuser\quantrocket\databases on the host OS and make it visible inside the container at /var/lib/quantrocket. This is referred to as bind mounting in Docker parlance.
Going forward, databases and code will be stored in a Docker-managed area of the filesystem, not under the user's home directory. The equivalent section of docker-compose.yml now looks like this:
volumes:
db:
services:
db:
image: 'quantrocket/db:1.1.0'
volumes:
- 'db:/var/lib/quantrocket'
This tells Docker to store the databases in a volume (that is, a persistent directory) named "db" inside the Docker-managed area of the filesystem.
Why the change
Using bind mounts to store data under the user's home directory was attractive because it makes it obvious to the user where the data is stored. Unfortunately, bind mounts come with significant problems, two of which are:
- On Windows, bind mounts don't work properly after restarting the computer (or restarting Docker), which leads to a confusing situation where QuantRocket containers can't see the code and databases in the user's home directory.
- On Mac, using bind mounts incurs a significant performance penalty in some cases. For example, historical data retrieval runs up to 60% faster when using Docker-managed volumes instead of using bind mounts.
Bind mounts on Linux are less likely to suffer adverse consequences since Docker is native to Linux.
Should I upgrade?
Existing users are encouraged to upgrade their deployments in order to address the above problems. However, the upgrade need not be done immediately if you aren't experiencing the above problems.
How to upgrade
The upgrade process involves stopping your existing deployment, generating and deploying a new docker-compose.yml, and copying your existing code and database from your home directory to the new deployment.
Step 1: stop your deployment
First, stop your deployment by running:
docker-compose -p quantrocket down
Step 2: generate a new docker-compose.yml
Use the configuration wizard to generate a new docker-compose.yml to replace your existing YAML file. In addition to the storage location changes, the new YAML file will also contain the latest versions of QuantRocket containers.
Step 3: start new deployment
Start your new deployment:
docker-compose -p quantrocket up -d
Step 4: copy your code and databases
The new deployment will point to the new storage locations in the Docker-managed area of the filesystem. You can copy your existing code and databases to the new location in order to pick up where you left off.
Windows users can run the following commands to copy their databases and code (replace "myuser" with your username):
docker cp C:\Users\myuser\quantrocket\databases. quantrocket_db_1:/var/lib/quantrocket/
docker cp C:\Users\myuser\quantrocket\codeload. quantrocket_codeload_1:/codeload/
Mac/Linux users can run the following commands to copy their databases and code:
docker cp ~/quantrocket/databases/. quantrocket_db_1:/var/lib/quantrocket/
docker cp ~/quantrocket/codeload/. quantrocket_codeload_1:/codeload/
To verify the code was copied over as expected, open JupyterLab (http://localhost:1969) and see if your files are there.
To verify that the databases were copied over as expected, run:
quantrocket db list
Where exactly are my code and databases stored now?
On Linux, your code and databases are stored in /var/lib/docker/volumes. On Windows and Mac, they are also stored in /var/lib/docker/volumes, but this file location is not on the host OS but rather inside the virtual machine which Docker for Mac and Docker for Windows run.
How do I export my code and databases from the new storage location?
The easiest way to export your code and databases is using docker cp:
docker cp quantrocket_db_1:/var/lib/quantrocket my_exported_dbs
docker cp quantrocket_codeload_1:/codeload my_exported_code
