Issues with database persistence on Windows after restart

[CMJariz]

Hi

I'm relatively new to Docker in general and QuantRocket specifically. I'm having sort of an issue with persistence of things like listings, universes, databases, etc. Whenever Docker needs to restart, Quantrocket cannot seem to find databases that I previously created and had saved in the db volume specified in docker-compose.yml, even when I can see those files are clearly there.

Every time it restarts I have to download the listings, create universes, and create and populate databases all over again. This doesn't seem right at all, I figured that the container would be able to pick up on files saved in the correct location locally.. Is there something I should be doing with Docker or Quantrocket to save the container's state in case of a crash or, you know, just wanting to shut off my computer?

Attached picture shows what I'm talking about. Quantrocket's DB list does not contain databases saved in the correct location that I can clearly see, and I have no idea how to get it to pick them up. I can also create new databases with the same name if it's not in that list, and nothing happens, and sometimes I create entirely new databases that I can't find the files for anywhere.

So confused, sorry, any help appreciated.

[Brian]

You're definitely not supposed to have to recreate your databases!

Check the db service definition in your docker-compose.yml. You want it to look something like this:

  db:
    image: 'quantrocket/db:1.1.0'
    volumes:
      - '/c/users/myuser/quantrocket/databases:/var/lib/quantrocket'

This will tell Docker to bind mount the db container to C:\Users\myuser\quantrocket\databases. Whatever's in that directory will be available inside the container (at /var/lib/quantrocket/) and vice versa. Judging by your screenshot it was running that way at some point.

On the other hand if you just have:

db:
  image: 'quantrocket/db:1.1.0'
  volumes:
    - '/var/lib/quantrocket'

Docker will still mount the container to a directory on the host machine, but it write create and manage the directory itself in a Docker-managed area of your computer.

You can check where the db service is mounted by running:

docker inspect quantrocket_db_1

And looking for the "Mounts" section:

   "Mounts": [
        {
            "Type": "bind",
            "Source": "/Users/myuser/quantrocket/databases",
            "Destination": "/var/lib/quantrocket",
            "Mode": "rw",
            "RW": true,
            "Propagation": "rprivate"
        }
    ],

[CMJariz]

Brian

Thanks for the quick response. Based on what I see in docker-compose.yml and after running docker inspect (see screenshot), it looks to be setup correctly. I definitely have not change docker-compose.yml. But it definitely is not seeing the databases in that volume.

[Brian]

You could hop into the container and see what databases it can see:

docker exec -ti quantrocket_db_1 bash
$ ls -l /var/lib/quantrocket

If you can see your databases then the mismatch is between the client and the container rather than between the container and the filesystem. In that case I would run docker ps and check if perhaps there are two separate deployments running (all containers within a deployment have a common prefix, in this case quantrocket_).

[CMJariz]

Brian

See attached screenshot. The container cannot see those databases. Docker ps does not appear to show two separate deployments.

I tried deleting the container, databases, and re-installing even the quantrocket client from scratch. It worked, correctly created and saw databases ... until I restarted my computer, at which point it no longer sees them. I really have no idea what could be causing this, it seems that the second that the Docker container shuts it cannot see what it is supposed to see, and can never see it again.

Sorry if this is such a weird issue, I just have a standard Windows 10 Pro x64 installation and this is obviously the only thing I have running on Docker, and I don't have any weird file system protection software or anything that I can see causing this.

[Brian]

It looks like the host directory not getting mounted after a restart is a known issue with Docker for Windows because it takes a few seconds for the host directories to become available and the container may have already started by then. Docker is tracking this internally and will hopefully devise a solution but in the meantime a workaround is to manually restart the deployment after restarting Windows or restarting Docker:

docker-compose -p quantrocket down
docker-compose -p quantrocket up -d

by which time the host directories will be available and should get mounted properly.

Going forward I recommend editing your YAML file and removing restart:always from each of the service definitions. restart:always tells Docker to autostart QuantRocket when Docker starts up, but since this doesn't work properly on Windows, it's better to disable this behavior. Then, whenever you restart Windows or Docker, you need to start QuantRocket manually:

cd ~\quantrocket\
docker-compose -p quantrocket up -d

We've modified the configuration wizard to not include restart:always for Windows and added a note about this to the tutorial.

[Brian]

This issue has been permanently addressed by changing from using bind mounts to volumes. See the announcement for details.