Postgres container failing after collect_ibkr_listings

JHamburg430 · July 12, 2023, 4:08pm

Following the sell-on-gap tutorial, I ran the collection of ibkr listings which was able to meet completion. However, my postgres container will no longer run due to:
'2023-07-12 11:31:00 2023-07-12 15:31:00.506 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device'

I know I have sufficient storage, so what would cause this issue? Do I need to allocate additional resources to the postgres container?

To note, I have tried the following:
-Restarting postgres container
-Docker Compose Down/Up
-Restarting PC

Assuming they are related, but logspout is consuming a lot CPU resources. This was previously fixed with compose up/down, but not since the postgres container issue.

Docker log from logspout:
2023-07-12 11:31:04 2023/07/12 15:31:04 # logspout v3.2.14 by gliderlabs
2023-07-12 11:31:04 2023/07/12 15:31:04 # adapters: raw syslog tcp tls udp multiline
2023-07-12 11:31:04 2023/07/12 15:31:04 # options :
2023-07-12 11:31:04 2023/07/12 15:31:04 backlog:false
2023-07-12 11:31:04 2023/07/12 15:31:04 persist:/mnt/routes
2023-07-12 11:31:04 2023/07/12 15:31:04 # jobs : pump routes http[health,logs,routes]:80
2023-07-12 11:31:04 2023/07/12 15:31:04 # routes :
2023-07-12 11:31:04 # ADAPTERADDRESSCONTAINERSSOURCESOPTIONS
2023-07-12 11:31:04 # syslog+udpflightlog:9021map[]
2023-07-12 11:31:04 # sysloglogs5.papertrailapp.com:47405houstonmap[]

Docker log from postgres container below:
2023-07-12 11:32:09
2023-07-12 11:33:01
2023-07-12 11:33:01 PostgreSQL Database directory appears to contain a database; Skipping initialization
2023-07-12 11:33:01
2023-07-12 11:34:02
2023-07-12 11:31:00 2023-07-12 15:31:00.506 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:31:04 2023-07-12 15:31:04.176 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:31:09 2023-07-12 15:31:09.160 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:31:11 2023-07-12 15:31:11.128 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:31:13 2023-07-12 15:31:13.502 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:31:16 2023-07-12 15:31:16.515 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:31:21 2023-07-12 15:31:21.062 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:31:28 2023-07-12 15:31:28.712 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:31:43 2023-07-12 15:31:43.113 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:32:09 2023-07-12 15:32:09.575 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:33:01 2023-07-12 15:33:01.632 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:34:02 2023-07-12 15:34:02.739 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:35:03 2023-07-12 15:35:03.702 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:36:04 2023-07-12 15:36:04.580 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:37:05 2023-07-12 15:37:05.432 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:38:06 2023-07-12 15:38:06.264 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:39:07 2023-07-12 15:39:07.073 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:40:07 2023-07-12 15:40:07.809 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:41:08 2023-07-12 15:41:08.635 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:42:09 2023-07-12 15:42:09.349 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:43:10 2023-07-12 15:43:10.248 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:44:10 2023-07-12 15:44:10.966 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:45:11 2023-07-12 15:45:11.721 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:46:12 2023-07-12 15:46:12.462 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:47:13 2023-07-12 15:47:13.262 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:48:14 2023-07-12 15:48:14.133 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device
2023-07-12 11:34:02 PostgreSQL Database directory appears to contain a database; Skipping initialization
2023-07-12 11:34:02
2023-07-12 11:35:03
2023-07-12 11:35:03 PostgreSQL Database directory appears to contain a database; Skipping initialization

Brian · July 12, 2023, 4:24pm

This means not enough disc space is allocated to Docker. This can be increased under the Resources tab in Docker Desktop. Look for "Virtual disk limit".

JHamburg430 · July 12, 2023, 7:20pm

At least in the Personal Tier of Docker Desktop, there is not a Virtual Disk Limit space, they direct you to use .wslconfig.

Referencing Microsoft documentation, there are not any options regarding vdisk size, only memory:
Available .wslconfig settings

I followed the instructions to increase WSL VHDX size:
How to expand the size of your WSL 2 Virtual Hard Disk

Increased size from 250 to 500:

Expanded the file system per instructions:

I see the block reflects this change:

Unless it has to be block 'sdc' for docker-desktop-user-distro? But this did not exist before enabling the setting in Docker Desktop, so I am fairly confident my files are in sdb. Please correct me if I am wrong and thanks again for the help!

Brian · July 12, 2023, 8:10pm

Right, the Virtual disk limit option is only in Docker Desktop for Mac. I haven't personally gone through the WSL resizing steps on Windows - does this GitHub comment help?

It's surprising that 250GB would not be enough space. Do you use Docker for other purposes? Have you tried running docker system prune to clean up any old stuff (if applicable)? The usstock minute bundle takes up about 60GB but other than that there's nothing that should add up to that much disk space.

JHamburg430 · July 12, 2023, 10:57pm

I do use Docker for other purposes as well (primarily Tensorflow), which I was planning to integrate into a QuantRocket strategy. This is why I was trying to avoid removing files as a solution since I still have plenty of storage left on my PC and will likely need even more for the complete strategy. This is most certainly a Docker block memory sizing issue so I will mark your response as a solution. Thanks!