Out of memory errors

Technical Help
#1

I'm running QR on GCP. It runs great for about 6-8 hours and then starts to run out of memory. One example log messages looks like this:

-- Jan 21 16:47:29 quantrocket kernel: Out of memory: Kill process 21485 (python) score 46 or sacrifice child

Jan 21 16:47:29 quantrocket kernel: Killed process 21485 (python), UID 0, total-vm:1131528kB, anon-rss:179056kB, file-rss:0kB, shmem-rss:0kB

Out of memory: Kill process 12153 (uwsgi-quantrock) score 38 or sacrifice child
Jan 21 19:23:27 quantrocket kernel: Killed process 13608 (uwsgi-quantrock), UID 0, total-vm:530320kB, anon-rss:147688kB, file-rss:0kB, shmem-rss:24kB
Jan 21 19:25:58 quantrocket rsyslogd: imjournal: journal reloaded... [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/0 ]
Jan 21 19:36:02 quantrocket kernel: google_cloud_op invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Jan 21 19:38:58 quantrocket kernel: google_cloud_op cpuset=/ mems_allowed=0
Jan 21 19:42:33 quantrocket kernel: CPU: 1 PID: 17799 Comm: google_cloud_op Tainted: G ------------ T 3.10.0-1160.80.1.el7.x86_64 #1
Jan 21 19:45:09 quantrocket kernel: Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/26/2022

QR is the only thing running on this box. What kind of information do I need to share to get to the bottom of this issue?

#2

The memory utilization of the killed processes shown in your output is not very big (1 GB max), so it seems that your overall system memory might be too low. The system should have at least 8 GB of memory, but 12-16 GB is better. You can run docker stats to see how much memory your Docker containers have and are using. For example, the following output shows that Docker has 11.7 GB of total memory available to use (to be shared among all containers), and the various containers are currently using a few hundred MB each (in this case).

CONTAINER ID   NAME                            CPU %     MEM USAGE / LIMIT    MEM %     NET I/O           BLOCK I/O         PIDS
f32cdc232b9f   quantrocket-ibg1-1              0.17%     153.4MiB / 11.7GiB   1.28%     18.2MB / 5.91MB   586kB / 9.08MB    23
d75e0fd4d892   quantrocket-history-1           0.01%     108.4MiB / 11.7GiB   0.90%     49.9kB / 26kB     606kB / 426kB     13
7dad1d848ab7   quantrocket-blotter-1           0.53%     234.7MiB / 11.7GiB   1.96%     83.1MB / 4.2MB    11.5MB / 207MB    9
7629d4203f1e   quantrocket-realtime-1          0.02%     114.2MiB / 11.7GiB   0.95%     58.6kB / 26.6kB   1.88MB / 16.4kB   23
84459e74f6c5   quantrocket-satellite-1         0.02%     38.03MiB / 11.7GiB   0.32%     30.9kB / 0B       1.91MB / 8.19kB   9
...

Some additional tips on memory errors are available in the usage guide.

#3

That fixed the problem!
Thanks,
/g