Hi Brian,
Getting 502's while running satellite. It's aperiodic and not 1 specific file. The entire countdown runs 2-3 days without issues, then reappears.
It also seems like if an error happens within satellite it hangs indefinitely and doesn't clean up without a manual restart. Sometimes it will run clean all day then the first cron kickoff the next day stalls. I don't think it's an HTTP concurrency issue but I'm unsure.
Cron
0 14 * * quantrocket history collect 'myprices' --priority && quantrocket history wait 'myprices' --timeout '2min' && quantrocket satellite exec 'codeload.project.pipeline.main'
Logs show history completed but quantrocket satellite exec doesn't run or fail.
2025-09-17 14:00:03 quantrocket.history: INFO [myprices] Collecting history from IBKR for 1 securities in myprices
2025-09-17 14:00:06 quantrocket.history: INFO [myprices] Saved 2220 total records for 1 total securities to quantrocket.v2.history.myprices.sqlite
When I do restart it manually I get logs like this
docker --context cloud compose restart satellite
>>
2025-09-17 15:24:42 quantrocket.countdown: ERROR Error running /opt/conda/bin/quantrocket satellite exec codeload.project.endofday.main
2025-09-17 15:24:42 quantrocket.countdown: ERROR msg: 'HTTPError(''502 Server Error: Bad Gateway for url: http://houston/satellite/commands?cmd=codeload.project.endofday.main'',
2025-09-17 15:24:42 quantrocket.countdown: ERROR ''please check the logs for more details'')'
2025-09-17 15:24:42 quantrocket.countdown: ERROR status: error
2025-09-17 15:24:42 quantrocket.countdown: ERROR Error running /opt/conda/bin/quantrocket satellite exec codeload.project.pipeline.main
2025-09-17 15:24:42 quantrocket.countdown: ERROR msg: 'HTTPError(''502 Server Error: Bad Gateway for url: http://houston/satellite/commands?cmd=codeload.project.pipeline.main'',
2025-09-17 15:24:42 quantrocket.countdown: ERROR ''please check the logs for more details'')'
2025-09-17 15:24:42 quantrocket.countdown: ERROR status: error
2025-09-17 15:24:42 quantrocket.countdown: ERROR Error running /opt/conda/bin/quantrocket satellite exec codeload.project.pipeline.main
2025-09-17 15:24:42 quantrocket.countdown: ERROR msg: 'HTTPError(''502 Server Error: Bad Gateway for url: http://houston/satellite/commands?cmd=codeload.project.pipeline.main'',
2025-09-17 15:24:42 quantrocket.countdown: ERROR ''please check the logs for more details'')'
2025-09-17 15:24:42 quantrocket.countdown: ERROR status: error
Checking docker while frozen.
docker --context cloud logs --tail 500 quantrocket_cloud-satellite-1
...The work of process 40 is done. Seeya!
corrupted double-linked list
worker 2 killed successfully (pid: 40)
Respawned uWSGI worker 2 (new pid: 267)
...The work of process 267 is done. Seeya!
worker 2 killed successfully (pid: 267)
Respawned uWSGI worker 2 (new pid: 269)
...The work of process 269 is done. Seeya!
worker 2 killed successfully (pid: 269)
Respawned uWSGI worker 2 (new pid: 298)
...The work of process 298 is done. Seeya!
!!! uWSGI process 298 got Segmentation Fault !!!
*** SIGNAL QUEUE IS FULL: buffer size 212992 bytes (you can tune it with --signal-bufsize) ***
could not deliver signal 0 to workers pool
*** SIGNAL QUEUE IS FULL: buffer size 212992 bytes (you can tune it with --signal-bufsize) ***
could not deliver signal 0 to workers pool
*** SIGNAL QUEUE IS FULL: buffer size 212992 bytes (you can tune it with --signal-bufsize) ***
could not deliver signal 0 to workers pool
*** SIGNAL QUEUE IS FULL: buffer size 212992 bytes (you can tune it with --signal-bufsize) ***
could not deliver signal 0 to workers pool
*** SIGNAL QUEUE IS FULL: buffer size 212992 bytes (you can tune it with --signal-bufsize) ***
...
Here I restarted then re-ran the pipeline. Even during successful runs I see SegFaults.
Run1:
quantrocket satellite exec 'codeload.project.pipeline.main'
[TRADE DETAILS] completed. <---- END OF MY CODE
...The work of process 28 is done. Seeya!
!!! uWSGI process 28 got Segmentation Fault !!!
Run2:
[TRADE DETAILS] completed. <---- END OF MY CODE
...The work of process 29 is done. Seeya!
!!! uWSGI process 29 got Segmentation Fault !!!
Inside quantrocket flightlog stream --details after a successful run.
quantrocket_cloud-satellite-1|corrupted double-linked list
quantrocket_cloud-satellite-1|worker 1 killed successfully (pid: 28)
quantrocket_cloud-satellite-1|Respawned uWSGI worker 1 (new pid: 150)
quantrocket_cloud-houston-1|172.18.0.21 - - [17/Sep/2025:20:20:22 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.31.0"
quantrocket_cloud-houston-1|172.18.0.10 - - [17/Sep/2025:20:20:22 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 9 "-" "python-urllib3/1.26.18"
quantrocket_cloud-countdown-1|From root@localhost Wed Sep 17 16:20:20 2025
quantrocket_cloud-countdown-1|Return-path: <root@localhost>
quantrocket_cloud-countdown-1|Envelope-to: root@localhost
quantrocket_cloud-countdown-1|Delivery-date: Wed, 17 Sep 2025 16:20:20 -0400
quantrocket_cloud-countdown-1|Received: from root by f3685c56c416 with local (Exim 4.94.2)
quantrocket_cloud-countdown-1| (envelope-from <root@localhost>)
quantrocket_cloud-countdown-1| id 1uyydY-0006ZW-80
quantrocket_cloud-countdown-1| for root@localhost; Wed, 17 Sep 2025 16:20:20 -0400
quantrocket_cloud-countdown-1|From: root@localhost (Cron Daemon)
quantrocket_cloud-countdown-1|To: root@localhost
quantrocket_cloud-countdown-1|Subject: Cron <root@f3685c56c416> quantrocket master isopen 'NYSE' --ago '35m' && quantrocket satellite exec 'codeload.project.pipeline.main' && quantrocket satellite exec 'codeload.project.merge.main'
quantrocket_cloud-countdown-1|MIME-Version: 1.0
quantrocket_cloud-countdown-1|Content-Type: text/plain; charset=UTF-8
quantrocket_cloud-countdown-1|Content-Transfer-Encoding: 8bit
quantrocket_cloud-countdown-1|X-Cron-Env: <SHELL=/bin/sh>
quantrocket_cloud-countdown-1|X-Cron-Env: <HOME=/root>
quantrocket_cloud-countdown-1|X-Cron-Env: <PATH=/usr/bin:/bin>
quantrocket_cloud-countdown-1|X-Cron-Env: <LOGNAME=root>
quantrocket_cloud-countdown-1|Message-Id: <E1uyydY-0006ZW-80@f3685c56c416>
quantrocket_cloud-countdown-1|Date: Wed, 17 Sep 2025 16:20:20 -0400
quantrocket_cloud-countdown-1|
quantrocket_cloud-countdown-1|status: success
quantrocket_cloud-countdown-1|status: success
quantrocket_cloud-countdown-1|
quantrocket_cloud-countdown-1|
I do async fetch options pricing within the pipeline but they're not raising errors. I also have asyncio.wait_for(timeout=15m) to confirm it's not timing out. They've never been the issue.