Logspout container performance impact

Technical Help
#1

Hi Brian,

I recently moved the Quantrokcet installation from a Windows-based to the MacOS environment to take advantage of the Apple M architecture and improve backtest performance, particularly within the Zipline libraries.

We noticed a significant performance impact from the logspout container to the extent that it was constantly running and impacting both continuous CPU performance and power consumption. This occurs even when all containers are idle as logspout seems to continuously use at least 3 cores worth of CPU resources. I had not noticed this under the Windows environment.

When I pause this container these issues go away and the backtest performance improves by almost 30%!

Do you have any guidance or insight as to why this occurs and if there is any other workaround besides pausing the container?

Many thanks.

#2

Someone on an Intel chip reported the same thing last year, so apparently it can happen but it's not specific to Apple Silicon. I use Apple Silicon and haven't had an issue with high CPU usage from logspout. I have, however, sometimes seem high CPU usage from various other containers on various other machines that seemingly couldn't be explained by anything QuantRocket was doing. Maybe something happening at the Docker level. Usually a reliable solution is:

docker compose down
docker compose up -d

If you don't want to take everything down and back up (e.g. because you've installed custom packages on certain containers, which the above commands would wipe out, requring you to install the custom packages again), you can alternatively try re-creating just the logspout container:

docker compose up -d --force-recreate logspout
#3

Thanks Brian,

I did recreate the logspout container. Still, the impact on the MacOS Docker world seems to be a higher level of background CPU usage while all containers are idle, and about a 33% difference (slower) when running a backtest. I see no obvious reason for this and there are no specific installation changes that should affect this.

In the Windows environment, the difference is there but less so. There is about a 6% performance improvement on a backtest when logpout is paused. However, I think this is because my ML code components run slower on the PC and hence the timing percentage impact seems muted.

I will keep digging but pausing the container seems to be a simple workaround for now.

#4

Did you also try docker compose down and docker compose up -d? Sometimes I’ve seen that work better than —force-recreate, perhaps because it makes Docker forget everything more completely than —force-recreate does.

#5

Thanks, Brian, That did fix the issue. The container CPU time is now down to minimal usage.

I am really confused as to what could have caused this since the logpout was the problem container and it was recreated.