Moonshot isn't logging to flightlogs stream --detail

Technical Help
#1

I noticed when attempt to debug a strategy a few days ago that flightlog wasn't logging anything for backtests. I use to see "beginning backtesting for X strategy" or something along those lines when the backtest started.

Now there's nothing which makes tracing back bugs a bit laborious.

I have tried restarting the moonshot, flight log, and logspout containers. I have also restarted the whole machine.

I can confirm I setup a logger, I can log custom logs that way.

Any ideas on how to get moonshot logs streaming correctly again?

#2

A message is printed to the detailed logs at the start of a backtest. If you're not seeing it, and you've restarted logspout, I would try removing your custom logger, as it's possible your overriding or redirecting the logs with that somehow.

#3

Also, for debugging Moonshot, you may want to check out a newly posted video in the usage guide that shows a good way to do that.

#4

Hey Brian,

I will have to check out your new video!

The custom logger was added a week or two after the logs stopped appearing. Any other ideas of what could be causing this behavior?

#5

Unfortunately, I’m not sure what other troubleshooting steps to suggest here.

#6

Hey Brian,

We are seeing a ton of errors in syslog and deamon.log. Its the same errors repeatedly. There are a huge amounts of these errors eating out our read/writes and cpu.

SYSLOG:


*May 31 18:16:14 debian dockerd[1277]: time="2024-05-31T18:16:14.280708531Z" level=error msg="Error streaming logs: invalid character '\\x00' looking for beginning of value" container=2c35f66d44b8 method="(*Daemon).ContainerLogs" module=daemon

May 31 18:16:14 debian dockerd[1277]: time="2024-05-31T18:16:14.284646861Z" level=error msg="Error streaming logs: invalid character '\\x00' looking for beginning of value" container=d839d5b87359 method="(*Daemon).ContainerLogs" module=daemon

May 31 18:16:14 debian dockerd[1277]: time="2024-05-31T18:16:14.286093371Z" level=error msg="Error streaming logs: invalid character '\\x00' looking for beginning of value" container=2c35f66d44b8 method="(*Daemon).ContainerLogs" module=daemon

@quantrocket:/var/log$  grep "Error streaming logs: " syslog |wc -l

5164667

DAEMON.LOG

May 31 18:19:02 debian dockerd[1277]: time="2024-05-31T18:19:02.563579066Z" level=error msg="Error streaming logs: invalid character '\\x00' looking for beginning of value" container=c5c155b3f890 method="(*Daemon).ContainerLogs" module=daemon

May 31 18:19:02 debian dockerd[1277]: time="2024-05-31T18:19:02.564294346Z" level=error msg="Error streaming logs: invalid character '\\x00' looking for beginning of value" container=f0c8b6554307 method="(*Daemon).ContainerLogs" module=daemon

@quantrocket:/var/log$ grep "Error streaming logs:" daemon.log | wc -l

5173115

Any ideas on how to either turn off this error logging or how to fix the issue causing the errors? I have a hunch this is causing our other system resource related errors.

Thanks!

#7

I don't have experience with that error, but googling the error message leads to this Docker issue on GitHub, which suggests corrupt log files may be the issue. I would suggest working through some of the comments in the GitHub issue. Most of the solutions consist of deleting the corrupt line from the log files or deleting the log files entirely.

#8

Hey Brian,

Thank you for that post. This does appear to be a docker issue that can be resolved permanently using log rotation.

I deleted all the json.log files in all the container directories in /var/lib/docker/containers, all the stacked up daemon and syslog directories and gave the machine a restart.

you can use the following command to see which of your log files has the offending character.

grep -r -l -a -P '\x00' /var/lib/docker/containers/

Here is a post on the log rotation: --log-opt max-size=50m doesn't have any effect on the logs generated by the image · Issue #35185 · moby/moby · GitHub
Is this something that could be addressed within quantrocket in a future update?

Thanks!

#9

This isn't a good candidate for automatic handling because it's only been seen once and it occurs on the host operating system, outside of the containers where QuantRocket runs.