Health check containers - missing messages in the detailed flightlog

Technical Help
#1

Hi, Is there any health checks on the containers? If they are running are they assumed to be running correctly?

I normally run the detailed flightlog in one of the Jupyter terminals to keep track of the data harvesting. But from time to time the logs for the history container isn't displayed. E.g. messages like:

Issuing to ibg2 historical data request for 138 D of 1 day TRADES for FSLR STK (conid 120271555) ending 20130720 04:59:59 GMT
Expected remaining runtime to fetch nasdaq-stk-1min history based on IB response times so far: 4 days, 18:08:53 (631601 more requests @ 1.95s/request across 3 gateways)

Running: "history queue" shows the job is still in the queue. "launchpad status" shows 3 running gateways. but still nothing appear to be collected. How do I check what requests are begin sent?

I have tried the following steps to resolve my issue but with no luck, the logs are still quiet.

Stopped and started all 3 gateways
Run "history cancel " and "history fetch "
Restarted the history, flightlog and launchpad containers

I would prefer to find a way to fix this issue without rebooting the host server, which resolved the issue last time. Please let me know if you need any log messages, I looked didn't find any error messages I could connect to this issue.

#2

If no logs at all are appearing you might need to right click > refresh terminal in Jupyter. Sometimes the Jupyter terminal itself can get stuck.

#3

I'm aware that Jupyter sometimes need a refresh this isn't the problem. Some log messages like the spooler messages from the blotter and Get Http 200 OK messages for the gateways from the houston appear in the log. It's only the messages from the history container there appear to be missing, or delayed. Connecting to the history container with docker logs indicate the logs aren't moving. The situation happened just after I stopped one of the gateways. Starting again and waiting around 2 hours appear to have "fixed" the issue and the data harvesting is running again.

#4

Related to the issue with missing log messages, I appear only to be collecting data from 2 of my 4 active gateways. How do I make sure all 4 gateways are used?