What to do if TimeoutError occurs during writing to database

Technical Help
#1

I have a cron job that runs at 1pm HKT (so that's 1am EST now) and it seems writing the execution records to blotter messes with the IB NA server overnight reset.

Orders are still generated/sent to IB/executed, but the execution records don't seem to be captured in the blotter record, with the following error:

quantrocket.blotter: ERROR Traceback (most recent call last): 
quantrocket.blotter: ERROR File "sym://qrocket_log_py", line 34, in wrapped 
quantrocket.blotter: ERROR File "sym://qrocket_spooler_py", line 89, in spooler_start_order_monitor 
quantrocket.blotter: ERROR File "sym://qrocket_blotter_monitor_py", line 106, in run 
quantrocket.blotter: ERROR File "sym://qrocket_blotter_monitor_py", line 283, in write_executions_to_db 
quantrocket.blotter: ERROR File "sym://qrocket_blotter_monitor_py", line 375, in invalidate_pnl 
quantrocket.blotter: ERROR TimeoutError: Timeout waiting for mule to write PNL invalidations to database 
quantrocket.blotter: ERROR

Anything I can do to avoid this happening again (it has been happening for the last few days with regularity), other than scheduling the cron job, for example, 30mins later?

Thank you.

#2

The same error occurred on my 3pm HKT cron job too, so it doesn't seem like this is due to the IB server reset.
Curiously this error occurs only to the FX (spot) strategies (so far).

#3

Did you try just restarting the blotter?

The worker that receives executions calls another worker which writes to the PNL database. The second worker is not finishing in a timely manner. Hopefully a restart should fix it. The only other scenario I can imagine would be if you were running a PNL query at the same time, as this might mean the worker was having to wait for a database lock held by the PNL query, thus causing the timeout.

#4

Ok, let me restart blotter and see if that fixes.

#5

It looks like the problem was actually the latter - I had multiple cron jobs (generating hundreds of orders) at the same time, which seems to have caused the timeout. The problem seems to have gone away with spacing out of those cron jobs.