Exception in message dispatch. Handler '_historical_data_handler' for 'historicalData'

Issues
#1

I'm getting a lot lot of these messages in my history log at the moment, at the same time data collection appears to have stopped.

Exception in message dispatch.  Handler '_historical_data_handler' for 'historicalData'
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/ibpythonic/dispatcher.py", line 43, in __call__
results.append(listener(message))
File "sym://qrocket_history_interactivebrokers_py", line 1014, in _historical_data_handler
KeyError: 14148
Exception in message dispatch.  Handler '_historical_data_handler' for 'historicalData'
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/ibpythonic/dispatcher.py", line 43, in __call__
results.append(listener(message))
File "sym://qrocket_history_interactivebrokers_py", line 1014, in _historical_data_handler
KeyError: 14148
Exception in message dispatch.  Handler '_historical_data_handler' for 'historicalData'
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/ibpythonic/dispatcher.py", line 43, in __call__
results.append(listener(message))
File "sym://qrocket_history_interactivebrokers_py", line 1014, in _historical_data_handler
KeyError: 14148
Exception in message dispatch.  Handler '_historical_data_handler' for 'historicalData'
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/ibpythonic/dispatcher.py", line 43, in __call__
results.append(listener(message))
File "sym://qrocket_history_interactivebrokers_py", line 1014, in _historical_data_handler
KeyError: 14148
Exception in message dispatch.  Handler '_historical_data_end_handler' for 'historicalDataEnd'
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/ibpythonic/dispatcher.py", line 43, in __call__
results.append(listener(message))
File "sym://qrocket_history_interactivebrokers_py", line 1024, in _historical_data_end_handler
KeyError: 14148
#2

Can you try to create a small reproducible example involving one or a few specific conids and provide the steps to reproduce? For this issue, a reproducible example should include, at minimum, the conid(s), database config, history container version, whether this happens when running a single gateway or only with multiple gateways, etc.

#3

Hi Brian,

It's hard to recreate in a lab scenario because it doesn't appear to be related to specific conid(s) or a specific gateway. But It has happened after pulling a large amount of data like all stocks on nasdaq over multiple gateways. I was getting an error for gateways 2,3 and 4 so decided to stop gateway 4 to try see if a stop/start would resolve the issue. This resulted in the above exception being thrown thousands of times. My best guess it's an error there is suppressed while the gateway is running but then on the stop command it releases all the exceptions and flood the logs with it.

container versions:
history latest
db is 1.2.0
gateway total of 4 gateways, gw 4 is running the new 973.1 image, the other 3 972.0. The version here doesn't appear to affect the issue.

#4

The errors I got before I stopped gateway4 was the following, as you can see I only get them for ibg2-4 not for ibg1. 1 and 4 are live accounts, 2 and 3 are paper accounts only. The errors continued up to the time when I stopped ibg4. And has continued for ibg2 and 3.

Aug 02 09:21:41 quantrocket.history: ERROR [nasdaq-stk-1min] No response received from ibg3 after 30 seconds for historical data request for 1 D of 1 min TRADES for AIRT STK (conid 4726067) ending 20180729 08:27:00 GMT, won't request any more data for this security at present to avoid data gaps, will try to get the data next time nasdaq-stk-1min is fetched 
Aug 02 09:21:41 quantrocket.history: ERROR [nasdaq-stk-1min] No response received from ibg4 after 30 seconds for historical data request for 1 D of 1 min TRADES for AIRT STK (conid 4726067) ending 20180730 08:27:00 GMT, won't request any more data for this security at present to avoid data gaps, will try to get the data next time nasdaq-stk-1min is fetched 
Aug 02 09:22:41 quantrocket.history: ERROR [nasdaq-stk-1min] No response received from ibg2 after 30 seconds for historical data request for 1 D of 1 min TRADES for AKAO STK (conid 146033470) ending 20180731 08:27:00 GMT, won't request any more data for this security at present to avoid data gaps, will try to get the data next time nasdaq-stk-1min is fetched 
Aug 02 09:22:42 quantrocket.history: ERROR [nasdaq-stk-1min] No response received from ibg4 after 30 seconds for historical data request for 1 D of 1 min TRADES for AKAO STK (conid 146033470) ending 20180802 08:27:00 GMT, won't request any more data for this security at present to avoid data gaps, will try to get the data next time nasdaq-stk-1min is fetched 
Aug 02 09:23:11 quantrocket.history: ERROR [nasdaq-stk-1min] No response received from ibg3 after 30 seconds for historical data request for 1 D of 1 min TRADES for AKBA STK (conid 146445446) ending 20180801 08:27:00 GMT, won't request any more data for this security at present to avoid data gaps, will try to get the data next time nasdaq-stk-1min is fetched 
Aug 02 09:23:11 quantrocket.history: ERROR [nasdaq-stk-1min] No response received from ibg4 after 30 seconds for historical data request for 1 D of 1 min TRADES for AKBA STK (conid 146445446) ending 20180802 08:27:00 GMT, won't request any more data for this security at present to avoid data gaps, will try to get the data next time nasdaq-stk-1min is fetched 
Aug 02 09:24:11 quantrocket.history: ERROR [nasdaq-stk-1min] No response received from ibg2 after 30 seconds for historical data request for 1 D of 1 min TRADES for AKER STK (conid 142928377) ending 20180731 08:27:00 GMT, won't request any more data for this security at present to avoid data gaps, will try to get the data next time nasdaq-stk-1min is fetched 
Aug 02 09:24:12 quantrocket.history: ERROR [nasdaq-stk-1min] No response received from ibg4 after 30 seconds for historical data request for 1 D of 1 min TRADES for AKER STK (conid 142928377) ending 20180802 08:27:00 GMT, won't request any more data for this security at present to avoid data gaps, will try to get the data next time nasdaq-stk-1min is fetched 
Aug 02 09:24:42 quantrocket.history: ERROR [nasdaq-stk-1min] No response received from ibg3 after 30 seconds for historical data request for 1 D of 1 min TRADES for AKRX STK (conid 265973) ending 20180801 08:27:00 GMT, won't request any more data for this security at present to avoid data gaps, will try to get the data next time nasdaq-stk-1min is fetched 
Aug 02 09:25:42 quantrocket.history: ERROR [nasdaq-stk-1min] No response received from ibg2 after 30 seconds for historical data request for 1 D of 1 min TRADES for AKTX STK (conid 206821772) ending 20180731 08:27:00 GMT, won't request any more data for this security at present to avoid data gaps, will try to get the data next time nasdaq-stk-1min is fetched 
Aug 02 09:25:42 quantrocket.history: ERROR [nasdaq-stk-1min] No response received from ibg4 after 30 seconds for historical data request for 1 D of 1 min TRADES for AKTX STK (conid 206821772) ending 20180802 08:27:00 GMT, won't request any more data for this security at present to avoid data gaps, will try to get the data next time nasdaq-stk-1min is fetched 
Aug 02 09:26:12 quantrocket.history: ERROR [nasdaq-stk-1min] No response received from ibg2 after 30 seconds for historical data request for 1 D of 1 min TRADES for ALBO STK (conid 254358124) ending 20180731 08:27:00 GMT, won't request any more data for this security at present to avoid data gaps, will try to get the data next time nasdaq-stk-1min is fetched 
Aug 02 09:26:12 quantrocket.history: ERROR [nasdaq-stk-1min] No response received from ibg4 after 30 seconds for historical data request for 1 D of 1 min TRADES for ALBO STK (conid 254358124) ending 20180802 08:27:00 GMT, won't request any more data for this security at present to avoid data gaps, will try to get the data next time nasdaq-stk-1min is fetched 
Aug 02 09:27:43 quantrocket.history: ERROR [nasdaq-stk-1min] No response received from ibg3 after 30 seconds for historical data request for 1 D of 1 min TRADES for ALDX STK (conid 148443223) ending 20180428 08:27:00 GMT, won't request any more data for this security at present to avoid data gaps, will try to get the data next time nasdaq-stk-1min is fetched 
Aug 02 09:28:13 quantrocket.history: ERROR [nasdaq-stk-1min] No response received from ibg4 after 30 seconds for historical data request for 1 D of 1 min TRADES for ALGN STK (conid 11459264) ending 20180802 08:27:00 GMT, won't request any more data for this security at present to avoid data gaps, will try to get the data next time nasdaq-stk-1min is fetched 
Aug 02 09:29:13 quantrocket.history: ERROR [nasdaq-stk-1min] No response received from ibg4 after 30 seconds for historical data request for 1 D of 1 min TRADES for ALIM STK (conid 74685913) ending 20180802 08:27:00 GMT, won't request any more data for this security at present to avoid data gaps, will try to get the data next time nasdaq-stk-1min is fetched 
Aug 02 09:29:43 quantrocket.history: ERROR [nasdaq-stk-1min] No response received from ibg4 after 30 seconds for historical data request for 1 D of 1 min TRADES for ALJJ STK (conid 41957917) ending 20180802 08:27:00 GMT, won't request any more data for this security at present to avoid data gaps, will try to get the data next time nasdaq-stk-1min is fetched 
Aug 02 09:31:13 quantrocket.history: ERROR [nasdaq-stk-1min] No response received from ibg3 after 30 seconds for historical data request for 1 D of 1 min TRADES for ALNA STK (conid 294729588) ending 20180801 08:27:00 GMT, won't request any more data for this security at present to avoid data gaps, will try to get the data next time nasdaq-stk-1min is fetched
#5

I'm looking into this

#6

Thanks. Please get in contact if you like to have a look into our setup to debug and extract logs.

#7

The root cause of the errors is a bottleneck in processing the data that is arriving from IB. Minute bars with multiple gateways triggers the bottleneck due to the greater volume of data. The history container is undergoing an internal redesign for the next release (1.3, targeting late August), and among other things that will address this issue. The redesign should result in notably faster data collection for intraday bars. I would suggest delaying 1-minute bar collection until the release of 1.3.

#8

Thanks Brian, looking forward to the update.

#10

Hi Brian, is there an update on when we can expect the new release?

#11

The release is in the testing and documentation phase. Should be out by the end of next week.

#12

The 1.3 release, with the fix for this issue, is now available.

#13

This topic was automatically closed after 5 days. New replies are no longer allowed.