IBKR error code 100: Max rate of messages per second has been exceeded

Technical Help
#1

Getting IBKR gateway error messages - started with no prior changes in code or environment, not clear where the issues came from / restarting IBG ... or changing IBKR accounts does not resolve the issue.

Nov 19 08:56:06 35.202.88.247 quantrocket.blotter WARNING IBKR executions not received from ibg1 in a timely manner, will try again shortly

Nov 19 08:56:16 35.202.88.247 quantrocket.blotter WARNING IBKR positions not received in a timely manner, will try again shortly

Nov 19 08:56:28 35.202.88.247 quantrocket.blotter WARNING ibg1 client 6000 got IBKR error code 100: Max rate of messages per second has been exceeded:max=50 rec=163 (1)

Nov 19 08:56:40 35.202.88.247 quantrocket.blotter WARNING ibg1 client 6000 got IBKR error code 100: Max rate of messages per second has been exceeded:max=50 rec=165 (2)

Nov 19 08:56:52 35.202.88.247 quantrocket.blotter WARNING ibg1 client 6000 got IBKR error code 100: Max rate of messages per second has been exceeded:max=50 rec=166 (3)

Nov 19 08:57:02 35.202.88.247 quantrocket.blotter WARNING IBKR executions not received from ibg1 in a timely manner, will try again shortly

Nov 19 08:57:12 35.202.88.247 quantrocket.blotter WARNING IBKR positions not received in a timely manner, will try again shortly

Nov 19 08:57:25 35.202.88.247 quantrocket.blotter WARNING ibg1 client 6000 got IBKR error code 100: Max rate of messages per second has been exceeded:max=50 rec=163 (1)

Nov 19 08:57:37 35.202.88.247 quantrocket.blotter WARNING ibg1 client 6000 got IBKR error code 100: Max rate of messages per second has been exceeded:max=50 rec=164 (2)

Nov 19 08:57:49 35.202.88.247 quantrocket.blotter WARNING ibg1 client 6000 got IBKR error code 100: Max rate of messages per second has been exceeded:max=50 rec=165 (3)

Nov 19 08:57:59 35.202.88.247 quantrocket.blotter WARNING IBKR executions not received from ibg1 in a timely manner, will try again shortly

Nov 19 08:58:09 35.202.88.247 quantrocket.blotter WARNING IBKR positions not received in a timely manner, will try again shortly

Nov 19 08:58:21 35.202.88.247 quantrocket.blotter WARNING ibg1 client 6000 got IBKR error code 100: Max rate of messages per second has been exceeded:max=50 rec=163 (1)

Nov 19 08:58:33 35.202.88.247 quantrocket.blotter WARNING ibg1 client 6000 got IBKR error code 100: Max rate of messages per second has been exceeded:max=50 rec=165 (2)

Nov 19 08:58:45 35.202.88.247 quantrocket.blotter WARNING ibg1 client 6000 got IBKR error code 100: Max rate of messages per second has been exceeded:max=50 rec=164 (3)

Nov 19 08:58:55 35.202.88.247 quantrocket.blotter WARNING IBKR executions not received from ibg1 in a timely manner, will try again shortly

Nov 19 08:59:05 35.202.88.247 quantrocket.blotter WARNING IBKR positions not received in a timely manner, will try again shortly

Nov 19 08:59:17 35.202.88.247 quantrocket.blotter WARNING ibg1 client 6000 got IBKR error code 100: Max rate of messages per second has been exceeded:max=50 rec=163 (1)

Nov 19 08:59:29 35.202.88.247 quantrocket.blotter WARNING ibg1 client 6000 got IBKR error code 100: Max rate of messages per second has been exceeded:max=50 rec=163 (2)

Nov 19 08:59:41 35.202.88.247 quantrocket.blotter WARNING ibg1 client 6000 got IBKR error code 100: Max rate of messages per second has been exceeded:max=50 rec=164 (3)

Nov 19 08:59:51 35.202.88.247 quantrocket.blotter WARNING IBKR executions not received from ibg1 in a timely manner, will try again shortly

Nov 19 09:00:01 35.202.88.247 quantrocket.blotter WARNING IBKR positions not received in a timely manner, will try again shortly

#2

This is a tough one.

IB Gateway accepts up to 50 API requests per second, in aggregate, from all connected clients (clients in this case are different QuantRocket containers). The error messages mean that the blotter is trying to make API requests to request orders and executions and IB Gateway is rejecting them because it’s saying there have already been ~163 messages sent within a 1-second timeframe (not necessarily all from the blotter).

Possibilities:

If there are many other things happening in your QuantRocket deployment, such as collecting historical data, collecting securities master details, I suppose in theory that could exceed the 50 message/sec limit. But everything is throttled and that’s never been reported by other users so I’m highly skeptical of that. You could open the IB Gateway GUI, check the box to show API messages, and click the various client tabs to see if one or more them seems highly active. Knowing which client IDs are sending lots of messages would be useful in troubleshooting. But I don’t find this theory all that plausible.

It’s a little suspicious that the number of messages that have already been received is always right around 163 (rec=163), as opposed to being more variable. Either some process is continually pelting IB Gateway with that many messages, as if running in an infinite loop, or IB Gateway is in some kind of stuck state where it think’s that happening even though it’s not. Again, just speculation.

Normally I would expect weird IB Gateway behavior (which does sometimes happen) to resolve after restarting IB Gateway and/or the Docker container.

If this was not happening to you before and is now happening, it's hard to know what to advise other than to suggest experimenting more to try to determine under what conditions it’s happening and under what conditions it’s not happening.

#3

I had this warning too, but after updating quantrocket from 2.2 to version 2.3 like 2 days ago I haven't seen single warning "ibg1 client 6000 got IBKR error code 100". And in IB Gateway GUI tab we can check client ID 6000 tab which was in warning message and see that many API messages are coming, in version 2.2 I saw growing list of messages there, but maybe in 2.3 it is smaller, idk.

#4

When viewing the API messages in the IB Gateway GUI, messages sent from QuantRocket to IBKR are indicated by a left pointing arrow at the beginning of the line:

<-

Responses from IBKR are indicated by a right pointing arrow:

->

Most of the volume will be from IBKR to QuantRocket, and those messages are not rate limited. The 50 msg/second limit only applies to messages sent from QuantRocket to IBKR. So the error message (assuming IB Gateway is not spuriously generating it) should indicate that there are more than 50 QuantRocket messages (with prefix <-) being sent in a single second. The 50 messages could come from any client, not just the client that gets the error message (client 6000 in this case).

If you get this error message, check the API messages and see if it looks like QuantRocket really is sending that many. If so, please export your API message logs and send it to brian [at] quantrocket [dot] com.

1 Like