Broken Pipe Error in Market Data Collection

Technical Help
#1

Not sure what to do with this error when trying to collect market data for the realtime database:

quantrocket_realtime_1|updating market data collections
    quantrocket_houston_1|172.18.0.5 - - [11/Sep/2020:17:51:01 +0000] "GET /master/securities.csv?exclude_delisted=True&exclude_expired=True&fields=ibkr_ConId&fields=ibkr_Symbol&fields=ibkr_LocalSymbol&fields=ibkr_SecType&fields=ibkr_Currency&fields=ibkr_PrimaryExchange&fields=Timezone&fields=ibkr_ValidExchanges&fields=ibkr_MdSizeMultiplier&fields=ibkr_ComboLegs HTTP/1.1" 200 35562 "-" "-"
  quantrocket_flightlog_1|2020-09-11 13:51:01 quantrocket.realtime: WARNING Ignoring 1 of 178 securities with no ibkr_ConId field: FIBBG00P2CHBM8. You must collect securities master listings from IBKR before before you can collect real-time data from IBKR, see http://qrok.it/h/ibrtsid for help.
    quantrocket_houston_1|172.18.0.5 - - [11/Sep/2020:17:51:01 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
    quantrocket_houston_1|172.18.0.15 - - [11/Sep/2020:17:51:01 +0000] "GET /ibg2/gateway HTTP/1.1" 200 22 "-" "python-requests/2.22.0"
    quantrocket_houston_1|172.18.0.15 - - [11/Sep/2020:17:51:01 +0000] "GET /ibg3/gateway HTTP/1.1" 200 22 "-" "python-requests/2.22.0"
    quantrocket_houston_1|172.18.0.15 - - [11/Sep/2020:17:51:01 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.22.0"
    quantrocket_houston_1|172.18.0.15 - - [11/Sep/2020:17:51:02 +0000] "GET /ibg4/gateway HTTP/1.1" 200 22 "-" "python-requests/2.22.0"
    quantrocket_houston_1|172.18.0.5 - - [11/Sep/2020:17:51:02 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 17 "-" "-"
   quantrocket_realtime_1|ibg1 client 7570 got IBKR error code 2107: HMDS data farm connection is inactive but should be available upon demand.ushmds
   quantrocket_realtime_1|Issuing to ibg1 market data request for ADM STK (sid FIBBG000BB6WG8)
   quantrocket_realtime_1|Issuing to ibg4 market data request for AEM STK (sid FIBBG000DLVDK3)
   quantrocket_realtime_1|ibg4 client 7570 got IBKR error code 2107: HMDS data farm connection is inactive but should be available upon demand.ushmds
   quantrocket_realtime_1|Issuing to ibg1 market data request for AEO STK (sid FIBBG000BGXZB5)
   quantrocket_realtime_1|Issuing to ibg4 market data request for AIMT STK (sid FIBBG005NMB2L4)
   quantrocket_realtime_1|Issuing to ibg1 market data request for ALB STK (sid FIBBG000BJ26K7)
   quantrocket_realtime_1|Issuing to ibg4 market data request for ALGN STK (sid FIBBG000BRNLL2)
   quantrocket_realtime_1|Issuing to ibg1 market data request for ALK STK (sid FIBBG000BBL0Y1)
   quantrocket_realtime_1|Issuing to ibg4 market data request for ALLY STK (sid FIBBG000BC2R71)
   quantrocket_realtime_1|Issuing to ibg1 market data request for AMCR STK (sid FIBBG00LNJRQ09)
   quantrocket_realtime_1|Exception in thread market_data_request_worker:
   quantrocket_realtime_1|Traceback (most recent call last):
   quantrocket_realtime_1|  File "/opt/conda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
   quantrocket_realtime_1|    self.run()
   quantrocket_realtime_1|  File "/opt/conda/lib/python3.7/threading.py", line 870, in run
   quantrocket_realtime_1|    self._target(*self._args, **self._kwargs)
   quantrocket_realtime_1|  File "sym://qrocket_realtime_collect_ibkr_workers_py", line 16, in wrapped
   quantrocket_realtime_1|  File "sym://qrocket_realtime_collect_ibkr_market_data_py", line 79, in run
   quantrocket_realtime_1|  File "sym://qrocket_realtime_collect_ibkr_market_data_py", line 152, in subscribe
   quantrocket_realtime_1|  File "/opt/conda/lib/python3.7/site-packages/ibapi/client.py", line 418, in reqMktData
   quantrocket_realtime_1|    self.sendMsg(msg)
   quantrocket_realtime_1|  File "/opt/conda/lib/python3.7/site-packages/ibapi/client.py", line 75, in sendMsg
   quantrocket_realtime_1|    self.conn.sendMsg(full_msg)
  quantrocket_flightlog_1|2020-09-11 13:51:02 quantrocket.realtime: ERROR Traceback (most recent call last):
   quantrocket_realtime_1|  File "/opt/conda/lib/python3.7/site-packages/ibapi/connection.py", line 80, in sendMsg
   quantrocket_realtime_1|    nSent = self.socket.send(msg)
   quantrocket_realtime_1|BrokenPipeError: [Errno 32] Broken pipe
   quantrocket_realtime_1|
    quantrocket_houston_1|172.18.0.5 - - [11/Sep/2020:17:51:02 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
  quantrocket_flightlog_1|2020-09-11 13:51:02 quantrocket.realtime: ERROR   File "sym://qrocket_realtime_collect_ibkr_workers_py", line 16, in wrapped
    quantrocket_houston_1|172.18.0.5 - - [11/Sep/2020:17:51:02 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
  quantrocket_flightlog_1|2020-09-11 13:51:02 quantrocket.realtime: ERROR   File "sym://qrocket_realtime_collect_ibkr_market_data_py", line 79, in run
    quantrocket_houston_1|172.18.0.5 - - [11/Sep/2020:17:51:02 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
  quantrocket_flightlog_1|2020-09-11 13:51:02 quantrocket.realtime: ERROR   File "sym://qrocket_realtime_collect_ibkr_market_data_py", line 152, in subscribe
    quantrocket_houston_1|172.18.0.5 - - [11/Sep/2020:17:51:02 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
  quantrocket_flightlog_1|2020-09-11 13:51:02 quantrocket.realtime: ERROR   File "/opt/conda/lib/python3.7/site-packages/ibapi/client.py", line 418, in reqMktData
    quantrocket_houston_1|172.18.0.5 - - [11/Sep/2020:17:51:02 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
  quantrocket_flightlog_1|2020-09-11 13:51:02 quantrocket.realtime: ERROR     self.sendMsg(msg)
    quantrocket_houston_1|172.18.0.5 - - [11/Sep/2020:17:51:02 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
  quantrocket_flightlog_1|2020-09-11 13:51:02 quantrocket.realtime: ERROR   File "/opt/conda/lib/python3.7/site-packages/ibapi/client.py", line 75, in sendMsg
    quantrocket_houston_1|172.18.0.5 - - [11/Sep/2020:17:51:02 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
  quantrocket_flightlog_1|2020-09-11 13:51:02 quantrocket.realtime: ERROR     self.conn.sendMsg(full_msg)
    quantrocket_houston_1|172.18.0.5 - - [11/Sep/2020:17:51:02 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
  quantrocket_flightlog_1|2020-09-11 13:51:02 quantrocket.realtime: ERROR   File "/opt/conda/lib/python3.7/site-packages/ibapi/connection.py", line 80, in sendMsg
    quantrocket_houston_1|172.18.0.5 - - [11/Sep/2020:17:51:02 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
  quantrocket_flightlog_1|2020-09-11 13:51:02 quantrocket.realtime: ERROR     nSent = self.socket.send(msg)
    quantrocket_houston_1|172.18.0.5 - - [11/Sep/2020:17:51:02 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
  quantrocket_flightlog_1|2020-09-11 13:51:02 quantrocket.realtime: ERROR BrokenPipeError: [Errno 32] Broken pipe
    quantrocket_houston_1|172.18.0.5 - - [11/Sep/2020:17:51:02 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
  quantrocket_flightlog_1|2020-09-11 13:51:02 quantrocket.realtime: ERROR 
    quantrocket_houston_1|172.18.0.5 - - [11/Sep/2020:17:51:02 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
   quantrocket_realtime_1|Worker exception detected, shutting down
  quantrocket_flightlog_1|2020-09-11 13:51:03 quantrocket.realtime: INFO Exiting IBKR market data collection due to errors

Please advise.

David

#2

Does the error keep happening and it is always the same sid? A Broken Pipe means the connection between QuantRocket and IB Gateway was lost or disconnected for some reason. This could be due to a transient network interruption in which case the error should pass. If it persists and is always the same sid, however, that would not be the likely explanation.

#3

Here's more detail on the error:

   2020-09-14 09:29:14 quantrocket.realtime: WARNING Ignoring 1 of 203 securities with no ibkr_ConId field: FIBBG00P2CHBM8. You must collect securities master listings from IBKR before before you can collect real-time data from IBKR, see http://qrok.it/h/ibrtsid for help.
    2020-09-14 09:29:14 quantrocket.realtime: WARNING ibg1 client 7023 got IBKR error code 320: Error reading request.Message id 0.  Unable to parse data. java.lang.NumberFormatException: For input string: "4065.0"
    2020-09-14 09:29:14 quantrocket.realtime: WARNING ibg4 client 7023 got IBKR error code 320: Error reading request.Message id 1.  Unable to parse data. java.lang.NumberFormatException: For input string: "4165.0"
    2020-09-14 09:29:15 quantrocket.realtime: ERROR Traceback (most recent call last):
    2020-09-14 09:29:15 quantrocket.realtime: ERROR   File "sym://qrocket_realtime_collect_ibkr_workers_py", line 16, in wrapped
    2020-09-14 09:29:15 quantrocket.realtime: ERROR   File "sym://qrocket_realtime_collect_ibkr_market_data_py", line 79, in run
    2020-09-14 09:29:15 quantrocket.realtime: ERROR   File "sym://qrocket_realtime_collect_ibkr_market_data_py", line 152, in subscribe
    2020-09-14 09:29:15 quantrocket.realtime: ERROR   File "/opt/conda/lib/python3.7/site-packages/ibapi/client.py", line 418, in reqMktData
    2020-09-14 09:29:15 quantrocket.realtime: ERROR     self.sendMsg(msg)
    2020-09-14 09:29:15 quantrocket.realtime: ERROR   File "/opt/conda/lib/python3.7/site-packages/ibapi/client.py", line 75, in sendMsg
    2020-09-14 09:29:15 quantrocket.realtime: ERROR     self.conn.sendMsg(full_msg)
    2020-09-14 09:29:15 quantrocket.realtime: ERROR   File "/opt/conda/lib/python3.7/site-packages/ibapi/connection.py", line 80, in sendMsg
    2020-09-14 09:29:15 quantrocket.realtime: ERROR     nSent = self.socket.send(msg)
    2020-09-14 09:29:15 quantrocket.realtime: ERROR BrokenPipeError: [Errno 32] Broken pipe
    2020-09-14 09:29:15 quantrocket.realtime: ERROR

I re-collected the securities master listings from IB using collect_ibkr_listings, but the error persists.

#4

This is the same issue that the history service was having, conids being treated as strings. I don't know why this seems to only be affecting you, but it can be fixed. (We actually already applied the fix when we did the history service fix, "just in case", but haven't yet released it.) In docker-compose.yml, replace:

image: 'quantrocket/realtime:2.2.0'

with

image: 'quantrocket/realtime:patch20200914'

and then run

docker-compose up -d realtime
QuantRocket 2.3.0 Release Notes
#5

That worked! All set now.

#6

This issue is fixed in version 2.3.0.

#8

This 'bkr_ConIdfield' is a warning, not an error, but it indicates that there is a missing field in the securities master listings from IBKR. Theibkr_ConId` field is necessary for real-time database enrichment from IBKR. The message suggests that you need to collect securities master listings from IBKR before you can collect real-time data from them. You can find more information about this issue and how to resolve it by visiting the link provided in the error message: Usage Guide