Docker compose-overrride.yml file

Technical Help
#1

I'm attempting to establish my second login to IBKR in preparation to download historic data and creating my "ibg2" account to speed the download process. Some context and questions:

  • I've attempted to put the "docker compose-override.yml" file in the same location as the "docker compose.yml" file but I don't know if I've done that correctly.
  • I've attempted to do this from Jupyter Notebook and it's not clear to me if I should have done this in my MAC Terminal or in Docker instead.
  • Below is what I've done from Jupyter Notebook and the errors I received.
%%writefile docker-compose.override.yml

# docker-compose.override.yml
services:
  ibg2:
    extends:
        file: docker-compose.yml
        service: ibg1

Overwriting docker-compose.override.yml

cd /path/to/docker-compose.yml
docker compose -p quantrocket up -d

  Cell In[34], line 2
    docker compose -p quantrocket up -d
           ^
SyntaxError: invalid syntax

from quantrocket.ibg import set_credentials
set_credentials("ibg2", username="dlehman1927", trading_mode="paper")

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[25], line 2
      1 from quantrocket.ibg import set_credentials
----> 2 set_credentials("ibg2", username="dlehman1927", trading_mode="paper")

File /opt/conda/lib/python3.11/site-packages/quantrocket/ibg.py:128, in set_credentials(gateway, username, password, trading_mode)
    126 statuses = list_gateway_statuses(gateways=[gateway])
    127 if not statuses:
--> 128     raise ValueError("no such IB Gateway: {0}".format(gateway))
    130 if username and not password:
    131     password = getpass.getpass(prompt="Enter IBKR Password: ")

ValueError: no such IB Gateway: ibg2

Any help or guidance would be appreciated. Thanks.

#2

All docker and docker compose commands must be run from the host operating system (via Terminal on Mac or Linux or Powershell on Windows). They cannot be run from inside a JupyterLab terminal. Docker compose YAML files must be stored in a directory on the host system, not in the JupyterLab filesystem. The installation tutorials provide guidance on where to put the docker compose files.

#3

@Brian I was able to successfully create the "compose-overrride.yml" file and as you can see below it is successfully showing up on my configuration. However, I'm still receiving errors when I attempt to "set_credentials" inside of Jupyter Notebook (I've also pasted that error below).

chris@Chriss-MBP-2 quantrocket % docker compose config  
name: quantrocket
services:
  account:
    depends_on:
      db:
        condition: service_started
        required: true
    image: quantrocket/account:2.9.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: db
        target: /var/lib/quantrocket
        volume: {}
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
  blotter:
    depends_on:
      db:
        condition: service_started
        required: true
    image: quantrocket/blotter:2.9.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: db
        target: /var/lib/quantrocket
        volume: {}
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
  codeload:
    environment:
      GIT_URL: https://github.com/quantrocket-codeload/quickstart.git
      OVERWRITE_EXISTING: "true"
    image: quantrocket/codeload:2.9.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: codeload
        target: /codeload
        volume: {}
  countdown:
    image: quantrocket/countdown:2.9.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
      - type: volume
        source: codeload
        target: /codeload
        volume: {}
  db:
    depends_on:
      postgres:
        condition: service_started
        required: true
    image: quantrocket/db:2.9.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: db
        target: /var/lib/quantrocket
        volume: {}
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
  flightlog:
    environment:
      SEND_CRASH_REPORTS: "true"
    image: quantrocket/flightlog:2.9.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: flightlog
        target: /var/log/flightlog
        volume: {}
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
  fundamental:
    depends_on:
      db:
        condition: service_started
        required: true
    image: quantrocket/fundamental:2.9.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: db
        target: /var/lib/quantrocket
        volume: {}
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
  history:
    depends_on:
      db:
        condition: service_started
        required: true
    image: quantrocket/history:2.9.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: db
        target: /var/lib/quantrocket
        volume: {}
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
  houston:
    environment:
      QUANTROCKET_VERSION: 2.9.2
    image: quantrocket/houston:2.9.0
    networks:
      default: null
    ports:
      - mode: ingress
        target: 80
        published: "1969"
        protocol: tcp
    restart: always
  ibg1:
    environment:
      INI_SETTINGS: --ExistingSessionDetectedAction=primary
    image: quantrocket/ibg:2.9.2
    networks:
      default: null
    platform: linux/x86_64
    restart: always
    volumes:
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
  ibg2:
    environment:
      INI_SETTINGS: --ExistingSessionDetectedAction=primary
    image: quantrocket/ibg:2.9.2
    networks:
      default: null
    platform: linux/x86_64
    restart: always
    volumes:
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
  ibgrouter:
    image: quantrocket/ibgrouter:2.9.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: codeload
        target: /codeload
        volume: {}
  jupyter:
    image: quantrocket/jupyter:2.9.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: codeload
        target: /codeload
        volume: {}
      - type: volume
        source: db
        target: /var/lib/quantrocket
        volume: {}
  license-service:
    image: quantrocket/license-service:2.9.1
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
  logspout:
    command:
      - syslog+udp://flightlog:9021,syslog://logs5.papertrailapp.com:47405?filter.name=*houston*
    depends_on:
      flightlog:
        condition: service_started
        required: true
      houston:
        condition: service_started
        required: true
    environment:
      BACKLOG: "false"
    image: quantrocket/logspout:2.9.0
    networks:
      default: null
    restart: always
    volumes:
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
        bind:
          create_host_path: true
  master:
    depends_on:
      db:
        condition: service_started
        required: true
    image: quantrocket/master:2.9.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: db
        target: /var/lib/quantrocket
        volume: {}
      - type: volume
        source: codeload
        target: /codeload
        volume: {}
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
  moonshot:
    image: quantrocket/moonshot:2.9.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: codeload
        target: /codeload
        volume: {}
  postgres:
    environment:
      PGDATA: /var/lib/quantrocket/postgresql12/data/pg_data
    image: quantrocket/postgres:2.9.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: db
        target: /var/lib/quantrocket
        volume: {}
  realtime:
    depends_on:
      db:
        condition: service_started
        required: true
    image: quantrocket/realtime:2.9.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: db
        target: /var/lib/quantrocket
        volume: {}
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
  satellite:
    depends_on:
      codeload:
        condition: service_started
        required: true
    image: quantrocket/satellite:2.9.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: codeload
        target: /codeload
        volume: {}
      - type: volume
        source: db
        target: /var/lib/quantrocket
        volume: {}
  zipline:
    image: quantrocket/zipline:2.9.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: codeload
        target: /codeload
        volume: {}
      - type: volume
        source: db
        target: /var/lib/quantrocket
        volume: {}
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
networks:
  default:
    name: quantrocket_default
volumes:
  codeload:
    name: quantrocket_codeload
  db:
    name: quantrocket_db
  flightlog:
    name: quantrocket_flightlog
  settings:
    name: quantrocket_settings
x-quantrocket-deploy-target: local
x-quantrocket-version: 2.9.2

When I attempt to "set_credentials" as follows I receive the below error:

from quantrocket.ibg import set_credentials
set_credentials("ibg2", username="dlehman1927", trading_mode="paper")

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[38], line 2
      1 from quantrocket.ibg import set_credentials
----> 2 set_credentials("ibg2", username="dlehman1927", trading_mode="paper")

File /opt/conda/lib/python3.11/site-packages/quantrocket/ibg.py:128, in set_credentials(gateway, username, password, trading_mode)
    126 statuses = list_gateway_statuses(gateways=[gateway])
    127 if not statuses:
--> 128     raise ValueError("no such IB Gateway: {0}".format(gateway))
    130 if username and not password:
    131     password = getpass.getpass(prompt="Enter IBKR Password: ")

ValueError: no such IB Gateway: ibg2

For what it's worth I have been able to successfully set my ibg1 credentials.

By the way, I don't know why the config is showing me as running Quantrocket Version 2.9, I upgraded to the latest (2.10.2) which is what shows in Jupyter Notebooks.

Guidance is appreciated.

Thanks!

#4

Sounds like you have multiple Compose files and are editing the wrong one.

#5

@Brian you were correct, I did have multiple Compose files. That said, I've eliminated the extras and I'm still receiving a value error indicating "no such IB Gateway: ibg2". A few comments / questions to provide more context:

  1. I deleted all compose files not in my Quantrocket folder. The only two that remain are the (1) "docker-compose.yml" file and (2) the "docker.compose.override.yml" file. I've made sure that the "docker-compose.yml" file is the latest version of Quantrocket 2.10.2.
  2. I ran "docker compose config" and it shows that both ibg1 and ibg2 are contained.
chris@Chriss-MBP-2 quantrocket % docker compose config
name: quantrocket
services:
  account:
    depends_on:
      db:
        condition: service_started
        required: true
    image: quantrocket/account:2.10.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: db
        target: /var/lib/quantrocket
        volume: {}
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
  blotter:
    depends_on:
      db:
        condition: service_started
        required: true
    image: quantrocket/blotter:2.10.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: db
        target: /var/lib/quantrocket
        volume: {}
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
  codeload:
    environment:
      GIT_BRANCH: "2.10"
      GIT_URL: https://github.com/quantrocket-codeload/quickstart.git
      OVERWRITE_EXISTING: "true"
    image: quantrocket/codeload:2.10.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: codeload
        target: /codeload
        volume: {}
  countdown:
    image: quantrocket/countdown:2.10.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
      - type: volume
        source: codeload
        target: /codeload
        volume: {}
  dash:
    image: quantrocket/dash:2.10.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: db
        target: /var/lib/quantrocket
        volume: {}
      - type: volume
        source: codeload
        target: /codeload
        volume: {}
  db:
    depends_on:
      postgres:
        condition: service_started
        required: true
    image: quantrocket/db:2.10.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: db
        target: /var/lib/quantrocket
        volume: {}
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
  flightlog:
    environment:
      SEND_CRASH_REPORTS: "true"
    image: quantrocket/flightlog:2.10.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: flightlog
        target: /var/log/flightlog
        volume: {}
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
  fundamental:
    depends_on:
      db:
        condition: service_started
        required: true
    image: quantrocket/fundamental:2.10.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: db
        target: /var/lib/quantrocket
        volume: {}
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
  history:
    depends_on:
      db:
        condition: service_started
        required: true
    image: quantrocket/history:2.10.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: db
        target: /var/lib/quantrocket
        volume: {}
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
  houston:
    environment:
      QUANTROCKET_VERSION: 2.10.2
    image: quantrocket/houston:2.10.0
    networks:
      default: null
    ports:
      - mode: ingress
        target: 80
        published: "1969"
        protocol: tcp
    restart: always
  ibg1:
    environment:
      INI_SETTINGS: --ExistingSessionDetectedAction=primary
    image: quantrocket/ibg:2.10.0
    networks:
      default: null
    platform: linux/x86_64
    restart: always
    volumes:
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
  ibg2:
    environment:
      INI_SETTINGS: --ExistingSessionDetectedAction=primary
    image: quantrocket/ibg:2.10.0
    networks:
      default: null
    platform: linux/x86_64
    restart: always
    volumes:
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
  ibgrouter:
    image: quantrocket/ibgrouter:2.10.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: codeload
        target: /codeload
        volume: {}
  jupyter:
    image: quantrocket/jupyter:2.10.1
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: codeload
        target: /codeload
        volume: {}
      - type: volume
        source: db
        target: /var/lib/quantrocket
        volume: {}
  license-service:
    image: quantrocket/license-service:2.10.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
  logspout:
    command:
      - syslog+udp://flightlog:9021,syslog://logs5.papertrailapp.com:47405?filter.name=*houston*
    depends_on:
      flightlog:
        condition: service_started
        required: true
      houston:
        condition: service_started
        required: true
    environment:
      BACKLOG: "false"
    image: quantrocket/logspout:2.10.0
    networks:
      default: null
    restart: always
    volumes:
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
        bind:
          create_host_path: true
  master:
    depends_on:
      db:
        condition: service_started
        required: true
    image: quantrocket/master:2.10.2
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: db
        target: /var/lib/quantrocket
        volume: {}
      - type: volume
        source: codeload
        target: /codeload
        volume: {}
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
  moonshot:
    image: quantrocket/moonshot:2.10.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: codeload
        target: /codeload
        volume: {}
  postgres:
    environment:
      PGDATA: /var/lib/quantrocket/postgresql12/data/pg_data
    image: quantrocket/postgres:2.10.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: db
        target: /var/lib/quantrocket
        volume: {}
  realtime:
    depends_on:
      db:
        condition: service_started
        required: true
    image: quantrocket/realtime:2.10.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: db
        target: /var/lib/quantrocket
        volume: {}
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
  satellite:
    depends_on:
      codeload:
        condition: service_started
        required: true
    image: quantrocket/satellite:2.10.0
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: codeload
        target: /codeload
        volume: {}
      - type: volume
        source: db
        target: /var/lib/quantrocket
        volume: {}
  zipline:
    image: quantrocket/zipline:2.10.1
    networks:
      default: null
    restart: always
    volumes:
      - type: volume
        source: codeload
        target: /codeload
        volume: {}
      - type: volume
        source: db
        target: /var/lib/quantrocket
        volume: {}
      - type: volume
        source: settings
        target: /etc/opt/quantrocket
        volume: {}
networks:
  default:
    name: quantrocket_default
volumes:
  codeload:
    name: quantrocket_codeload
  db:
    name: quantrocket_db
  flightlog:
    name: quantrocket_flightlog
  settings:
    name: quantrocket_settings
x-quantrocket-deploy-target: local
x-quantrocket-version: 2.10.2
chris@Chriss-MBP-2 quantrocket %
  1. Below is a the "docker-compose.override.yml" file contents in case this helps.
# docker-compose.override.yml
services:
  ibg2:
    extends:
      file: docker-compose.yml
      service: ibg1
  1. After running the config I redeployed:
docker compose -p quantrocket up -d master
  1. I then attempted to start the gateway:
from quantrocket.ibg import start_gateways
start_gateways(gateways=["ibg2"])
{'status': 'the gateways will be started asynchronously'}
  1. Then I attempt to set credentials but still receive the error:
from quantrocket.ibg import set_credentials
set_credentials("ibg2", username="dlehman1927", trading_mode="paper")
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[71], line 2
      1 from quantrocket.ibg import set_credentials
----> 2 set_credentials("ibg2", username="dlehman1927", trading_mode="paper")

File /opt/conda/lib/python3.11/site-packages/quantrocket/ibg.py:128, in set_credentials(gateway, username, password, trading_mode)
    126 statuses = list_gateway_statuses(gateways=[gateway])
    127 if not statuses:
--> 128     raise ValueError("no such IB Gateway: {0}".format(gateway))
    130 if username and not password:
    131     password = getpass.getpass(prompt="Enter IBKR Password: ")

ValueError: no such IB Gateway: ibg2

Thoughts?

Thanks!

#6

@Brian I've made progress. I put my user name and password for ibg2 directly into the "override" file and now ibg2 is being recognized. I'm still working through the connections inside the IBG GUI but I'm hoping I'm past the issue I outlined above. Thanks.