Illegal instruction error on houston on Apple M1 chip

sebastian · April 30, 2021, 5:47pm

After following the docker and quantrocket installations instructions on an Apple M1 machine, we're seeing houston restart indefinitely with an Illegal instruction error in the logs like this:

e[36mhouston_1          |e[0m /opt/quantrocket/bin/houston.sh: line 15:    18 Illegal instruction     /usr/local/openresty/bin/openresty -g "daemon off;"
e[36mhouston_1          |e[0m no BASIC_AUTH_USER found so not creating .htpasswd file
e[36mhouston_1          |e[0m LETSENCRYPT_DOMAIN not set so not installing SSL
e[36mhouston_1          |e[0m qemu: uncaught target signal 4 (Illegal instruction) - core dumped
e[36mhouston_1          |e[0m /opt/quantrocket/bin/houston.sh: line 15:    18 Illegal instruction     /usr/local/openresty/bin/openresty -g "daemon off;"
e[36mhouston_1          |e[0m no BASIC_AUTH_USER found so not creating .htpasswd file
e[36mhouston_1          |e[0m LETSENCRYPT_DOMAIN not set so not installing SSL
e[36mhouston_1          |e[0m qemu: uncaught target signal 4 (Illegal instruction) - core dumped
e[36mhouston_1          |e[0m /opt/quantrocket/bin/houston.sh: line 15:    17 Illegal instruction     /usr/local/openresty/bin/openresty -g "daemon off;"
e[36mhouston_1          |e[0m no BASIC_AUTH_USER found so not creating .htpasswd file
e[36mhouston_1          |e[0m LETSENCRYPT_DOMAIN not set so not installing SSL
e[36mhouston_1          |e[0m qemu: uncaught target signal 4 (Illegal instruction) - core dumped
e[36mhouston_1          |e[0m /opt/quantrocket/bin/houston.sh: line 15:    17 Illegal instruction     /usr/local/openresty/bin/openresty -g "daemon off;"

We've tried forcing linux/amd64 as a platform without success.

There's an issue in docker for mac which shows a possible workaround:

github.com/docker/for-mac

for M1 Macs: the qemu-x86_64 emulates an ancient CPU model that doesn't support modern features, breaks Swift compiler / LLVM / ...

opened 10:19PM - 10 Apr 21 UTC

closed 08:19AM - 07 May 21 UTC

weissi

lifecycle/locked area/m1

- [x] I have tried with the latest version of Docker Desktop - [x] I have tri…ed disabling enabled experimental features - [ ] I have uploaded Diagnostics ### Expected behavior I can run modern software that uses Intel instructions like `pshufb` in Docker for Mac (on M1 Macs) because the qemu-x86_64 that's installed using the `binfmt_misc` support can emulate it. ### Actual behavior I can't run modern software that makes use of newer Intel instructions because the qemu-x86_64 that gets run using `binfmt_misc` doesn't get passed a `-cpu core2duo` (or newer such as `-cpu Skylake-Client`/`-cpu max`) argument. ### Information The problem is that the `qemu-x86_64` doesn't get passed a `-cpu core2duo` (or even better something like `-cpu Skylake-Client` or maybe `-cpu max` ("Enables all features supported by the accelerator in the current host")) argument. Really, I would recommend to just pass the newest CPU that qemu can emulate (probably `-cpu max`). I got the Swift compiler to work just fine by compiling my own qemu with this patch applied: ``` sed -i 's/cpu_model = NULL/cpu_model = "core2duo"/g' linux-user/main.c ``` which just sets the default CPU model to `core2duo`. I then installed as the Docker for Mac default `binfmt_misc` handler for `x86_64` using ``` /containers/services/binfmt/rootfs/usr/bin/binfmt -uninstall qemu-x86_64 echo ":qemu-x86_64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x3e\x00:\xff\xff\xff\xff\xff\xfe\xfe\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/var/lib/docker/overlay2/80s11o66od7msjkcmt4hal5ui/diff/amd64/usr/bin/qemu-x86_64-static-c2d:OCF" > /proc/sys/fs/binfmt_misc/register ``` in the actual VM which I entered using [`justincormack/nsenter`](https://github.com/justincormack/nsenter1). ### Steps to reproduce the behavior The quickest way to reproduce the problem is: 1. Get an M1 Mac and install Docker for Mac for M1 Macs 2. Run `docker run -it --rm --platform linux/amd64 swift:5.3 bash -c 'echo "print(\"hello world\")" > /tmp/test.swift && cd /tmp && swiftc test.swift && ./test'` Expected: prints "hello world" Actual: Output something similar to ``` Please submit a bug report (https://swift.org/contributing/#reporting-bugs) and include the project and the crash backtrace. Stack dump: 0. Program arguments: /swift-DEVELOPMENT-SNAPSHOT-2021-02-09-a-ubuntu18.04/usr/bin/swift-frontend -frontend -c -primary-file /tmp//test.swift -target x86_64-unknown-linux-gnu -disable-objc-interop -color-diagnostics -module-name test -o /tmp/test-d8e021.o 1. Swift version 5.4-dev (LLVM 1e4181b99f530d2, Swift 842cc9c88f5463b) #0 0x0000000005cb5c13 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/swift-DEVELOPMENT-SNAPSHOT-2021-02-09-a-ubuntu18.04/usr/bin/swift-frontend+0x5cb5c13) #1 0x0000000005cb396e llvm::sys::RunSignalHandlers() (/swift-DEVELOPMENT-SNAPSHOT-2021-02-09-a-ubuntu18.04/usr/bin/swift-frontend+0x5cb396e) #2 0x0000000005cb5f9c SignalHandler(int) (/swift-DEVELOPMENT-SNAPSHOT-2021-02-09-a-ubuntu18.04/usr/bin/swift-frontend+0x5cb5f9c) #3 0x0000004000a48980 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x12980) #4 0x00000000018a7f63 std::vector<swift::DiagnosticState::Behavior, std::allocator<swift::DiagnosticState::Behavior> >::_M_fill_insert(__gnu_cxx::__normal_iterator<swift::DiagnosticState::Behavior*, std::vector<swift::DiagnosticState::Behavior, std::allocator<swift::DiagnosticState::Behavior> > >, unsigned long, swift::DiagnosticState::Behavior const&) (/swift-DEVELOPMENT-SNAPSHOT-2021-02-09-a-ubuntu18.04/usr/bin/swift-frontend+0x18a7f63) #5 0x00000000018a237f swift::DiagnosticState::DiagnosticState() (/swift-DEVELOPMENT-SNAPSHOT-2021-02-09-a-ubuntu18.04/usr/bin/swift-frontend+0x18a237f) #6 0x00000000006ae240 swift::CompilerInstance::CompilerInstance() (/swift-DEVELOPMENT-SNAPSHOT-2021-02-09-a-ubuntu18.04/usr/bin/swift-frontend+0x6ae240) #7 0x0000000000513165 swift::performFrontend(llvm::ArrayRef<char const*>, char const*, void*, swift::FrontendObserver*) (/swift-DEVELOPMENT-SNAPSHOT-2021-02-09-a-ubuntu18.04/usr/bin/swift-frontend+0x513165) #8 0x00000000004ad906 main (/swift-DEVELOPMENT-SNAPSHOT-2021-02-09-a-ubuntu18.04/usr/bin/swift-frontend+0x4ad906) #9 0x0000004002013bf7 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21bf7) #10 0x00000000004ad57a _start (/swift-DEVELOPMENT-SNAPSHOT-2021-02-09-a-ubuntu18.04/usr/bin/swift-frontend+0x4ad57a) qemu: uncaught target signal 4 (Illegal instruction) - core dumped <unknown>:0: error: unable to execute command: Illegal instruction <unknown>:0: error: compile command failed due to signal 4 (use -v to see invocation) ``` Please note the `qemu: uncaught target signal 4 (Illegal instruction) - core dumped` which qemu prints when it encounters the `pshufb` Intel instruction (which needs the Intel SSE3 feature which qemu (correctly) only emulates if you tell it to emulate a new-enough CPU using `-cpu core2duo` or better). The problem here is that the Swift compiler runs a `pshufb` instruction which isn't available in the ancient CPU that qemu emulates if you don't pass `-cpu core2duo` or newer. See [Swift compiler bug SR-14186](https://bugs.swift.org/browse/SR-14186) for all the details. For a somewhat more complete test, it'd be probably good to run ``` docker run -it --rm --platform linux/amd64 swift:5.3 bash -c 'git clone https://github.com/apple/swift-nio.git && cd swift-nio && swift test' ``` which clones [SwiftNIO](https://github.com/apple/swift-nio), compiles it and runs its test suite.

Brian · April 30, 2021, 7:48pm

Does it work if you set that environment variable in your docker-compose.override.yml?

version: 2.4
services
  houston:
    environment:
      QEMU_CPU: max

sebastian · April 30, 2021, 8:03pm

I tried setting houston in the quantrocket provided docker-compose.yml this way:

  houston:
    image: "quantrocket/houston:2.5.0"
    platform: "linux/amd64"
    ports:
      - "1969:80"
    environment:
      QUANTROCKET_VERSION: "2.5.0"
      QEMU_CPU: "max"
    restart: always

But that didn't help.

(btw, the mac install documentation shows jupyterlab running in port 1969, which confused a couple workmates).

sebastian · May 3, 2021, 8:18am

@Brian any updates on this?

Brian · May 3, 2021, 12:34pm

Docker has had many issues with M1 and only recently went GA. Now that it has, we'll try to get hold of an M1 and see if QuantRocket needs to do something. I assume you followed the special Docker instructions for M1?

In the meantime the clear workaround is to install on a different workstation or in the cloud.

Brian · May 3, 2021, 2:04pm

Can you try restoring your docker-compose.yml to its original state and then replacing

image: 'quantrocket:houston:2.5.0'

with

image: 'quantrocket/houston:multiarch20210503'

and then running

docker-compose up -d houston

This is a multi-architecture build of houston which should support ARM. But it has not been tested because AWS doesn’t yet have M1 instances and Scaleway, the only cloud vendor that does, is sold out.

Even if this fixes houston, I can’t promise there won’t be issues with other containers besides houston, but this would be a good place to start.

sebastian · May 3, 2021, 3:21pm

Thanks @Brian, we tried that out and found this new error in the logs:

nginx: [error] failed to initialize Lua VM in /usr/local/openresty/nginx/conf/nginx.conf:72

Brian · May 3, 2021, 8:02pm

Unfortunately then this will have to be on hold until we can gain access to an M1.

sebastian · May 3, 2021, 8:31pm

sounds good! please keep us posted of the progress...

Brian · May 4, 2021, 4:18pm

One other thing to try pending M1 access…

The 2.5.0 houston is on an somewhat old version of openresty/nginx, and it’s possible a newer version would help (houston is basically just an openresty server). With a clean docker-compose.yml, please try replacing

quantrocket/houston:2.5.0

with

quantrocket/houston:latest

which has the latest version of openresty, and see if that will run on M1.

sebastian · May 4, 2021, 5:09pm

That seems to make it!

Seeing something weird with port 80 not responding but 1969 in mac but it's probably a different issue. We get jupyter lab on the 1969 port correctly... Will test some API calls for a while and report back if we find issues.

Brian · May 5, 2021, 3:31pm

Please be aware that houston:latest with the updated version of OpenResty/Nginx breaks the zipline container's ability to log to flightlog. You are welcome to keep using it but it will not be ready for general release until that is resolved.

sebastian · May 5, 2021, 7:32pm

Thanks for the pointer! We'll keep an eye on this.

Brian · June 21, 2021, 8:48pm

M1 support is still in a holding pattern as we wait for updates to some of the build pipelines QuantRocket depends on. Specifically, although Docker Desktop now supports running ARM-based images, Docker Hub doesn't yet support building them.

brent.simpson · February 22, 2022, 5:02pm

Thank you, Brian. Is there any recent M1 update or workaround?

Brian · February 23, 2022, 3:05pm

Unfortunately M1 is still not supported because certain necessary packages for building uWSGI, the web server QuantRocket uses, are not yet available for arm64. But getting this resolved is still a priority. (I own an Apple M1 myself.) For now it's necessary to deploy QuantRocket on Apple Intel, Windows, Linux, or in the cloud.