Build for hardware you do not have

You have a shell on a target. It is an old Debian, or an ARM appliance, and it has no compiler, no gcc, no make, and no internet. You have a code that needs to run on it. Historically this meant install cross compilers and messing with toolchains.

This is the lesson where Docker starts getting useful. We will mount a directory into a container, build inside it, and pick the binary back up on the host.

Getting files in

A bind mount maps a directory on your host to a path inside the container. -v host:container, and -w sets the working directory so you land in it:

$ ls
main.c

$ docker run --rm -it -v "$PWD:/w" -w /w ubuntu:22.04 bash
root@ce0e2c61b5ec:/w# ls -la
total 12
drwxrwxr-x 2 1000 1000 4096 Aug 18 02:42 .
drwxr-xr-x 1 root root 4096 Aug 18 02:42 ..
-rw-rw-r-- 1 1000 1000  117 Aug 18 02:42 main.c

That is not a copy. It is the same inode. Edit main.c in your editor on the host, and the container sees the change immediately, which means your normal tooling stays on the host and only the build happens inside.

Quote $PWD. A space in a path turns an unquoted mount into an argument parsing error at best, and a mount of the wrong directory at worst.

Note the ownership. The container has no idea who user 1000 is, because your /etc/passwd is not in there. It shows the raw UID.

Building

Let’s look at a basic hello world.

#include <stdio.h>

int main(void) {
    printf("hello from a machine that does not have a compiler\n");
    return 0;
}

Build it in a container that has a toolchain:

$ docker run --rm -v "$PWD:/w" -w /w ubuntu:22.04 bash -c \
    'apt-get update -qq && apt-get install -y -qq build-essential file && gcc -static -o hello main.c && file hello'
hello: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=3b13c55c600a503d13471b5f6d7a29285866c015, for GNU/Linux 3.2.0, not stripped

-static is the important flag. Dynamically linked, that binary needs the target’s glibc to be new enough, and “version GLIBC_2.34 not found” is how you learn that it is not. Statically linked, it carries everything it needs. It gets bigger, 900KB here for a printf, and on a target with a strict egress budget that trade is usually still worth it.

The binary is on your host, because the directory was never a copy:

$ file hello
hello: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=3b13c55c600a503d13471b5f6d7a29285866c015, for GNU/Linux 3.2.0, not stripped

The root-owned file trap

Look at what the build left behind:

$ ls -la
total 892
-rwxr-xr-x 1 root root 900344 Aug 18 02:42 hello
-rw-rw-r-- 1 evan evan    117 Aug 18 02:42 main.c

The container ran as root, so the file it wrote is owned by root on your host.

If you want to run as yourself instead:

$ docker run --rm --user "$(id -u):$(id -g)" -v "$PWD:/w" -w /w builder gcc -static -o hello main.c
$ ls -la hello
-rwxr-xr-x 1 evan evan 900344 Aug 18 02:52 hello

One distro specific note for mounts: if Docker came from a snap, the daemon can only see non-hidden paths under your home directory. Bind mounting /tmp/work or ~/.cache/work silently gives you an empty directory instead of an error. Keep lab directories in ~/something.

Cross compiling

That binary is x86-64. The appliance you are building for is ARM, and you do not own an ARM machine. Docker can build for it anyway, once the host knows how to run foreign-architecture binaries.

Two Ubuntu packages set that up:

$ sudo apt-get install -y qemu-user-static binfmt-support

qemu-user-static is a set of statically-linked CPU emulators, one per architecture. binfmt-support wires them into binfmt_misc, a kernel feature that picks an interpreter based on a binary’s header. Installing the first registers handlers through the second, so there is nothing to configure by hand.

Confirm the aarch64 handler is live. It shows up as a file in /proc/sys/fs/binfmt_misc/:

$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /usr/libexec/qemu-binfmt/aarch64-binfmt-P
flags: POF
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

enabled, with interpreter pointing at the QEMU emulator, is all you need to see. Now the kernel runs an aarch64 binary by handing it to QEMU, and Docker gets that for free. Ask for an ARM container and it works:

$ docker run --rm --platform linux/arm64 ubuntu:22.04 uname -m
aarch64

An ARM userland, running on your x86-64 machine. --platform tells Docker which architecture’s image to pull, and the binfmt handler takes care of actually running it. The registration lives in the running kernel, so it holds until you reboot, and systemd-binfmt re-applies it on boot. Docker Desktop ships this already configured.

Now the same static build as before, aimed at ARM. The only change is --platform linux/arm64:

$ time docker run --rm --platform linux/arm64 -v "$PWD:/w" -w /w ubuntu:22.04 bash -c \
    'apt-get update -qq && apt-get install -y -qq build-essential && gcc -static -o hello-arm64 main.c'

real	0m56.978s

$ file hello-arm64
hello-arm64: ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=8317f5484be88fa26166c99fa527925982ae4cc0, for GNU/Linux 3.7.0, not stripped

An aarch64 static binary, built on a machine with no ARM in it. The same binfmt handler that let Docker run the ARM container will run the binary you just built, straight from your host shell:

$ ./hello-arm64
hello from a machine that does not have a compiler

That is QEMU emulating an entire ARM binary transparently, which is convenient for a quick smoke test and, as the next section covers, the reason the build was slow.

The catch, and the faster road

Every instruction in that build ran under emulation, including all of apt and all of GCC. Fifty seven seconds for a five line program. On a real codebase that becomes minutes, and QEMU user mode has rough edges: threading and some syscalls behave differently, and the occasional configure script hangs.

The alternative is a real cross toolchain, which runs natively and just emits foreign code. dockcross packages one per target. The image prints a wrapper script, and the wrapper handles the mount, the UID mapping, and the toolchain environment:

$ docker run --rm dockcross/linux-arm64 > ./dockcross-linux-arm64
$ chmod +x ./dockcross-linux-arm64
$ time ./dockcross-linux-arm64 bash -c '$CC -static -o hello-dockcross main.c'

real	0m1.235s

$ file hello-dockcross
hello-dockcross: ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=d68d44a80dff26ed0f90a59970bd4aa27cefc6c2, for GNU/Linux 6.1.35, with debug_info, not stripped

Same architecture, 46 times faster, and no emulator involved. dockcross has images for armv5 through arm64, MIPS both endians, PowerPC, s390x, RISC-V, plus Windows and Android targets, which covers most of what you meet in embedded work.

Use emulation when you need the target’s own userland and package manager, because linking against the target distro’s libraries is exactly what it gives you. Use dockcross when you need speed and are building something self contained.

Which architectures can you even ask for

--platform only helps if someone published an image for that platform. The official images cover amd64, arm64, armv6, armv7, i386, ppc64le, s390x, mips64le and riscv64, though coverage per image varies. Check before you plan around it:

$ docker buildx imagetools inspect ubuntu:22.04 | grep Platform
  Platform:    linux/amd64
  Platform:    unknown/unknown
  Platform:    linux/arm/v7
  Platform:    unknown/unknown
  Platform:    linux/arm64/v8
  Platform:    unknown/unknown
  Platform:    linux/ppc64le
  Platform:    unknown/unknown
  Platform:    linux/riscv64
  Platform:    unknown/unknown
  Platform:    linux/s390x
  Platform:    unknown/unknown

The unknown/unknown entries are attestation manifests, build provenance riding along in the same index. Ignore them, the real platform list is the other six.

Next we stop retyping apt-get install on every run and bake our own image.