ARM legacy system alongside arm64 replacement

Table of Contents

General situation

For some reason or another, you might have a legacy system that should theoretically be replaced by a newer system, especially when changing hardware/architecture anyway -- but you want to keep it running all the same. Or even need to, as it might be infrastructure for other hosts (e.g., needed to build Debian packages for them), which a newer system couldn't accomplish by itself.

Luckily, we're living in the future already, and can use many kinds of different technology to let the old system live on in a new one -- be it virtual machines (VMs; accelerated by hardware support like Linux KVM, or plain qemu without acceleration enabled), Linux Containers (LXC; the standalone one or that one provided by libvirt …), systemd-nspawn (chroot on steroids; possibly used via systemd-machined and machinectl), or anything else that you can imagine.

More specific situation

Years ago, I put my previous, 32-bit home server into an LXC container on the new, 64-bit system. Except for that it's still running although I wanted to retire it one day, it's working great. This was on amd64 (aka x86_64, or today simply "PC", as in Personal Computer) hardware.

So now I wanted to do mostly the same thing with a Raspbian armhf (meant for running on Raspberry Pi) development environment (32-bit) on a "new" RPi 3B+, which got Debian (the real thing) buster (upcoming Debian 10) arm64 (64-bit) running on it. One thing, I was astonished that it could do such a thing, being ARM-based and not PC, and all. Another thing, it was not such a smooth ride as on PC hardware!

The deprecated ARM instructions

So I set up the outer system based on a preview image (created by Gunnar Wolf), and copied the previous development environment to /var/lib/machines/devenv-raspbian. cd to there, do a quick test:

root@devenv-arm64:/var/lib/machines/devenv-raspbian# chroot . bin/bash
root@devenv-arm64:/# ls
bin  boot  dev  etc  [...]
root@devenv-arm64:/#

Some things to note here:

  1. It works.

  2. It spams the dmesg/journal/syslog with messages such as:

     Apr 25 14:17:38 devenv-arm64 kernel: "bash" (2612) uses deprecated CP15 Barrier instruction at 0xf7e88b50
 Apr 25 14:17:38 devenv-arm64 kernel: "bash" (2612) uses deprecated CP15 Barrier instruction at 0xf7e88b50
 Apr 25 14:17:38 devenv-arm64 kernel: "bash" (2612) uses deprecated CP15 Barrier instruction at 0xf7e88b50
 [...]
 Apr 25 14:17:39 devenv-arm64 kernel: "bash" (2612) uses deprecated setend instruction at 0xf7e666f4
 Apr 25 14:17:39 devenv-arm64 kernel: "bash" (2612) uses deprecated setend instruction at 0xf7e66ca4
 [...]
 Apr 25 14:18:05 devenv-arm64 kernel: cp15barrier_handler: 143 callbacks suppressed
 Apr 25 14:18:05 devenv-arm64 kernel: "bash" (2612) uses deprecated CP15 Barrier instruction at 0xf7d032b8
 Apr 25 14:18:05 devenv-arm64 kernel: "bash" (2612) uses deprecated CP15 Barrier instruction at 0xf7d03394
 Apr 25 14:18:05 devenv-arm64 kernel: "bash" (2612) uses deprecated setend instruction at 0xf7e666f4
 Apr 25 14:18:05 devenv-arm64 kernel: "bash" (2612) uses deprecated setend instruction at 0xf7e66bd8
 [...]
 Apr 25 14:18:17 devenv-arm64 kernel: "ls" (2613) uses deprecated CP15 Barrier instruction at 0xf7cbfb50
 Apr 25 14:18:17 devenv-arm64 kernel: "ls" (2613) uses deprecated CP15 Barrier instruction at 0xf7cbfb50
 Apr 25 14:18:17 devenv-arm64 kernel: compat_setend_handler: 2 callbacks suppressed
 Apr 25 14:18:17 devenv-arm64 kernel: "bash" (2612) uses deprecated setend instruction at 0xf7e666f4
 Apr 25 14:18:17 devenv-arm64 kernel: "bash" (2612) uses deprecated setend instruction at 0xf7e66ca4
 [...]

Point 2 nearly overshadowed point 1, here. Luckily, it was all easily fixed by doing:

root@devenv-arm64:~# echo 2 >/proc/sys/abi/setend
root@devenv-arm64:~# echo 2 >/proc/sys/abi/cp15_barrier

Apr 25 15:03:19 devenv-arm64 kernel: Removed setend emulation handler
Apr 25 15:03:19 devenv-arm64 kernel: Enabled setend support
Apr 25 15:03:40 devenv-arm64 kernel: Removed cp15_barrier emulation handler
Apr 25 15:03:40 devenv-arm64 kernel: Enabled cp15_barrier support

As you may guess from the dmesg messages, this didn't just silence the warnings, but did something different: It actually turned off the in-kernel support for those deprecated ARM instructions, and moved responsibility for supporting the functionality to the bare hardware. Luckily, our Raspberry Pi 3B+ is backwards-compatible to running an ARM userland compiled for the more limited Raspberry Pi 1, and still has the deprecated instructions implemented even in aarch64 mode running a 32bit personality. But if we would have liked to run an arbitrary 32bit ARM userland on an arbitrary 64bit ARM CPU running in 64bit mode, this could have been a show-stopper, here.

Research

Some details of my research into this weird situation: Initial suggestion for using sysctls abi.* found, given by someone at linaro in 2017; reference to previous discussion from 2014 made by someone at Arm Ltd in the same thread in 2017; proposed timeline for ARM instruction deprecation by someone else at Arm Ltd in 2014. There, it says how it's all meant to work together for the goal of finally getting rid of certain instructions. (?)

As a side note, those emulation warnings are coming up for other people doing slightly similar things (though in a much more professional way) as well; e.g., directhex wrangling with automated build infrastructure. In section "When is a superset not a superset", it says:

CP15 memory barrier emulation is slow. My friend Vince Sanders, who helped with some of this analysis, suggested a cost of order 1000 cycles per emulated call. How many was I looking at? According to dmesg, about a million per second.

I guess that number has been taken by looking at the result of the emulation warning dmesg rate-limiting. Let's hope it wasn't millions of log lines, instead …

Running mono and the syscall filter

Having just a chroot would be lame, nowadays; as you can easily boot another operating system (as LXC or systemd-nspawn or even docker container, as long as it's a Linux distribution and/or copes with using the same kernel as the host system; or, otherwise, in a virtual machine, e.g. based on qemu and kvm). For this situation, where I thought the guest system would cope with the newer, foreign-architecture kernel of the host system with the deprecated ARM instructions put out of our way, I was going to use systemd-nspawn container.

So for our previous /var/lib/machines/devenv-raspbian, I set up a companion configuration file /etc/systemd/nspawn/devenv-raspbian.nspawn (for unsetting some systemd-nspawn defaults when used from systemd-machined, and to set up networking), which apparently already got used when I cd'd to the machine(/container) root directory and issued a simple:

root@devenv-arm64:/var/lib/machines/devenv-raspbian# systemd-nspawn
root@devenv-raspbian:~#

Again, some things to note:

  1. It works.
  2. This time, we're not inside a simple chroot, but in a container already, with the uts namespace unshared and the hostname set according to the container name. (No tricks with /etc/debian_chroot and a PS1 which uses it needed!)
  3. It even picked up our inner root's home directory, and ended up there instead of in the container root directory.

With this running I tried the next step of booting into the container operating system. For the most part, it was a simple:

root@devenv-arm64:~# machinectl start devenv-raspbian

This gave the guest's boot messages in the host's journal. When it's ready, a simple machinectl without arguments, from an unprivileged user, says what OS/version the container OS is, and what IP address is being used -- an information which is readily available as it's a container, not a VM.

When all works out well, you can enable the container for start on next (host) boot:

root@devenv-arm64:~# machinectl enable devenv-raspbian

So far, so good.

The problem

After some time I realized everything mono (.NET) was instantly failing at program start. Initial thoughts were on maybe another missing instruction, at the hardware/kernel level; that wasn't the case, though. So I ran the thing through strace.. and noticed loads of cacheflush() failing with EPERM. After some fruitless web search into the Linux source, it appeared to me as if this could be part of some restriction placed upon us by the systemd-nspawn containering. -- A quick test running the csharp interactive C# shell just inside a chroot instead of a systemd-nspawn container succeeded: (Transcript from brain memory.)

root@devenv-arm64:/var/lib/machines/devenv-raspbian# chroot . bin/bash
root@devenv-arm64:/# csharp
[some message about mono needing /proc]
root@devenv-arm64:/# mount -t proc proc proc
root@devenv-arm64:/# csharp
[some assertion not met, seems to explode much as before]
root@devenv-arm64:/# linux32 csharp
[works]
root@devenv-arm64:/# umount proc  # Don't forget; or systemd-nspawn
                                  # later will error out when started
                                  # from "machinectl start devenv-raspbian".

So it was able to work, just wouldn't with systemd-nspawn.

Finally, I found out about systemd-nspawn having an automatic system-call-filter implemented as a white-list. So this was where the cacheflush() EPERM was coming from! Additionally to forcing the personality to "arm" (32-bit) in the .nspawn file mentioned above, we need to put cacheflush into the syscall-filter white-list. To avoid trial-and-error until the minimum necessary would be found, and as our containerization was for the old and new system to coexist -- and not to de-root the system being used as container --, I assumed that cacheflush was missing from the white-list due to the fact that it's being ARM-private, and simply put every ARM-private syscall there that I could find in an armhf header file. (Most probably what I was looking at was asm/unistd.h from the Linux kernel sources.) Specifically, that were: breakpoint cacheflush usr26 usr32 set_tls. (TODO: Report as issue / wishlist bug on systemd and link that here?)

Power down the container, start it again (remember to umount manually mounted proc first), et voila: mono runtime-based software from the old development environment was running, too!

Conclusion

It is possible to run an older Raspberry Pi compatible OS as a container inside a newer such OS on a newer hardware version of the Raspberry Pi. It's just unfortunate to seem impossible in practice at first; with both the deprecated ARM instructions filling up the journal, and mono runtime-based software of the older OS not running looking like a show-stopper. These problems can be overcome, though, which I hope to be able to communicate with this blog post.