UEFI on AARCH64

In the previous post we covered how an EFI application can load another binary in memory. That’s basically what a bootloader does. In this short post I will show that basically the same code will work on a different architecture.

As usual all the sources are available on GitHub.

The Board

UEFI Specification supports multiple different architectures, including aarch64. aarch64 is basically a 64-bit version of ARM architecture.

It appears that in the ARM ecosystem there is quite a variety of differnt turn-up methods and approaches, so there is no guarantees that a particular board firmware implements UEFI Specification.

I happened to have a board that does have a version of firmware implementing the UEFI Specification - HiKey960. Given that I have this board, I though that it might be curious to see if the same code I have will work for aarch64.

I will not cover the board in any depth here. That being said, I had this board for quite a while, so I wasn’t sure if it’s still working of if it has the reasonably new version of the firmware.

Here are a few links that helped me to update the firmware and bring up the board:

1. board recovery instructions are available here
2. instructions to setup ser2net to connect to the device console are availbe here
3. instructions to install Debian on the board are here.

Those instructions overlap significantly, but somehow none of them is complete. From now on I assume that the board has the latest UEFI firmware and by default starts GRUB and boots into Debian when you switch the board on.

UEFI Shell

HiKey960 has internal storage where all the firmware, Debian image and root filesystem live. As normal for UEFI firmware on this internal storage we also have a FAT-formatted partition.

You can explore this FAT partition if you boot the board into Debian:

root@linaro-developer:~# cat /etc/fstab | grep /boot/efi
UUID=6D97-E8F4	/boot/efi	vfat	rw,nofail	0	2

root@linaro-developer:~# ls -l /boot/efi/EFI/BOOT
total 816
-rwxr-xr-x 1 root root 120832 Jul 20  2020 BOOTAA64.EFI
-rwxr-xr-x 1 root root 702464 Jul 20  2020 GRUBAA64.EFI

As you can see one of the binaries in the partition is the GRUB itself. I suspect that it’s a regular EFI binary just like the one I built, though with quite a bit more functionality.

If you want to prevent the board from booting into GRUB right away by default you can just rename those two binaries:

root@linaro-developer:~# mv /boot/efi/EFI/BOOT/GRUBAA64.EFI /boot/efi/EFI/BOOT/GRUBAA64.EFI.old
root@linaro-developer:~# mv /boot/efi/EFI/BOOT/BOOTAA64.EFI /boot/efi/EFI/BOOT/BOOTAA64.EFI.old

If you do that the firmware would not find the expected GRUB binary to load and will fail to load the system. In my case when it happens the I see the following prompt:

Press RETURN or SPACE key to quit.

When I press ENTER the board shows some kind of a BIOS-like firmware UI. In that UI you can select the Boot Manager item in the menu and then UEFI Shell to start the shell.

In the EFI getting started post I covered how to load and exectue a binary from UEFI Shell. There is one caveat though, the binaries were not built yet for the aarch64 and we didn’t store them into the internal storage of the board.

Building for ARM

With GCC toolchain to cross-compile a binary for a different architecture we’d need to download or build a toolchain for the target platform. LLVM-based toolchain might actually have a built-in support for multiple architectures, so all it takes to cross-compile is to specify the target platform.

Here is the complete Makefile content that works for me:

CC := clang
LD := lld
CFLAGS := \
	-ffreestanding -MMD -mno-red-zone -std=c11 \
	-target aarch64-unknown-windows -Wall -Werror -pedantic
LDFLAGS := -flavor link -subsystem:efi_application -entry:efi_main

KERNEL_CFLAGS := \
	-ffreestanding -MMD -mno-red-zone -std=c11 \
	-target aarch64-unknown-gnu -Wall -Werror -pedantic
KERNEL_LDFLAGS := \
	-flavor ld -e main

SRCS := main.c clib.c kernel.c

default: all

%.o: %.c
	$(CC) $(CFLAGS) -c $< -o $@

boot.efi: clib.o main.o
	$(LD) $(LDFLAGS) $^ -out:$@

kernel.elf: kernel.c
	$(CC) $(KERNEL_CFLAGS) -c $< -o kernel.o
	$(LD) $(KERNEL_LDFLAGS) kernel.o -o $@

-include $(SRCS:.c=.d)

.PHONY: clean all default

all: boot.efi kernel.elf

clean:
	rm -rf *.efi *.elf *.o *.d *.lib

You might notice that this Makefile is quite similar to the one in EFI getting started post. The only important difference is the -target parameter.

The -target flag for the EFI binary now contains aarch64-unknown-windows. And the -target flag for the ELF binary now contains aarch64-unknown-gnu. So all it took in this case is it to tell clang the target architecture. In general the flags between different architectures might not really match.

For example, I was surprised to find out that -mno-red-zone option works for aarch64 architecture. Even though red zone describes a relatively generic optimization that makes sense for different architectures, it was surprising to find out that a similar optimization for aarch64 is also named red zone.

NOTE: the version of the Makefile in the repository is slightly more generic and supports building for both x86-64 and aarch64, so it’s different from the code in the post.

If we try to build the code now it will fail in the following way:

clang -ffreestanding -MMD -mno-red-zone -std=c11 -target aarch64-unknown-windows -Wall -Werror -pedantic -c main.c -o main.o
main.c:457:22: error: 'sysv_abi' calling convention is not supported for this target [-Werror,-Wignored-attributes]
        int (__attribute__((sysv_abi)) *entry)(void);
                            ^
main.c:465:31: error: 'sysv_abi' calling convention is not supported for this target [-Werror,-Wignored-attributes]
        entry = (int (__attribute__((sysv_abi)) *)(void))elf->image_entry;
                                     ^
2 errors generated.
make: *** [Makefile:18: main.o] Error 1

In the main.c when I’m calling into the code just loaded from an ELF file I have to specify a calling convention explicitly. That’s because on x86-64 UEFI calling convention doesn’t match the calling convention used by the code in the ELF binary.

However, when compiling for aarch64 the sysv_abi attribute is not supported. Fortunately enough it appears that calling conventions for aarch64 do not diverge as much, so we don’t really need to explicitly tell compiler to use a different calling convention and that part of the code might be deleted.

The way I handled it in the repository is that I introduce a file with compiler specific magic, macroses, etc. I called this file compiler.h and defined there ELFABI macro in the following way:

#ifndef __COMPILER_H__
#define __COMPILER_H__

#ifdef __x86_64__
#define ELFABI __attribute__((sysv_abi))
#else
#define ELFABI
#endif

#endif  // __COMPILER_H__

So now where I need to explicitly specify the calling convention I can just use ELFABI macro. After this change I managed to get the two binaries:

• boot.efi - the EFI binary
• kernel.efl - the ELF binary.

Copying to the Board

There are a few ways to make the newly built binaries accesible on the board. One simple way is to use an micro SD card.

You can just rename GRUBAA64.EFI.old and BOOTAA64.EFI.old to their original names (GRUBAA64.EFI and BOOTAA64.EFI) using UEFI Shell command mv and boot into Debian. Alternatively, you can just load GRUBAA64.EFI.old directly from the UEFI Shell without renaming.

In Debian you can insert the micro SD card into the slot on the board and mount it. Depending on how the micro SD card was formatted the commands to do that might look something like this:

root@linaro-developer:~# cd /mnt/
root@linaro-developer:/mnt# mkdir boot
root@linaro-developer:/mnt# mount -t vfat /dev/mmcblk0p1 /mnt/boot

After that just copy the files from /mnt/boot to /boot/efi/EFI/BOOT where the rest of the EFI binaries live:

root@linaro-developer:/mnt# cp boot/boot.efi /boot/efi/EFI/BOOT/boot.efi
root@linaro-developer:/mnt# cp boot/kernel.elf /boot/efi/EFI/BOOT/kernel

NOTE: it doesn’t matter how we call the EFI binary since we explicitly call it by name in the UEFI Shell, but the path to the ELF binary is hardcoded in the EFI binary, so it must live in efi/boot/kernel.

UEFI firmware actually also recognizes micro SD card if it’s in the slot. So instead of booting into Debian you can just copy the files using the UEFI Shell.

Now when all the files are in place refer to the EFI getting started post for actually running the binary.

Bonus: testing in QEMU

If you don’t have an aarch64 board (or even if you do, but working with the actual hardware is too bothersome) you can use QEMU. First you need to download aarch64 EFI firmware for QEMU:

sudo apt-get install qemu-efi-aarch64

This package contains the aarch64 EFI image for QEMU. You can copy it into the working directory:

cd ~/ws/efi
cp /usr/share/qemu-efi-aarch64/QEMU_EFI.fd OVMF_aarch64.fd

You can use this file in the same way as described in the EFI getting started post, but with the QEMU emulating aarch64:

qemu-system-aarch64 -machine virt -cpu max \
	-drive if=pflash,format=raw,file=/home/kmu/ws/efi/OVMF_aarch64.fd \
	-drive format=raw,file=fat:rw:/home/kmu/ws/efi/root \
	-net none \
	-nographic

NOTE: /home/kmu/ws/efi is the working directory on my computor, the path might be different on yours.

NOTE: as in EFI getting started root is the root of the EFI file system, so all the files we want to be available in the virtual machine should be copied there.

NOTE: for some reason I couldn’t make it work without explicitly specifing -cpu option.

Instead of conclusion

Rather suprisingly making it work on ARM turned out to be almost problem-free.