ARMs PL011 UART on HiKey960 board

In the previous post we managed to make PL011 UART controller as emulated by QEMU work. Emulation is a useful tool, but it’s just never going to be perfect. So naturally I wanted to try it on the real hardware and used HiKey960 board that I have and that happen to have a PL011 compatible UART controller.

The sources for this post are available on GitHub.

Finding the parameters

In the previous post we covered that in simple case we basically need to know two parameters:

• base memory address where the memory mapped registers of PL011 controller are available
• frequency of the base clock given to the PL011 controller.

For QEMU we found those parameters by looking at the Device Tree dump, so it makes sense to do the same for the HiKey960 board.

Device Tree for HiKey960 board is available as part of the Linux Kernel source code (see arch/arm/arm64/boot/dts/hisilicon/hi3660-hikey960.dts). However Device Tree for the real board is slightly more complicated one than the one we saw for the QEMU emulated board, so it’s hard to find what we want there. One specific complication is that it has 7 different UARTs there, so it’s not clear which one we need.

Fortunately figuring out that part turned out to be easy. The UART section of the HiKey960 Hardware User Manual explicitly says that the UART that goes under number 6 is the one we need.

Now in the arch/arm/arm64/boot/dts/hisilicon/hi3660-hikey960.dts there isn’t much information about the UART 6, however we can find more in one of the included files: arch/arm64/boot/dts/hisilicon/hi3660.dtsi. Specifically, you can find that the base address we need is 0x0xfff32000:

uart6: serial@fff32000 {
	compatible = "arm,pl011", "arm,primecell";
	reg = <0x0 0xfff32000 0x0 0x1000>;
	interrupts = <GIC_SPI 79 IRQ_TYPE_LEVEL_HIGH>;
	clocks = <&crg_ctrl HI3660_CLK_UART6>,
		 <&crg_ctrl HI3660_PCLK>;
	clock-names = "uartclk", "apb_pclk";
	pinctrl-names = "default";
	pinctrl-0 = <&uart6_pmx_func &uart6_cfg_func>;
	status = "disabled";
};

What I couldn’t find from just looking at the Device Tree is the frequency of the clock and had to google a little bit for HI3660_CLK_UART6. I was a bit afraid that the clock frequency is also configurable and is not specified anywhere explicitly. Fortunately, it turned out not to be the case and the frequency is fixed to 19200000.

NOTE: I didn’t honestly look at the code to understand if the random number I found (19200000) is the actual frequency I need, but the experiment showed that it appear to be the case.

Problems

Here I though that I had everything figured out. After all how badly can I misconfigure a UART controller? Succesful test in QEMU also gave me a boost of confidence.

So I went and changed the code:

#include "pl011.h"

void main(void)
{
    struct pl011 serial;

    pl011_setup(
         &serial, /* base_address = */0xfff32000, /* base_clock = */19200000);

    // QEMU PL011 parameters:
    //
    // pl011_setup(
    //     &serial, /* base_address = */0x9000000, /* base_clock = */24000000);

    pl011_send(&serial, "Hello, World\n", sizeof("Hello, World\n"));

    // Hang in there
    while (1) {}
}

I built the code, copied the binaries on the device and was ready to move on after the successful test, but here is what I got trying to load my binary instead of the expected “Hello World!”:

Program headers:
p_type: PT_PHDR, p_flags: PF_R, p_offset: 0x40, p_vaddr: 0x200040, p_paddr: 0x20
0040, p_filesz: 0xE0, p_memsz: 0xE0, p_align: 0x8
p_type: PT_LOAD, p_flags: PF_R, p_offset: 0x0, p_vaddr: 0x200000, p_paddr: 0x200
000, p_filesz: 0x12E, p_memsz: 0x12E, p_align: 0x10000
p_type: PT_LOAD, p_flags: PF_X | PF_R, p_offset: 0x130, p_vaddr: 0x210130, p_pad
dr: 0x210130, p_filesz: 0x408, p_memsz: 0x408, p_align: 0x10000
p_type: 0x6474E551, p_flags: PF_W | PF_R, p_offset: 0x0, p_vaddr: 0x0, p_paddr: 
0x0, p_filesz: 0x0, p_memsz: 0x0, p_align: 0x0
Lo

So that’s definitely not a success. I see that the loader manages to read the binary and then magically everything stops. How come?

Initially I though that there is something wrong with the loader I use (see github.com/krinkinmu/efi). After all it’s not like I rigourously tested the code, so it’s easy to imagine that a different ELF binary might have surfaced some problems in the code.

However after poking a bit into the loader code and putting a few debug prints here and there to understand what’s going on I realized that the Lo part in the output above is actually part of the Loading ELF image... message. So it’s clear that the problem happens in the middle of outputing the message.

Normally I would suspect the worst here: that I somehow corrupted the memory somehwere in the code and it only surfaced somewhere deep in the UEFI firmware internals. However trying to understand at what point the corruption happens with the debug print didn’t point me to any reasonable direction.

At this point I started to suspect that the problem doesn’t happen when I make a particular call, but rather time related instead. So I started adding delays to the various places in the code like this:

for (volatile int i = 0; i < 1000000000; ++i) {}

And what a surprise, I actually managed to make it work and got my “Hello World”:

Program headers:
p_type: PT_PHDR, p_flags: PF_R, p_offset: 0x40, p_vaddr: 0x200040, p_paddr: 0x20
0040, p_filesz: 0xE0, p_memsz: 0xE0, p_align: 0x8
p_type: PT_LOAD, p_flags: PF_R, p_offset: 0x0, p_vaddr: 0x200000, p_paddr: 0x200
000, p_filesz: 0x12E, p_memsz: 0x12E, p_align: 0x10000
p_type: PT_LOAD, p_flags: PF_X | PF_R, p_offset: 0x130, p_vaddr: 0x210130, p_pad
dr: 0x210130, p_filesz: 0x440, p_memsz: 0x440, p_align: 0x10000
p_type: 0x6474E551, p_flags: PF_W | PF_R, p_offset: 0x0, p_vaddr: 0x0, p_paddr: 
0x0, p_filesz: 0x0, p_memsz: 0x0, p_align: 0x0
Loading ELF image...
Loaded ELF image
Starting ELF image...
Hello, World

What I found interesting is that to make it work I had to add a delay after the code that prints “Starting ELF image…” message. That made me think that maybe the problem is not really in the loader, but rather in the way the loaded code interracts with the PL011 controller.

The first thing it does is resetting the device to put it into a know state. So I figured, what if UEFI firmware uses PL011 internal buffers (FIFOs) and doesn’t wait for the data to actually to be sent out. That would certainly explain why message prints were interrupted half way.

So I went on and modied the pl011_setup function slightly to add a wait for any ongoing transfers to complete before doing a reset when I setup the device:

int pl011_setup(
    struct pl011 *dev, uint64_t base_address, uint64_t base_clock)
{
    dev->base_address = base_address;
    dev->base_clock = base_clock;
    dev->baudrate = 115200;
    dev->data_bits = 8;
    dev->stop_bits = 1;
    wait_tx_complete(dev);
    return pl011_reset(dev);
}

And that turned out to also “solve” the problem.

I still don’t understand why reset puts the device in the weird state. Reset indeed should interrupt any ongoing transfers and flush all FIFOs, so that’s not unexpected. What is unexpected is why we cannot see the “Hello World” message after that.

Maybe reset itself gets stuck or maybe it puts the device in such a state so it cannot send anything out.

Instead of conclusion

I cannot claim that I actually solved the problem or just masked it, as I do not understand the exact mechanism that triggers the problematic behavior.

However I can say that while emulators are useful you should always test you code on the real hardware.