NuKVM/memtest86plus
3
0
Fork 0
mirror of https://github.com/memtest86plus/memtest86plus.git synced 2026-09-03 20:53:06 -05:00
Code Issues Packages Projects Releases Wiki Activity

Add support for aarch64 (ARM64) architecture (#626)

Browse Source 
* Initial commit for ARM64 (header.S & startup.S)

* Add Aarch64 interrupt handler

* Adapt lib/ to Aarch64

* Small ASM addition for AArch64

* Rewrite Aarch64 system/
- cpuid.c/cpuinfo.c for CPU detection (Qualcomm, Apple, Nvidia, etc)
- Add ARM Power State Coordination Interface (PSCI) functions
- vmem.c for Virtual Memory MAnagement on ARM64
- Various functions placeholders

* Add pseudo-TSC correction for Aarch64

* Add pseudo-tsc measurement for Aarch64

* Add Aarch64 if/elif on basic IO/MMIO/MEM RW functions

* Add support for Surface 13 / Snapdragon X KB USB
Fix an issue with the XHCI.c drivers with packets > 8 bytes

* Add MMIO Serial support for ARM64

* Add ARM64 ASM for block_move & mov_inv_fixed. Fix VM start pointer

* Add ARM Serial MMIO configuration

* Revert previous commit & reintroduce ARM64 cache functions

* Add ACPI MCFG Table description & SMP parsing. This requires careful review/testing to avoid BC issues!

* Revert previous commmit and reintroduce relocation and heap change for ARM64

* ARM64 does not support CPU PNS. Use DMI type4 parsing instead. Add SMBIOS v3 support (ARM64 does not support v2)

* Fix typo in pci.h

* Move Aarch64 load-limit computation from main.c to pmem.c

* Add .arm64 to version string. Use PMU for clock computation display. Fix D5 Temp NULL not handled

* Add Makefile and linker scripts for aarch64. Set cross-compiler from x86 by default

* Attempt to add aarch64 to workflow

* Remove a remaining internal debug flag

* Correct typos

* Fix ARM64 SMP freeze
assign heap when RAM is one contiguous region, keep it below 3GB for DMA (RPi5), add DSB before SEV to avoid missed barrier wakeups
Sync I-cache with D-cache after program relocation to prevent stale instruction fetch

* Validate MADT entry lengths to avoid reading past the table (or looping forever) on malformed ACPI tables

* aarch64: add NEON kernels for the vector PRSG tests
Tests 3 (bus stress) and 5 (moving inversions, random) now dispatch to 128-bit NEON fill/scan instead of scalar.

* Fix ARM64 data abort from the per-second DMI check & fix unbounded strstr()

* Fix ARM64 data abort when saving results to USB after a relocation

* Update README.md with AArch64 (ARM64) support

* aarch64: shrink static string tables, harden MCFG/paging
 Replace pointer-based name tables with inline char arrays (saves ~1.5KB + relocs)
alidate MCFG length/bus ranges, and bound pm_map/vm_map fills against overflows

* Add NVIDIA & latest Apple ARM SoC to cpuid.c

* Fix stale pointers read by the report writer after a program relocation
cpu_model gets a static init (creates its reloc record)SPD type becomes an inline char array
This commit is contained in:
Sam Demeulemeester
2026-07-12 13:27:45 +02:00
committed by GitHub
parent 33a5eeeffa
commit 7fc312d5b0
71 changed files with 3918 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/Linux.yml
+10 -1
View File
@@ -21,7 +21,7 @@ jobs:
matrix:
compiler: [gcc]
os: [ubuntu-22.04, ubuntu-24.04]
arch: [i586, x86_64, la64]
arch: [i586, x86_64, la64, aarch64]
steps:
- uses: actions/checkout@v7
@@ -32,6 +32,9 @@ jobs:
run: |
sudo apt-get update
sudo apt-get install build-essential gcc-multilib clang libc6-dev-i386-cross dosfstools mtools xorriso -y
if [ ${{ matrix.arch }} == 'aarch64' ]; then
sudo apt-get install gcc-aarch64-linux-gnu binutils-aarch64-linux-gnu -y
fi
if [ ${{ matrix.arch }} == 'la64' ]; then
sudo mkdir /opt/LoongArch_Toolchains -p; cd /opt/LoongArch_Toolchains
sudo wget https://github.com/YongbaoOS/Yongbao-Toolchains/releases/download/2024.8.6/x86_64-cross-tools-loongarch64-binutils_git60d4fed4e364-gcc_14.2.0.tar.xz
@@ -50,6 +53,8 @@ jobs:
cd build/x86_64
elif [ ${{ matrix.arch }} == 'la64' ]; then
cd build/loongarch64
elif [ ${{ matrix.arch }} == 'aarch64' ]; then
cd build/aarch64
fi
make clean
@@ -63,5 +68,9 @@ jobs:
elif [ ${{ matrix.arch }} == 'la64' ]; then
export PATH=/opt/LoongArch_Toolchains/cross-tools/bin/:$PATH
cd build/loongarch64
elif [ ${{ matrix.arch }} == 'aarch64' ]; then
cd build/aarch64
make -j 2 CC=aarch64-linux-gnu-gcc LD=aarch64-linux-gnu-ld OBJCOPY=aarch64-linux-gnu-objcopy iso
exit 0
fi
make -j 2 CC="${{matrix.compiler}}" iso
.gitignore
+1
View File
@@ -17,6 +17,7 @@ mt86plus
memtest_shared
memtest_shared.bin
*.bin
*.elf
*.efi
*.img
*.iso
README.md
+29 -16
View File
@@ -1,8 +1,8 @@
# Memtest86+
Memtest86+ is a free, open-source, stand-alone memory tester for x86, x86-64
and LoongArch64 architecture computers. It provides a much more thorough memory
check than that provided by BIOS memory tests.
Memtest86+ is a free, open-source, stand-alone memory tester for x86, x86-64,
ARM AArch64 and LoongArch64 architecture computers. It provides a much more
thorough memory check than that provided by BIOS memory tests.
It is also able to access almost all the computer's memory, not being
restricted by the memory used by the operating system and not depending
@@ -11,8 +11,8 @@ on any underlying software like UEFI libraries.
Memtest86+ can be loaded and run either directly by a PC BIOS (legacy or UEFI)
or via an intermediate bootloader that supports the Linux 16-bit, 32-bit,
64-bit, or EFI handover boot protocol. It should work on most x86, x86-64 CPU
(Pentium class or later 32-bit or 64-bit) and most LoongArch64 CPU (Loongson 3
and Loongson 2 family).
(Pentium class or later 32-bit or 64-bit), most ARMv8-A AArch64 CPU (UEFI
boot only) and most LoongArch64 CPU (Loongson 3 and Loongson 2 family).
Binary releases (both stable and nightly dev builds) are available on
[memtest.org](https://memtest.org).
@@ -85,6 +85,16 @@ an intermediate bootloader using the Linux 16-bit boot protocol. The image
can be also booted by an intermediate bootloader using the Linux 32-bit,
64-bit, or 64-bit EFI handover boot protocols.
### To build an ARM AArch64 64-bit image
Change directory into the `build/aarch64` directory and type `make`. When not
building on an AArch64 host, the Makefile defaults to the `aarch64-linux-gnu-`
cross toolchain; pass `CC=`, `LD=` and `OBJCOPY=` overrides if yours is named
differently. The result is a `mt86plus` binary image file which can be booted
directly by a 64-bit UEFI firmware (if named `mt86plus.efi`, or as
`EFI/BOOT/BOOTAA64.EFI` on removable media). UEFI is the only supported boot
method on this architecture.
### To build a LoongArch64 64-bit image
#### x86-64 Linux environment
@@ -142,9 +152,10 @@ GRUB as an intermediate bootloader. See the individual `Makefile` under the
`build` directory for details. The ISO image is both legacy and UEFI bootable,
so you need GRUB modules for both legacy and EFI boot installed on your
build system (e.g. on Debian, the required GRUB modules are located in
packages `grub-pc-bin`, `grub-efi-ia32-bin`, `grub-efi-amd64-bin` and
`grub-efi-loong64-bin`). You may need to adjust some path and file names
in the Makefile to match the naming on your system.<br>
packages `grub-pc-bin`, `grub-efi-ia32-bin`, `grub-efi-amd64-bin`,
`grub-efi-arm64-bin` and `grub-efi-loong64-bin`). You may need to adjust
some path and file names in the Makefile to match the naming on your
system.<br>
**P.S.** LoongArch64 GRUB ISO can only be created on LoongArch64 Linux
environment.
@@ -643,10 +654,11 @@ the memory interface. The burst length changes on every round, and every other
round inserts short idle gaps into the traffic to generate load transients in
the module power delivery. This targets DDR5-era interface faults that on-die
ECC cannot correct: link signal-integrity errors, marginal XMP/EXPO training
and PMIC voltage droop. On x86_64 the bursts use the same non-temporal
SIMD kernels as test 5; other builds fall back to cached scalar accesses, which
exercise the interface less aggressively. Errors from this test are typically
transient and will not necessarily repeat at the same address; they do not
and PMIC voltage droop. On x86_64 and AArch64 the bursts use the same
non-temporal SIMD kernels as test 5; other builds fall back to cached scalar
accesses, which exercise the interface less aggressively. Errors from this
test are typically transient and will not necessarily repeat at the same
address; they do not
contribute to BadRAM patterns, memmap regions, or bad page regions.
### Test 4 : Moving inversions, ones & zeros
@@ -658,10 +670,11 @@ inversions algorithm with patterns of all ones and all zeros.
In each memory region in turn, uses the moving inversions algorithm with a
vector-wide pseudo-random sequence and its complement. On x86_64 the fill and
check loops use AVX2 (256-bit) or SSE2 (128-bit) instructions, selected
automatically at boot and shown in the test name, with each vector lane
running an independent pseudo-random stream. Writes use non-temporal stores,
which bypass the cache and maximise the stress on the memory bus. Every
check loops use AVX2 (256-bit) or SSE2 (128-bit) instructions, and on AArch64
NEON (128-bit) instructions, selected automatically at boot and shown in the
test name, with each vector lane running an independent pseudo-random stream.
Writes use non-temporal stores, which bypass the cache and maximise the
stress on the memory bus. Every
fourth round a single random value is broadcast to all lanes, reproducing the
uniform-background fault model of the classic random pattern test. The random
sequences are different on each round and each test pass, so multiple passes
app/aarch64/interrupt.c
+128
View File
@@ -0,0 +1,128 @@
// SPDX-License-Identifier: GPL-2.0
// Copyright (C) 2026 Sam Demeulemeester
//
// The C part of the ARM64 (AArch64) exception handler. The assembler part
// in boot/aarch64/startup.S saves the interrupted context in a trap_regs
// struct on the stack and calls interrupt(). We don't use interrupts on
// this architecture (cross-CPU wakeups use the event mechanism), so every
// exception that gets here is reported as fatal
#include <stdint.h>
#include "hwctrl.h"
#include "keyboard.h"
#include "screen.h"
#include "smp.h"
#include "display.h"
#include "interrupt.h"
//------------------------------------------------------------------------------
// Types
//------------------------------------------------------------------------------
// Match the context frame layout in boot/aarch64/startup.S.
struct trap_regs {
uint64_t x[31];
uint64_t sp;
uint64_t elr;
uint64_t spsr;
uint64_t esr;
uint64_t far;
uint64_t vec;
uint64_t pad;
};
//------------------------------------------------------------------------------
// Constants
//------------------------------------------------------------------------------
static const char vector_type_name[][12] = {
"Synchronous",
"IRQ",
"FIQ",
"SError",
};
static const char vector_origin_name[][7] = {
"EL1t",
"EL1h",
"EL0/64",
"EL0/32",
};
//------------------------------------------------------------------------------
// Private Functions
//------------------------------------------------------------------------------
static const char *exception_class_name(uint64_t esr)
{
switch ((esr >> 26) & 0x3F) {
case 0x00: return "Unknown";
case 0x0E: return "Illegal execution state";
case 0x15: return "SVC";
case 0x18: return "Sysreg trap";
case 0x20: return "Instruction abort (lower EL)";
case 0x21: return "Instruction abort";
case 0x22: return "PC alignment fault";
case 0x24: return "Data abort (lower EL)";
case 0x25: return "Data abort";
case 0x26: return "SP alignment fault";
case 0x2F: return "SError";
case 0x30:
case 0x31: return "Breakpoint";
case 0x32:
case 0x33: return "Software step";
case 0x34:
case 0x35: return "Watchpoint";
case 0x3C: return "BRK";
default: return "Other";
}
}
//------------------------------------------------------------------------------
// Public Functions
//------------------------------------------------------------------------------
void interrupt(struct trap_regs *trap_regs)
{
spin_lock(error_mutex);
clear_message_area();
display_pinned_message(0, 0, "Unexpected exception on CPU %i", smp_my_cpu_num());
display_pinned_message(2, 0, "Type: %s %s (%s)", vector_type_name[trap_regs->vec & 0x3],
vector_origin_name[(trap_regs->vec >> 2) & 0x3],
exception_class_name(trap_regs->esr));
display_pinned_message(3, 0, " PC: %016x", (uintptr_t)trap_regs->elr);
display_pinned_message(4, 0, " FAR: %016x", (uintptr_t)trap_regs->far);
display_pinned_message(5, 0, " ESR: %016x", (uintptr_t)trap_regs->esr);
display_pinned_message(6, 0, "SPSR: %016x", (uintptr_t)trap_regs->spsr);
display_pinned_message(3, 25, "LR: %016x", (uintptr_t)trap_regs->x[30]);
display_pinned_message(4, 25, "SP: %016x", (uintptr_t)trap_regs->sp);
display_pinned_message(5, 25, "X0: %016x", (uintptr_t)trap_regs->x[0]);
display_pinned_message(6, 25, "X1: %016x", (uintptr_t)trap_regs->x[1]);
display_pinned_message(7, 25, "X2: %016x", (uintptr_t)trap_regs->x[2]);
display_pinned_message(8, 25, "X3: %016x", (uintptr_t)trap_regs->x[3]);
display_pinned_message(9, 25, "X4: %016x", (uintptr_t)trap_regs->x[4]);
display_pinned_message(10, 25, "X5: %016x", (uintptr_t)trap_regs->x[5]);
display_pinned_message(11, 25, "X6: %016x", (uintptr_t)trap_regs->x[6]);
display_pinned_message(12, 25, "X7: %016x", (uintptr_t)trap_regs->x[7]);
display_pinned_message(13, 25, "X8: %016x", (uintptr_t)trap_regs->x[8]);
display_pinned_message(0, 50, "Stack:");
for (int i = 0; i < 12; i++) {
uintptr_t addr = trap_regs->sp + sizeof(uint64_t)*(11 - i);
uint64_t data = *(uint64_t *)addr;
display_pinned_message(1 + i, 50, "%012x %016x", addr, (uintptr_t)data);
}
clear_screen_region(ROW_FOOTER, 0, ROW_FOOTER, SCREEN_WIDTH - 1);
prints(ROW_FOOTER, 0, "Press any key to reboot...");
while (get_key() == 0) { }
reboot();
}
app/config.c
+16
View File
@@ -118,6 +118,7 @@ bool tty_new_line = false;
uint32_t tty_mmio_ref_clk = UART_REF_CLK_MMIO; // Reference clock for MMIO (in Hz)
int tty_mmio_stride = 4; // Stride for MMIO (register width in bytes)
bool tty_pl011 = false; // UART is an ARM PL011 rather than a 16550
bool err_banner_redraw = false; // Redraw banner on new errors
@@ -134,6 +135,18 @@ static void parse_serial_params(const char *params)
return;
}
// Check for an ARM PL011 UART ("console=ttyAMA,0x9000000")
if (strncmp(params, "ttyAMA,0x", 9) == 0) {
uintptr_t pl011_adr = hexstr2int(params+9);
if (pl011_adr > 0xFFFF) {
tty_pl011 = true;
tty_address = pl011_adr;
} else {
enable_tty = false;
}
return;
}
// Check if console is MMIO and grab address and stride
uintptr_t mmio_adr = 0;
if (strncmp(params, "mmio,0x", 7) == 0) {
@@ -280,6 +293,9 @@ static void parse_option(const char *option, const char *params)
{
if (option[0] == '\0') return;
// Options may be given without parameters.
if (params == NULL) params = "";
if (strncmp(option, "console", 8) == 0) {
parse_serial_params(params);
} else if (strncmp(option, "newline", 7) == 0) {
app/config.h
+1
View File
@@ -78,6 +78,7 @@ extern bool tty_new_line;
extern uint32_t tty_mmio_ref_clk;
extern int tty_mmio_stride;
extern bool tty_pl011;
extern bool err_banner_redraw;
app/display.c
+10 -1
View File
@@ -187,6 +187,8 @@ void display_init(void)
prints(ROW_FOOTER, 74, ".x32");
#elif defined (__loongarch_lp64)
prints(ROW_FOOTER, 74, ".la64");
#elif defined (__aarch64__)
prints(ROW_FOOTER, 74, ".arm64");
#endif
set_foreground_colour(palette.foreground);
@@ -195,9 +197,16 @@ void display_init(void)
if (cpu_model) {
display_cpu_model(cpu_model);
}
#if defined(__aarch64__)
// Generic timer does not run at CPU clock. Use the PMU instead
if (cpu_clk_mhz) {
display_cpu_clk((int)cpu_clk_mhz);
}
#else
if (clks_per_msec) {
display_cpu_clk((int)(clks_per_msec / 1000));
}
#endif
#if TESTWORD_WIDTH < 64
if (cpuid_info.flags.lm) {
display_cpu_addr_mode(" [LM]");
@@ -431,7 +440,7 @@ void display_temperature(void)
if (enable_temp_ram) {
// Display RAM Temperature (DDR5+ Only) - LA64 unsupported yet
if (dmi_memory_device->type == DMI_DDR5 && !strstr(cpuid_info.vendor_id.str, "Loongson")) {
if (dmi_memory_device_type == DMI_DDR5 && !strstr(cpuid_info.vendor_id.str, "Loongson")) {
for (int i = 0; i < MAX_SPD_SLOT; i++) {
app/main.c
+35
View File
@@ -65,9 +65,16 @@
#define TEST_INTERRUPT 0
#endif
#if defined(__aarch64__)
// RAM may start well above physical address 0, so the load limits are
// computed at run time, relative to the start of RAM (check global_init)
#define LOW_LOAD_LIMIT low_load_limit
#define HIGH_LOAD_LIMIT high_load_limit
#else
#define LOW_LOAD_LIMIT SIZE_C(4,MB) // must be a multiple of the page size
#define HIGH_LOAD_LIMIT (VM_PINNED_SIZE << PAGE_SHIFT)
#endif
//------------------------------------------------------------------------------
// Private Variables
@@ -165,6 +172,10 @@ static void run_at(uintptr_t addr, int my_cpu)
memcpy((void *)(addr + locals_offset), (void *)(_start + locals_offset), LOCALS_SIZE);
locals_offset += AP_STACK_SIZE;
}
#if defined(__aarch64__)
// Make the copied code visible to instruction fetch.
cache_sync_code_range((void *)addr, (void *)(addr + (_stacks - _start)));
#endif
}
LONG_BARRIER;
@@ -173,6 +184,10 @@ static void run_at(uintptr_t addr, int my_cpu)
// The 32-bit startup code needs to know where it is located.
__asm__ __volatile__("movl %0, %%edi; jmp *%0" : : "r" (new_start_addr));
__builtin_unreachable();
#elif defined(__aarch64__)
// Discard any instructions speculatively fetched before the I-cache invalidation.
__asm__ __volatile__("isb");
((void (*)(void))new_start_addr)();
#else
((void (*)(void))new_start_addr)(); // Formerly a non-portable construct: goto *new_start_addr;
#endif
@@ -192,12 +207,20 @@ static void relocate_to(uintptr_t addr)
memset((void *)(addr + locals_offset), 0, LOCALS_SIZE);
locals_offset += AP_STACK_SIZE;
}
#if defined(__aarch64__)
// Make the copied code visible to instruction fetch.
cache_sync_code_range((void *)addr, (void *)(addr + (_stacks - _start)));
#endif
// Jump to new_start_addr.
#ifdef __i386__
// The 32-bit startup code needs to know where it is located.
__asm__ __volatile__("movl %0, %%edi; jmp *%0" : : "r" (new_start_addr));
__builtin_unreachable();
#elif defined(__aarch64__)
// Discard any instructions speculatively fetched before the I-cache invalidation.
__asm__ __volatile__("isb");
((void (*)(void))new_start_addr)();
#else
((void (*)(void))new_start_addr)();
#endif
@@ -362,6 +385,9 @@ static void global_init(void)
for (int i = 0; i < pm_map_size; i++) {
trace(0, "pm %0*x - %0*x", 2*sizeof(uintptr_t), pm_map[i].start, 2*sizeof(uintptr_t), pm_map[i].end);
}
if (paging_incomplete) {
trace(0, "WARNING: page table pool exhausted, some address ranges are not mapped");
}
if (acpi_config.rsdp_addr != 0) {
trace(0, "ACPI RSDP (v%u.%u) found in %s at %0*x", acpi_config.ver_maj, acpi_config.ver_min, rsdp_source, 2*sizeof(uintptr_t), acpi_config.rsdp_addr);
trace(0, "ACPI FADT found at %0*x", 2*sizeof(uintptr_t), acpi_config.fadt_addr);
@@ -463,6 +489,9 @@ static void setup_vm_map(uintptr_t win_start, uintptr_t win_end)
uint64_t new_end;
while (1) {
if (vm_map_size >= MAX_MEM_SEGMENTS) {
break;
}
if (smp_narrow_to_proximity_domain(orig_start, orig_end, &proximity_domain_idx, &new_start, &new_end)) {
// Create a new entry in the virtual memory map.
num_mapped_pages += (new_end - new_start) >> PAGE_SHIFT;
@@ -580,7 +609,13 @@ static void test_all_windows(int my_cpu)
break;
case 1:
window_start = (LOW_LOAD_LIMIT >> PAGE_SHIFT);
#if defined(__aarch64__)
// LOW_LOAD_LIMIT may be above VM_WINDOW_SIZE. End the window
// at the next window boundary to avoid recheck the region containing the low copy.
window_end = (window_start + VM_WINDOW_SIZE) & ~(VM_WINDOW_SIZE - 1);
#else
window_end = VM_WINDOW_SIZE;
#endif
break;
default:
window_start = window_end;
boot/aarch64/header.S
+191
View File
@@ -0,0 +1,191 @@
// SPDX-License-Identifier: GPL-2.0
//
// header.S provides the MS-DOS & PE/COFF headers needed to boot directly
// from an ARM64 (AArch64) UEFI BIOS. There is no legacy boot path on this
// architecture.
//
// Copyright (C) 2026 Sam Demeulemeester
//
#define __ASSEMBLY__
#include "boot.h"
#include "build_version.h"
#include "peimage.h"
// The EFI loader loads the header at ImageBase, so we have to locate the main program
// after that. This means we cant load the main program at HIGH_LOAD_ADDR. Pick a load
// address far away from HIGH_LOAD_ADDR, to avoid overlap when relocating the code.
#define IMAGE_BASE 0x200000
.section ".header", "ax", %progbits
.globl head
head:
// "MZ", the MS-DOS header signature.
.byte 0x4d
.byte 0x5a
// The PE header pointer.
.org 0x3c
.long pe_header
pe_header:
.ascii "PE"
.short 0
coff_header:
.short IMAGE_FILE_MACHINE_ARM64 // Machine (AArch64)
.short 5 // NumberOfSections
.long 0 // TimeDateStamp
.long 0 // PointerToSymbolTable
.long 0 // NumberOfSymbols
.short section_table - optional_header // SizeOfOptionalHeader
.short IMAGE_FILE_DEBUG_STRIPPED \
| IMAGE_FILE_LOCAL_SYMS_STRIPPED \
| IMAGE_FILE_LINE_NUMS_STRIPPED \
| IMAGE_FILE_EXECUTABLE_IMAGE // Characteristics
optional_header:
.short IMAGE_NT_OPTIONAL_HDR64_MAGIC // PE32+ format
.byte 0x02 // MajorLinkerVersion
.byte 0x14 // MinorLinkerVersion
.long _virt_text_size // SizeOfCode
.long _virt_init_size // SizeOfInitializedData
.long 0 // SizeOfUninitializedData
.long _virt_text_start + 0x400 // AddressOfEntryPoint
.long _virt_text_start // BaseOfCode
extra_header_fields:
.quad IMAGE_BASE // ImageBase
.long 4096 // SectionAlignment
.long 512 // FileAlignment
.short 0 // MajorOperatingSystemVersion
.short 0 // MinorOperatingSystemVersion
.short 0 // MajorImageVersion
.short 0 // MinorImageVersion
.short 0 // MajorSubsystemVersion
.short 0 // MinorSubsystemVersion
.long 0 // Win32VersionValue
.long _virt_img_size // SizeOfImage
.long _file_head_size // SizeOfHeaders
.long 0 // CheckSum
.short 10 // Subsystem (EFI application)
.short IMAGE_DLLCHARACTERISTICS_NX_COMPAT // DllCharacteristics
.quad 0 // SizeOfStackReserve
.quad 0 // SizeOfStackCommit
.quad 0 // SizeOfHeapReserve
.quad 0 // SizeOfHeapCommit
.long 0 // LoaderFlags
.long IMAGE_DIRECTORY_ENTRY_DEBUG // NumberOfRvaAndSizes
.long 0 // DataDirectory.Export.VirtualAddress
.long 0 // DataDirectory.Export.Size
.long 0 // DataDirectory.Import.VirtualAddress
.long 0 // DataDirectory.Import.Size
.long 0 // DataDirectory.Resource.VirtualAddress
.long 0 // DataDirectory.Resource.Size
.long 0 // DataDirectory.Exception.VirtualAddress
.long 0 // DataDirectory.Exception.Size
.long 0 // DataDirectory.Certs.VirtualAddress
.long 0 // DataDirectory.Certs.Size
.long _virt_reloc_start // DataDirectory.BaseReloc.VirtualAddress
.long _real_reloc_size // DataDirectory.BaseReloc.Size
// Section table
section_table:
.ascii ".text"
.byte 0
.byte 0
.byte 0
.long _virt_text_size // VirtualSize
.long _virt_text_start // VirtualAddress
.long _file_text_size // SizeOfRawData
.long _file_text_start // PointerToRawData
.long 0 // PointerToRelocations
.long 0 // PointerToLineNumbers
.short 0 // NumberOfRelocations
.short 0 // NumberOfLineNumbers
.long IMAGE_SCN_MEM_READ \
| IMAGE_SCN_MEM_EXECUTE \
| IMAGE_SCN_CNT_CODE // Characteristics (section flags)
.ascii ".data"
.byte 0
.byte 0
.byte 0
.long _virt_data_size // VirtualSize
.long _virt_data_start // VirtualAddress
.long _file_data_size // SizeOfRawData
.long _file_data_start // PointerToRawData
.long 0 // PointerToRelocations
.long 0 // PointerToLineNumbers
.short 0 // NumberOfRelocations
.short 0 // NumberOfLineNumbers
.long IMAGE_SCN_MEM_READ \
| IMAGE_SCN_MEM_WRITE \
| IMAGE_SCN_CNT_INITIALIZED_DATA // Characteristics (section flags)
.ascii ".reloc"
.byte 0
.byte 0
.long _virt_reloc_size // VirtualSize
.long _virt_reloc_start // VirtualAddress
.long _file_reloc_size // SizeOfRawData
.long _file_reloc_start // PointerToRawData
.long 0 // PointerToRelocations
.long 0 // PointerToLineNumbers
.short 0 // NumberOfRelocations
.short 0 // NumberOfLineNumbers
.long IMAGE_SCN_MEM_READ \
| IMAGE_SCN_CNT_INITIALIZED_DATA // Characteristics (section flags)
.ascii ".sbat"
.byte 0
.byte 0
.byte 0
.long _virt_sbat_size // VirtualSize
.long _virt_sbat_start // VirtualAddress
.long _file_sbat_size // SizeOfRawData
.long _file_sbat_start // PointerToRawData
.long 0 // PointerToRelocations
.long 0 // PointerToLineNumbers
.short 0 // NumberOfRelocations
.short 0 // NumberOfLineNumbers
.long IMAGE_SCN_MEM_READ \
| IMAGE_SCN_CNT_INITIALIZED_DATA // Characteristics (section flags)
.ascii ".osrel"
.byte 0
.byte 0
.long _virt_osrel_size // VirtualSize
.long _virt_osrel_start // VirtualAddress
.long _file_osrel_size // SizeOfRawData
.long _file_osrel_start // PointerToRawData
.long 0 // PointerToRelocations
.long 0 // PointerToLineNumbers
.short 0 // NumberOfRelocations
.short 0 // NumberOfLineNumbers
.long IMAGE_SCN_MEM_READ \
| IMAGE_SCN_CNT_INITIALIZED_DATA // Characteristics (section flags)
.org 512
.section ".reloc"
.long 0 // Page RVA
.long 10 // Block Size (2*4+2)
.short (IMAGE_REL_BASED_ABSOLUTE << 12) + 0 // reloc 0 -> 0
.section ".sbat", "a", %progbits
.incbin "../boot/sbat.csv"
.section ".osrel", "a", %progbits
.ascii "ID=memtest86plus\n"
.ascii "NAME=\"Memtest86+\"\n"
.ascii "VERSION_ID=\"" , MT_VERSION , "\"\n"
.ascii "PRETTY_NAME=\"Memtest86+ v" , MT_VERSION , "\"\n"
boot/aarch64/startup.S
+371
View File
@@ -0,0 +1,371 @@
// SPDX-License-Identifier: GPL-2.0
//
// startup.S contains the 64-bit startup code for both the BSP and APs on
// ARM64 (AArch64). It initialises the MMU with our own identity-mapped page
// tables, drops from EL2 to EL1 if necessary, sets up the exception vectors
// and stacks, completes relocation, and finally calls the main application.
//
// The BSP is entered from the UEFI firmware at efi_boot with the MMU on
// (using the firmware page tables) at either EL2 or EL1. The APs are
// entered via PSCI CPU_ON at startup64 with the MMU off, again at either
// EL2 or EL1, with our CPU number in x0. After each program relocation,
// all CPUs re-enter at startup
//
// Copyright (C) 2026 Sam Demeulemeester
//
#define __ASSEMBLY__
#include "boot.h"
// Register initialisation values.
#define SCTLR_VALUE 0x30D01805 // RES1 bits, MMU on, D-cache on, I-cache on
#define TCR_VALUE 0x80903510 // T0SZ=16 IRGN0=WBWA ORGN0=WBWA SH0=inner TG0=4K
// T1SZ=16 EPD1=1 TG1=4K (IPS filled in at run time)
#define MAIR_VALUE 0x4400FF // idx0 = Normal WB WA, idx1 = Device-nGnRnE,
// idx2 = Normal non-cacheable
#define HCR_VALUE 0x30080000000 // RW=1 (EL1 is AArch64), API=1 APK=1 (no PAuth traps)
#define CPTR_VALUE 0x33FF // RES1 bits, no FP/SIMD traps
#define SPSR_EL1H_MASKED 0x3C5 // EL1h, all interrupt sources masked
// Exception context frame layout. This must match struct trap_regs in
// app/aarch64/interrupt.c
#define CTX_X(n) ((n) * 8)
#define CTX_SP 248
#define CTX_ELR 256
#define CTX_SPSR 264
#define CTX_ESR 272
#define CTX_FAR 280
#define CTX_VEC 288
#define CTX_SIZE 304
.macro mov_q, rd, val
movz \rd, #((\val) & 0xffff)
movk \rd, #(((\val) >> 16) & 0xffff), lsl #16
movk \rd, #(((\val) >> 32) & 0xffff), lsl #32
movk \rd, #(((\val) >> 48) & 0xffff), lsl #48
.endm
.text
.globl startup32
startup32:
brk #0 // unreachable (or at least should be)
// The EFI PE32+ boot entry point.
.org 0x400
.globl efi_boot
efi_boot:
// Preserve the EFI image handle and system table pointer while we
// apply our dynamic relocations. This must be done before calling
// any other C code, as we cannot rely on the linker eliminating
// GOT indirection for data references
mov x19, x0
mov x20, x1
bl reloc
mov x0, x19
mov x1, x20
mov x2, xzr // boot params not yet allocated
bl efi_setup
// Save the boot params pointer.
adrp x1, boot_params_addr
str x0, [x1, :lo12:boot_params_addr]
// Build our identity-mapped page tables. We are still running on the
// firmwares page tables (at EL1 or EL2), which per the UEFI spec
// identity-map all memory regions, so plain C code is fine here
bl paging_init
// Record which copy of the program owns the current page tables
adrp x0, _start
add x0, x0, :lo12:_start
adrp x1, tables_for_base
str x0, [x1, :lo12:tables_for_base]
mov x0, xzr // the BSP is CPU 0
b startup64
// The entry point for AP boot (via PSCI CPU_ON with the MMU off, CPU number
// in x0) and for the BSP after EFI setup (with the MMU on, using either the
// firmware page tables at EL1 or the firmware EL2 translation regime)
.globl startup64
startup64:
msr daifset, #0xF
mov x19, x0 // our CPU number
// Enable FP/SIMD at EL1 and EL0 (the compiler may use SIMD registers).
mov x0, #(3 << 20) // CPACR_EL1.FPEN
msr cpacr_el1, x0
// Set up the EL1 translation regime to use our page tables. These
// registers are writable from both EL1 and EL2.
mov_q x0, MAIR_VALUE
msr mair_el1, x0
mov_q x0, TCR_VALUE
mrs x1, id_aa64mmfr0_el1
and x1, x1, #0xF // PARange
cmp x1, #5
b.ls 1f
mov x1, #5 // cap the intermediate PA size at 48 bits
1: bfi x0, x1, #32, #3 // TCR_EL1.IPS
msr tcr_el1, x0
adrp x0, ttbr0_table
add x0, x0, :lo12:ttbr0_table
msr ttbr0_el1, x0
adrp x0, vectors
add x0, x0, :lo12:vectors
msr vbar_el1, x0
mrs x0, CurrentEL
cmp x0, #0x8
b.eq el2_entry
// Entered at EL1. The MMU may be on (BSP, firmware page tables) or
// off (AP). Switch to our page tables and make sure the MMU and
// caches are enabled. Both mappings identity-map this code, so the
// transition is safe
isb
tlbi vmalle1
dsb nsh
isb
mov_q x0, SCTLR_VALUE
msr sctlr_el1, x0
isb
b el1_entry
el2_entry:
// Initialise the EL2 registers that affect EL1, then drop to EL1
// with the MMU and caches already enabled.
// If the firmware runs EL2 with VHE (HCR_EL2.E2H set), the EL2
// system register layouts differ and clearing E2H redefines the
// active translation regime, so turn the EL2 MMU off first (this
// code runs from an identity mapping, so that is safe)
mrs x0, hcr_el2
tbz x0, #34, 1f // HCR_EL2.E2H
mrs x0, sctlr_el2
bic x0, x0, #1 // clear SCTLR_EL2.M
msr sctlr_el2, x0
isb
mov_q x0, HCR_VALUE // clears E2H
msr hcr_el2, x0
isb
1:
mrs x0, midr_el1
msr vpidr_el2, x0
mrs x0, mpidr_el1
msr vmpidr_el2, x0
mov x0, #3 // EL1PCEN | EL1PCTEN
msr cnthctl_el2, x0
msr cntvoff_el2, xzr
mov_q x0, CPTR_VALUE
msr cptr_el2, x0
msr hstr_el2, xzr
msr mdcr_el2, xzr
mov_q x0, HCR_VALUE
msr hcr_el2, x0
mov_q x0, SCTLR_VALUE
msr sctlr_el1, x0
tlbi vmalle1
dsb nsh
mov x0, sp
msr sp_el1, x0 // continue on the same stack
adr x0, el1_entry
msr elr_el2, x0
mov_q x0, SPSR_EL1H_MASKED
msr spsr_el2, x0
isb
eret
el1_entry:
msr tpidr_el1, x19 // save our CPU number
b startup
// The main entry point and the entry point for restart after relocation.
.globl startup
startup:
msr daifset, #0xF
// Some of the startup actions are not thread safe. Use a mutex
// to protect this section of code
adrp x9, startup_mutex
add x9, x9, :lo12:startup_mutex
mov w10, #1
0: ldaxr w11, [x9]
cbnz w11, 0b
stxr w11, w10, [x9]
cbnz w11, 0b
// Set up our stack.
mrs x10, tpidr_el1
mov x11, #AP_STACK_SIZE
mul x10, x10, x11
mov x11, #(BSP_STACK_SIZE - LOCALS_SIZE)
add x10, x10, x11
adrp x11, _stacks
add x11, x11, :lo12:_stacks
add x11, x11, x10
mov sp, x11
bl reloc
// If this copy of the program doesnt own the current page tables,
// rebuild them in this copy's memory (safe: we are still running on
// the previous copy's tables, which identity-map everything)
adrp x0, _start
add x0, x0, :lo12:_start
adrp x9, tables_for_base
ldr x1, [x9, :lo12:tables_for_base]
cmp x0, x1
b.eq 1f
str x0, [x9, :lo12:tables_for_base]
bl paging_init
1:
// Every CPU (re)loads TTBR0 with *this copy's* page tables
adrp x0, ttbr0_table
add x0, x0, :lo12:ttbr0_table
msr ttbr0_el1, x0
isb
tlbi vmalle1
dsb nsh
isb
// Point VBAR at *this copy's* exception vectors.
adrp x0, vectors
add x0, x0, :lo12:vectors
msr vbar_el1, x0
isb
// Release the startup mutex
adrp x9, startup_mutex
add x9, x9, :lo12:startup_mutex
stlr wzr, [x9]
// Run Memtest86+.
bl main
2: wfi
b 2b
// The exception vector table.
.macro vec_entry, idx
.align 7
sub sp, sp, #CTX_SIZE
stp x0, x1, [sp, #CTX_X(0)]
mov x0, #\idx
b exception_common
.endm
.align 11
.globl vectors
vectors:
vec_entry 0 // current EL with SP0: synchronous
vec_entry 1 // current EL with SP0: IRQ
vec_entry 2 // current EL with SP0: FIQ
vec_entry 3 // current EL with SP0: SError
vec_entry 4 // current EL with SPx: synchronous
vec_entry 5 // current EL with SPx: IRQ
vec_entry 6 // current EL with SPx: FIQ
vec_entry 7 // current EL with SPx: SError
vec_entry 8 // lower EL (AArch64): synchronous
vec_entry 9 // lower EL (AArch64): IRQ
vec_entry 10 // lower EL (AArch64): FIQ
vec_entry 11 // lower EL (AArch64): SError
vec_entry 12 // lower EL (AArch32): synchronous
vec_entry 13 // lower EL (AArch32): IRQ
vec_entry 14 // lower EL (AArch32): FIQ
vec_entry 15 // lower EL (AArch32): SError
.align 2
exception_common:
stp x2, x3, [sp, #CTX_X(2)]
stp x4, x5, [sp, #CTX_X(4)]
stp x6, x7, [sp, #CTX_X(6)]
stp x8, x9, [sp, #CTX_X(8)]
stp x10, x11, [sp, #CTX_X(10)]
stp x12, x13, [sp, #CTX_X(12)]
stp x14, x15, [sp, #CTX_X(14)]
stp x16, x17, [sp, #CTX_X(16)]
stp x18, x19, [sp, #CTX_X(18)]
stp x20, x21, [sp, #CTX_X(20)]
stp x22, x23, [sp, #CTX_X(22)]
stp x24, x25, [sp, #CTX_X(24)]
stp x26, x27, [sp, #CTX_X(26)]
stp x28, x29, [sp, #CTX_X(28)]
str x30, [sp, #CTX_X(30)]
add x1, sp, #CTX_SIZE
str x1, [sp, #CTX_SP]
mrs x1, elr_el1
str x1, [sp, #CTX_ELR]
mrs x1, spsr_el1
str x1, [sp, #CTX_SPSR]
mrs x1, esr_el1
str x1, [sp, #CTX_ESR]
mrs x1, far_el1
str x1, [sp, #CTX_FAR]
str x0, [sp, #CTX_VEC]
mov x0, sp
bl interrupt
// If the handler returns, restore the interrupted context
ldr x1, [sp, #CTX_ELR]
msr elr_el1, x1
ldr x1, [sp, #CTX_SPSR]
msr spsr_el1, x1
ldp x2, x3, [sp, #CTX_X(2)]
ldp x4, x5, [sp, #CTX_X(4)]
ldp x6, x7, [sp, #CTX_X(6)]
ldp x8, x9, [sp, #CTX_X(8)]
ldp x10, x11, [sp, #CTX_X(10)]
ldp x12, x13, [sp, #CTX_X(12)]
ldp x14, x15, [sp, #CTX_X(14)]
ldp x16, x17, [sp, #CTX_X(16)]
ldp x18, x19, [sp, #CTX_X(18)]
ldp x20, x21, [sp, #CTX_X(20)]
ldp x22, x23, [sp, #CTX_X(22)]
ldp x24, x25, [sp, #CTX_X(24)]
ldp x26, x27, [sp, #CTX_X(26)]
ldp x28, x29, [sp, #CTX_X(28)]
ldr x30, [sp, #CTX_X(30)]
ldp x0, x1, [sp, #CTX_X(0)]
add sp, sp, #CTX_SIZE
eret
.previous
// Variables.
.data
.align 4
.globl ap_startup_addr
ap_startup_addr:
.quad 0 // filled in at run time
.globl boot_params_addr
boot_params_addr:
.quad 0
tables_for_base:
.quad 0
startup_mutex:
.long 0
.previous
// Main stack area.
.section ".stacks", "aw", %nobits
.align 16
. = . + STACKS_SIZE
.previous
boot/boot.h
+4
View File
@@ -91,7 +91,11 @@ extern uintptr_t boot_params_addr;
extern uint8_t ap_trampoline[];
#if defined(__loongarch_lp64) || defined(__aarch64__)
extern uintptr_t ap_startup_addr;
#else
extern uint32_t ap_startup_addr;
#endif
extern uint8_t ap_trampoline_end[];
boot/efisetup.c
+11 -1
View File
@@ -1,10 +1,10 @@
// SPDX-License-Identifier: GPL-2.0-only
// Copyright (C) 2020-2024 Martin Whitaker.
// Copyright 2011 Intel Corporation; author Matt Fleming
//
// Derived from Linux 5.6 arch/x86/boot/compressed/eboot.c and extracts
// from drivers/firmware/efi/libstub:
//
// Copyright 2011 Intel Corporation; author Matt Fleming
#include <stdbool.h>
@@ -179,6 +179,10 @@ static int get_cmd_line_length(efi_loaded_image_t *image)
efi_char16_t *cmd_line = (efi_char16_t *)image->load_options;
int max_length = image->load_options_size / sizeof(efi_char16_t);
int length = 0;
// When booted directly by the firmware there are no load options.
if (cmd_line == NULL || max_length == 0) {
return 0;
}
// Skip Unicode byte order mark if present
if (cmd_line[0] == u'\uFEFF') {
cmd_line = &cmd_line[1];
@@ -193,6 +197,10 @@ static int get_cmd_line_length(efi_loaded_image_t *image)
static void get_cmd_line(efi_loaded_image_t *image, int num_chars, char *buffer)
{
efi_char16_t *cmd_line = (efi_char16_t *)image->load_options;
if (cmd_line == NULL || num_chars == 0) {
buffer[0] = '\0';
return;
}
if (cmd_line[0] == u'\uFEFF') {
cmd_line = &cmd_line[1];
}
@@ -810,6 +818,8 @@ fail:
__asm__("hlt");
#elif defined(__loongarch_lp64)
__asm__("idle 0");
#elif defined(__aarch64__)
__asm__("wfi");
#endif
}
}
build/aarch64/Makefile
+276
View File
@@ -0,0 +1,276 @@
# Default to the cross toolchain when not building on an AArch64 host.
# Overridable from the command line (make CC=... LD=... OBJCOPY=...).
ifeq ($(shell uname -m),aarch64)
CC = gcc
LD = ld
OBJCOPY = objcopy
else
CC = aarch64-linux-gnu-gcc
LD = aarch64-linux-gnu-ld
OBJCOPY = aarch64-linux-gnu-objcopy
endif
GIT ?= git
ifeq ($(GIT),none)
GIT_AVAILABLE = false
else
GIT_AVAILABLE = true
endif
CFLAGS = -std=gnu11 -Wall -Wextra -Wshadow -march=armv8-a -mno-outline-atomics -fpic \
-fno-builtin -ffreestanding -fomit-frame-pointer -fno-stack-protector -DARCH_BITS=64
LDFLAGS = -z max-page-size=4096
ifeq ($(DEBUG), 1)
CFLAGS+=-ggdb3 -DDEBUG_GDB
OPT_SMALL=-Og
OPT_FAST=-Og
MS_LDS=memtest_shared_debug.lds
else
OPT_SMALL=-Os
OPT_FAST=-O3
MS_LDS=ldscripts/memtest_shared.lds
endif
INC_DIRS = -I../../boot -I../../system -I ../../system/imc -I../../system/aarch64 -I../../lib -I../../tests -I../../app -Iapp
SYS_OBJS = system/acpi.o \
system/cpulocal.o \
system/ehci.o \
system/fat32.o \
system/font.o \
system/heap.o \
system/hwquirks.o \
system/keyboard.o \
system/ohci.o \
system/pmem.o \
system/reloc.o \
system/screen.o \
system/serial.o \
system/smbios.o \
system/spd.o \
system/smp.o \
system/timers.o \
system/uhci.o \
system/usbhcd.o \
system/usbmsd.o \
system/xhci.o \
system/aarch64/i2c.o \
system/aarch64/cpuid.o \
system/aarch64/cpuinfo.o \
system/aarch64/hwctrl.o \
system/aarch64/memctrl.o \
system/aarch64/simd.o \
system/aarch64/pci.o \
system/aarch64/psci.o \
system/aarch64/temperature.o \
system/aarch64/vmem.o
IMC_SRCS = $(wildcard ../../system/imc/aarch64/*.c)
IMC_OBJS = $(subst ../../,,$(IMC_SRCS:.c=.o))
LIB_OBJS = lib/barrier.o \
lib/print.o \
lib/read.o \
lib/string.o \
lib/unistd.o
TST_OBJS = tests/addr_walk1.o \
tests/bit_fade.o \
tests/block_move.o \
tests/bus_stress.o \
tests/modulo_n.o \
tests/mov_inv_fixed.o \
tests/mov_inv_rng.o \
tests/vec_prsg.o \
tests/aarch64/vec_prsg_neon.o \
tests/mov_inv_walk1.o \
tests/own_addr.o \
tests/rowhammer.o \
tests/test_helper.o \
tests/tests.o
APP_OBJS = app/badram.o \
app/config.o \
app/display.o \
app/error.o \
app/main.o \
app/reports.o \
app/aarch64/interrupt.o
OBJS = boot/startup.o boot/efisetup.o $(SYS_OBJS) $(IMC_OBJS) $(LIB_OBJS) $(TST_OBJS) $(APP_OBJS)
all: mt86plus
check:
@if [ -z ${DEBUG} ]; then\
echo "Macro DEBUG is not defined. Run debug_memtest.sh to invoke debug target";\
exit 1;\
fi
debug: check memtest.debug
-include boot/efisetup.d
-include $(subst .o,.d,$(SYS_OBJS))
-include $(subst .o,.d,$(IMC_OBJS))
-include $(subst .o,.d,$(LIB_OBJS))
-include $(subst .o,.d,$(TST_OBJS))
-include $(subst .o,.d,$(APP_OBJS))
boot/header.o : | ../../boot/sbat.csv
boot/startup.o: ../../boot/aarch64/startup.S ../../boot/boot.h
@mkdir -p boot
$(CC) -x assembler-with-cpp -c -I../../boot -I../../system/aarch64 -o $@ $<
boot/%.o: ../../boot/%.S ../../boot/boot.h app/build_version.h
@mkdir -p boot
$(CC) -x assembler-with-cpp -c -I../../boot -Iapp -o $@ $<
boot/aarch64/%.o: ../../boot/aarch64/%.S ../../boot/boot.h app/build_version.h
@mkdir -p boot/aarch64
$(CC) -x assembler-with-cpp -c -I../../boot -Iapp -o $@ $<
boot/efisetup.o: ../../boot/efisetup.c
@mkdir -p boot
$(CC) -c $(CFLAGS) $(OPT_SMALL) $(INC_DIRS) -o $@ $< -MMD -MP -MT $@ -MF $(@:.o=.d)
system/reloc.o: ../../system/reloc64.c
@mkdir -p system
$(CC) -c $(CFLAGS) -fno-strict-aliasing $(OPT_SMALL) $(INC_DIRS) -o $@ $< -MMD -MP -MT $@ -MF $(@:.o=.d)
system/%.o: ../../system/%.c
@mkdir -p system
$(CC) -c $(CFLAGS) $(OPT_SMALL) $(INC_DIRS) -o $@ $< -MMD -MP -MT $@ -MF $(@:.o=.d)
system/aarch64/%.o: ../../system/aarch64/%.c
@mkdir -p system/aarch64/
$(CC) -c $(CFLAGS) $(OPT_SMALL) $(INC_DIRS) -o $@ $< -MMD -MP -MT $@ -MF $(@:.o=.d)
system/imc/aarch64/%.o: ../../system/imc/aarch64/%.c
@mkdir -p system/imc/aarch64/
$(CC) -c $(CFLAGS) $(OPT_SMALL) $(INC_DIRS) -o $@ $< -MMD -MP -MT $@ -MF $(@:.o=.d)
lib/%.o: ../../lib/%.c
@mkdir -p lib
$(CC) -c $(CFLAGS) $(OPT_SMALL) $(INC_DIRS) -o $@ $< -MMD -MP -MT $@ -MF $(@:.o=.d)
tests/%.o: ../../tests/%.c
@mkdir -p tests
$(CC) -c $(CFLAGS) $(OPT_FAST) $(INC_DIRS) -o $@ $< -MMD -MP -MT $@ -MF $(@:.o=.d)
tests/aarch64/%.o: ../../tests/aarch64/%.c
@mkdir -p tests/aarch64
$(CC) -c $(CFLAGS) $(OPT_FAST) $(INC_DIRS) -o $@ $< -MMD -MP -MT $@ -MF $(@:.o=.d)
app/%.o: ../../app/%.c app/build_version.h
@mkdir -p app
$(CC) -c $(CFLAGS) $(OPT_SMALL) $(INC_DIRS) -o $@ $< -MMD -MP -MT $@ -MF $(@:.o=.d)
app/aarch64/%.o: ../../app/aarch64/%.c app/build_version.h
@mkdir -p app/aarch64
$(CC) -c $(CFLAGS) $(OPT_SMALL) $(INC_DIRS) -o $@ $< -MMD -MP -MT $@ -MF $(@:.o=.d)
app/build_version.h: FORCE
@mkdir -p app
@( \
cp -f ../../app/version.h $@.tmp; \
if $(GIT_AVAILABLE) && test -d ../../.git ; then \
hash=`git rev-parse HEAD | cut -c1-7`; \
sed -i 's/GIT_HASH\s\".*"/GIT_HASH "'$$hash'"/' $@.tmp; \
else \
sed -i 's/GIT_HASH\s\".*"/GIT_HASH "unknown"/' $@.tmp; \
fi; \
cmp $@ $@.tmp 2>/dev/null || cp -f $@.tmp $@; \
rm -f $@.tmp; \
)
FORCE:
# Link it statically once so I know I don't have undefined symbols and
# then link it dynamically so I have full relocation information.
memtest_shared: $(OBJS) $(MS_LDS) Makefile
$(LD) $(LDFLAGS) --warn-common -static -T $(MS_LDS) -o $@ $(OBJS) && \
$(LD) $(LDFLAGS) -shared -Bsymbolic -T $(MS_LDS) -o $@ $(OBJS)
memtest_shared.bin: memtest_shared
$(OBJCOPY) -O binary $< memtest_shared.bin
memtest.debug: memtest_shared
objcopy --only-keep-debug memtest_shared memtest.debug
strip -R .eh_frame memtest_shared
strip -R .comment memtest_shared
# Build the PE/EFI binary with W^X sections
# The flat binary is split at the _etext_ro boundary (page-aligned in
# memtest_shared.lds) into an RX half (.text) and an RW half (.data)
# Both are then used by ld as raw binary inputs. The EFI linker script
# push them into separate PE sections via filename match
mt86plus: memtest_shared.bin boot/aarch64/header.o ldscripts/memtest_efi.lds
$(eval SIZES=$(shell size -B -d memtest_shared | grep memtest_shared))
$(eval SPLIT=$(shell nm -t d memtest_shared | awk '/ _etext_ro$$/{print $$1+0}'))
dd if=memtest_shared.bin of=memtest_text.bin bs=$(SPLIT) count=1 2>/dev/null
dd if=memtest_shared.bin of=memtest_data.bin bs=$(SPLIT) skip=1 2>/dev/null
$(LD) --defsym=_bss_size=$(word 3,$(SIZES)) -T ldscripts/memtest_efi.lds boot/aarch64/header.o -b binary memtest_text.bin memtest_data.bin -o mt86plus.elf
$(OBJCOPY) -O binary mt86plus.elf mt86plus
esp.img: mt86plus
@mkdir -p iso/EFI/BOOT
cp mt86plus iso/EFI/BOOT/BOOTAA64.EFI
@rm -f esp.img
/sbin/mkdosfs -n MEMTEST-ESP -F12 -C esp.img 4096
mcopy -s -i esp.img iso/EFI ::
memtest.iso: esp.img
xorrisofs -pad -R -J -volid MT86PLUS_64_EFI_ONLY -no-emul-boot -graft-points \
-append_partition 2 0xef ./esp.img \
-o ./memtest.iso /EFI=./iso/EFI
iso: memtest.iso
clean:
rm -rf boot system lib tests app *.img *.iso *.elf memtest* mt86plus iso grub-*
# grub-memtest.iso uses GRUB as an intermediate bootloader to allow Memtest86+
# to be started with the native USB keyboard drivers either enabled or disabled,
# or to be started in a fail-safe mode with SMP and memory identification &
# speed testing disabled.
GRUB_CFG ?= grub
GRUB_FONT_DIR ?= /usr/share/grub
GRUB_LIB_DIR ?= /usr/lib/grub
GRUB_MKIMAGE := $(shell command -v grub2-mkimage || command -v grub-mkimage)
GRUB_MODULES = iso9660 fat part_msdos part_gpt all_video font gfxterm gfxmenu \
boot chain configfile echo ls
grub-bootaa64.efi:
$(GRUB_MKIMAGE) --output $@ --prefix /EFI/BOOT/grub --format arm64-efi $(GRUB_MODULES)
grub-esp.img: mt86plus grub-bootaa64.efi ../../grub/${GRUB_CFG}-efi.cfg
@mkdir -p grub-iso/EFI/BOOT/grub/arm64-efi grub-iso/EFI/BOOT/grub/fonts
cp mt86plus grub-iso/EFI/BOOT/memtest
cp grub-bootaa64.efi grub-iso/EFI/BOOT/BOOTAA64.EFI
cp ../../grub/${GRUB_CFG}-efi.cfg grub-iso/EFI/BOOT/grub/grub.cfg
cp $(GRUB_FONT_DIR)/unicode.pf2 grub-iso/EFI/BOOT/grub/fonts/
cp $(GRUB_LIB_DIR)/arm64-efi/*.mod grub-iso/EFI/BOOT/grub/arm64-efi/
@rm -f grub-esp.img
/sbin/mkdosfs -n MT86P_ESP -F12 -C grub-esp.img 8192
mcopy -s -i grub-esp.img grub-iso/EFI ::
grub-memtest.iso: mt86plus grub-esp.img
@mkdir -p grub-iso/boot/grub/fonts
cp $(GRUB_FONT_DIR)/unicode.pf2 grub-iso/boot/grub/fonts/
xorrisofs -pad -R -J -volid MT86PLUS_64_EFI_ONLY -no-emul-boot -graft-points \
-boot-load-size 4 -boot-info-table --grub2-boot-info \
-append_partition 2 0xef ./grub-esp.img \
-o ./grub-memtest.iso /EFI=./grub-iso/EFI
grub-iso: grub-memtest.iso
build/aarch64/ldscripts/memtest_efi.lds
+93
View File
@@ -0,0 +1,93 @@
OUTPUT_FORMAT("elf64-littleaarch64")
OUTPUT_ARCH(aarch64)
ENTRY(head);
/* This ELF is only a staging container that gets objcopy'd to the flat PE
image; the PE section table in boot/aarch64/header.S is what defines the
run-time permissions. Explicit PHDRS keep the staging segments W^X so ld
doesn't warn about an RWX LOAD segment. */
PHDRS {
header PT_LOAD FLAGS(4); /* R__ */
text PT_LOAD FLAGS(5); /* R_E */
data PT_LOAD FLAGS(6); /* RW_ */
rodata PT_LOAD FLAGS(4); /* R__ */
}
SECTIONS {
. = 0;
.header : {
*(.header)
} :header
. = ALIGN(4096);
.text : {
_file_text_start = . ;
*memtest_text*(.data)
_real_text_end = . ;
. = ALIGN(512);
_file_text_end = . ;
} :text
.data : {
_file_data_start = . ;
*memtest_data*(.data)
_real_data_end = . ;
. = ALIGN(512);
_file_data_end = . ;
} :data
.reloc : {
_file_reloc_start = . ;
*(.reloc)
_real_reloc_end = . ;
. = ALIGN(512);
_file_reloc_end = . ;
} :rodata
.sbat : {
_file_sbat_start = . ;
*(.sbat)
_real_sbat_end = . ;
. = ALIGN(512);
_file_sbat_end = . ;
} :rodata
.osrel : {
_file_osrel_start = . ;
*(.osrel)
_real_osrel_end = . ;
. = ALIGN(512);
_file_osrel_end = . ;
} :rodata
.shstrtab : { *(.shstrtab) }
/DISCARD/ : { *(*) }
_real_text_size = _real_text_end - _file_text_start;
_real_data_size = _real_data_end - _file_data_start;
_real_reloc_size = _real_reloc_end - _file_reloc_start;
_real_sbat_size = _real_sbat_end - _file_sbat_start;
_real_osrel_size = _real_osrel_end - _file_osrel_start;
_file_head_size = _file_text_start;
_file_text_size = _file_text_end - _file_text_start;
_file_data_size = _file_data_end - _file_data_start;
_file_reloc_size = _file_reloc_end - _file_reloc_start;
_file_sbat_size = _file_sbat_end - _file_sbat_start;
_file_osrel_size = _file_osrel_end - _file_osrel_start;
_sys_size = (_real_text_size + _real_data_size + 15) >> 4;
_init_size = _real_text_size + _real_data_size + _bss_size;
_virt_head_size = ((_file_head_size + 4095) >> 12) << 12;
_virt_text_size = ((_real_text_size + 4095) >> 12) << 12;
_virt_data_size = ((_real_data_size + _bss_size + 4095) >> 12) << 12;
_virt_reloc_size = ((_file_reloc_size + 4095) >> 12) << 12;
_virt_sbat_size = ((_file_sbat_size + 4095) >> 12) << 12;
_virt_osrel_size = ((_file_osrel_size + 4095) >> 12) << 12;
_virt_text_start = _virt_head_size;
_virt_data_start = _virt_text_start + _virt_text_size;
_virt_reloc_start = _virt_data_start + _virt_data_size;
_virt_sbat_start = _virt_reloc_start + _virt_reloc_size;
_virt_osrel_start = _virt_sbat_start + _virt_sbat_size;
_virt_img_size = _virt_osrel_start + _virt_osrel_size;
_virt_init_size = _virt_data_size + _virt_reloc_size + _virt_sbat_size + _virt_osrel_size;
}
build/aarch64/ldscripts/memtest_shared.lds
+76
View File
@@ -0,0 +1,76 @@
OUTPUT_FORMAT("elf64-littleaarch64")
OUTPUT_ARCH(aarch64);
ENTRY(startup64);
/* Explicit PHDRS so ld doesn't auto-merge the read-execute and read-write
sections into a single RWX LOAD segment. The output ELF is later
objcopy'd to a flat binary, so these segments are purely cosmetic, but
any warning is silenced and the layout is explicit. */
PHDRS {
text PT_LOAD FLAGS(5); /* R_E */
data PT_LOAD FLAGS(6); /* RW_ */
dynamic PT_DYNAMIC FLAGS(6);
}
SECTIONS {
. = 0;
.text : {
_start = .;
*(.text)
*(.text.*)
*(.plt)
_etext = . ;
} :text = 0x00000000
.rodata : {
*(.rodata)
*(.rodata.*)
} :text
.dynsym : { *(.dynsym) } :text
.dynstr : { *(.dynstr) } :text
.hash : { *(.hash) } :text
.gnu.hash : { *(.gnu.hash) } :text
.shstrtab : { *(.shstrtab) }
.rela.dyn : { *(.rela.dyn) } :text
.rela.text : { *(.rela.text .rela.text.*) } :text
.rela.rodata : { *(.rela.rodata .rela.rodata.*) } :text
.rela.data : { *(.rela.data .rela.data.*) } :text
.rela.got : { *(.rela.got .rela.got.*) } :text
.rela.plt : { *(.rela.plt .rela.plt.*) } :text
/* Page-align the boundary between read-only and writable sections.
PE SectionAlignment is 4096, so this ensures the split point
matches the PE virtual address layout exactly. */
. = ALIGN(4096);
_etext_ro = .;
.data : {
_data = .;
*(.data)
*(.data.*)
} :data
.got : {
*(.got.plt)
*(.got)
} :data
/* Define _DYNAMIC here so the sanity-check static link (which has no
dynamic section) still resolves the reference in reloc64.c. */
.dynamic : { _DYNAMIC = .; *(.dynamic) } :data :dynamic
_edata = . ;
. = ALIGN(4);
.bss : {
_bss = .;
*(.dynbss)
*(.bss)
*(.bss.*)
*(COMMON)
. = ALIGN(16);
_stacks = .;
*(.stacks)
/* _end must be at least 256 byte aligned */
. = ALIGN(256);
_end = .;
} :data
/DISCARD/ : { *(*) }
}
lib/assert.h
+2
View File
@@ -22,6 +22,8 @@ static inline void assert(int expr)
__asm__ __volatile__ ("int $3");
#elif defined(__loongarch_lp64)
__asm__ __volatile__ ("break 0");
#elif defined(__aarch64__)
__asm__ __volatile__ ("brk #0");
#endif
}
}
lib/barrier.c
+12
View File
@@ -58,6 +58,8 @@ void barrier_spin_wait(barrier_t *barrier)
"nop \n\t" \
"nop \n\t" \
);
#elif defined(__aarch64__)
__asm__ __volatile__ ("yield");
#endif
}
return;
@@ -119,6 +121,16 @@ void barrier_halt_wait(barrier_t *barrier)
: "$t0", "t1", "$t2"
: end
);
#elif defined(__aarch64__)
// On ARM64, use event instead of an interrupt. The event register is sticky,
// so SEV before WFE still wakes immediately, avoiding missed wakeups.
if (__sync_sub_and_fetch(&barrier->count, 1) != 0) {
volatile bool *i_am_blocked = &waiting_flags[my_cpu].flag;
while (*i_am_blocked) {
__asm__ __volatile__ ("wfe" ::: "memory");
}
goto end;
}
#endif
// Last one here, so reset the barrier and wake the others.
barrier->count = barrier->num_threads;
lib/spinlock.h
+10 -2
View File
@@ -34,6 +34,14 @@ static inline void cpu_pause (void)
"nop \n\t" \
);
}
#elif defined(__aarch64__)
/**
* ARM64 CPU pause.
*/
static inline void cpu_pause (void)
{
__asm__ __volatile__ ("yield");
}
#endif
/**
@@ -45,7 +53,7 @@ static inline void spin_wait(spinlock_t *lock)
while (*lock) {
#if defined(__x86_64) || defined(__i386__)
__builtin_ia32_pause();
#elif defined (__loongarch_lp64)
#elif defined (__loongarch_lp64) || defined(__aarch64__)
cpu_pause();
#endif
}
@@ -62,7 +70,7 @@ static inline void spin_lock(spinlock_t *lock)
do {
#if defined(__x86_64) || defined(__i386__)
__builtin_ia32_pause();
#elif defined (__loongarch_lp64)
#elif defined (__loongarch_lp64) || defined(__aarch64__)
cpu_pause();
#endif
} while (*lock);
lib/string.c
+5 -1
View File
@@ -53,7 +53,7 @@ void *memmove(void *dest, const void *src, size_t n)
return dest;
}
#if (defined(DEBUG_GDB) || defined(__loongarch_lp64) || defined(__clang__))
#if (defined(DEBUG_GDB) || defined(__loongarch_lp64) || defined(__aarch64__) || defined(__clang__))
void *memcpy (void *dest, const void *src, size_t len)
{
@@ -79,6 +79,10 @@ char *strstr(const char *haystack, const char *needle)
size_t haystack_len = strlen(haystack);
size_t needle_len = strlen(needle);
if (needle_len > haystack_len) {
return NULL;
}
size_t max_idx = haystack_len - needle_len;
for (size_t idx = 0; idx <= max_idx; idx++) {
lib/string.h
+2 -2
View File
@@ -36,7 +36,7 @@ static inline int memcmp(const void *s1, const void *s2, size_t n)
* not overlap.
* void *memcpy(void *dst, const void *src, size_t n);
*/
#if !(defined(DEBUG_GDB) || defined(__loongarch_lp64) || defined(__clang__))
#if !(defined(DEBUG_GDB) || defined(__loongarch_lp64) || defined(__aarch64__) || defined(__clang__))
#define memcpy(d, s, n) __builtin_memcpy((d), (s), (n))
#else
void *memcpy (void *dest, const void *src, size_t len);
@@ -54,7 +54,7 @@ void *memmove(void *dest, const void *src, size_t n);
* value c.
* void *memset(void *s, int c, size_t n);
*/
#if !(defined(DEBUG_GDB) || defined(__loongarch_lp64) || defined(__clang__))
#if !(defined(DEBUG_GDB) || defined(__loongarch_lp64) || defined(__aarch64__) || defined(__clang__))
#define memset(s, c, n) __builtin_memset((s), (c), (n))
#else
void *memset (void *dest, int val, size_t len);
lib/unistd.c
+3 -1
View File
@@ -32,10 +32,12 @@ void usleep(unsigned int usec)
"nop \n\t" \
"nop \n\t" \
);
#elif defined(__aarch64__)
__asm__ __volatile__ ("yield");
#endif
} while ((get_tsc() - t0) < cycles);
} else {
// This will be highly inaccurate, but should give at least the requested delay.
// Highly inaccurate, but should give *at least* the requested delay
volatile uint64_t count = (uint64_t)usec * 1000;
while (count > 0) {
count--;
system/aarch64/cpuid.c
+265
View File
@@ -0,0 +1,265 @@
// SPDX-License-Identifier: GPL-2.0
// Copyright (C) 2026 Sam Demeulemeester
//
// ARM64 (AArch64) CPU identification via the MIDR_EL1 register
#include <stdbool.h>
#include <stdint.h>
#include "cpuid.h"
#include "string.h"
#include "registers.h"
//------------------------------------------------------------------------------
// Types
//------------------------------------------------------------------------------
typedef struct {
uint16_t part_num;
const char name[16]; // 15 chars + NUL max; GCC < 15 doesn't warn if the NUL gets dropped
} cpu_part_t;
//------------------------------------------------------------------------------
// Constants
//------------------------------------------------------------------------------
static const cpu_part_t arm_parts[] = {
{ 0xD00, "Foundation" },
{ 0xD03, "Cortex-A53" },
{ 0xD04, "Cortex-A35" },
{ 0xD05, "Cortex-A55" },
{ 0xD07, "Cortex-A57" },
{ 0xD08, "Cortex-A72" },
{ 0xD09, "Cortex-A73" },
{ 0xD0A, "Cortex-A75" },
{ 0xD0B, "Cortex-A76" },
{ 0xD0C, "Neoverse N1" },
{ 0xD0D, "Cortex-A77" },
{ 0xD0E, "Cortex-A76AE" },
{ 0xD0F, "AEMv8" },
{ 0xD40, "Neoverse V1" },
{ 0xD41, "Cortex-A78" },
{ 0xD42, "Cortex-A78AE" },
{ 0xD44, "Cortex-X1" },
{ 0xD46, "Cortex-A510" },
{ 0xD47, "Cortex-A710" },
{ 0xD48, "Cortex-X2" },
{ 0xD49, "Neoverse N2" },
{ 0xD4B, "Cortex-A78C" },
{ 0xD4C, "Cortex-X1C" },
{ 0xD4D, "Cortex-A715" },
{ 0xD4E, "Cortex-X3" },
{ 0xD4F, "Neoverse V2" },
{ 0xD80, "Cortex-A520" },
{ 0xD81, "Cortex-A720" },
{ 0xD82, "Cortex-X4" },
{ 0xD83, "Neoverse V3AE"},
{ 0xD84, "Neoverse V3" },
{ 0xD85, "Cortex-X925" },
{ 0xD87, "Cortex-A725" },
{ 0xD89, "Cortex-A720AE"},
{ 0xD8E, "Neoverse N3" },
{ 0, "" }
};
static const cpu_part_t qcom_parts[] = {
{ 0x001, "Oryon X1" },
{ 0x800, "Kryo 2xx Au" },
{ 0x801, "Kryo 2xx Ag" },
{ 0x802, "Kryo 3xx Au" },
{ 0x803, "Kryo 3xx Ag" },
{ 0x804, "Kryo 4xx Au" },
{ 0x805, "Kryo 4xx Ag" },
{ 0xC00, "Falkor" },
{ 0, "" }
};
static const cpu_part_t apple_parts[] = {
{ 0x022, "M1 Icestorm" },
{ 0x023, "M1 Firestorm" },
{ 0x024, "M1 Ice Pro" },
{ 0x025, "M1 Fire Pro" },
{ 0x028, "M1 Ice Max" },
{ 0x029, "M1 Fire Max" },
{ 0x032, "M2 Blizzard" },
{ 0x033, "M2 Avalanche" },
{ 0x034, "M2 Bliz Pro" },
{ 0x035, "M2 Aval Pro" },
{ 0x038, "M2 Bliz Max" },
{ 0x039, "M2 Aval Max" },
{ 0x042, "M3 Sawtooth" },
{ 0x043, "M3 Everest" },
{ 0x044, "M3 Saw Pro" },
{ 0x045, "M3 Ever Pro" },
{ 0x048, "M3 Saw Max" },
{ 0x049, "M3 Ever Max" },
{ 0x052, "M4 Sawtooth" },
{ 0x053, "M4 Everest" },
{ 0x054, "M4 Saw Pro" },
{ 0x055, "M4 Ever Pro" },
{ 0x058, "M4 Saw Max" },
{ 0x059, "M4 Ever Max" },
{ 0, "" }
};
static const cpu_part_t nvidia_parts[] = {
{ 0x000, "Denver" },
{ 0x003, "Denver 2" },
{ 0x004, "Carmel" },
{ 0x010, "Olympus" },
{ 0, "" }
};
static const cpu_part_t microsoft_parts[] = {
{ 0xD49, "Cobalt 100" },
{ 0, "" }
};
static const cpu_part_t cavium_parts[] = {
{ 0x0A1, "ThunderX" },
{ 0x0A2, "ThunderX 81" },
{ 0x0A3, "ThunderX 83" },
{ 0x0AF, "ThunderX2" },
{ 0x0B1, "OcteonTx2 98" },
{ 0x0B2, "OcteonTx2 96" },
{ 0x0B3, "OcteonTx2 95" },
{ 0x0B4, "OcteonTx2 95N"},
{ 0x0B5, "OcteonTx2 95M"},
{ 0x0B6, "OcteonTx2 95O"},
{ 0, "" }
};
//------------------------------------------------------------------------------
// Public Variables
//------------------------------------------------------------------------------
cpuid_info_t cpuid_info;
//------------------------------------------------------------------------------
// Private Functions
//------------------------------------------------------------------------------
static const char *lookup_part(const cpu_part_t *parts, uint16_t part_num)
{
for (int i = 0; parts[i].name[0] != '\0'; i++) {
if (parts[i].part_num == part_num) {
return parts[i].name;
}
}
return NULL;
}
static size_t append_str(char *buffer, size_t offset, size_t size, const char *str)
{
while (*str != '\0' && offset < (size - 1)) {
buffer[offset++] = *str++;
}
buffer[offset] = '\0';
return offset;
}
static size_t append_hex(char *buffer, size_t offset, size_t size, uint32_t value, int digits)
{
for (int i = digits - 1; i >= 0 && offset < (size - 1); i--) {
uint32_t digit = (value >> (i * 4)) & 0xF;
buffer[offset++] = digit < 10 ? '0' + digit : 'A' + digit - 10;
}
buffer[offset] = '\0';
return offset;
}
//------------------------------------------------------------------------------
// Public Functions
//------------------------------------------------------------------------------
void cpuid_init(void)
{
uint64_t midr = read_sysreg(midr_el1);
const char *vendor = NULL;
const char *part = NULL;
switch (MIDR_IMPLEMENTER(midr)) {
case 0x41:
vendor = "Arm";
part = lookup_part(arm_parts, MIDR_PART_NUM(midr));
break;
case 0x51:
vendor = "Qualcomm";
part = lookup_part(qcom_parts, MIDR_PART_NUM(midr));
break;
case 0x61:
vendor = "Apple";
part = lookup_part(apple_parts, MIDR_PART_NUM(midr));
break;
case 0xC0:
vendor = "Ampere";
break;
case 0x4E:
vendor = "NVIDIA";
part = lookup_part(nvidia_parts, MIDR_PART_NUM(midr));
break;
case 0x46:
vendor = "Fujitsu";
break;
case 0x43:
vendor = "Cavium";
part = lookup_part(cavium_parts, MIDR_PART_NUM(midr));
break;
case 0x6D:
vendor = "Microsoft";
part = lookup_part(microsoft_parts, MIDR_PART_NUM(midr));
break;
case 0x00:
vendor = "QEMU";
break;
default:
vendor = NULL;
break;
}
(void)append_str(cpuid_info.vendor_id.str, 0, CPUID_VENDOR_STR_LENGTH,
vendor != NULL ? vendor : "Unknown");
// Build the brand string.
char *brand = cpuid_info.brand_id.str;
size_t brand_size = CPUID_BRAND_STR_LENGTH;
size_t len = 0;
brand[0] = '\0';
if (vendor != NULL) {
len = append_str(brand, len, brand_size, vendor);
len = append_str(brand, len, brand_size, " ");
}
if (part != NULL) {
len = append_str(brand, len, brand_size, part);
} else {
// Fall back to the raw part number.
len = append_str(brand, len, brand_size, "0x");
len = append_hex(brand, len, brand_size, MIDR_PART_NUM(midr), 3);
}
cpuid_info.version.raw[0] = (uint32_t)midr;
cpuid_info.version.stepping = MIDR_REVISION(midr);
cpuid_info.flags.htt = false;
// CNTVCT_EL0 (read by get_tsc()) is always available, so advertise a TSC;
// this lets the random-pattern tests seed their PRSG from a live counter.
cpuid_info.flags.rdtsc = true;
cpuid_info.topology.core_count = -1;
cpuid_info.topology.thread_count = -1;
cpuid_info.topology.is_hybrid = 0;
cpuid_info.topology.ecore_count = -1;
cpuid_info.topology.pcore_count = -1;
}
core_type_t get_ap_hybrid_type(void)
{
return CORE_PCORE;
}
system/aarch64/cpuinfo.c
+280
View File
@@ -0,0 +1,280 @@
// SPDX-License-Identifier: GPL-2.0
// Copyright (C) 2020-2022 Martin Whitaker.
// Copyright (C) 2004-2026 Sam Demeulemeester
//
// ARM64 (AArch64) CPU information.
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
#include "cpuid.h"
#include "tsc.h"
#include "boot.h"
#include "config.h"
#include "pmem.h"
#include "memsize.h"
#include "cpuinfo.h"
#include "registers.h"
//------------------------------------------------------------------------------
// Constants
//------------------------------------------------------------------------------
// RAM may start well above address 0 on this architecture, so the minimum
// benchmark address is an offset from the base of RAM, not an absolute adr.
#define BENCH_MIN_START_OFFSET 0x10000000 // 256MB
//------------------------------------------------------------------------------
// Public Variables
//------------------------------------------------------------------------------
const char *cpu_model = ""; // static init required: creates the reloc record that rebases it
int l1_cache = 0;
int l2_cache = 0;
int l3_cache = 0;
uint32_t l1_cache_speed = 0;
uint32_t l2_cache_speed = 0;
uint32_t l3_cache_speed = 0;
uint32_t ram_speed = 0;
uint32_t clks_per_msec = 0;
uint32_t cpu_clk_mhz = 0;
//------------------------------------------------------------------------------
// Private Functions
//------------------------------------------------------------------------------
static void determine_cache_size(void)
{
uint64_t clidr = read_sysreg(clidr_el1);
bool has_ccidx = ((read_sysreg(id_aa64mmfr2_el1) >> 20) & 0xF) != 0;
for (int level = 1; level <= 3; level++) {
int cache_type = (clidr >> (3 * (level - 1))) & 0x7;
if (cache_type == 0) {
break;
}
// Select the data or unified cache at this level
write_sysreg((level - 1) << 1, csselr_el1);
__asm__ __volatile__ ("isb");
uint64_t ccsidr = read_sysreg(ccsidr_el1);
uint64_t line_size, ways, sets;
line_size = UINT64_C(1) << ((ccsidr & 0x7) + 4);
if (has_ccidx) {
ways = ((ccsidr >> 3) & 0x1FFFFF) + 1;
sets = ((ccsidr >> 32) & 0xFFFFFF) + 1;
} else {
ways = ((ccsidr >> 3) & 0x3FF) + 1;
sets = ((ccsidr >> 13) & 0x7FFF) + 1;
}
int size = (line_size * ways * sets) / 1024;
switch (level) {
case 1:
l1_cache = size;
break;
case 2:
l2_cache = size;
break;
case 3:
l3_cache = size;
break;
}
}
}
// Copy wlen * 64 bytes from src to dst. With wlen == 0, just executes the
// loop entry/exit overhead.
static inline void copy_block(uintptr_t src, uintptr_t dst, uintptr_t wlen)
{
__asm__ __volatile__ (
"mov x9, %0\n\t"
"mov x10, %1\n\t"
"mov x11, %2\n\t"
"2:\n\t"
"cbz x11, 1f\n\t"
"ldp x12, x13, [x9]\n\t"
"ldp x14, x15, [x9, #16]\n\t"
"stp x12, x13, [x10]\n\t"
"stp x14, x15, [x10, #16]\n\t"
"ldp x12, x13, [x9, #32]\n\t"
"ldp x14, x15, [x9, #48]\n\t"
"stp x12, x13, [x10, #32]\n\t"
"stp x14, x15, [x10, #48]\n\t"
"add x9, x9, #64\n\t"
"add x10, x10, #64\n\t"
"sub x11, x11, #1\n\t"
"b 2b\n\t"
"1:\n\t"
:: "r" (src), "r" (dst), "r" (wlen)
: "x9", "x10", "x11", "x12", "x13", "x14", "x15", "memory"
);
}
static void pmu_cycle_counter_enable(void)
{
write_sysreg(read_sysreg(pmcr_el0) | 0x5, pmcr_el0); // enable, reset cycle counter
write_sysreg(UINT64_C(1) << 31, pmcntenset_el0); // enable the cycle counter
write_sysreg(0, pmccfiltr_el0); // count cycles at EL1
__asm__ __volatile__ ("isb");
}
static void pmu_cycle_counter_disable(void)
{
write_sysreg(read_sysreg(pmcr_el0) & ~UINT64_C(1), pmcr_el0);
}
static uint32_t memspeed(uintptr_t src, uint32_t len, int iter)
{
uintptr_t dst;
uintptr_t wlen;
uint64_t start_time, end_time, run_time_clk, overhead;
int i;
// get_tsc() counts generic timer ticks (not CPU clocks, unlike
// clks_per_msec), so express time in generic timer ticks throughout.
uint32_t ticks_per_msec = read_sysreg(cntfrq_el0) / 1000;
if (ticks_per_msec == 0) {
return 0;
}
dst = src + len;
wlen = len / 64;
// Get number of clock cycles due to overhead
start_time = get_tsc();
for (i = 0; i < iter; i++) {
copy_block(src, dst, 0);
}
end_time = get_tsc();
overhead = (end_time - start_time);
// Prime the cache
copy_block(src, dst, wlen);
// Copy these bytes
uint64_t start_cycles = read_sysreg(pmccntr_el0);
start_time = get_tsc();
for (i = 0; i < iter; i++) {
copy_block(src, dst, wlen);
}
end_time = get_tsc();
uint64_t cycles = read_sysreg(pmccntr_el0) - start_cycles;
if ((end_time - start_time) > overhead) {
run_time_clk = (end_time - start_time) - overhead;
} else {
return 0;
}
// Suppress results when cycles roughly match timer ticks,
// indicating Snapdragon X core clock clamped near 19.2MHz under load
if (cpu_clk_mhz > 0 && cycles < 4 * run_time_clk) {
return 0;
}
run_time_clk = (((uint64_t)len * iter) / (double)run_time_clk) * ticks_per_msec * 2;
return run_time_clk;
}
static void measure_memory_bandwidth(void)
{
uintptr_t bench_start_adr = 0;
size_t mem_test_len;
if (l3_cache) {
mem_test_len = 4*l3_cache*1024;
} else if (l2_cache) {
mem_test_len = 4*l2_cache*1024;
} else {
return; // If we're not able to detect L2, don't start benchmark
}
uintptr_t bench_min_adr = (pm_map[0].start << PAGE_SHIFT) + BENCH_MIN_START_OFFSET;
// Locate enough free space for testing.
for (int i = 0; i < pm_map_size; i++) {
uintptr_t try_start = pm_map[i].start << PAGE_SHIFT;
uintptr_t try_end = try_start + mem_test_len * 2;
// No start address below bench_min_adr
if (try_start < bench_min_adr) {
if ((pm_map[i].end << PAGE_SHIFT) >= (bench_min_adr + mem_test_len * 2)) {
try_start = bench_min_adr;
try_end = bench_min_adr + mem_test_len * 2;
} else {
continue;
}
}
// Avoid the memory region where the program is currently located.
if (try_start < (uintptr_t)_end && try_end > (uintptr_t)_start) {
try_start = (uintptr_t)_end;
try_end = try_start + mem_test_len * 2;
}
uintptr_t end_limit = pm_map[i].end << PAGE_SHIFT;
if (try_end <= end_limit) {
bench_start_adr = try_start;
break;
}
}
if (bench_start_adr == 0) {
return;
}
pmu_cycle_counter_enable();
// Measure L1 BW using 1/3rd of the total L1 cache size
if (l1_cache) {
l1_cache_speed = memspeed(bench_start_adr, (l1_cache/3)*1024, 50);
}
// Measure L2 BW using half the L2 cache size
if (l2_cache) {
l2_cache_speed = memspeed(bench_start_adr, l2_cache/2*1024, 50);
}
// Measure L3 BW using half the L3 cache size
if (l3_cache) {
l3_cache_speed = memspeed(bench_start_adr, l3_cache/2*1024, 50);
}
// Measure RAM BW
ram_speed = memspeed(bench_start_adr, mem_test_len, 25);
#if defined(BENCH_DEBUG)
bench_diag_collect(bench_start_adr, mem_test_len);
#endif
pmu_cycle_counter_disable();
}
//------------------------------------------------------------------------------
// Public Functions
//------------------------------------------------------------------------------
void cpuinfo_init(void)
{
determine_cache_size();
cpu_model = cpuid_info.brand_id.str;
}
void membw_init(void)
{
if (enable_bench) {
measure_memory_bandwidth();
}
}
system/aarch64/hwctrl.c
+68
View File
@@ -0,0 +1,68 @@
// SPDX-License-Identifier: GPL-2.0
// Copyright (C) 2020-2022 Martin Whitaker.
// Copyright (C) 2026 Sam Demeulemeester
#include "stddef.h"
#include "boot.h"
#include "bootparams.h"
#include "efi.h"
#include "unistd.h"
#include "vmem.h"
#include "psci.h"
#include "hwctrl.h"
//------------------------------------------------------------------------------
// Private Variables
//------------------------------------------------------------------------------
static efi_runtime_services_t *efi_rs_table = NULL;
//------------------------------------------------------------------------------
// Public Functions
//------------------------------------------------------------------------------
void hwctrl_init(void)
{
boot_params_t *boot_params = (boot_params_t *)boot_params_addr;
if (boot_params->efi_info.loader_signature == EFI64_LOADER_SIGNATURE) {
uintptr_t system_table_addr = (uintptr_t)boot_params->efi_info.sys_tab_hi << 32 | boot_params->efi_info.sys_tab;
if (system_table_addr != 0) {
// The EFI tables live in runtime services memory, which is not
// included in our page tables, so map them before use.
system_table_addr = map_region(system_table_addr, sizeof(efi64_system_table_t), false);
if (system_table_addr != 0) {
efi64_system_table_t *sys_table = (efi64_system_table_t *)system_table_addr;
uintptr_t rs_table_addr = map_region(sys_table->runtime_services, sizeof(efi_runtime_services_t), false);
efi_rs_table = (efi_runtime_services_t *)rs_table_addr;
}
}
}
}
void reboot(void)
{
// Prefer PSCI. This only returns if PSCI is unavailable.
psci_system_reset();
if (efi_rs_table != NULL) {
efi_rs_table->reset_system(EFI_RESET_COLD, 0, 0);
usleep(1000000);
} else {
while (1);
}
}
void floppy_off()
{
// No floppy on this architecture.
}
void cursor_off()
{
// No text-mode cursor on this architecture.
}
system/aarch64/i2c.c
+20
View File
@@ -0,0 +1,20 @@
// SPDX-License-Identifier: GPL-2.0
// Copyright (C) 2004-2026 Sam Demeulemeester
//
// SPD/I2C access is not supported yet on ARM64.
#include <stdbool.h>
#include <stdint.h>
#include "i2c_x86.h"
uint8_t get_spd(uint8_t slot_idx __attribute__((unused)), uint16_t spd_adr __attribute__((unused)))
{
return 0;
}
int print_spd_startup_info(void)
{
// No SPD access on ARM64; return 0 so the caller uses the SMBIOS Type 17 info.
return 0;
}
system/aarch64/memctrl.c
+36
View File
@@ -0,0 +1,36 @@
// SPDX-License-Identifier: GPL-2.0
// Copyright (C) 2004-2026 Sam Demeulemeester
//
// ------------------------
//
// Platform-specific code for IMC configuration, ECC support, etc.
//
#include <stdbool.h>
#include "config.h"
#include "cpuinfo.h"
#include "memctrl.h"
#include "imc/imc.h"
#include "display.h"
imc_info_t imc = {"UNDEF", 0, 0, 0, 0, 0, 0, 0, 0};
ecc_info_t ecc_status = {false, ECC_ERR_NONE, 0, 0, 0, 0};
// ---------------------
// -- Public function --
// ---------------------
void memctrl_init(void)
{
// Memory controller detection and ECC are not supported yet on ARM64.
ecc_status.ecc_enabled = false;
}
void memctrl_poll_ecc(void)
{
// Nothing to do.
}
system/aarch64/pci.c
+346
View File
@@ -0,0 +1,346 @@
// SPDX-License-Identifier: GPL-2.0
// Copyright (C) 2020-2022 Martin Whitaker.
// Copyright (C) 2026 Sam Demeulemeester.
//
// PCI configuration space access on ARM64 via the ECAM region described by
// the ACPI MCFG table. If no MCFG table is present (eg platform devices
// only, all reads return all-1 and writes are ignored
#include <stdbool.h>
#include <stdint.h>
#include "acpi.h"
#include "memsize.h"
#include "mmio.h"
#include "pmem.h"
#include "vmem.h"
#include "registers.h"
#include "pci.h"
//------------------------------------------------------------------------------
// Types
//------------------------------------------------------------------------------
typedef struct __attribute__((packed)) {
uint64_t base_addr;
uint16_t segment;
uint8_t start_bus;
uint8_t end_bus;
uint32_t reserved;
} mcfg_entry_t;
//------------------------------------------------------------------------------
// Private Variables
//------------------------------------------------------------------------------
static uintptr_t ecam_base = 0;
static int ecam_start_bus = 0;
static int ecam_end_bus = -1;
static uint64_t ecam_phys = 0;
// Ranges handed out by pci_alloc_mmio, to avoid handing out overlaps
#define MAX_MMIO_ALLOCS 8
static struct {
uintptr_t base;
uintptr_t size;
} mmio_allocs[MAX_MMIO_ALLOCS];
static int num_mmio_allocs = 0;
//------------------------------------------------------------------------------
// Private Functions
//------------------------------------------------------------------------------
static void ecam_init(void);
static bool ecam_initialized = false;
static uintptr_t ecam_addr(int bus, int dev, int func, int reg)
{
// pci_init() is called before acpi_init(), so the MCFG table is not
// known at that point. Initialize lazily instead
if (!ecam_initialized) {
ecam_init();
}
if (ecam_base == 0 || bus < ecam_start_bus || bus > ecam_end_bus) {
return 0;
}
return ecam_base + (((uintptr_t)(bus - ecam_start_bus) << 20)
| ((uintptr_t)dev << 15)
| ((uintptr_t)func << 12)
| (reg & 0xFFF));
}
//------------------------------------------------------------------------------
// Public Functions
//------------------------------------------------------------------------------
static bool overlaps_ram(uintptr_t base, uintptr_t size)
{
for (int i = 0; i < pm_map_size; i++) {
uint64_t region_start = (uint64_t)pm_map[i].start << PAGE_SHIFT;
uint64_t region_end = (uint64_t)pm_map[i].end << PAGE_SHIFT;
if (base < region_end && region_start < (base + size)) {
return true;
}
}
return false;
}
// Allocate top-down within [win_start, win_end), aligned to size (BAR
// alignment equals BAR size), avoiding our own previous allocations.
// Firmware allocates bottom-up, so the top of a window is most likely free.
static uintptr_t alloc_in_window(uintptr_t win_start, uintptr_t win_end, uintptr_t size)
{
if (num_mmio_allocs == MAX_MMIO_ALLOCS || size == 0 || win_end < size) {
return 0;
}
uintptr_t candidate = (win_end - size) & ~(size - 1);
bool conflict = true;
while (conflict && candidate >= win_start) {
conflict = false;
for (int i = 0; i < num_mmio_allocs; i++) {
if (candidate < (mmio_allocs[i].base + mmio_allocs[i].size)
&& mmio_allocs[i].base < (candidate + size)) {
conflict = true;
candidate = (mmio_allocs[i].base - size) & ~(size - 1);
break;
}
}
}
if (candidate < win_start || overlaps_ram(candidate, size)) {
return 0;
}
mmio_allocs[num_mmio_allocs].base = candidate;
mmio_allocs[num_mmio_allocs].size = size;
num_mmio_allocs++;
return candidate;
}
// Find the bridge whose secondary bus is the given bus, and make sure its
// memory window and command register allow access to the devices behind it.
static bool find_bridge_window(int bus, uintptr_t *win_start, uintptr_t *win_end)
{
for (int b = 0; b < PCI_MAX_BUS; b++) {
for (int d = 0; d < PCI_MAX_DEV; d++) {
for (int f = 0; f < PCI_MAX_FUNC; f++) {
uint16_t vendor_id = pci_config_read16(b, d, f, 0x00);
uint8_t hdr_type = pci_config_read8 (b, d, f, 0x0e);
if (vendor_id == 0xffff) {
if (f == 0) break;
continue;
}
if ((hdr_type & 0x7f) == 1 && pci_config_read8(b, d, f, 0x19) == bus) {
uint16_t mem_base = pci_config_read16(b, d, f, 0x20);
uint16_t mem_limit = pci_config_read16(b, d, f, 0x22);
uintptr_t base = (uintptr_t)(mem_base & 0xFFF0) << 16;
uintptr_t limit = ((uintptr_t)(mem_limit & 0xFFF0) << 16) | 0xFFFFF;
if (base == 0 || base > limit) {
return false;
}
// Enable memory decode and bus mastering on the bridge.
uint16_t command = pci_config_read16(b, d, f, 0x04);
pci_config_write16(b, d, f, 0x04, command | 0x0006);
*win_start = base;
*win_end = limit + 1;
return true;
}
if (f == 0 && (hdr_type & 0x80) == 0) break;
}
}
}
return false;
}
uintptr_t pci_alloc_mmio(int bus, int dev, int func, int bar_reg, uintptr_t size)
{
if (ecam_base == 0 || size == 0 || (size & (size - 1)) != 0) {
return 0;
}
uintptr_t win_start = 0;
uintptr_t win_end = 0;
if (!find_bridge_window(bus, &win_start, &win_end)) {
// Device on a root bus: the host bridge MMIO aperture is only
// described in AML, which we can't parse. Use known windows.
uint32_t host_bridge_id = pci_config_read32(0, 0, 0, 0x00);
if (host_bridge_id == 0x00081b36) {
// Fixed low MMIO window for QEMU (DBG))
win_start = 0x10000000;
win_end = 0x3EFF0000;
} else if (ecam_phys != 0 && ecam_phys <= 0x100000000ULL) {
// Heuristic: on many platforms with a low ECAM, the MMIO window
// sits directly below it. overlaps_ram() guards against RAM.
win_end = ecam_phys;
win_start = win_end >= 0x10000000 ? win_end - 0x10000000 : 0;
} else {
return 0;
}
}
uintptr_t base = alloc_in_window(win_start, win_end, size);
if (base == 0) {
return 0;
}
// Program the BAR (and the upper half for a 64-bit BAR)
bool is_64bit = (pci_config_read32(bus, dev, func, bar_reg) & 0x6) == 0x4;
pci_config_write32(bus, dev, func, bar_reg, base);
if (is_64bit) {
pci_config_write32(bus, dev, func, bar_reg + 4, 0);
}
return base;
}
static void ecam_init(void)
{
if (acpi_config.mcfg_addr == 0) {
return;
}
ecam_initialized = true;
// Qualcomm PCIe root complexes are not ECAM-compliant: config space
// accesses to unclocked controllers or unimplemented devices stall the
// interconnect (hard hang, then watchdog reset), even though the
// firmware publishes an MCFG table. Nothing we need lives on PCI on
// these SoCs (USB is a platform device), so leave the ECAM disabled.
if (MIDR_IMPLEMENTER(read_sysreg(midr_el1)) == 0x51) {
return;
}
rsdt_header_t *mcfg = (rsdt_header_t *)map_region(acpi_config.mcfg_addr, sizeof(rsdt_header_t), true);
if (mcfg == NULL) return;
// Reject truncated tables, so the entry count below can't underflow.
if (mcfg->length < sizeof(rsdt_header_t) + 8) {
return;
}
mcfg = (rsdt_header_t *)map_region(acpi_config.mcfg_addr, mcfg->length, true);
if (mcfg == NULL) return;
if (acpi_checksum(mcfg, mcfg->length) != 0) {
return;
}
// The MCFG entries start after the table header and an 8 byte reserved
// field. Use the first entry for PCI segment 0.
uintptr_t first_entry = (uintptr_t)mcfg + sizeof(rsdt_header_t) + 8;
int num_entries = (mcfg->length - sizeof(rsdt_header_t) - 8) / sizeof(mcfg_entry_t);
for (int i = 0; i < num_entries; i++) {
mcfg_entry_t *entry = (mcfg_entry_t *)(first_entry + i * sizeof(mcfg_entry_t));
if (entry->segment == 0) {
if (entry->start_bus > entry->end_bus) {
continue; // malformed entry
}
size_t ecam_size = ((size_t)(entry->end_bus - entry->start_bus) + 1) << 20;
ecam_phys = entry->base_addr;
ecam_base = map_region(entry->base_addr, ecam_size, false);
ecam_start_bus = entry->start_bus;
ecam_end_bus = entry->end_bus;
break;
}
}
}
void pci_init(void)
{
// The ACPI tables have not been parsed yet when this is called, so the
// ECAM setup is done lazily on the first config space access instead.
}
// Qualcomm Snapdragon X (X1E80100 family): the USB controllers are DWC3
// platform devices, which expose the standard XHCI register interface at
// the core base address when operating in host mode. Base addresses from
// the Linux devicetree (arch/arm64/boot/dts/qcom/x1e80100.dtsi). Only the
// two primary USB-C controllers for now: accessing an unclocked block
// stalls the interconnect, and these two are the most likely to have been
// initialized by the firmware (boot keyboard support).
static const uintptr_t x1e80100_usb_bases[] = {
0x0a600000, // usb_1_ss0 (USB-C port 0)
0x0a800000, // usb_1_ss1 (USB-C port 1)
0x0a400000, // usb_mp (multiport: USB-A)
// 0x0aa00000, // usb_1_ss2 (USB-C port 2, not on all machines)
// 0x0a200000, // usb_2
};
bool platform_usb_controller(int index, uintptr_t *base_addr)
{
uint64_t midr = read_sysreg(midr_el1);
if (MIDR_IMPLEMENTER(midr) != 0x51 || MIDR_PART_NUM(midr) != 0x001) {
return false;
}
if (index < 0 || index >= (int)(sizeof(x1e80100_usb_bases) / sizeof(x1e80100_usb_bases[0]))) {
return false;
}
*base_addr = x1e80100_usb_bases[index];
return true;
}
uint8_t pci_config_read8(int bus, int dev, int func, int reg)
{
uintptr_t addr = ecam_addr(bus, dev, func, reg);
if (addr == 0) return 0xFF;
return mmio_read8((uint8_t *)addr);
}
uint16_t pci_config_read16(int bus, int dev, int func, int reg)
{
uintptr_t addr = ecam_addr(bus, dev, func, reg);
if (addr == 0) return 0xFFFF;
return mmio_read16((uint16_t *)addr);
}
uint32_t pci_config_read32(int bus, int dev, int func, int reg)
{
uintptr_t addr = ecam_addr(bus, dev, func, reg);
if (addr == 0) return 0xFFFFFFFF;
return mmio_read32((uint32_t *)addr);
}
void pci_config_write8(int bus, int dev, int func, int reg, uint8_t value)
{
uintptr_t addr = ecam_addr(bus, dev, func, reg);
if (addr == 0) return;
mmio_write8((uint8_t *)addr, value);
}
void pci_config_write16(int bus, int dev, int func, int reg, uint16_t value)
{
uintptr_t addr = ecam_addr(bus, dev, func, reg);
if (addr == 0) return;
mmio_write16((uint16_t *)addr, value);
}
void pci_config_write32(int bus, int dev, int func, int reg, uint32_t value)
{
uintptr_t addr = ecam_addr(bus, dev, func, reg);
if (addr == 0) return;
mmio_write32((uint32_t *)addr, value);
}
// -------------
// LPC Functions
// -------------
// There is no LPC bus on this architecture.
void lpc_outb(uint8_t cmd __attribute__((unused)), uint8_t data __attribute__((unused)))
{
}
uint8_t lpc_inb(uint8_t reg __attribute__((unused)))
{
return 0xFF;
}
system/aarch64/psci.c
+59
View File
@@ -0,0 +1,59 @@
// SPDX-License-Identifier: GPL-2.0
// Copyright (C) 2026 Sam Demeulemeester.
#include <stdint.h>
#include "acpi.h"
#include "psci.h"
//------------------------------------------------------------------------------
// Constants
//------------------------------------------------------------------------------
// FADT ARM_BOOT_ARCH flags.
#define FADT_ARM_PSCI_COMPLIANT (1 << 0)
#define FADT_ARM_PSCI_USE_HVC (1 << 1)
//------------------------------------------------------------------------------
// Public Functions
//------------------------------------------------------------------------------
int64_t psci_call(uint64_t fn, uint64_t arg0, uint64_t arg1, uint64_t arg2)
{
register uint64_t x0 __asm__ ("x0") = fn;
register uint64_t x1 __asm__ ("x1") = arg0;
register uint64_t x2 __asm__ ("x2") = arg1;
register uint64_t x3 __asm__ ("x3") = arg2;
if (acpi_config.arm_boot_arch & FADT_ARM_PSCI_USE_HVC) {
__asm__ __volatile__ ("hvc #0"
: "+r" (x0), "+r" (x1), "+r" (x2), "+r" (x3)
: /* no other inputs */
: "x4", "x5", "x6", "x7", "x8", "x9", "x10", "x11",
"x12", "x13", "x14", "x15", "x16", "x17", "memory"
);
} else {
__asm__ __volatile__ ("smc #0"
: "+r" (x0), "+r" (x1), "+r" (x2), "+r" (x3)
: /* no other inputs */
: "x4", "x5", "x6", "x7", "x8", "x9", "x10", "x11",
"x12", "x13", "x14", "x15", "x16", "x17", "memory"
);
}
return (int64_t)x0;
}
int64_t psci_cpu_on(uint64_t mpidr, uintptr_t entry_point, uint64_t context_id)
{
return psci_call(PSCI_FN_CPU_ON, mpidr, entry_point, context_id);
}
void psci_system_reset(void)
{
if (acpi_config.arm_boot_arch & FADT_ARM_PSCI_COMPLIANT) {
(void)psci_call(PSCI_FN_SYSTEM_RESET, 0, 0, 0);
}
}
system/aarch64/psci.h
+45
View File
@@ -0,0 +1,45 @@
// SPDX-License-Identifier: GPL-2.0
#ifndef PSCI_H
#define PSCI_H
/**
* \file
*
* Provides an interface to the ARM Power State Coordination Interface
* (PSCI) firmware functions.
*
*//*
// Copyright (C) 2026 Sam Demeulemeester.
*/
#include <stdint.h>
// PSCI function IDs
#define PSCI_FN_VERSION UINT32_C(0x84000000)
#define PSCI_FN_CPU_ON UINT32_C(0xC4000003)
#define PSCI_FN_SYSTEM_OFF UINT32_C(0x84000008)
#define PSCI_FN_SYSTEM_RESET UINT32_C(0x84000009)
// PSCI return codes
#define PSCI_RET_SUCCESS 0
#define PSCI_RET_NOT_SUPPORTED -1
#define PSCI_RET_ALREADY_ON -4
/**
* Makes a PSCI firmware call using the conduit indicated by the ACPI FADT
*/
int64_t psci_call(uint64_t fn, uint64_t arg0, uint64_t arg1, uint64_t arg2);
/**
* Starts the CPU core identified by mpidr at the given entry point address.
* The entry point is entered with the MMU off and context_id in x0
*/
int64_t psci_cpu_on(uint64_t mpidr, uintptr_t entry_point, uint64_t context_id);
/**
* Resets the system. Only returns if PSCI is not availabel or the call fails
*/
void psci_system_reset(void);
#endif // PSCI_H
system/aarch64/registers.h
+42
View File
@@ -0,0 +1,42 @@
// SPDX-License-Identifier: GPL-2.0
#ifndef ARM64_REGISTERS_H
#define ARM64_REGISTERS_H
/**
* \file
*
* Provides access to the ARM64 (AArch64) system registers.
*
*//*
// Copyright (C) 2026 Sam Demeulemeester.
*/
#include <stdint.h>
#define read_sysreg(reg) \
({ \
uint64_t value; \
__asm__ __volatile__ ("mrs %0, " #reg \
: "=r" (value) \
); \
value; \
})
#define write_sysreg(value, reg) \
__asm__ __volatile__ ("msr " #reg ", %0" \
: /* no outputs */ \
: "r" ((uint64_t)(value)) \
)
// MIDR_EL1 fields
#define MIDR_IMPLEMENTER(midr) (((midr) >> 24) & 0xFF)
#define MIDR_VARIANT(midr) (((midr) >> 20) & 0xF)
#define MIDR_ARCHITECTURE(midr) (((midr) >> 16) & 0xF)
#define MIDR_PART_NUM(midr) (((midr) >> 4) & 0xFFF)
#define MIDR_REVISION(midr) (((midr) >> 0) & 0xF)
// MPIDR_EL1 affinity fields. This mask extract Aff3/2/1/0
#define MPIDR_AFFINITY_MASK UINT64_C(0xFF00FFFFFF)
#endif // ARM64_REGISTERS_H
system/aarch64/simd.c
+31
View File
@@ -0,0 +1,31 @@
// SPDX-License-Identifier: GPL-2.0
// Copyright (C) 2004-2026 Sam Demeulemeester.
#include <stdint.h>
#include "registers.h"
#include "simd.h"
//------------------------------------------------------------------------------
// Public Variables
//------------------------------------------------------------------------------
simd_tier_t simd_tier = SIMD_NONE;
//------------------------------------------------------------------------------
// Public Functions
//------------------------------------------------------------------------------
void simd_init(void)
{
// Advanced SIMD (NEON) is architecturally mandatory on ARMv8-A, but honour
// the ID register: ID_AA64PFR0_EL1.AdvSIMD == 0xF means "not implemented".
// The ID registers are readable at EL1, where memtest runs.
uint64_t pfr0 = read_sysreg(id_aa64pfr0_el1);
uint64_t advsimd = (pfr0 >> 20) & 0xF;
// SVE (bits [35:32]) is detected here only to reserve a future tier; no SVE
// kernels exist yet, so we never select SIMD_SVE.
simd_tier = (advsimd == 0xF) ? SIMD_NONE : SIMD_NEON;
}
system/aarch64/temperature.c
+37
View File
@@ -0,0 +1,37 @@
// SPDX-License-Identifier: GPL-2.0
// Copyright (C) 2026 Sam Demeulemeester
#include <stdint.h>
#include "config.h"
#include "cpuid.h"
#include "cpuinfo.h"
#include "hwquirks.h"
#include "memctrl.h"
#include "temperature.h"
//------------------------------------------------------------------------------
// Public Variables
//------------------------------------------------------------------------------
float cpu_temp_offset = 0;
//------------------------------------------------------------------------------
// Public Functions
//------------------------------------------------------------------------------
void cpu_temp_init(void)
{
// Temperature reporting not yet supported on ARM64.
}
int get_cpu_temp(void)
{
return TEMP_INVALID;
}
int get_ram_temp(uint8_t slot __attribute__((unused)))
{
return TEMP_INVALID;
}
system/aarch64/vmem.c
+357
View File
@@ -0,0 +1,357 @@
// SPDX-License-Identifier: GPL-2.0
// Copyright (C) 2020-2022 Martin Whitaker.
// Copyright (C) 2026 Sam Demeulemeester.
//
// ARM64 (AArch64) virtual memory management.
//
// We build our own identity-mapped page tables covering all the memory
// regions described by the BIOS/EFI memory map, so all mapping functions
// simply return the physical address unchanged. RAM is mapped as Normal
// write-back cacheable memory, the frame buffer as Normal non-cacheable
// memory, and anything else mapped on demand (MMIO) as Device-nGnRnE.
//
// The tables use the 4KB granule with a 48-bit input address range and
// are built from 1GB and 2MB block descriptors only. paging_init() is
// called from the startup code, both on first boot and after the program
// has relocated itself, in the latter case while still running on the
// previous copy's page tables.
#include <stdbool.h>
#include <stdint.h>
#include "boot.h"
#include "bootparams.h"
#include "string.h"
#include "vmem.h"
//------------------------------------------------------------------------------
// Constants
//------------------------------------------------------------------------------
#define BLOCK_2M (UINT64_C(1) << 21)
#define BLOCK_1G (UINT64_C(1) << 30)
#define NUM_L1_TABLES 8 // each covers 512GB
#define NUM_L2_TABLES 128 // each covers 1GB
// Descriptor types
#define PTE_TYPE_BLOCK UINT64_C(0x1)
#define PTE_TYPE_TABLE UINT64_C(0x3)
#define PTE_VALID UINT64_C(0x1)
#define PTE_ADDR_MASK UINT64_C(0x0000FFFFFFFFF000)
// Lower and upper block attributes. The MAIR indices must match the
// MAIR_EL1 value set in boot/aarch64/startup.S.
#define PTE_ATTRINDX(idx) ((uint64_t)(idx) << 2)
#define PTE_SH_INNER (UINT64_C(3) << 8)
#define PTE_AF (UINT64_C(1) << 10)
#define PTE_PXN (UINT64_C(1) << 53)
#define PTE_UXN (UINT64_C(1) << 54)
#define PTE_ATTR_NORMAL (PTE_ATTRINDX(0) | PTE_SH_INNER | PTE_AF | PTE_UXN)
#define PTE_ATTR_DEVICE (PTE_ATTRINDX(1) | PTE_AF | PTE_PXN | PTE_UXN)
#define PTE_ATTR_NORMAL_NC (PTE_ATTRINDX(2) | PTE_SH_INNER | PTE_AF | PTE_PXN | PTE_UXN)
#define MAX_DEVICE_REGIONS 64
//------------------------------------------------------------------------------
// Types
//------------------------------------------------------------------------------
typedef struct {
uint64_t start;
uint64_t end;
} device_region_t;
//------------------------------------------------------------------------------
// Private Variables
//------------------------------------------------------------------------------
// The level 1 and level 2 table pools. Tables are allocated on demand.
static uint64_t l1_tables[NUM_L1_TABLES][512] __attribute__((aligned(PAGE_SIZE)));
static uint64_t l2_tables[NUM_L2_TABLES][512] __attribute__((aligned(PAGE_SIZE)));
static int num_l1_used = 0;
static int num_l2_used = 0;
// The device (MMIO) regions mapped so far. The page tables are rebuilt from
// scratch each time the program relocates itself, so we must remember these
// regions in order to replay the mappings.
static device_region_t device_regions[MAX_DEVICE_REGIONS];
static int num_device_regions = 0;
// A snapshot of the information we need from the boot params. The boot
// params live in memory that will be overwritten once testing starts, but
// the page tables are rebuilt on every relocation, so we must keep our own
// copy. These variables are part of the program image, and so are preserved
// by relocation.
static bool boot_info_saved = false;
static e820_entry_t e820_copy[E820_MAP_SIZE];
static int e820_copy_entries = 0;
static uint64_t lfb_start = 0;
static uint64_t lfb_end = 0;
static uint64_t cmd_line_start = 0;
static uint64_t cmd_line_end = 0;
//------------------------------------------------------------------------------
// Public Variables
//------------------------------------------------------------------------------
// The level 0 (top) translation table. Loaded into TTBR0_EL1 by at startup
uint64_t ttbr0_table[512] __attribute__((aligned(PAGE_SIZE)));
bool paging_incomplete = false;
//------------------------------------------------------------------------------
// Private Functions
//------------------------------------------------------------------------------
static uint64_t *alloc_l1_table(void)
{
if (num_l1_used >= NUM_L1_TABLES) {
return NULL;
}
uint64_t *table = l1_tables[num_l1_used++];
memset(table, 0, PAGE_SIZE);
return table;
}
static uint64_t *alloc_l2_table(void)
{
if (num_l2_used >= NUM_L2_TABLES) {
return NULL;
}
uint64_t *table = l2_tables[num_l2_used++];
memset(table, 0, PAGE_SIZE);
return table;
}
// Returns the table pointed to by a table descriptor, creating a new table
// if the descriptor is empty. Returns NULL if the descriptor is a block
// (the range is already mapped at a coarser granularity) or if the table
// pool is exhausted.
static uint64_t *get_l1_table(uint64_t *entry)
{
if (*entry & PTE_VALID) {
return (uint64_t *)(uintptr_t)(*entry & PTE_ADDR_MASK);
}
uint64_t *table = alloc_l1_table();
if (table != NULL) {
*entry = (uintptr_t)table | PTE_TYPE_TABLE;
}
return table;
}
static uint64_t *get_l2_table(uint64_t *entry)
{
if (*entry & PTE_VALID) {
if ((*entry & PTE_TYPE_TABLE) != PTE_TYPE_TABLE) {
return NULL; // already mapped by a 1GB block
}
return (uint64_t *)(uintptr_t)(*entry & PTE_ADDR_MASK);
}
uint64_t *table = alloc_l2_table();
if (table != NULL) {
*entry = (uintptr_t)table | PTE_TYPE_TABLE;
}
return table;
}
// Identity-maps the given physical address range with the given attributes,
// rounding it out to 2MB boundaries. Existing mappings are left untouched,
// so the first mapping of a region wins. Returns false if we ran out of
// translation tables.
static bool map_range(uint64_t start, uint64_t end, uint64_t attrs)
{
// Reject inverted ranges and anything beyond the 48-bit input address
// range, before the rounding below can wrap for an end near 2^64.
if (start > end || end > (UINT64_C(1) << 48)) {
return false;
}
start = start & ~(BLOCK_2M - 1);
end = (end + BLOCK_2M - 1) & ~(BLOCK_2M - 1);
uint64_t addr = start;
while (addr < end) {
uint64_t *l1_table = get_l1_table(&ttbr0_table[(addr >> 39) & 511]);
if (l1_table == NULL) {
return false;
}
uint64_t *l1_entry = &l1_table[(addr >> 30) & 511];
if ((addr & (BLOCK_1G - 1)) == 0 && (end - addr) >= BLOCK_1G && !(*l1_entry & PTE_VALID)) {
*l1_entry = addr | attrs | PTE_TYPE_BLOCK;
addr += BLOCK_1G;
continue;
}
if ((*l1_entry & PTE_VALID) && (*l1_entry & PTE_TYPE_TABLE) != PTE_TYPE_TABLE) {
// Already mapped by a 1GB block.
addr += BLOCK_2M;
continue;
}
uint64_t *l2_table = get_l2_table(l1_entry);
if (l2_table == NULL) {
return false;
}
uint64_t *l2_entry = &l2_table[(addr >> 21) & 511];
if (!(*l2_entry & PTE_VALID)) {
*l2_entry = addr | attrs | PTE_TYPE_BLOCK;
}
addr += BLOCK_2M;
}
// Make sure the table updates are visible to the table walker.
__asm__ __volatile__ ("dsb ishst" ::: "memory");
return true;
}
//------------------------------------------------------------------------------
// Public Functions
//------------------------------------------------------------------------------
// Builds the identity-mapped page tables from the boot params. Called from
// the startup code with the MMU still using either the firmware's page
// tables (first boot) or the previous program copy's page tables (after reloc)
void paging_init(void)
{
// The first time we are called (from the boot code, before any memory
// has been touched), take a snapshot of the boot information we need.
// The boot params themselves live in memory that gets overwritten once
// testing starts, so they cannot be relied upon after that.
if (!boot_info_saved) {
const boot_params_t *boot_params = (const boot_params_t *)boot_params_addr;
const screen_info_t *si = &boot_params->screen_info;
if (si->orig_video_isVGA == VIDEO_TYPE_EFI) {
lfb_start = si->lfb_base;
if (LFB_CAPABILITY_64BIT_BASE & si->capabilities) {
lfb_start |= (uint64_t)si->ext_lfb_base << 32;
}
lfb_end = lfb_start + (uint64_t)si->lfb_linelength * si->lfb_height;
}
e820_copy_entries = boot_params->e820_entries;
if (e820_copy_entries > E820_MAP_SIZE) {
e820_copy_entries = E820_MAP_SIZE;
}
memcpy(e820_copy, boot_params->e820_map, e820_copy_entries * sizeof(e820_entry_t));
if (boot_params->cmd_line_ptr != 0) {
cmd_line_start = boot_params->cmd_line_ptr;
cmd_line_end = cmd_line_start + boot_params->cmd_line_size;
}
boot_info_saved = true;
}
num_l1_used = 0;
num_l2_used = 0;
memset(ttbr0_table, 0, sizeof(ttbr0_table));
// Map the frame buffer first, as Normal non-cacheable memory. On some
// systems it is allocated from RAM, and we must make sure the display
// controller sees our writes.
if (lfb_end != 0) {
paging_incomplete |= !map_range(lfb_start, lfb_end, PTE_ATTR_NORMAL_NC);
}
// Map all the memory regions described by the BIOS memory map as Normal
// write-back cacheable memory.
for (int i = 0; i < e820_copy_entries; i++) {
const e820_entry_t *entry = &e820_copy[i];
if (entry->type == E820_RAM || entry->type == E820_ACPI) {
paging_incomplete |= !map_range(entry->addr, entry->addr + entry->size, PTE_ATTR_NORMAL);
}
}
// The boot command line may live outside the mapped regions if the boot
// loader put it somewhere unusual.
if (cmd_line_start != 0) {
paging_incomplete |= !map_range(cmd_line_start, cmd_line_end, PTE_ATTR_NORMAL);
}
// Replay the device mappings made via map_region.
for (int i = 0; i < num_device_regions; i++) {
paging_incomplete |= !map_range(device_regions[i].start, device_regions[i].end, PTE_ATTR_DEVICE);
}
}
uintptr_t map_region(uintptr_t base_addr, size_t size, bool only_for_startup __attribute__((unused)))
{
if (size == 0) {
size = 1;
}
// Reject sizes that would wrap the address space.
if (base_addr + size < base_addr) {
return 0;
}
// Record the region (at block granularity) so the mapping can be
// replayed when the page tables are rebuilt after relocation.
uint64_t start = base_addr & ~(BLOCK_2M - 1);
uint64_t end = (base_addr + size + BLOCK_2M - 1) & ~(BLOCK_2M - 1);
int i;
for (i = 0; i < num_device_regions; i++) {
if (start >= device_regions[i].start && end <= device_regions[i].end) {
break;
}
}
if (i == num_device_regions) {
if (num_device_regions >= MAX_DEVICE_REGIONS) {
return 0;
}
device_regions[num_device_regions].start = start;
device_regions[num_device_regions].end = end;
num_device_regions++;
}
// Anything not already mapped is assumed to be MMIO and gets a Device
// mapping. Regions mapped earlier (e.g. RAM, the frame buffer) keep
// their original attributes.
if (!map_range(base_addr, base_addr + size, PTE_ATTR_DEVICE)) {
return 0;
}
__asm__ __volatile__ ("isb" ::: "memory");
return base_addr;
}
bool map_window(uintptr_t start_page __attribute__((unused)))
{
// All of physical memory is permanently identity mapped.
return true;
}
void *first_word_mapping(uintptr_t page)
{
return (void *)(page << PAGE_SHIFT);
}
void *last_word_mapping(uintptr_t page, size_t word_size)
{
return (uint8_t *)first_word_mapping(page) + (PAGE_SIZE - word_size);
}
uintptr_t page_of(void *addr)
{
return (uintptr_t)addr >> PAGE_SHIFT;
}
system/acpi.c
+14 -2
View File
@@ -1,6 +1,6 @@
// SPDX-License-Identifier: GPL-2.0
// Copyright (C) 2020-2022 Martin Whitaker.
// Copyright (C) 2004-2022 Sam Demeulemeester.
// Copyright (C) 2004-2026 Sam Demeulemeester.
#include "boot.h"
#include "bootparams.h"
@@ -41,6 +41,8 @@
#define SLITSignature ('S' | ('L' << 8) | ('I' << 16) | ('T' << 24)) // System Locality Information Table (NUMA)
#define SRATSignature ('S' | ('R' << 8) | ('A' << 16) | ('T' << 24)) // System Resource Affinity Table (NUMA)
#define MCFGSignature ('M' | ('C' << 8) | ('F' << 16) | ('G' << 24)) // PCIe ECAM Description Table
//------------------------------------------------------------------------------
// Types
//------------------------------------------------------------------------------
@@ -78,7 +80,7 @@ static const efi_guid_t EFI_ACPI_2_RDSP_GUID = { 0x8868e871, 0xe4f1, 0x11d3, {0x
const char *rsdp_source = "";
acpi_t acpi_config = {0, 0, 0, 0, 0, /*0,*/ 0, 0, 0, false};
acpi_t acpi_config = {0, 0, 0, 0, 0, /*0,*/ 0, 0, 0, 0, 0, false};
//------------------------------------------------------------------------------
// Private Functions
@@ -284,6 +286,14 @@ static bool parse_fadt(uintptr_t fadt_addr)
acpi_config.pm_addr = *(uint32_t *)(fadt_addr+FADT_PM_TMR_BLK_OFFSET);
acpi_config.pm_is_io = true;
#if defined(__aarch64__)
// Get the ARM boot architecture flags (PSCI availability and conduit).
if (fadt->length > FADT_ARM_BOOT_ARCH_OFFSET + 1) {
acpi_config.arm_boot_arch = *(uint8_t *)(fadt_addr+FADT_ARM_BOOT_ARCH_OFFSET)
| *(uint8_t *)(fadt_addr+FADT_ARM_BOOT_ARCH_OFFSET+1) << 8;
}
#endif
#if (ARCH_BITS == 64)
acpi_gen_addr_struct *rt;
@@ -337,5 +347,7 @@ void acpi_init(void)
acpi_config.srat_addr = find_acpi_table(SRATSignature);
acpi_config.mcfg_addr = find_acpi_table(MCFGSignature);
//acpi_config.slit_addr = find_acpi_table(SLITSignature);
}
system/acpi.h
+4 -1
View File
@@ -8,13 +8,14 @@
*
*//*
* Copyright (C) 2020-2022 Martin Whitaker.
* Copyright (C) 2004-2022 Sam Demeulemeester.
* Copyright (C) 2004-2026 Sam Demeulemeester.
*/
#include <stdbool.h>
#include <stdint.h>
#define FADT_PM_TMR_BLK_OFFSET 76
#define FADT_ARM_BOOT_ARCH_OFFSET 129
#define FADT_MINOR_REV_OFFSET 131
#define FADT_X_PM_TMR_BLK_OFFSET 208
@@ -29,7 +30,9 @@ typedef struct __attribute__ ((packed)) {
uintptr_t hpet_addr;
uintptr_t srat_addr;
//uintptr_t slit_addr;
uintptr_t mcfg_addr;
uintptr_t pm_addr;
uint16_t arm_boot_arch;
uint8_t ver_maj;
uint8_t ver_min;
bool pm_is_io;
system/cache.h
+119 -2
View File
@@ -8,6 +8,7 @@
*
*//*
* Copyright (C) 2020-2022 Martin Whitaker.
* Copyright (C) 2026 Sam Demeulemeester.
*/
#ifdef __loongarch_lp64
@@ -21,6 +22,63 @@
static inline void cache_flush(void);
#endif
#ifdef __aarch64__
#include <stdbool.h>
#include <stdint.h>
#include "registers.h"
static inline void cache_flush(void);
/**
* Cleans the D-cache by virtual address to the point of coherency over the
* given range. Needed before starting other CPUs (they come up with their
* MMU and caches disabled!)
*/
static inline void cache_clean_range(const void *start, const void *end)
{
uint64_t ctr = read_sysreg(ctr_el0);
uintptr_t line_size = UINT64_C(4) << ((ctr >> 16) & 0xF); // DminLine
uintptr_t addr = (uintptr_t)start & ~(line_size - 1);
while (addr < (uintptr_t)end) {
__asm__ __volatile__ ("dc cvac, %0" : : "r" (addr) : "memory");
addr += line_size;
}
__asm__ __volatile__ ("dsb sy" ::: "memory");
}
/**
* Makes newly written code in the given range visible to instruction fetch,
* which is not coherent with the data caches on this architecture. Each CPU
* must still execute an ISB before executing the new code.
*/
static inline void cache_sync_code_range(const void *start, const void *end)
{
uint64_t ctr = read_sysreg(ctr_el0);
// Clean the D-cache to the point of unification, unless CTR_EL0.IDC says this is unneeded.
if (!(ctr & (UINT64_C(1) << 28))) {
uintptr_t line_size = UINT64_C(4) << ((ctr >> 16) & 0xF); // DminLine
uintptr_t addr = (uintptr_t)start & ~(line_size - 1);
while (addr < (uintptr_t)end) {
__asm__ __volatile__ ("dc cvau, %0" : : "r" (addr) : "memory");
addr += line_size;
}
}
__asm__ __volatile__ ("dsb ish" ::: "memory");
// Invalidate the I-cache (broadcast), unless CTR_EL0.DIC says this is unneeded.
if (!(ctr & (UINT64_C(1) << 29))) {
uintptr_t line_size = UINT64_C(4) << (ctr & 0xF); // IminLine
uintptr_t addr = (uintptr_t)start & ~(line_size - 1);
while (addr < (uintptr_t)end) {
__asm__ __volatile__ ("ic ivau, %0" : : "r" (addr) : "memory");
addr += line_size;
}
}
__asm__ __volatile__ ("dsb ish; isb" ::: "memory");
}
#endif
/**
* Disable the CPU caches.
*/
@@ -49,6 +107,12 @@ static inline void cache_off(void)
#elif defined(__loongarch_lp64)
cache_flush();
__csrxchg_d(0, 3 << 4, 0x181);
#elif defined(__aarch64__)
uint64_t sctlr = read_sysreg(sctlr_el1);
sctlr &= ~(UINT64_C(1) << 2); /* Clear C */
write_sysreg(sctlr, sctlr_el1);
__asm__ __volatile__ ("isb" ::: "memory");
cache_flush();
#endif
}
@@ -78,6 +142,11 @@ static inline void cache_on(void)
#elif defined(__loongarch_lp64)
cache_flush();
__csrxchg_d(1 << 4, 3 << 4, 0x181);
#elif defined(__aarch64__)
uint64_t sctlr = read_sysreg(sctlr_el1);
sctlr |= (UINT64_C(1) << 2) | (UINT64_C(1) << 12); /* Set C and I */
write_sysreg(sctlr, sctlr_el1);
__asm__ __volatile__ ("isb" ::: "memory");
#endif
}
@@ -146,6 +215,49 @@ static inline void cache_flush(void)
va += line_size;
}
}
#elif defined(__aarch64__)
// Clean and invalidate the whole D-cache hierarchy by set/way.
__asm__ __volatile__ ("dsb sy" ::: "memory");
uint64_t clidr = read_sysreg(clidr_el1);
bool has_ccidx = ((read_sysreg(id_aa64mmfr2_el1) >> 20) & 0xF) != 0;
for (int level = 0; level < 7; level++) {
int cache_type = (clidr >> (3 * level)) & 0x7;
if (cache_type == 0) {
break; // no more cache levels
}
if (cache_type < 2) {
continue; // no data or unified cache at this level
}
// Select the data or unified cache at this level.
write_sysreg(level << 1, csselr_el1);
__asm__ __volatile__ ("isb");
uint64_t ccsidr = read_sysreg(ccsidr_el1);
uint32_t line_shift = (ccsidr & 0x7) + 4;
uint32_t max_way, max_set;
if (has_ccidx) {
max_way = (ccsidr >> 3) & 0x1FFFFF;
max_set = (ccsidr >> 32) & 0xFFFFFF;
} else {
max_way = (ccsidr >> 3) & 0x3FF;
max_set = (ccsidr >> 13) & 0x7FFF;
}
uint32_t way_shift = max_way ? __builtin_clz(max_way) : 0;
for (uint32_t set = 0; set <= max_set; set++) {
for (uint32_t way = 0; way <= max_way; way++) {
uint64_t set_way = ((uint64_t)way << way_shift)
| ((uint64_t)set << line_shift)
| (level << 1);
__asm__ __volatile__ ("dc cisw, %0" : : "r" (set_way) : "memory");
}
}
}
__asm__ __volatile__ ("dsb sy; isb" ::: "memory");
#endif
}
@@ -175,7 +287,8 @@ static inline void cache_line_flush(const volatile void *addr)
/**
* A full memory barrier ordering all prior loads and stores before any that
* follow. On x86 this is MFENCE (SSE2 baseline), so only execute it once SSE2
* is known to be present; on LoongArch it is a full data barrier.
* is known to be present; on LoongArch it is a full data barrier; on AArch64 a
* DSB ISH (stronger than DMB) also drains any preceding DC CIVAC maintenance.
*/
static inline void mem_barrier(void)
{
@@ -183,12 +296,14 @@ static inline void mem_barrier(void)
__asm__ __volatile__ ("mfence" : : : "memory");
#elif defined(__loongarch_lp64)
__asm__ __volatile__ ("dbar 0" : : : "memory");
#elif defined(__aarch64__)
__asm__ __volatile__ ("dsb ish" : : : "memory");
#endif
}
/**
* A load fence used to bracket rdtsc timing so reads are not reordered across
* the measurement. On x86 this is LFENCE (SSE2 baseline).
* the measurement. On x86 this is LFENCE (SSE2 baseline); on AArch64 a DSB LD.
*/
static inline void load_fence(void)
{
@@ -196,6 +311,8 @@ static inline void load_fence(void)
__asm__ __volatile__ ("lfence" : : : "memory");
#elif defined(__loongarch_lp64)
__asm__ __volatile__ ("dbar 0" : : : "memory");
#elif defined(__aarch64__)
__asm__ __volatile__ ("dsb ld" : : : "memory");
#endif
}
system/cpuinfo.h
+7 -1
View File
@@ -8,7 +8,7 @@
*
*//*
* Copyright (C) 2020-2022 Martin Whitaker.
* Copyright (C) 2004-2023 Sam Demeulemeester.
* Copyright (C) 2004-2026 Sam Demeulemeester.
*/
#include <stdbool.h>
@@ -136,6 +136,12 @@ extern uint32_t ram_speed;
*/
extern uint32_t clks_per_msec;
/**
* The measured CPU clock speed in MHz, on architectures where the timebase
* used by get_tsc() does not run at the CPU clock frequency. 0 if unknown.
*/
extern uint32_t cpu_clk_mhz;
/**
* Determines the CPU info and stores it in the exported variables.
*/
system/heap.c
+40 -1
View File
@@ -84,10 +84,40 @@ void heap_init(void)
uintptr_t max_segment_size = 0;
#if defined(__i386__) || defined (__x86_64__)
uintptr_t low_memory_heap = PAGE_C(1, MB);
uintptr_t heap_limit = PAGE_C(4,GB);
#elif defined(__loongarch_lp64)
uintptr_t low_memory_heap = PAGE_C(256, MB);
uintptr_t heap_limit = PAGE_C(4,GB);
#elif defined(__aarch64__)
// RAM may start well above 0, so make the heap limits relative to the
// start of RAM. If RAM starts below 3GB, keep the heap below an absolute
// 3GB: some SoCs cannot DMA above it (e.g. BCM2712 USB on Raspberry Pi 5).
uintptr_t low_memory_heap = pm_map[0].start + PAGE_C(256, MB);
uintptr_t heap_limit;
if (pm_map[0].start < PAGE_C(3,GB)) {
heap_limit = PAGE_C(3,GB);
} else {
heap_limit = pm_map[0].start + PAGE_C(4,GB);
}
// If a memory segment straddles the heap limit, split it at the limit, so
// the scan below can assign a heap even when RAM is one contiguous region.
for (int i = 0; i < pm_map_size; i++) {
if (pm_map[i].start < heap_limit && pm_map[i].end > heap_limit) {
if (pm_map_size < MAX_MEM_SEGMENTS) {
for (int j = pm_map_size; j > i + 1; j--) {
pm_map[j] = pm_map[j - 1];
}
pm_map[i + 1].start = heap_limit;
pm_map[i + 1].end = pm_map[i].end;
pm_map[i].end = heap_limit;
pm_map_size++;
}
break;
}
}
#endif
for (int i = 0; i < pm_map_size && pm_map[i].end <= PAGE_C(4,GB); i++) {
for (int i = 0; i < pm_map_size && pm_map[i].end <= heap_limit; i++) {
uintptr_t try_heap_start = pm_map[i].start;
uintptr_t try_heap_end = pm_map[i].end;
if (program_start >= try_heap_start && program_end <= try_heap_end) {
@@ -106,4 +136,13 @@ void heap_init(void)
heaps[HEAP_TYPE_HM_1].end = try_heap_end;
}
}
#if defined(__aarch64__)
// There is no low-memory addressing constraint on this architecture, so
// if no segment small enough for the low-memory heap was found (RAM is
// typically one big contiguous region), just use the high-memory heap.
if (heaps[HEAP_TYPE_LM_1].segment < 0) {
heaps[HEAP_TYPE_LM_1] = heaps[HEAP_TYPE_HM_1];
}
#endif
}
system/io.h
+22
View File
@@ -202,6 +202,28 @@ __OUT(l, "w");
#define outl(val,port) __outl(val,port)
#define inl(port) __inl(port)
#elif defined(__aarch64__)
// There is no port-mapped I/O on ARM64. These stubs make legacy device
// probes (PS/2 keyboard, port-mapped UARTs, PIT) fail gracefully.
#include <stdint.h>
static inline uint8_t __inb(unsigned short port) { (void)port; return 0xFF; }
static inline uint16_t __inw(unsigned short port) { (void)port; return 0xFFFF; }
static inline uint32_t __inl(unsigned short port) { (void)port; return 0xFFFFFFFF; }
static inline void __outb(uint8_t value, unsigned short port) { (void)value; (void)port; }
static inline void __outw(uint16_t value, unsigned short port) { (void)value; (void)port; }
static inline void __outl(uint32_t value, unsigned short port) { (void)value; (void)port; }
#define outb(val,port) __outb(val,port)
#define inb(port) __inb(port)
#define outw(val,port) __outw(val,port)
#define inw(port) __inw(port)
#define outl(val,port) __outl(val,port)
#define inl(port) __inl(port)
#endif
#endif // IO_H
system/loongarch/cpuinfo.c
+1 -1
View File
@@ -32,7 +32,7 @@
// Public Variables
//------------------------------------------------------------------------------
const char *cpu_model = NULL;
const char *cpu_model = ""; // static init required: creates the reloc record that rebases it
int l1_cache = 0;
int l2_cache = 0;
system/loongarch/vmem.c
+6
View File
@@ -30,6 +30,12 @@ extern uint8_t highest_map_bit;
#define PCIE40_HIGH_ADDRESS 0xC000000000ULL
#define PCIE64_LOW_ADDRESS 0x8000000000ULL
#define PCIE64_HIGH_ADDRESS 0xFD00000000ULL
//------------------------------------------------------------------------------
// Public Variables
//------------------------------------------------------------------------------
bool paging_incomplete = false; // never set on this architecture
//------------------------------------------------------------------------------
// Public Functions
//------------------------------------------------------------------------------
system/memrw.h
+17
View File
@@ -35,6 +35,23 @@
#define __MEMRW_WRITE_INSTRUCTIONS(bitwidth) "st." __MEMRW_SUFFIX_##bitwidth##BIT " %1, %0"
#define __MEMRW_FLUSH_INSTRUCTIONS(bitwidth) "st." __MEMRW_SUFFIX_##bitwidth##BIT " %1, %0; dbar 0"
#elif defined(__aarch64__)
#define __MEMRW_SUFFIX_8BIT "b"
#define __MEMRW_SUFFIX_16BIT "h"
#define __MEMRW_SUFFIX_32BIT ""
#define __MEMRW_SUFFIX_64BIT ""
#define __MEMRW_REG_8BIT "w"
#define __MEMRW_REG_16BIT "w"
#define __MEMRW_REG_32BIT "w"
#define __MEMRW_REG_64BIT "x"
#define __MEMRW_READ_INSTRUCTIONS(bitwidth) \
"ldr" __MEMRW_SUFFIX_##bitwidth##BIT " %" __MEMRW_REG_##bitwidth##BIT "0, %1"
#define __MEMRW_WRITE_INSTRUCTIONS(bitwidth) \
"str" __MEMRW_SUFFIX_##bitwidth##BIT " %" __MEMRW_REG_##bitwidth##BIT "1, %0"
#define __MEMRW_FLUSH_INSTRUCTIONS(bitwidth) \
"str" __MEMRW_SUFFIX_##bitwidth##BIT " %" __MEMRW_REG_##bitwidth##BIT "1, %0; dsb sy"
#endif
#define __MEMRW_READ_FUNC(bitwidth) \
system/mmio.h
+24
View File
@@ -10,6 +10,7 @@
* memory spaces.
*//*
* Copyright (C) 2024 Loongson Technology Corporation Limited. All rights reserved.
* Copyright (C) 2026 Sam Demeulemeester
*
*/
@@ -30,6 +31,29 @@
#define __MMIORW_READ_WRITE_CLOBBER "$t0", "memory"
#elif defined(__aarch64__)
#define __MMIORW_SUFFIX_8BIT "b"
#define __MMIORW_SUFFIX_16BIT "h"
#define __MMIORW_SUFFIX_32BIT ""
#define __MMIORW_SUFFIX_64BIT ""
#define __MMIORW_REG_8BIT "w"
#define __MMIORW_REG_16BIT "w"
#define __MMIORW_REG_32BIT "w"
#define __MMIORW_REG_64BIT "x"
// The barriers order device accesses relative to normal memory accesses
// (e.g. DMA buffer setup before ringing a doorbell register).
#define __MMIORW_READ_INSTRUCTIONS(bitwidth) \
"ldr" __MMIORW_SUFFIX_##bitwidth##BIT " %" __MMIORW_REG_##bitwidth##BIT "0, %1; dmb sy"
#define __MMIORW_WRITE_INSTRUCTIONS(bitwidth) \
"dmb sy; str" __MMIORW_SUFFIX_##bitwidth##BIT " %" __MMIORW_REG_##bitwidth##BIT "1, %0"
#define __MMIORW_READ_WRITE_CLOBBER "memory"
#endif
#define __MMIORW_READ_FUNC(bitwidth) \
system/pci.h
+10
View File
@@ -9,6 +9,7 @@
*//*
* Copyright (C) 2020-2022 Martin Whitaker.
* Copyright (C) 2024 Loongson Technology Corporation Limited. All rights reserved.
* Copyright (C) 2026 Sam Demeulemeester.
*/
#include <stdint.h>
@@ -99,6 +100,15 @@ void pci_config_write16(int bus, int dev, int func, int reg, uint16_t value);
*/
void pci_config_write32(int bus, int dev, int func, int reg, uint32_t value);
#if defined(__aarch64__)
/** Allocates MMIO space for an unassigned memory BAR and writes it to the BAR.
* Returns the allocated base address, or 0 if no suitable range exists. */
uintptr_t pci_alloc_mmio(int bus, int dev, int func, int bar_reg, uintptr_t size);
/** Finds a platform USB host controller with fixed MMIO, such as DWC3/XHCI.
* Returns true and stores its base address if the given index exists. */
bool platform_usb_controller(int index, uintptr_t *base_addr);
#endif
/**
* Basic LPC Functions
system/pmem.c
+18 -1
View File
@@ -20,6 +20,8 @@
#include "pmem.h"
#include "vmem.h"
//------------------------------------------------------------------------------
// Constants
//------------------------------------------------------------------------------
@@ -39,6 +41,11 @@ int pm_map_size = 0;
size_t num_pm_pages = 0;
#if defined(__aarch64__)
uintptr_t low_load_limit = 0;
uintptr_t high_load_limit = 0;
#endif
//------------------------------------------------------------------------------
// Private Functions
//------------------------------------------------------------------------------
@@ -248,8 +255,18 @@ void pmem_init(void)
const boot_params_t *boot_params = (boot_params_t *)boot_params_addr;
int sanitized_entries = sanitize_e820_map(sanitized_map, boot_params->e820_map, boot_params->e820_entries);
int e820_entries = boot_params->e820_entries;
if (e820_entries > E820_MAP_SIZE) {
e820_entries = E820_MAP_SIZE;
}
int sanitized_entries = sanitize_e820_map(sanitized_map, boot_params->e820_map, e820_entries);
init_pm_map(sanitized_map, sanitized_entries);
sort_pm_map();
#if defined(__aarch64__)
low_load_limit = (pm_map[0].start << PAGE_SHIFT) + SIZE_C(4,MB);
high_load_limit = (pm_map[0].start << PAGE_SHIFT) + (VM_PINNED_SIZE << PAGE_SHIFT);
#endif
}
system/pmem.h
+8 -1
View File
@@ -13,7 +13,7 @@
#include <stddef.h>
#include <stdint.h>
#define MAX_MEM_SEGMENTS 127
#define MAX_MEM_SEGMENTS 128 // must be >= E820_MAP_SIZE (boot.h)
typedef struct {
uintptr_t start;
@@ -25,6 +25,13 @@ extern int pm_map_size;
extern size_t num_pm_pages;
#if defined(__aarch64__)
// Load limits computed at run time from the start of physical RAM, which may
// begin well above address 0. Set by pmem_init().
extern uintptr_t low_load_limit;
extern uintptr_t high_load_limit;
#endif
void pmem_init(void);
#endif /* PMEM_H */
system/reloc32.c
+1
View File
@@ -112,6 +112,7 @@ static void get_dynamic_info(Elf32_Dyn *dyn_section, Elf32_Addr load_offs, Elf32
}
}
// Slot contents must be image pointers only - see the note in reloc64.c.
static void do_relocation(Elf32_Addr load_addr, Elf32_Addr load_offs, const Elf32_Rel *rel)
{
Elf32_Addr *target_addr = (Elf32_Addr *)(load_addr + rel->r_offset);
system/reloc64.c
+41 -2
View File
@@ -34,6 +34,8 @@
#define R_X86_64_RELATIVE 8
#define R_LARCH_NONE 0
#define R_LARCH_RELATIVE 3
#define R_AARCH64_NONE 0
#define R_AARCH64_RELATIVE 1027
//------------------------------------------------------------------------------
// Types
@@ -86,6 +88,14 @@ static inline Elf64_Addr __attribute__ ((unused)) get_load_address(void)
:
: "memory"
);
#elif defined(__aarch64__)
__asm__ __volatile__ (
"adrp %0, _start \n\t"
"add %0, %0, :lo12:_start"
: "=r" (addr)
:
: "memory"
);
#endif
return addr;
}
@@ -112,6 +122,27 @@ static inline Elf64_Addr __attribute__ ((unused)) get_dynamic_section_offset(voi
:
: "$t0", "memory"
);
#elif defined(__aarch64__)
// On AArch64 the GOT[0] convention is not reliable (the entry itself
// needs a dynamic relocation), so compute the link-time offset of
// _DYNAMIC PC-relatively instead. The program is linked at address 0,
// so the offset is basically the distance from _start
Elf64_Addr start_addr;
__asm__ __volatile__ (
"adrp %0, _DYNAMIC \n\t"
"add %0, %0, :lo12:_DYNAMIC"
: "=r" (offs)
:
: "memory"
);
__asm__ __volatile__ (
"adrp %0, _start \n\t"
"add %0, %0, :lo12:_start"
: "=r" (start_addr)
:
: "memory"
);
offs -= start_addr;
#endif
return offs;
}
@@ -138,11 +169,17 @@ static void get_dynamic_info(Elf64_Dyn *dyn_section, Elf64_Addr load_offs, Elf64
}
}
/*
* Slots are adjusted by the move delta, so a statically initialised pointer
* variable must only ever hold pointers into the program image; anything
* else (firmware tables, heap, MMIO) belongs in a BSS variable.
*/
static void do_relocation(Elf64_Addr load_addr, Elf64_Addr load_offs, const Elf64_Rela *rel)
{
Elf64_Addr *target_addr = (Elf64_Addr *)(load_addr + rel->r_offset);
if ((ELF64_R_TYPE(rel->r_info) == R_X86_64_RELATIVE) ||
(ELF64_R_TYPE(rel->r_info) == R_LARCH_RELATIVE)) {
(ELF64_R_TYPE(rel->r_info) == R_LARCH_RELATIVE) ||
(ELF64_R_TYPE(rel->r_info) == R_AARCH64_RELATIVE)) {
if (load_offs == load_addr) {
*target_addr = load_addr + rel->r_addend;
} else {
@@ -193,7 +230,9 @@ void reloc(void)
do_relocations(load_addr, load_offs, dyn_info[DT_RELA]->d_un.d_ptr, dyn_info[DT_RELASZ]->d_un.d_val);
if (dyn_info[DT_PLTREL]->d_un.d_val == DT_RELA) {
// The PLT-related tags are absent if there are no PLT relocations
// (e.g. on AArch64).
if (dyn_info[DT_PLTREL] != NULL && dyn_info[DT_PLTREL]->d_un.d_val == DT_RELA) {
do_relocations(load_addr, load_offs, dyn_info[DT_JMPREL]->d_un.d_ptr, dyn_info[DT_PLTRELSZ]->d_un.d_val);
}
}
system/serial.c
+33
View File
@@ -18,6 +18,11 @@
#include <larchintrin.h>
#endif
#ifdef __aarch64__
#include "mmio.h"
#include "vmem.h"
#endif
static struct serial_port console_serial;
//------------------------------------------------------------------------------
@@ -76,6 +81,13 @@ static void serial_echo_print(const char *p)
/* Now, do each character */
while (*p) {
/* Send the character out. */
#ifdef __aarch64__
if (port->is_pl011) {
while (mmio_read32((uint32_t *)(port->base_addr + PL011_FR)) & PL011_FR_TXFF) { }
mmio_write32((uint32_t *)(port->base_addr + PL011_DR), *p++);
continue;
}
#endif
serial_wait_for_xmit(port);
serial_write_reg(port, UART_TX, *p++);
}
@@ -110,6 +122,9 @@ void tty_init(void)
console_serial.base_addr = map_region(tty_address, 0x0, false);
// By default, CPU UART0 is used, which uses the stable counter as the clock.
tty_mmio_ref_clk = (__cpucfg(0x4) * (__cpucfg(0x5) & 0xFFFF)) / ((__cpucfg(0x5) >> 16) & 0xFFFF);
#elif defined(__aarch64__)
console_serial.base_addr = map_region(tty_address, 0x1000, false);
console_serial.is_pl011 = tty_pl011;
#else
console_serial.base_addr = tty_address;
#endif
@@ -131,6 +146,14 @@ void tty_init(void)
console_serial.refclk = UART_REF_CLK_IO;
}
if (console_serial.is_pl011) {
// The PL011 has already been configured by the firmware; just use
// it as-is.
tty_clear_screen();
tty_disable_cursor();
return;
}
/* read the Divisor Latch */
uart_status = serial_read_reg(&console_serial, UART_LCR);
serial_write_reg(&console_serial, UART_LCR, uart_status | UART_LCR_DLAB);
@@ -249,6 +272,16 @@ char tty_get_char(int max_wait_frames)
{
int wait_time = max_wait_frames * console_serial.frame_time;
do {
#ifdef __aarch64__
if (console_serial.is_pl011) {
if (!(mmio_read32((uint32_t *)(console_serial.base_addr + PL011_FR)) & PL011_FR_RXFE)) {
return mmio_read32((uint32_t *)(console_serial.base_addr + PL011_DR)) & 0xFF;
}
usleep(10);
wait_time -= 10;
continue;
}
#endif
int uart_status = serial_read_reg(&console_serial, UART_LSR);
if (uart_status & UART_LSR_DR) {
return serial_read_reg(&console_serial, UART_RX);
system/serial.h
+11
View File
@@ -23,6 +23,7 @@ static const uint16_t serial_io_ports[] = { 0x3f8, 0x2f8, 0x3e8, 0x2e8 };
struct serial_port {
bool enable;
bool is_mmio;
bool is_pl011;
int parity;
int bits;
int baudrate;
@@ -32,6 +33,16 @@ struct serial_port {
uintptr_t base_addr;
};
/*
* Definitions for the ARM PL011 UART registers
*/
#define PL011_DR 0x00 /* Data Register */
#define PL011_FR 0x18 /* Flag Register */
#define PL011_FR_TXFF 0x20 /* Transmit FIFO full */
#define PL011_FR_RXFE 0x10 /* Receive FIFO empty */
/*
* Definitions for VT100 commands
*/
system/simd.h
+7 -1
View File
@@ -18,7 +18,9 @@
typedef enum {
SIMD_NONE, // scalar only
SIMD_SSE2, // x86 128-bit
SIMD_AVX2 // x86 256-bit
SIMD_AVX2, // x86 256-bit
SIMD_NEON, // aarch64 128-bit
SIMD_SVE // aarch64 scalable (reserved; no kernels yet)
} simd_tier_t;
/**
@@ -42,6 +44,10 @@ static inline const char *simd_tier_name(void)
return "AVX2";
case SIMD_SSE2:
return "SSE2";
case SIMD_NEON:
return "NEON";
case SIMD_SVE:
return "SVE";
default:
return NULL;
}
system/smbios.c
+9 -6
View File
@@ -25,15 +25,13 @@ static const efi_guid_t SMBIOS2_GUID = { 0xeb9d2d31, 0x2d88, 0x11d3, {0x9a, 0x16
// Some firmware (e.g. QEMU virt, ARM laptops) only publishes the 64-bit SMBIOS v3 entry point.
static const efi_guid_t SMBIOS3_GUID = { 0xf2fd1544, 0x9794, 0x4a2c, {0x99, 0x2e, 0xe5, 0xbb, 0xcf, 0x20, 0xe3, 0x94} };
// Consumers of dmi_memory_device read ->type without a NULL check, so point
// it at an all-zero struct (type 0 = undefined) until a real one is found.
static struct mem_dev null_mem_dev;
struct system_info *dmi_system_info;
struct baseboard_info *dmi_baseboard_info;
struct mem_dev *dmi_memory_device = &null_mem_dev;
struct mem_dev *dmi_memory_device; // no static initialiser: reloc() would rebase it (see reloc64.c)
struct cpu_info *dmi_cpu_info;
uint8_t dmi_memory_device_type;
struct mem_dev *dmi_memory_devices[MAX_DMI_MEM_DEVICES];
int dmi_num_memory_devices = 0;
@@ -179,8 +177,9 @@ static int parse_dmi(uint16_t numstructs)
// Multiple type 17 structs are allowed, with unpopulated slots sometimes
// reported as type 2 (unknown). If type is 0 (uninitialized) or 1/2 (previously
// initialized with unknown value) => set or overwrite the struct
if (dmi_memory_device->type <= 2) {
if (dmi_memory_device_type <= 2) {
dmi_memory_device = mdev;
dmi_memory_device_type = mdev->type;
}
// Collect every populated device (size 0 means empty socket,
// 0xFFFF means populated with unknown size, so keep the latter).
@@ -195,6 +194,8 @@ static int parse_dmi(uint16_t numstructs)
dmi_system_info = NULL;
dmi_baseboard_info = NULL;
dmi_cpu_info = NULL;
dmi_memory_device = NULL;
dmi_memory_device_type = 0;
dmi_num_memory_devices = 0;
return -1;
}
@@ -209,6 +210,8 @@ static int parse_dmi(uint16_t numstructs)
dmi_system_info = NULL;
dmi_baseboard_info = NULL;
dmi_cpu_info = NULL;
dmi_memory_device = NULL;
dmi_memory_device_type = 0;
dmi_num_memory_devices = 0;
return -1;
}
system/smbios.h
+7
View File
@@ -163,6 +163,13 @@ struct mem_dev {
extern struct mem_dev *dmi_memory_device;
/**
* Cached copy of dmi_memory_device->type (0 = undefined). Use this after
* startup: pointers into the SMBIOS table do not survive relocation.
*/
extern uint8_t dmi_memory_device_type;
/**
* Maximum number of SMBIOS Type 17 (Memory Device) structs collected.
* Pointers only, so this is cheap; 32 covers 2-socket servers with
system/smp.c
+116 -5
View File
@@ -19,6 +19,12 @@
#include <larchintrin.h>
#endif
#if defined(__aarch64__)
#include "registers.h"
#include "cache.h"
#include "psci.h"
#endif
#include "acpi.h"
#include "boot.h"
#include "macros.h"
@@ -100,14 +106,21 @@
#define MADT_PROCESSOR 0
#define MADT_LAPIC_ADDR 5
#define MADT_PROCESSOR_X2APIC 9
#define MADT_GICC 11
#define MADT_CORE_PIC 17
#define MADT_PROCESSOR_X2APIC 9
// MADT processor flag values
#define MADT_PF_ENABLED 0x1
#define MADT_PF_ONLINE_CAPABLE 0x2
// MADT GICC flag values
#define MADT_GICC_ENABLED 0x1
#define MADT_GICC_ONLINE_CAPABLE 0x8
// SRAT entry types
#define SRAT_PROCESSOR_APIC_AFFINITY 0
@@ -131,6 +144,16 @@
typedef volatile uint32_t apic_register_t[4];
// The type used to identify a CPU core. On most architectures this is the
// local APIC ID or equivalent. On ARM64 it is the (up to 40-bit) MPIDR
// affinity value.
#if defined(__aarch64__)
typedef uint64_t cpu_apic_id_t;
#else
typedef uint32_t cpu_apic_id_t;
#endif
typedef struct __attribute__((packed)) {
uint32_t proximity_domain_idx;
uint64_t start;
@@ -248,6 +271,26 @@ typedef struct {
} madt_processor_entry_t;
#pragma pack ()
#elif defined(__aarch64__)
typedef struct __attribute__((packed)) {
uint8_t type;
uint8_t length;
uint16_t reserved1;
uint32_t cpu_interface_num;
uint32_t acpi_processor_uid;
uint32_t flags;
uint32_t parking_version;
uint32_t performance_gsiv;
uint64_t parked_address;
uint64_t gicc_base;
uint64_t gicv_base;
uint64_t gich_base;
uint32_t vgic_gsiv;
uint64_t gicr_base;
uint64_t mpidr;
} madt_gicc_entry_t;
#endif
typedef struct {
@@ -312,11 +355,13 @@ typedef struct __attribute__((packed)) {
// Private Variables
//------------------------------------------------------------------------------
#if !defined(__aarch64__)
static apic_register_t *apic = NULL;
#endif
static uint32_t cpu_num_to_proximity_domain_idx[MAX_CPUS];
static uint32_t cpu_num_to_apic_id[MAX_CPUS];
static cpu_apic_id_t cpu_num_to_apic_id[MAX_CPUS];
static memory_affinity_t memory_affinity_ranges[MAX_APIC_IDS];
@@ -329,7 +374,9 @@ static uintptr_t smp_heap_page = 0;
static uintptr_t alloc_addr = 0;
#if !defined(__aarch64__)
static bool apic_x2apic = false;
#endif
//------------------------------------------------------------------------------
// Variables
@@ -345,6 +392,7 @@ uint8_t highest_map_bit = 0;
// Private Functions
//------------------------------------------------------------------------------
#if !defined(__aarch64__)
static int my_apic_id(void)
{
#if defined(__i386__) || defined(__x86_64__)
@@ -358,6 +406,7 @@ static int my_apic_id(void)
return ((int)__csrrd_w(0x20));
#endif
}
#endif
#if defined(__i386__) || defined(__x86_64__)
static void apic_write(int reg, uint32_t val)
@@ -533,10 +582,21 @@ static bool find_cpus_in_madt(void)
int found_cpus = 0;
#if defined(__aarch64__)
uint64_t bsp_mpidr = read_sysreg(mpidr_el1) & MPIDR_AFFINITY_MASK;
cpu_num_to_apic_id[0] = bsp_mpidr;
#endif
uint8_t *tab_entry_ptr = (uint8_t *)mpc + sizeof(*mpc);
uint8_t *mpc_table_end = (uint8_t *)mpc + mpc->h.length;
while (tab_entry_ptr < mpc_table_end) {
while (tab_entry_ptr + sizeof(madt_entry_header_t) <= mpc_table_end) {
madt_entry_header_t *entry_header = (madt_entry_header_t *)tab_entry_ptr;
// Reject malformed entries that could make us read past the end of
// the table or loop forever.
if (entry_header->length < sizeof(madt_entry_header_t)
|| tab_entry_ptr + entry_header->length > mpc_table_end) {
return false;
}
#if defined(__i386__) || defined(__x86_64__)
if (entry_header->type == MADT_PROCESSOR) {
if (entry_header->length != sizeof(madt_processor_entry_t)) {
@@ -591,10 +651,32 @@ static bool find_cpus_in_madt(void)
found_cpus++;
}
}
#elif defined(__aarch64__)
if (entry_header->type == MADT_GICC) {
// GICC entries are 76 bytes or longer, depending on the ACPI
// revision. All variants have the MPIDR at the same offset.
if (entry_header->length < 76) {
return false;
}
madt_gicc_entry_t *entry = (madt_gicc_entry_t *)tab_entry_ptr;
if (entry->flags & (MADT_GICC_ENABLED|MADT_GICC_ONLINE_CAPABLE)) {
uint64_t mpidr = entry->mpidr & MPIDR_AFFINITY_MASK;
if (mpidr != bsp_mpidr && num_available_cpus < MAX_CPUS) {
cpu_num_to_apic_id[num_available_cpus] = mpidr;
num_available_cpus++;
}
found_cpus++;
}
}
#endif
tab_entry_ptr += entry_header->length;
}
#if defined(__aarch64__)
// There is no memory-mapped local interrupt controller to map.
(void)apic_addr;
(void)found_cpus;
#else
if (!apic_x2apic) {
apic = (volatile apic_register_t *)map_region(apic_addr, APIC_REGS_SIZE, false);
if (apic == NULL) {
@@ -602,6 +684,7 @@ static bool find_cpus_in_madt(void)
return false;
}
}
#endif
return true;
}
@@ -783,6 +866,7 @@ static bool parse_slit(uintptr_t slit_addr)
}
#endif
#if !defined(__aarch64__)
static inline void send_ipi(int apic_id, int trigger __attribute__((unused)), int level __attribute__((unused)), int mode, uint8_t vector)
{
#if defined(__i386__) || defined(__x86_64__)
@@ -836,6 +920,7 @@ static bool send_ipi_and_wait(int apic_id, int trigger, int level, int mode, uin
return true;
#endif
}
#endif // !defined(__aarch64__)
#if defined(__i386__) || defined(__x86_64__)
static uint32_t read_apic_esr(bool is_p5)
@@ -914,6 +999,14 @@ static bool start_cpu(int cpu_num)
return true;
}
#elif defined(__aarch64__)
static bool start_cpu(int cpu_num)
{
// The AP enters startup64 with the MMU off and its CPU number in x0.
int64_t status = psci_cpu_on(cpu_num_to_apic_id[cpu_num], (uintptr_t)ap_startup_addr, cpu_num);
return (status == PSCI_RET_SUCCESS);
}
#endif
#if defined(__loongarch_lp64)
@@ -1077,7 +1170,7 @@ void smp_init(bool smp_enable)
#if defined(__i386__) || defined(__x86_64__)
alloc_addr = HEAP_BASE_ADDR + (ap_trampoline_end - ap_trampoline);
#elif defined(__loongarch_lp64)
#elif defined(__loongarch_lp64) || defined(__aarch64__)
alloc_addr = HEAP_BASE_ADDR;
#endif
}
@@ -1094,6 +1187,12 @@ int smp_start(cpu_state_t cpu_state[MAX_CPUS])
memcpy((uint8_t *)HEAP_BASE_ADDR, ap_trampoline, ap_trampoline_end - ap_trampoline);
#elif defined(__loongarch_lp64)
ap_startup_addr = (uintptr_t)startup64;
#elif defined(__aarch64__)
ap_startup_addr = (uintptr_t)startup64;
// The APs boot with MMU and caches off, so make the (possibly relocated)
// program image visible at the point of coherency before waking them.
cache_clean_range(_start, _end);
#endif
cpu_state[0] = CPU_STATE_RUNNING; // we don't support disabling the boot CPU
@@ -1135,18 +1234,29 @@ int smp_start(cpu_state_t cpu_state[MAX_CPUS])
#endif
}
void smp_send_nmi(int cpu_num)
void smp_send_nmi(int cpu_num __attribute__((unused)))
{
#if defined(__aarch64__)
// Wake up all CPUs waiting in WFE. The waiters recheck their wakeup flag,
// so waking more CPUs than necessary is harmless. The DSB ensures the flag
// update is visible before the event, or the wakeup could be missed.
__asm__ __volatile__ ("dsb ish; sev" ::: "memory");
#else
#if defined(__i386__) || defined(__x86_64__)
while (apic_read(APIC_REG_ICRLO) & APIC_ICR_BUSY) {
__builtin_ia32_pause();
}
#endif
send_ipi(cpu_num_to_apic_id[cpu_num], 0, 0, APIC_DELMODE_NMI, 0);
#endif
}
int smp_my_cpu_num(void)
{
#if defined(__aarch64__)
// Our CPU number was stored in TPIDR_EL1 by the startup code.
return (int)read_sysreg(tpidr_el1);
#else
if (num_available_cpus <= 1) return 0;
int apic_id = my_apic_id();
@@ -1156,6 +1266,7 @@ int smp_my_cpu_num(void)
}
}
return 0;
#endif
}
uint32_t smp_get_proximity_domain_idx(int cpu_num)
system/spd.c
+12 -8
View File
@@ -23,6 +23,10 @@ ram_info_t ram = { 0, 0, 0, 0, 0, 0, "N/A"};
ram_slot_info_t ram_slot_info[MAX_SPD_SLOT];
spd_info spd_slot_cache[MAX_SPD_SLOT];
// spd_info.type is an inline array, not a pointer: a literal pointer stored in this
// BSS cache would have no relocation record and go stale after a program relocation.
#define set_spd_type(spdi, t) memcpy((spdi)->type, t, sizeof(t))
static inline uint8_t bcd_to_ui8(uint8_t bcd)
{
return bcd - 6 * (bcd >> 4);
@@ -89,7 +93,7 @@ void print_spdi(spd_info spdi, uint8_t row)
}
// Populate global ram var
ram.type = spdi.type;
memcpy(ram.type, spdi.type, sizeof(ram.type));
if (ram.freq == 0 || ram.freq > spdi.freq) {
ram.freq = spdi.freq;
}
@@ -128,7 +132,7 @@ static void read_sku(char *sku, uint8_t slot_idx, uint16_t offset, uint8_t max_l
static void parse_spd_ddr5(spd_info *spdi, uint8_t slot_idx)
{
spdi->type = "DDR5";
set_spd_type(spdi, "DDR5");
// Compute module size for symmetric & asymmetric configuration
for (int sbyte_adr = 1; sbyte_adr <= 2; sbyte_adr++) {
@@ -311,7 +315,7 @@ static void parse_spd_ddr5(spd_info *spdi, uint8_t slot_idx)
static void parse_spd_ddr4(spd_info *spdi, uint8_t slot_idx)
{
spdi->type = "DDR4";
set_spd_type(spdi, "DDR4");
// Compute module size in MB with shifts
spdi->module_size = 1U << (
@@ -432,7 +436,7 @@ static void parse_spd_ddr4(spd_info *spdi, uint8_t slot_idx)
static void parse_spd_ddr3(spd_info *spdi, uint8_t slot_idx)
{
spdi->type = "DDR3";
set_spd_type(spdi, "DDR3");
// Compute module size in MB with shifts
spdi->module_size = 1U << (
@@ -591,7 +595,7 @@ static void parse_spd_ddr3(spd_info *spdi, uint8_t slot_idx)
static void parse_spd_ddr2(spd_info *spdi, uint8_t slot_idx)
{
spdi->type = "DDR2";
set_spd_type(spdi, "DDR2");
// Compute module size in MB
switch (get_spd(slot_idx, 31)) {
@@ -730,7 +734,7 @@ static void parse_spd_ddr2(spd_info *spdi, uint8_t slot_idx)
static void parse_spd_ddr(spd_info *spdi, uint8_t slot_idx)
{
spdi->type = "DDR";
set_spd_type(spdi, "DDR");
// Compute module size in MB
switch (get_spd(slot_idx, 31)) {
@@ -821,7 +825,7 @@ static void parse_spd_ddr(spd_info *spdi, uint8_t slot_idx)
static void parse_spd_rdram(spd_info *spdi, uint8_t slot_idx)
{
spdi->type = "RDRAM";
set_spd_type(spdi, "RDRAM");
// Compute module size in MB
uint8_t tbyte = get_spd(slot_idx, 5);
@@ -896,7 +900,7 @@ static void parse_spd_rdram(spd_info *spdi, uint8_t slot_idx)
static void parse_spd_sdram(spd_info *spdi, uint8_t slot_idx)
{
spdi->type = "SDRAM";
set_spd_type(spdi, "SDRAM");
uint8_t spd_byte3 = get_spd(slot_idx, 3) & 0x0F; // Number of Row Addresses (2 x 4 bits, upper part used if asymmetrical banking used)
uint8_t spd_byte4 = get_spd(slot_idx, 4) & 0x0F; // Number of Column Addresses (2 x 4 bits, upper part used if asymmetrical banking used)
system/spd.h
+2 -2
View File
@@ -32,7 +32,7 @@ typedef struct spd_infos {
uint8_t slot_num;
uint16_t jedec_code;
uint32_t module_size;
char *type;
char type[6]; // inline, not a pointer: the BSS cache has no reloc records to rebase it
char sku[SPD_SKU_LEN + 1];
uint8_t XMP;
uint16_t freq;
@@ -55,7 +55,7 @@ typedef struct ram_infos {
uint16_t tRCD;
uint16_t tRP;
uint16_t tRAS;
char *type;
char type[6];
} ram_info_t;
typedef struct {
system/timers.c
+46 -1
View File
@@ -1,6 +1,6 @@
// SPDX-License-Identifier: GPL-2.0
// Copyright (C) 2020-2022 Martin Whitaker.
// Copyright (C) 2004-2022 Sam Demeulemeester.
// Copyright (C) 2004-2026 Sam Demeulemeester.
// Copyright (C) 2024 Loongson Technology Corporation Limited. All rights reserved.
#include <stdbool.h>
@@ -19,6 +19,10 @@
#include "registers.h"
#endif
#if defined(__aarch64__)
#include "registers.h"
#endif
//------------------------------------------------------------------------------
// Constants
//------------------------------------------------------------------------------
@@ -129,6 +133,47 @@ static void correct_tsc(void)
clks_per_msec = (end - start) / num_millisec;
}
#elif defined(__aarch64__)
static void correct_tsc(void)
{
// The generic timer has a known, constant frequency, and get_tsc()
// returns the generic timer count, so all time calculations must use
// the generic timer frequency. Note this is NOT the CPU clock frequency
// on this architecture (typically 19.2MHz - 1GHz).
uint64_t timer_freq = read_sysreg(cntfrq_el0);
uint32_t ticks_per_msec = timer_freq / 1000;
clks_per_msec = ticks_per_msec;
// Use the PMU cycle counter to measure the actual CPU clock frequency
// over a 50ms interval. This is only used for display purposes.
uint64_t num_millisec = 50;
write_sysreg(read_sysreg(pmcr_el0) | 0x5, pmcr_el0); // enable, reset cycle counter
write_sysreg(UINT64_C(1) << 31, pmcntenset_el0); // enable the cycle counter
write_sysreg(0, pmccfiltr_el0); // count cycles at EL1
__asm__ __volatile__ ("isb");
uint64_t expected_ticks, current_ticks;
__asm__ __volatile__ ("isb; mrs %0, cntvct_el0" : "=r" (current_ticks));
expected_ticks = current_ticks + ticks_per_msec * num_millisec;
uint64_t start = read_sysreg(pmccntr_el0);
do {
__asm__ __volatile__ ("isb; mrs %0, cntvct_el0" : "=r" (current_ticks));
} while (current_ticks < expected_ticks);
uint64_t end = read_sysreg(pmccntr_el0);
write_sysreg(read_sysreg(pmcr_el0) & ~UINT64_C(1), pmcr_el0); // disable the PMU
cpu_clk_mhz = (end - start) / num_millisec / 1000;
// If the PMU is unimplemented or not counting, leave the CPU clock
// frequency unknown rather than displaying the timer frequency.
if (cpu_clk_mhz <= ticks_per_msec / 1000) {
cpu_clk_mhz = 0;
}
}
#endif
//------------------------------------------------------------------------------
system/tsc.h
+28
View File
@@ -16,6 +16,9 @@
// LoongArch GCC builtin function
#include <larchintrin.h>
#endif
#if defined(__aarch64__)
#include "cpuinfo.h"
#endif
#define rdtsc(low, high) \
__asm__ __volatile__("rdtsc" \
@@ -60,6 +63,31 @@ static inline uint64_t get_tsc(void)
return (val * (clks_per_msec / stable_count_freq));
}
#elif defined(__aarch64__)
static inline uint64_t get_tsc(void)
{
uint64_t count;
static uint64_t timer_freq_khz = 0;
if (!timer_freq_khz) {
uint64_t freq;
__asm__ __volatile__ ("mrs %0, cntfrq_el0" : "=r" (freq));
timer_freq_khz = freq / 1000;
if (!timer_freq_khz) {
timer_freq_khz = 1;
}
}
// Scale the generic timer count to CPU clock cycles, so that intervals
// measured with get_tsc() can be converted to time using clks_per_msec.
__asm__ __volatile__ (
"isb \n\t"
"mrs %0, cntvct_el0 \n\t"
: "=r" (count)
);
return count * (clks_per_msec / timer_freq_khz);
}
#endif
#endif // TSC_H
system/usbhcd.c
+100 -5
View File
@@ -19,6 +19,11 @@
#include "string.h"
#include "unistd.h"
#if defined(USB_DEBUG_HOLD)
#include "cpuinfo.h"
#include "tsc.h"
#endif
#include "usbhcd.h"
//------------------------------------------------------------------------------
@@ -48,6 +53,7 @@ typedef enum {
typedef struct {
hci_type_t type;
bool platform; // a non-PCI controller at a fixed MMIO address
uint8_t bus;
uint8_t dev;
uint8_t func;
@@ -101,6 +107,10 @@ static bool usb_runtime_scan = false;
static usb_msd_t usb_msd_info;
// Index into hcd_list of the controller hosting the MSD. Stored as an index because
// a cached usb_hcd_t pointer would go stale on relocation (see the note in reloc64.c).
static int usb_msd_hcd_idx = -1;
//------------------------------------------------------------------------------
// Public Variables
//------------------------------------------------------------------------------
@@ -264,8 +274,9 @@ static void get_keyboard_info_from_descriptors(const uint8_t *desc_buffer, int d
} else if (header->type == USB_DESC_ENDPOINT && header->length == sizeof(usb_endpoint_desc_t)) {
usb_endpoint_desc_t *endpoint = (usb_endpoint_desc_t *)curr_ptr;
if (usb_init_options & USB_DEBUG_KBD) {
print_usb_info("endpoint addr 0x%02x attr 0x%02x",
(uintptr_t)endpoint->address, (uintptr_t)endpoint->attributes);
print_usb_info("endpoint addr 0x%02x attr 0x%02x maxpkt %i interval %i",
(uintptr_t)endpoint->address, (uintptr_t)endpoint->attributes,
(uintptr_t)endpoint->max_packet_size, (uintptr_t)endpoint->interval);
sleep(1);
}
if (kbd && (endpoint->address & 0x80) && (endpoint->attributes & 0x3) == 0x3) {
@@ -463,7 +474,7 @@ static bool check_for_usb_msd(const usb_hcd_t *hcd, const usb_ep_t *ep0, usb_spe
}
ep_out.driver_data = *(uintptr_t *)hcd->ws->data_buffer;
usb_msd_info.hcd = hcd;
usb_msd_hcd_idx = hcd - hcd_list;
usb_msd_info.ep0 = *ep0;
usb_msd_info.ep_in = ep_in;
usb_msd_info.ep_out = ep_out;
@@ -583,6 +594,7 @@ static int find_usb_controllers(hci_info_t hci_list[])
hci_type_t controller_type = pci_config_read8(bus, dev, func, 0x09) >> 4;
if (controller_type < MAX_HCI_TYPE) {
hci_list[num_hci].type = controller_type;
hci_list[num_hci].platform = false;
hci_list[num_hci].bus = bus;
hci_list[num_hci].dev = dev;
hci_list[num_hci].func = func;
@@ -606,6 +618,18 @@ static int find_usb_controllers(hci_info_t hci_list[])
}
}
}
#if defined(__aarch64__)
// Add platform (non-PCI) XHCI controllers at fixed MMIO addresses.
uintptr_t plat_base;
for (int i = 0; num_hci < MAX_HCI && platform_usb_controller(i, &plat_base); i++) {
hci_list[num_hci].type = XHCI;
hci_list[num_hci].platform = true;
hci_list[num_hci].pm_base_addr = plat_base;
num_hci++;
}
#endif
return num_hci;
}
@@ -613,6 +637,36 @@ static void reset_usb_controller(hci_info_t *hci)
{
hci_type_t controller_type = hci->type;
#if defined(__aarch64__)
if (hci->platform) {
// A platform controller has no PCI config space; the firmware has
// already powered and clocked it. Just map and reset it.
print_usb_info("Found platform %s controller at %08x",
hci_name[controller_type], hci->pm_base_addr);
uintptr_t plat_addr = map_region(hci->pm_base_addr, 0x20000, false);
if (plat_addr == 0 || controller_type != XHCI) {
print_usb_info(" Failed to map device into virtual memory");
hci->type = NOT_HCI; // mark this controller as unusable
return;
}
#if defined(USB_DEBUG_HOLD)
// Identify the controller (Synopsys DWC3/DWC_usb31 ID register,
// 0x5533xxxx/0x3331xxxx; 0 or all-ones = unclocked or wrong
// address) and its port direction (GCTL[13:12], 1 = host mode).
uint32_t gsnpsid = read32((uint32_t *)(plat_addr + 0xC120));
uint32_t gctl = read32((uint32_t *)(plat_addr + 0xC110));
print_usb_info(" DWC3 id %08x prtcapdir %i", gsnpsid, (gctl >> 12) & 3);
#endif
if (!xhci_reset(plat_addr)) {
print_usb_info(" Controller reset failed");
hci->type = NOT_HCI; // mark this controller as unusable
return;
}
hci->vm_base_addr = plat_addr;
return;
}
#endif
int bus = hci->bus;
int dev = hci->dev;
int func = hci->func;
@@ -653,6 +707,20 @@ static void reset_usb_controller(hci_info_t *hci)
mmio_size &= ~(uintptr_t)0xf;
mmio_size = ~mmio_size + 1;
#if defined(__aarch64__)
// Firmware often leaves the BARs of devices it didn't use unassigned
// on this architecture, so allocate an MMIO address ourselves.
if (!in_io_space && base_addr == 0 && mmio_size != 0) {
base_addr = pci_alloc_mmio(bus, dev, func, bar, mmio_size);
if (base_addr == 0) {
print_usb_info("Found %s controller %04x:%04x with no MMIO address; allocation failed",
hci_name[controller_type], (uintptr_t)vendor_id, (uintptr_t)device_id);
hci->type = NOT_HCI; // mark this controller as unusable
return;
}
}
#endif
// Restore access to the device and set the bus master flag in case the BIOS hasn't.
pci_config_write16(bus, dev, func, 0x04, pci_command | (in_io_space ? 0x0005 : 0x0006));
@@ -950,6 +1018,8 @@ bool find_attached_usb_keyboards(const usb_hcd_t *hcd, const usb_hub_t *hub, int
}
usb_device_desc_t *device = (usb_device_desc_t *)hcd->ws->data_buffer;
bool is_hub = (device->class == USB_CLASS_HUB);
// Saved before the configuration descriptors below overwrite the device descriptor.
uint8_t product_str_index = device->product_str;
uint8_t num_configs = device->num_configs;
uint16_t vendor_id = device->vendor_id;
@@ -1011,7 +1081,8 @@ bool find_attached_usb_keyboards(const usb_hcd_t *hcd, const usb_hub_t *hub, int
}
if (!configure_keyboard(hcd, &ep0, kbd->interface_num)) break;
print_usb_info(" Keyboard found on port %i interface %i endpoint %i",
print_usb_info(" Keyboard %04x:%04x found on port %i interface %i endpoint %i",
(uintptr_t)vendor_id, (uintptr_t)product_id,
port_num, kbd->interface_num, kbd->endpoint_num);
keyboard_found = true;
@@ -1060,6 +1131,12 @@ bool process_usb_keyboard_report(const usb_hcd_t *hcd, const hid_kbd_rpt_t *repo
void find_usb_keyboards(bool pause_if_none)
{
#if defined(USB_DEBUG_HOLD)
// Diagnostic builds: also dump the interface and endpoint descriptors
// of each device encountered during enumeration.
usb_init_options |= USB_DEBUG_KBD;
#endif
clear_screen();
print_usb_info("Scanning for USB keyboards & Mass Storage Devices...");
@@ -1107,6 +1184,23 @@ void find_usb_keyboards(bool pause_if_none)
print_row--; // overwrite message
}
}
#if defined(USB_DEBUG_HOLD)
// Diagnostic builds: echo raw keycodes for a while so keyboard traffic
// (or its absence) can be observed, then continue unattended.
print_usb_info("USB_DEBUG_HOLD: type on the keyboard(s) now (20 seconds)...");
{
uint64_t end_time = get_tsc() + 20 * 1000 * (uint64_t)clks_per_msec;
int echoed = 0;
while (get_tsc() < end_time) {
uint8_t keycode = get_usb_keycode();
if (keycode != 0 && echoed < 20) {
print_usb_info(" got keycode %02x", (uintptr_t)keycode);
echoed++;
}
}
}
#endif
}
uint8_t get_usb_keycode(void)
@@ -1137,10 +1231,11 @@ void usb_rearm_keyboards(void)
bool find_usb_mass_storage(usb_msd_t *msd)
{
if (!usb_mass_storage_found) {
if (!usb_mass_storage_found || usb_msd_hcd_idx < 0) {
return false;
}
*msd = usb_msd_info;
msd->hcd = &hcd_list[usb_msd_hcd_idx];
msd->block_count = 0;
msd->block_size = 512;
return true;
system/vmem.h
+6
View File
@@ -34,6 +34,12 @@
*/
#define VM_WINDOW_SIZE PAGE_C(1,GB)
/**
* Set if any range could not be mapped when the page tables were built or
* rebuilt (never cleared). Accessing an unmapped range will fault.
*/
extern bool paging_incomplete;
/**
* Maps a physical memory region into the upper 2GB of virtual memory. The
* virtual address will have the same alignment within a page as the physical
system/x86/cpuinfo.c
+1 -1
View File
@@ -33,7 +33,7 @@
// Public Variables
//------------------------------------------------------------------------------
const char *cpu_model = NULL;
const char *cpu_model = ""; // static init required: creates the reloc record that rebases it
int l1_cache = 0;
int l2_cache = 0;
system/x86/i2c.c
+4 -4
View File
@@ -406,7 +406,7 @@ static bool ich5_get_smb(void)
// Enable I2C Host Controller Interface if disabled
// Use SMBUS Mode for DDR5 to allow bank switch using Proc Call
uint8_t temp = pci_config_read8(smbbus, smbdev, smbfun, 0x40);
if ((temp & 4) == 0 && dmi_memory_device->type != DMI_DDR5) {
if ((temp & 4) == 0 && dmi_memory_device_type != DMI_DDR5) {
pci_config_write8(smbbus, smbdev, smbfun, 0x40, temp | 0x04);
}
@@ -553,7 +553,7 @@ uint8_t get_spd(uint8_t slot_idx, uint16_t spd_adr)
uint8_t get_spd_hub_register(uint8_t slot_idx, uint8_t spd_hub_adr)
{
if(dmi_memory_device->type == DMI_DDR5) {
if(dmi_memory_device_type == DMI_DDR5) {
return ich5_read_spd_byte(slot_idx, spd_hub_adr | 0xFF00);
}
@@ -572,7 +572,7 @@ static uint8_t ich5_read_spd_byte(uint8_t smbus_adr, uint16_t spd_adr)
{
smbus_adr += 0x50;
if (dmi_memory_device->type == DMI_DDR4) {
if (dmi_memory_device_type == DMI_DDR4) {
// Switch page if needed (DDR4)
if (spd_adr > 0xFF && spd_page != 1) {
__outb((0x37 << 1) | I2C_WRITE, SMBHSTADD);
@@ -592,7 +592,7 @@ static uint8_t ich5_read_spd_byte(uint8_t smbus_adr, uint16_t spd_adr)
if (spd_adr > 0xFF) {
spd_adr -= 0x100;
}
} else if (dmi_memory_device->type == DMI_DDR5) {
} else if (dmi_memory_device_type == DMI_DDR5) {
// For DDR5, choose between reading from the SPD EEPROM (which may require a bank switch)
// and reading from the DDR5 SPD Hub Register (where we added a 0xFF00 offset).
system/x86/temperature.c
+1 -1
View File
@@ -165,7 +165,7 @@ int get_ram_temp(uint8_t slot)
float ram_temp;
// RAM Temperature is only supported on DDR5
if (dmi_memory_device->type != DMI_DDR5 || slot >= MAX_SPD_SLOT)
if (dmi_memory_device_type != DMI_DDR5 || slot >= MAX_SPD_SLOT)
return TEMP_INVALID;
// Check if slot has a temp sensor
system/x86/vmem.c
+6
View File
@@ -37,6 +37,12 @@
#define VM_REGION_END (VM_REGION_START + MAX_REGION_PAGES * VM_PAGE_SIZE - 1)
#define VM_SPACE_END 0xffffffff
//------------------------------------------------------------------------------
// Public Variables
//------------------------------------------------------------------------------
bool paging_incomplete = false; // never set on this architecture
//------------------------------------------------------------------------------
// Private Variables
//------------------------------------------------------------------------------
system/xhci.c
+49 -10
View File
@@ -317,7 +317,16 @@ typedef struct {
// Keyboard data transfer rings
ep_tr_t kbd_tr [MAX_KEYBOARDS];
// Keyboard data transfer buffers.
// Keyboard data transfer buffers. Sized for the largest full-speed interrupt
// packet: some keyboards send packets longer than the 8 byte boot protocol report,
// and a transfer shorter than the packet cause a babble error that halts the endpoint.
// Only the first 8 bytes (boot report) are used.
uint8_t kbd_xfer[MAX_KEYBOARDS][64] __attribute__ ((aligned (64)));
// Requested transfer length for each keyboard endpoint.
uint8_t kbd_xfer_len[MAX_KEYBOARDS];
// Last received keyboard reports.
hid_kbd_rpt_t kbd_rpt [MAX_KEYBOARDS];
// Saved keyboard reports.
@@ -921,6 +930,17 @@ static bool configure_kbd_ep(const usb_hcd_t *hcd, const usb_ep_t *ep, int kbd_i
ws->kbd_slot_id[kbd_idx] = ep->device_id;
ws->kbd_ep_id [kbd_idx] = 2 * ep->endpoint_num + 1; // EP <N> IN
// Request whole packets, as some devices send packets longer than the
// boot protocol report (a shorter request would cause a babble error).
size_t xfer_len = ep->max_packet_size;
if (xfer_len < sizeof(hid_kbd_rpt_t)) {
xfer_len = sizeof(hid_kbd_rpt_t);
}
if (xfer_len > sizeof(ws->kbd_xfer[kbd_idx])) {
xfer_len = sizeof(ws->kbd_xfer[kbd_idx]);
}
ws->kbd_xfer_len[kbd_idx] = xfer_len;
// Configure the controller.
return configure_interrupt_endpoint(ws, ep, 0, 0, 0, (uintptr_t)(&ws->kbd_tr[kbd_idx]), sizeof(hid_kbd_rpt_t));
}
@@ -1050,17 +1070,19 @@ static void rearm_keyboards(const usb_hcd_t *hcd)
ws->kbd_rearm_needed[kbd_idx] = false;
ep_tr_t *kbd_tr = &ws->kbd_tr[kbd_idx];
hid_kbd_rpt_t *kbd_rpt = &ws->kbd_rpt[kbd_idx];
// The discarded transfer completed, so its report is already in the buffer.
// Process it, otherwise a key release goes unnoticed and the next press of
// the same key is treated as a repeat and dropped.
// The discarded transfer's report is already in the DMA buffer; copy out
// the 8-byte boot report and process it so a key release is not missed.
hid_kbd_rpt_t *kbd_rpt = &ws->kbd_rpt[kbd_idx];
*kbd_rpt = *(hid_kbd_rpt_t *)ws->kbd_xfer[kbd_idx];
hid_kbd_rpt_t *prev_kbd_rpt = &ws->prev_kbd_rpt[kbd_idx];
if (process_usb_keyboard_report(hcd, kbd_rpt, prev_kbd_rpt)) {
*prev_kbd_rpt = *kbd_rpt;
}
issue_normal_trb(kbd_tr, kbd_rpt, XHCI_TRB_DIR_IN, sizeof(hid_kbd_rpt_t));
// Re-arm into the whole-packet buffer to avoid babble errors on >8-byte reports.
issue_normal_trb(kbd_tr, ws->kbd_xfer[kbd_idx], XHCI_TRB_DIR_IN, ws->kbd_xfer_len[kbd_idx]);
ring_device_doorbell(ws->db_regs, ws->kbd_slot_id[kbd_idx], ws->kbd_ep_id[kbd_idx]);
}
}
@@ -1072,12 +1094,30 @@ static void poll_keyboards(const usb_hcd_t *hcd)
xhci_trb_t event;
while (get_xhci_event(ws, &event)) {
if (event_type(&event) != XHCI_TRB_TRANSFER_EVENT || event_cc(&event) != XHCI_EVENT_CC_SUCCESS) continue;
if (event_type(&event) != XHCI_TRB_TRANSFER_EVENT) continue;
// A short packet is normal: the transfer requests a whole packet,
// but boot protocol reports may be shorter.
int cc = event_cc(&event);
if (cc != XHCI_EVENT_CC_SUCCESS && cc != XHCI_EVENT_CC_SHORT_PACKET) {
#if defined(USB_DEBUG_HOLD)
// Diagnostic builds: make discarded transfer errors (e.g. a
// babble error, which halts the endpoint) visible.
static int error_prints = 0;
if (error_prints < 5) {
print_usb_info(" kbd transfer event error cc %i", cc);
error_prints++;
}
#endif
continue;
}
int kbd_idx = identify_keyboard(ws, event_slot_id(&event), event_ep_id(&event));
if (kbd_idx < 0) continue;
// Only the first 8 bytes hold the boot protocol report.
hid_kbd_rpt_t *kbd_rpt = &ws->kbd_rpt[kbd_idx];
*kbd_rpt = *(hid_kbd_rpt_t *)ws->kbd_xfer[kbd_idx];
hid_kbd_rpt_t *prev_kbd_rpt = &ws->prev_kbd_rpt[kbd_idx];
if (process_usb_keyboard_report(hcd, kbd_rpt, prev_kbd_rpt)) {
@@ -1085,7 +1125,7 @@ static void poll_keyboards(const usb_hcd_t *hcd)
}
ep_tr_t *kbd_tr = &ws->kbd_tr[kbd_idx];
issue_normal_trb(kbd_tr, kbd_rpt, XHCI_TRB_DIR_IN, sizeof(hid_kbd_rpt_t));
issue_normal_trb(kbd_tr, ws->kbd_xfer[kbd_idx], XHCI_TRB_DIR_IN, ws->kbd_xfer_len[kbd_idx]);
ring_device_doorbell(ws->db_regs, ws->kbd_slot_id[kbd_idx], ws->kbd_ep_id[kbd_idx]);
}
}
@@ -1481,8 +1521,7 @@ bool xhci_probe(uintptr_t base_addr, usb_hcd_t *hcd)
ep_tr_t *kbd_tr = &ws->kbd_tr[kbd_idx];
kbd_tr->enqueue_state = EP_TR_SIZE; // cycle = 1, index = 0
hid_kbd_rpt_t *kbd_rpt = &ws->kbd_rpt[kbd_idx];
issue_normal_trb(kbd_tr, kbd_rpt, XHCI_TRB_DIR_IN, sizeof(hid_kbd_rpt_t));
issue_normal_trb(kbd_tr, ws->kbd_xfer[kbd_idx], XHCI_TRB_DIR_IN, ws->kbd_xfer_len[kbd_idx]);
ring_device_doorbell(ws->db_regs, ws->kbd_slot_id[kbd_idx], ws->kbd_ep_id[kbd_idx]);
}
tests/aarch64/vec_prsg_neon.c
+155
View File
@@ -0,0 +1,155 @@
// SPDX-License-Identifier: GPL-2.0
// Copyright (C) 2004-2026 Sam Demeulemeester.
//
// NEON kernels for the PRSG vector fill/check primitives. See
// tests/vec_prsg.h for the kernel contract and tests/vec_prsg.c for the
// dispatch wrappers and scalar fallbacks.
//
// The kernels are written in inline assembly, so the compiler needs no
// intrinsics headers. On a 64-bit build a vector block is VEC_LANES(4) x
// 64-bit = 256 bits = two 128-bit NEON registers (q0 = {lane0,lane1},
// q1 = {lane2,lane3}), matching the two-xmm SSE2 layout.
//
// Forward xorshift: x ^= x << 13; x ^= x >> 7; x ^= x << 17;
// The backward step inverts each stage in reverse order by repeated squaring:
// y = x ^ (x << s) is undone by x = y; x ^= x << s; x ^= x << 2s; x ^= x << 4s;
// ... doubling the shift until it exceeds the word width.
//
// Stores use STNP (store pair, non-temporal hint): the closest ARM analogue to
// the x86 non-temporal stores, avoiding cache allocation on write-heavy fills.
// The hint is advisory; actual eviction to DRAM between sweeps is still owned by
// flush_caches() / cache_flush(), not by the store type. Each kernel ends on a
// DSB ISH so no store is left buffered (the analogue of the x86 sfence).
#if defined(__aarch64__)
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
#include "vec_prsg.h"
// One forward xorshift step (x ^= x<<13; x ^= x>>7; x ^= x<<17) on the two
// 64-bit lanes in state register s, using t as a scratch register.
#define NEON_STEP_FWD(s, t) \
"shl " t ".2d, " s ".2d, #13 \n\t" \
"eor " s ".16b, " s ".16b, " t ".16b \n\t" \
"ushr " t ".2d, " s ".2d, #7 \n\t" \
"eor " s ".16b, " s ".16b, " t ".16b \n\t" \
"shl " t ".2d, " s ".2d, #17 \n\t" \
"eor " s ".16b, " s ".16b, " t ".16b \n\t"
// One repeated-squaring inverse stage: undoes x ^= x <op> sh.
#define NEON_UNDO(s, t, op, sh) \
op " " t ".2d, " s ".2d, #" #sh " \n\t" \
"eor " s ".16b, " s ".16b, " t ".16b \n\t"
// One backward xorshift step: inverts <<17, >>7, <<13 in that order.
#define NEON_STEP_BACK(s, t) \
NEON_UNDO(s, t, "shl", 17) \
NEON_UNDO(s, t, "shl", 34) \
NEON_UNDO(s, t, "ushr", 7) \
NEON_UNDO(s, t, "ushr", 14) \
NEON_UNDO(s, t, "ushr", 28) \
NEON_UNDO(s, t, "ushr", 56) \
NEON_UNDO(s, t, "shl", 13) \
NEON_UNDO(s, t, "shl", 26) \
NEON_UNDO(s, t, "shl", 52)
void vec_fill_neon(vec_state_t *st, testword_t *p, size_t nblocks, bool splat)
{
size_t i = 0;
__asm__ __volatile__ (
"ldp q0, q1, [%[st]] \n\t"
"0: \n\t"
"stnp q0, q1, [%[p]] \n\t"
"cbnz %w[splat], 1f \n\t"
NEON_STEP_FWD("v0", "v4")
NEON_STEP_FWD("v1", "v4")
"1: \n\t"
"add %[p], %[p], #32 \n\t"
"add %[i], %[i], #1 \n\t"
"cmp %[i], %[n] \n\t"
"b.lo 0b \n\t"
"stp q0, q1, [%[st]] \n\t"
"dsb ish \n\t"
: [p] "+r" (p), [i] "+r" (i)
: [st] "r" (st), [n] "r" (nblocks), [splat] "r" ((uint32_t)splat)
: "v0", "v1", "v4", "memory", "cc"
);
}
size_t vec_scan_fwd_neon(vec_state_t *st, testword_t *p, size_t nblocks, bool splat)
{
size_t done = 0;
__asm__ __volatile__ (
"ldp q0, q1, [%[st]] \n\t"
"0: \n\t"
"ldp q2, q3, [%[p]] \n\t"
"cmeq v2.2d, v2.2d, v0.2d \n\t" // all-ones per matching lane
"cmeq v3.2d, v3.2d, v1.2d \n\t"
"and v2.16b, v2.16b, v3.16b \n\t"
"uminv b5, v2.16b \n\t" // 0xFF iff every lane matched
"umov w8, v5.b[0] \n\t"
"cmp w8, #0xff \n\t"
"b.ne 2f \n\t"
"mvn v2.16b, v0.16b \n\t" // complement = ~state
"mvn v3.16b, v1.16b \n\t"
"stnp q2, q3, [%[p]] \n\t"
"cbnz %w[splat], 1f \n\t"
NEON_STEP_FWD("v0", "v4")
NEON_STEP_FWD("v1", "v4")
"1: \n\t"
"add %[p], %[p], #32 \n\t"
"add %[done], %[done], #1 \n\t"
"cmp %[done], %[n] \n\t"
"b.lo 0b \n\t"
"2: \n\t"
"stp q0, q1, [%[st]] \n\t"
"dsb ish \n\t"
: [p] "+r" (p), [done] "+r" (done)
: [st] "r" (st), [n] "r" (nblocks), [splat] "r" ((uint32_t)splat)
: "v0", "v1", "v2", "v3", "v4", "v5", "x8", "memory", "cc"
);
return done;
}
size_t vec_scan_rev_neon(vec_state_t *st, testword_t *q, size_t nblocks, bool splat)
{
size_t done = 0;
__asm__ __volatile__ (
"ldp q0, q1, [%[st]] \n\t"
"0: \n\t"
"cbnz %w[splat], 1f \n\t"
NEON_STEP_BACK("v0", "v4")
NEON_STEP_BACK("v1", "v4")
"1: \n\t"
"mvn v2.16b, v0.16b \n\t" // expected = ~state
"mvn v3.16b, v1.16b \n\t"
"ldp q6, q7, [%[q]] \n\t"
"cmeq v6.2d, v6.2d, v2.2d \n\t"
"cmeq v7.2d, v7.2d, v3.2d \n\t"
"and v6.16b, v6.16b, v7.16b \n\t"
"uminv b5, v6.16b \n\t" // 0xFF iff every lane matched
"umov w8, v5.b[0] \n\t"
"cmp w8, #0xff \n\t"
"b.ne 2f \n\t"
"stnp q0, q1, [%[q]] \n\t" // restore the original pattern
"sub %[q], %[q], #32 \n\t"
"add %[done], %[done], #1 \n\t"
"cmp %[done], %[n] \n\t"
"b.lo 0b \n\t"
"2: \n\t"
"stp q0, q1, [%[st]] \n\t"
"dsb ish \n\t"
: [q] "+r" (q), [done] "+r" (done)
: [st] "r" (st), [n] "r" (nblocks), [splat] "r" ((uint32_t)splat)
: "v0", "v1", "v2", "v3", "v4", "v5", "v6", "v7", "x8", "memory", "cc"
);
return done;
}
#endif // __aarch64__
tests/block_move.c
+36
View File
@@ -243,6 +243,42 @@ int test_block_move(int my_cpu, int iterations)
:: "r" (p), "r" (pm), "r" (half_length)
: "$t0", "$t1", "$t2", "$t3"
);
#elif defined(__aarch64__)
__asm__ __volatile__ (
// Move first half to second half
"mov x0, %0 \n\t" // Source, p (start point)
"mov x1, %1 \n\t" // Destination, pm (mid point)
"mov x2, %2 \n\t" // Length, half_length (size of a half in words)
"0: \n\t"
"ldr x3, [x0], #8 \n\t"
"str x3, [x1], #8 \n\t"
"subs x2, x2, #1 \n\t"
"b.ne 0b \n\t"
// Move the second half, less the last 64 bytes, to the first half, offset plus 64 bytes
"mov x0, %1 \n\t" // Source, pm (mid point)
"add x1, %0, #64 \n\t" // Destination, p (start point) plus 64 bytes
"sub x2, %2, #8 \n\t" // Length, half_length minus 8 words (64 bytes)
"1: \n\t"
"ldr x3, [x0], #8 \n\t"
"str x3, [x1], #8 \n\t"
"subs x2, x2, #1 \n\t"
"b.ne 1b \n\t"
// Move the last 8 words (64 bytes) of the second half to the start of the first half
"mov x1, %0 \n\t" // Destination, p (start point)
// Source, the 8 words of the second half left over by the last loop
"mov x2, #8 \n\t"
"2: \n\t"
"ldr x3, [x0], #8 \n\t"
"str x3, [x1], #8 \n\t"
"subs x2, x2, #1 \n\t"
"b.ne 2b \n\t"
:: "r" (p), "r" (pm), "r" (half_length)
: "x0", "x1", "x2", "x3", "memory", "cc"
);
#endif
do_tick(my_cpu);
BAILOUT;
tests/mov_inv_fixed.c
+13
View File
@@ -95,6 +95,19 @@ int test_mov_inv_fixed(int my_cpu, int iterations, testword_t pattern1, testword
: "memory"
);
p = pe;
#elif defined(__aarch64__)
uint64_t length = pe - p + 1;
testword_t *dst = p;
__asm__ __volatile__ ("\t"
"0: \n\t"
"str %2, [%1], #8 \n\t"
"subs %0, %0, #1 \n\t"
"b.ne 0b \n\t"
: "+r" (length), "+r" (dst)
: "r" (pattern1)
: "memory", "cc"
);
p = pe;
#endif
#else
do {
tests/tests.c
+13 -3
View File
@@ -130,12 +130,22 @@ int run_test(int my_cpu, int test, int stage, int iterations)
if (my_cpu == master_cpu) {
if (window_num == 0) {
// First window, so we need to test all selected lower memory.
vm_map[0].start = first_word_mapping(pm_limit_lower);
uintptr_t *first_word = first_word_mapping(pm_limit_lower);
// For USB_WORKAROUND.
if (vm_map[0].start < (uintptr_t *)0x500) {
vm_map[0].start = (uintptr_t *)0x500;
if (first_word < (uintptr_t *)0x500) {
first_word = (uintptr_t *)0x500;
}
#if defined(__aarch64__)
// RAM starts well above physical address 0, so never move the
// start below the first mapped RAM page.
if (first_word > vm_map[0].start) {
vm_map[0].start = first_word;
}
#else
vm_map[0].start = first_word;
#endif
}
/* Update display of memory segments being tested */
tests/vec_prsg.c
+46
View File
@@ -185,6 +185,11 @@ void vec_fill(vec_state_t *st, testword_t *p, size_t nblocks, bool splat)
vec_fill_sse2(st, p, nblocks, splat);
return;
}
#elif defined(__aarch64__)
if (simd_tier == SIMD_NEON) {
vec_fill_neon(st, p, nblocks, splat);
return;
}
#endif
vec_fill_scalar(st, p, nblocks, splat);
}
@@ -209,6 +214,23 @@ void vec_check_fwd(vec_state_t *st, testword_t *p, size_t nblocks, bool splat, b
}
return;
}
#elif defined(__aarch64__)
if (simd_tier == SIMD_NEON) {
while (nblocks > 0) {
size_t done = vec_scan_fwd_neon(st, p, nblocks, splat);
p += done * VEC_LANES;
nblocks -= done;
if (nblocks == 0) {
break;
}
// The scan stopped at a mismatching block. Report and fix it up,
// then resume the scan on the remaining blocks.
vec_check_fwd_scalar(st, p, 1, splat, use_for_badram);
p += VEC_LANES;
nblocks--;
}
return;
}
#endif
vec_check_fwd_scalar(st, p, nblocks, splat, use_for_badram);
}
@@ -243,6 +265,30 @@ void vec_check_rev(vec_state_t *st, testword_t *p, size_t nblocks, bool splat, b
}
return;
}
#elif defined(__aarch64__)
if (simd_tier == SIMD_NEON) {
testword_t *q = p + (nblocks - 1) * VEC_LANES;
while (nblocks > 0) {
size_t done = vec_scan_rev_neon(st, q, nblocks, splat);
q -= done * VEC_LANES;
nblocks -= done;
if (nblocks == 0) {
break;
}
// The scan stopped at a mismatching block.
for (int l = 0; l < VEC_LANES; l++) {
testword_t expect = ~st->lane[l];
testword_t actual = read_word(&q[l]);
if (unlikely(actual != expect)) {
data_error(&q[l], expect, actual, use_for_badram);
}
write_word(&q[l], st->lane[l]);
}
q -= VEC_LANES;
nblocks--;
}
return;
}
#endif
vec_check_rev_scalar(st, p, nblocks, splat, use_for_badram);
}
tests/vec_prsg.h
+13
View File
@@ -62,6 +62,19 @@ size_t vec_scan_fwd_avx2(vec_state_t *st, testword_t *p, size_t nblocks, bool sp
size_t vec_scan_rev_avx2(vec_state_t *st, testword_t *q, size_t nblocks, bool splat);
#endif
#if defined(__aarch64__)
/*
* NEON kernels (see tests/aarch64/vec_prsg_neon.c). Same contract as the x86
* kernels above: start from the lane states in *st, leave the updated states
* back in *st, and end with a DSB ISH so no non-temporal store is left
* buffered. The scan kernels stop at the first mismatching block and return
* the number of blocks completed, positioned at the failing block.
*/
void vec_fill_neon(vec_state_t *st, testword_t *p, size_t nblocks, bool splat);
size_t vec_scan_fwd_neon(vec_state_t *st, testword_t *p, size_t nblocks, bool splat);
size_t vec_scan_rev_neon(vec_state_t *st, testword_t *q, size_t nblocks, bool splat);
#endif
/**
* Initialises the lane states from seed. For splat rounds all lanes hold the
* same value and are never stepped, giving a uniform background pattern.