We Reduced Cold-Boot Time From ~4ms to ~60µs Without Touching AArch64

A flight-control computer recovering from a hardware reset has a limited window to become operational again. For safety-critical hardware, cold-boot latency is a vital Key Performance Indicator (KPI) for on-field system performance. When an airborne flight-control computer undergoes a sudden hardware reset mid-flight, the window to recover and regain control is exceptionally tight, making rapid initialization a core functional requirement rather than a luxury. 

We set out to answer one question: on real ARM hardware, where does a formally verified microkernel's cold-boot time actually go, and how much of it is necessary? To find out, we instrumented the kernel boot path on a Raspberry Pi 5 (BCM2712, 4× Cortex-A76 @ 2.4GHz) and measured every stage between firmware hand-off and scheduler readiness—removing work one category at a time, before opening the Cortex-A76 optimization manual.

The result was not the final number. It was where the reductions came from: nearly all of the improvement landed before any architecture-specific tuning. This post covers those three phases.

What we measured

For every figure in this article, boot time is defined as the interval between the first kernel instruction at EL2 (firmware hand-off) and the point where all four CPUs are online and the scheduler can dispatch the first thread.

Not power-on. Not the bootloader. From kernel_entry to scheduler-ready.

BL31 hand-off
↓
kernel_entry
↓
early assembly initialization
↓
MMU setup
↓
kernel initialization
↓
SMP bring-up
↓
scheduler ready

Timing was collected using the ARM generic timer (CNTVCT_EL0, 54MHz—one tick per ~18.5ns). All figures are cold boots over the same execution window, with no warm-cache shortcuts. We report rounded values because at this resolution a single boot cannot validate a sub-microsecond change; the numbers describe order-of-magnitude shifts, not fractions of a microsecond.

One caveat before the breakdown: a large part of the 4ms baseline was never real boot time. The first phase does not make the kernel faster. It corrects what we were measuring—and that correction is the single largest line in the table.

Phase 1: Transitioning to the Production Configuration

~4,000µs → ~800µs

Our initial benchmark started with a comprehensive debug configuration. Clocking in at just ~4ms while running full diagnostic test suites, this baseline is already highly competitive compared to standard open-source RTOS frameworks.

To find the true operational speed of the core kernel, we stripped out the development and diagnostic code that does not ship in production:

• Kernel self-tests & SMP validation frameworks
• Scheduler test workloads
• Verbose boot-time UART logging

The largest driver of overhead in the debug build was synchronous serial logging. Every stage wrote status messages over a PL011 UART. Once the 16-byte transmit FIFO filled up, the next write busy-waited for space to clear—costing roughly 100ns per MMIO poll. Across dozens of multi-line messages, this busy-waiting added massive latency.

By switching to the production build configuration, disabling the test suites, stripping out the logging loops, and enabling release-level compiler optimization, boot latency dropped from ~4,000µs to ~800µs.

This transition establishes our real-world baseline, moving from an instrumented development build to the actual lightweight executable that deploys to the field.

The first rule of boot optimization is to measure the binary you will ship, not a debug build with the test harness still wired in.

Phase 2: Removing Unnecessary Boot Work

~800µs → ~250µs

With the debug overhead gone, the remaining ~800µs reflected real kernel initialization. The question was no longer how to make that work faster—it was whether it needed to run at all. Four paths did not.

Allocator initialization

The page allocator tracks free memory with a bitmap: one bit per page, 0 = free, 1 = allocated. The original init path "freed" every page individually through kfree()—acquire lock, validate, update, release—once per page in memory. That is O(N) lock round-trips before the kernel does anything useful.

But the bitmap lives in .bss, which the kernel's own early-boot assembly zeroes before C initialization runs. A zeroed bitmap already means "all pages free." The correct initial state existed before the first kfree(). The loop was not optimized; it was deleted—O(N) lock round-trips became O(1).

Permanent kernel objects

Idle threads and per-CPU workers live for the lifetime of the system; they are never freed. The original code still allocated them from the page allocator at boot, paying lock contention and cold page faults for objects with static lifetime—eight needless allocations on a four-core system, at the coldest point in the memory lifecycle. They belong in static storage:

static kthread_t idle_tcb[NCPU];     /* TCBs in BSS, not the heap */
static kthread_t worker_tcb[NCPU];

Boot-time string processing

Thread names were generated with printf-style formatting. The CPU present-mask was worse: it was built by formatting the string "0-3" and then parsing it back—formatting a value, then parsing it, to recover the value you already had. Both were replaced with direct construction, where the mask is simply an integer known at compile time.

Allocator scan locality

Every allocation began its bitmap scan at page zero. As allocation counts grew, so did scan length: allocation #201 walked past 200 occupied pages before finding a free one—O(N) per call. A single integer remembering where the last allocation ended—a next-fit hint—makes this O(1) amortized.

Individually, none of these was dramatic. Together they took boot from ~800µs to ~250µs, with no assembly, no compiler flags, and no pipeline analysis—only the removal of work the kernel had no reason to do.

The fastest code is the code that never runs. Audit what the boot path does before tuning how it does it.

Phase 3: Whole-Program Optimization

~250µs → ~60µs

At ~250µs the unnecessary work was gone. What remained was genuine kernel logic the boot sequence requires. The question shifted again: given that this work must run, what was preventing the compiler from compiling it optimally?

The answer was visibility. RedKill is a small, closed, bare-metal binary—no shared libraries, no dynamic linker, nothing the compiler cannot see. That is precisely the shape whole-program optimization is built for.

Single-partition link-time optimization

By default, compilation optimizes each source file in isolation; cross-file inlining happens only at link time, after the optimizer has lost full context. A single-partition LTO build optimizes the entire kernel as one unit. get_cpu_num(), called constantly during SMP bring-up, collapsed from a full call/return sequence into two or three inlined instructions at each call site. From there the effect compounds: with callees inlined, the constants flowing into them—CPU count, clock frequency, MMIO base addresses—fold into literals, which expose dead branches, which are then eliminated, which shrinks the binary and lowers instruction-cache pressure.

# Bare-metal AArch64, GCC — release flags.
ARCH_CFLAGS += -flto=auto -flto-partition=one -fno-fat-lto-objects
ARCH_CFLAGS += -fno-pic -fno-pie -fno-plt      # no dynamic linker exists
ARCH_CFLAGS += -fno-semantic-interposition     # bare-metal: no interposition
ARCH_CFLAGS += -finline-stringops              # inline memset/memcpy

Inline thresholds tuned for a small kernel

GCC's default inline limits assume a codebase of hundreds of thousands of lines, where unbounded inlining would inflate binary size and cache pressure. A 30KB kernel is the opposite case: the entire boot call graph fits comfortably inside the Cortex-A76's 64KB L1 instruction cache, so aggressive inlining costs almost nothing and removes call overhead wholesale.

# Raise inline limits for a small whole-program build.
ARCH_CFLAGS += --param inline-unit-growth=100   # default ~20
ARCH_CFLAGS += --param max-inline-insns-auto=70 # default ~15
ARCH_CFLAGS += -fipa-pta                         # interprocedural points-to
ARCH_CFLAGS += -fira-loop-pressure               # better reg alloc in loops

One flag that prevents a crash, not a slowdown

-fno-tree-loop-distribute-patterns belongs in a different category from the others—it is a correctness requirement. At -O3, GCC pattern-matches loops that resemble memset and rewrites them into a call to memset. In userspace, libc provides one. This kernel is its own memset: without this flag, GCC finds the loop inside that implementation, rewrites it into a call to itself, and produces infinite recursion. The boot hangs. On any bare-metal target that supplies its own memory primitives, the flag is mandatory.

Code layout

-ffunction-sections -fdata-sections places each function in its own section, letting the linker script order them deliberately: boot assembly at offset 0, hot C functions next, cold error handlers last. On a cold boot every cache is empty, and the core's linear prefetcher only helps when the next instruction is physically close to the current one. A hot function 40KB away gets no prefetch benefit; one 64 bytes ahead does. At this scale, layout is part of the cache behavior, not a cosmetic concern.

One related fix surfaced here at no cost: an interrupt-handler table had a single initialized entry, which forced the entire multi-kilobyte array into the loadable .data segment. Zero-initializing it moves the array to .bss, which is not stored in the image at all—a smaller binary for free.

Given full visibility into a 30KB closed binary, the compiler removed roughly three-quarters of the remaining time on its own: ~250µs → ~60µs, with no instruction written by hand.

On a closed bare-metal target, the compiler is not a tool you use—it is a component you configure. Give it whole-program visibility, raise its limits, and own your layout.

Where this leaves us

A 66× reduction, with no performance counter, no pipeline analysis, and no assembly. The order matters as much as the result. The high-leverage work lived in the unglamorous layers—measuring the right binary, deleting work that did not need to run, and handing the compiler the whole program—and it came first for a concrete reason: tuning instructions earlier would have meant hand-optimizing code that Phase 2 was about to delete and Phase 3 was about to inline out of existence. You earn the right to optimize the machine by first removing everything that should never reach it.

What comes next

At ~60µs, the kernel is doing only work it genuinely needs, and the compiler has exhausted what it can do at the C level. Everything that remains is the Cortex-A76 itself—its pipeline, its memory-ordering rules, and the firmware that hands control to it.

The first thing we found inside the core was a single instruction, ISB (Instruction Synchronization Barrier), scattered across the boot path. Each one is a full pipeline flush, and most of the occurrences turned out to be unnecessary. But before removing a single barrier, we had to prove it was the bottleneck—and the first reading from the performance-monitoring unit eliminated three of our working theories at once, with a single table of numbers.

That is where the next post begins: the microarchitecture, the counters that falsify your hypotheses, and the firmware boundary where this finally stops being a software problem.

Hardware: Raspberry Pi 5 · BCM2712 · 4× Cortex-A76 @ 2.4GHz · EL2 entry via TF-A BL31. All measurements: cold boot, CNTVCT_EL0 @ 54MHz. Code excerpts are simplified, representative illustrations of publicly documented Arm and GCC techniques.