NightRun: Open-Source UEFI Runtime Boots LLMs Without Operating System Overhead
Key Takeaways
- ▸NightRun eliminates OS overhead by booting UEFI directly into LLM inference, freeing up RAM and memory bandwidth
- ▸Supports consumer hardware including x86_64 PCs and Raspberry Pi 5, with optimized kernels for each architecture
- ▸Production-ready with parity-tested tokenization, built-in chat UI, and hardware-specific acceleration (AVX2, NEON)
Summary
NightRun is an experimental open-source project that radically rethinks how to run large language models on resource-constrained devices. Rather than booting into a conventional operating system, the UEFI-based runtime loads a quantized LLM directly into RAM, eliminating the memory and bandwidth overhead of a full OS kernel. This approach frees up significant resources for inference: the Rust-based runtime supports models like Llama 3.2, Granite 4.1, and Qwen on hardware ranging from x86_64 PCs to Raspberry Pi 5 boards.
The runtime uses hardware-optimized kernels (AVX2/FMA for x86, NEON for ARM) and a zero-copy design to run quantized models directly from RAM without dequantization overhead. NightRun includes a built-in framebuffer chat interface and supports model CRC verification during boot. On a real 8GB Raspberry Pi 5, Granite 4.1 3B generates approximately 3.0 tokens per second; on x86 QEMU, Llama 3.2 1B reaches around 20 tokens per second.
The project, developed using Anthropic's Claude Code and Fable 5 model, has been released under the MIT license on GitHub with a user-friendly installer that includes safeguards to prevent accidental system drive overwrites.
- Open-source (MIT license) alternative to lightweight Linux distributions for privacy-preserving local LLM deployment
Editorial Opinion
NightRun represents a creative approach to the practical challenge of running LLMs on consumer hardware with minimal overhead. By leveraging UEFI as a runtime foundation rather than adopting yet another lightweight Linux distribution, the project demonstrates how rethinking the stack can yield meaningful efficiency gains. The inclusion of first-class safety features (CRC verification, storage sealing, installation safeguards) suggests this is designed with real-world deployment in mind, not just academic proof-of-concept.
