Overview

Colibrì is a groundbreaking inference engine designed to bring the power of massive Mixture-of-Experts (MoE) models, ranging from 744B to an astonishing 2.8T parameters, to your existing hardware. It achieves this feat by meticulously treating storage (NVMe), RAM, and VRAM as a unified inference hierarchy, streaming expert weights directly from disk as needed. This "AI memory multitiering" approach liberates users from the necessity of scarce, high-end hardware, making cutting-edge AI accessible and affordable. Written in pure C with zero dependencies, Colibrì offers an incredibly small engine capable of handling immense models.

Key Features

• Massive Model Support: Run frontier MoE models like GLM-5.2/5.3 (744B), Inkling (975B), Kimi K3 (2.8T), and others on consumer-grade hardware.
• Unified Memory Hierarchy: Integrates VRAM, RAM, and NVMe storage into a single, seamless hierarchy for expert weight placement, drastically reducing hardware requirements.
• Pure C, Zero Dependencies: A lean and efficient engine built entirely in C, ensuring broad compatibility and minimal overhead.
• Disk Streaming for Experts: Experts are streamed from disk on demand, allowing enormous models to run even when they far exceed available RAM or VRAM.
• Research Platform: Serves as an open research platform focused on pushing inference-side performance across the entire software/hardware boundary, from model formats to I/O and scheduling.
• Transparent and Measurable: Emphasizes reproducible end-to-end measurements and guarantees semantic correctness over speculative speed gains.
• Dynamic Caching and Prefetching: Implements a "JIT for weights" approach, learning workload patterns to cache hot experts and prefetch upcoming ones.
• Heterogeneous Execution: Supports CPU, CUDA, Metal, and Vulkan backends, allowing for flexible execution across diverse hardware.
• Compressed State: Achieves significant memory savings with features like a 57x smaller KV state and persistent warm conversations.

Typical Use Cases

Colibrì empowers a wide range of users and applications, including:

• Individual AI Enthusiasts: Run state-of-the-art large language models locally on personal computers, laptops, or existing workstations without needing to rent cloud resources.
• Researchers and Developers: Experiment with frontier MoE models, probe their internal workings, and contribute to research on inference optimization and memory management.
• Prototyping and Development: Quickly iterate on AI applications that leverage large language models without the cost and latency of API calls.
• Edge Computing and Resource-Constrained Environments: Deploy powerful AI capabilities in scenarios where high-end hardware is not feasible, by leveraging clever storage and memory management.
• Educational Purposes: Provide hands-on access to and understanding of how massive AI models operate, demystifying their complexity.