Overview

DwarfStar is a highly optimized, native inference engine purpose-built for DeepSeek V4 Flash and PRO models, with support for GLM 5.2. It focuses on delivering exceptional performance for local LLM deployment across diverse hardware, including Macs with Metal, NVIDIA CUDA, and AMD ROCm systems. Unlike general-purpose GGUF runners, DwarfStar is a self-contained and specialized solution, meticulously integrating model loading, prompt rendering, tool calls, KV state management, and HTTP serving for maximum efficiency.

Key Features

• Specialized for DeepSeek V4 Flash & PRO: Engineered from the ground up to maximize the performance of DeepSeek V4 models, including advanced quantization techniques.
• Broad Hardware Acceleration: Leverages Metal for Macs (even with SSD streaming for memory-constrained systems), NVIDIA CUDA (including multi-GPU and DGX Spark), and ROCm for AMD systems like Strix Halo.
• Aggressive Quantization Support: Excels with routed-expert quantization, enabling powerful models to run on consumer hardware.
• SSD Streaming: Empowers users to run models larger than available RAM by intelligently streaming expert weights from fast SSDs, offering a viable path for high-memory models on personal machines.
• Pipeline & Tensor Parallelism: Supports distributed inference, allowing models to be split across multiple machines for increased capacity or to accelerate prefill operations.
• DSpark Speculative Decoding: Integrates an experimental, opt-in speculative decoding mechanism to potentially accelerate generation for predictable outputs like code.
• Deep Integration: Features a tightly coupled system where model loading, prompt handling, and inference components are built and tested together for reliability and speed.
• Community-Driven: Built upon the foundation of llama.cpp and GGML, acknowledging significant contributions from the open-source community.

Typical Use Cases

• Local LLM Deployment: Run state-of-the-art open-weight models on your personal computer, workstation, or even high-end laptops, empowering individual developers and researchers.
• Enterprise LLM Servers: Transform older CUDA-equipped servers into multi-user LLM platforms by utilizing multi-GPU support and micro-batching for efficient shared access.
• Cross-Machine LLM Collaboration: Leverage pipeline parallelism to combine the RAM of multiple machines (e.g., two MacBooks) to run larger models that would otherwise be impossible on a single device.
• Accelerated Development & Prototyping: Rapidly iterate on LLM applications with fast local inference, benefiting from optimized performance on a variety of popular hardware.
• Memory-Constrained Inference: Utilize SSD streaming to run substantial models on systems with limited RAM, making advanced AI accessible on a wider range of hardware.