Overview

oMLX is a high-performance LLM inference server specifically engineered for Apple Silicon Macs, delivering powerful AI capabilities directly to your desktop. It seamlessly integrates continuous batching and a smart SSD caching system for lightning-fast LLM operations, all managed conveniently from your macOS menu bar. Designed for developers and power users, oMLX makes running sophisticated AI models locally practical and efficient, with a focus on ease of use and comprehensive control.

Key Features

• Apple Silicon Optimized: Built from the ground up to leverage the unique architecture of M1/M2/M3/M4 chips for maximum inference speed.
• Continuous Batching: Maximizes throughput by dynamically batching incoming requests, ensuring efficient GPU utilization.
• Tiered KV Caching (RAM + SSD): Utilizes a sophisticated caching mechanism with a fast in-memory "hot" tier and a larger "cold" SSD tier. This allows context to persist and be reused even across conversations or server restarts, dramatically improving performance for interactive use cases.
• macOS Menu Bar App: Provides intuitive control and monitoring directly from your system's menu bar. Start, stop, and track your LLM server without ever opening a terminal.
• Comprehensive Model Support: Serves text LLMs, Vision-Language Models (VLMs), OCR models, embedding models, and rerankers.
• Admin Dashboard: A rich, offline-capable web UI for real-time monitoring, model management, direct chat, benchmarking, and easy model downloading from HuggingFace.
• OpenAI/Anthropic API Compatibility: Acts as a drop-in replacement for popular inference APIs, making integration with existing tools and workflows straightforward.
• Multi-Model Serving: Load and manage multiple LLMs, VLMs, and other model types concurrently with intelligent eviction policies and manual pinning.
• Tool Calling & Structured Output: Robust support for function calling and structured output generation, including integration with MCP tools.
• Easy Installation: Available as a native macOS app, via Homebrew for seamless integration with your command line, or from source for developers.

Typical Use Cases

• Local Development & Prototyping: Rapidly iterate on LLM-powered applications without relying on cloud APIs, leveraging continuous batching and persistent caching for a fluid development loop.
• On-Device AI for Creative Professionals: Power creative tools with local LLMs for content generation, code assistance, image analysis, and more, directly on your Mac.
• Privacy-Focused AI Workflows: Process sensitive data locally with the assurance that your information never leaves your machine.
• Advanced Chatbot and Assistant Development: Build sophisticated conversational agents that can maintain context across long interactions thanks to the tiered KV cache.
• Research and Experimentation: Explore the capabilities of various LLMs, VLMs, and embedding models on your own hardware with detailed performance benchmarking tools.