Overview

MTPLX is a groundbreaking native application and command-line interface designed to unlock the full potential of local language models on Apple Silicon. It revolutionizes LLM inference by natively leveraging Multi-Token Prediction (MTP) heads found in modern models like Qwen, achieving speeds up to 3x faster than traditional methods. This means significantly quicker responses and more fluid interactions with your AI models, directly on your Mac, without relying on external drafter models or compromising output quality.

Key Features

• Native MTP Speculative Decoding: Utilizes the model's built-in MTP capabilities for drafting and batched verification, resulting in remarkable speed improvements (e.g., 1.6x to 2.24x on recent Apple Silicon).
• Exact Rejection Sampling: Guarantees identical output distributions to standard decoding methods, preserving model integrity and quality with temperature and top_p settings behaving as expected.
• RAM Efficiency: No need for secondary drafter models, keeping your system memory free for other tasks.
• User-Friendly Mac App: An intuitive application handles hardware checks, model recommendations, downloads, Python engine setup, fan control, and path configuration, making LLM deployment effortless.
• Optimized Model Catalog: Recommends and serves high-performance models like Qwen 3.8 27B (optimized for speed and coding), Qwen 3.6, and Gemma, with variants tailored for different memory configurations and use cases.
• Auto-Tuning: Automatically measures and selects the optimal MTP draft depth for your specific Mac hardware, ensuring peak performance.
• Forge: MTP Model Creation: A powerful tool to convert existing Hugging Face models into MTPLX-ready MTP models, including adapter training, verification, and publishing.
• OpenAI/Anthropic Compatible API Server: Serves a comprehensive API allowing seamless integration with popular LLM frameworks, UIs, and clients.
• Retrieval Model Serving: Extends the API to serve embedding and reranker models, simplifying RAG and agent memory setups without additional inference servers.
• Comprehensive CLI Tools: Provides extensive command-line utilities for model management, benchmarking, tuning, and server control.
• Transparent Compatibility Reporting: mtplx inspect clearly classifies model compatibility, ensuring users are informed before attempting to run a model.

Typical Use Cases

• Accelerated Local Chatbots: Experience incredibly responsive chat interactions with local LLMs for personal assistance, creative writing, or brainstorming.
• High-Performance Coding Assistants: Utilize optimized models like Qwen 3.8 27B for faster code generation, completion, and debugging directly on your machine.
• Streamlined RAG and Agent Development: Integrate local LLMs with retrieval-augmented generation and agent frameworks, leveraging the built-in embedding and reranking capabilities.
• Model Development and Experimentation: Forge allows researchers and developers to create and benchmark their own MTP-enabled models, pushing the boundaries of LLM performance.
• Educational and Hobbyist Exploration: Easily set up and experiment with powerful local LLMs on Apple Silicon thanks to the user-friendly Mac app and comprehensive documentation.
• Benchmarking and Performance Analysis: Utilize built-in benchmarking tools like AIME to objectively measure LLM performance on your hardware.