Overview

The Bonsai Demo repository provides a streamlined environment for running advanced, parameter-efficient language models locally. It supports the innovative Bonsai (1-bit) and Ternary-Bonsai models, offering powerful AI capabilities across a range of hardware, including Mac (Metal), Linux/Windows (CUDA, Vulkan, ROCm), and CPU. This demo facilitates easy setup and deployment, allowing users to explore the latest advancements in AI model compression and performance.

Key Features

• Advanced Model Compression: Leverage highly efficient 1-bit and Ternary (2-bit) quantization for models like Bonsai 27B, Bonsai 8B, 4B, and 1.7B. This significantly reduces memory footprint and computational requirements.
• Vision-Language Capabilities: The latest Bonsai 27B models introduce robust vision capabilities, allowing interaction with images, screenshots, and PDFs.
• Agentic Tool Calling: Native support for OpenAI-style tool calls enables models to interact with external services and perform complex tasks.
• Exceptional Context Handling: Experience long context windows (256k+ tokens) for extended conversations and detailed analysis.
• Cross-Platform Compatibility: Run models seamlessly on macOS (Metal), Linux, and Windows, with support for various hardware accelerators (CUDA, Vulkan, ROCm) and CPU.
• Simplified Setup: A single setup.sh (or setup.ps1 for Windows) script automates dependency installation, model downloads, and binary compilation/fetching.
• Interactive Demos: Includes built-in chat interfaces and optional Open WebUI integration for a full agentic experience with vision, tool calling, and code interpretation.
• Flexible Model Selection: Easily switch between different model families (Bonsai vs. Ternary-Bonsai) and sizes (27B, 8B, 4B, 1.7B) via environment variables.

Typical Use Cases

• Local LLM Deployment: Run sophisticated language models on your own hardware for privacy, cost-efficiency, and offline access.
• AI Development and Prototyping: Quickly set up and test advanced AI models for applications requiring natural language understanding, generation, and reasoning.
• Vision-Based AI Applications: Develop and deploy applications that can analyze and interpret visual information alongside text.
• Agentic AI Systems: Build intelligent agents capable of interacting with external tools and executing complex workflows.
• Research and Experimentation: Explore the frontiers of model compression and efficient AI inference with state-of-the-art architectures.
• Edge AI and Mobile Applications: The highly compressed models are ideal for deployment on resource-constrained devices, including smartphones.