Overview

This repository provides a high-performance recipe for deploying the DeepSeek-V4-Flash-Vision-Exp model on a 2x DGX Spark cluster. It leverages vLLM with Tensor Parallelism (TP=2), speculative decoding, and a 1 million token context window, optimized for efficiency and speed.

Key Features

• Massive Context Window: Supports an impressive 1 million token context ceiling, enabling advanced long-context reasoning.
• Vision Capabilities: Native support for image input via image_url or <image>path</image> formats, powered by ViT and Aligner from the Vision-Exp checkpoint. (Note: GIF support is limited to still frames, and video encoding is not available in the official weights).
• Optimized Performance: Employs vLLM with TP=2 and DSpark speculative decoding for accelerated inference.
• Efficient KV Cache: Utilizes nvfp4_ds_mla KV cache, with substantial capacity allowing for 2,331,430 tokens on the cluster.
• Flexible Deployment: Offers options for local checkpoint copying or worker weights over NFS for streamlined resource management.
• Configurable: Extensive .env.dspark switches allow fine-tuning of various parameters including model version, context length, concurrency, and performance knobs.
• Robust Hotfixes: Includes several on-by-default hotfixes and diagnostic tools for enhanced stability and troubleshooting.

Typical Use Cases

This recipe is ideal for advanced AI applications requiring:

• Long-context analysis: Processing and reasoning over extensive documents, codebases, or lengthy conversations.
• Multimodal understanding: Applications that integrate visual understanding with text generation, such as detailed image captioning or visual question answering.
• High-throughput inference: Deploying large language models in production environments demanding significant concurrency and low latency, especially for tasks involving extended interactions.
• Research and Development: Experimenting with cutting-edge vision-language models on powerful hardware configurations.