DeepSeek releases its new V3 language model runnable on consumer hardware for free

Chinese AI company DeepSeek recently released its new large language model, DeepSeek-V3-0324. The 641-gigabyte model was released on the AI platform Hugging Face with minimal pre-announcement, consistent with the company's practice of subdued product launches.

The model is unique in its MIT license, permitting free commercial use. Early benchmarks indicate that DeepSeek-V3-0324 is capable of running on consumer-grade hardware, such as Apple's Mac Studio with the M3 Ultra chip. AI scientist Awni Hannun reported it was achievable to reach over 20 tokens per second speed of processing using this setup. This capability of running a large language model on local off-the-shelf hardware is the exact opposite of the conventional way of utilizing a huge data center infrastructure to support high-end AI models.

According to DeepSeek, early trials have shown a significant improvement over previous versions. The model has been rigorously tested by internal stakeholders and has performed excellently, possibly outperforming all other competing models and even beating Anthropic's Claude Sonnet 3.5 on non-reasoning tasks. However, unlike Sonnet-style subscription models, DeepSeek-V3-0324 is free to download and use.

Technically, the model is a mixture-of-experts (MoE) architecture. It selectively uses about 37 billion of its 685 billion parameters per task, encouraging efficiency by reducing computational needs while maintaining performance. The model also employs Multi-Head Latent Attention (MLA) and Multi-Token Prediction (MTP) technologies that contribute to improved context retention and faster output speeds.

The model can be accessed via Hugging Face, OpenRouter's API and chat interface, and DeepSeek's chat platform if desired. Inference provider Hyperbolic Labs offers access to the model, too.