Meta releases much more powerful Code Llama 70B to rival GitHub Copilot

In August 2023, Meta released Code Llama, a large-scale language model explicitly designed for coding tasks, building on the previously Llama 2 model. Today, the company unveiled "a new, more powerful version," Code Llama 70B.

Code Llama 70B has been trained on 500 billion tokens of code and associated data. It can process and generate longer code sequences thanks to a context window of 100,000 tokens.

According to Meta, Code Llama 70B uses a technique called self-attention to understand code structures. It can implement algorithms, sorting, searches, and more from text or snippet prompts across many languages like Python, C++, Javascript, and Java.

The AI-powered tool includes variants that have been fine-tuned for specific tasks. One such variant is CodeLlama-70B-Instruct, which has been trained to understand natural language instructions. There is also a Python-focused version called CodeLlama-70B-Python. With additional training on 100 billion Python code tokens, it generates Python with "unmatched fluency and accuracy," according to Meta.

"Writing and editing code has emerged as one of the most important uses of AI models today," wrote Meta CEO Mark Zuckerberg in a post on Facebook. "The ability to code has also proven to be important for AI models to process information in other domains more rigorously and logically."

Code Llama 70B can be downloaded freely under the same open license as earlier Code Llama models. Meta says this permissive license allows both academic and commercial users to modify the model.

In case you missed it, Microsoft-owned GitHub launched Copilot Chat in July 2023. Developers can ask Copilot questions about their code, get explanations for specific parts of the code, and even have Copilot fix bugs in the code. Recently, Google also made Duet AI and Gemini Pro generally available, its AI-powered code completion and generation tool.