Code Llama is an AI model built on top of Llama 2, fine-tuned for generating and discussing code. It’s free for research and commercial use.
Code Llama, a large language model (LLM) that can use text prompts to generate and discuss code. Code Llama is state-of-the-art for publicly available LLMs on coding tasks.
Code Llama is state-of-the-art for publicly available LLMs on coding tasks. It has the potential to make workflows faster and more efficient for developers and lower the barrier to entry for people who are learning to code. Code Llama has the potential to be used as a productivity and educational tool to help programmers write more robust, well-documented software.
Code Llama is a code-specialized version of Llama 2 that was created by further training Llama 2 on its code-specific datasets, sampling more data from that same dataset for longer. Essentially, Code Llama features enhanced coding capabilities. It can generate code and natural language about code, from both code and natural language prompts (e.g., “Write me a function that outputs the fibonacci sequence”). It can also be used for code completion and debugging. It supports many of the most popular programming languages used today, including Python, C++, Java, PHP, Typescript (Javascript), C#, Bash and more.
FaceBook has released three sizes of Code Llama with 7B, 13B and 34B parameters respectively. Each of these models is trained with 500B tokens of code and code-related data. The 7B and 13B base and instruct models have also been trained with fill-in-the-middle (FIM) capability, allowing them to insert code into existing code, meaning they can support tasks like code completion right out of the box.
The three models address different serving and latency requirements. The 7B model, for example, can be served on a single GPU. The 34B model returns the best results and allows for better coding assistance, but the smaller 7B and 13B models are faster and more suitable for tasks that require low latency, like real-time code completion.
Meta further fine-tuned two additional variations of Code Llama: Code Llama – Python and Code Llama – Instruct.
Code Llama – Python is a language specialized variation of Code Llama, further fine-tuned on 100B tokens of Python code. Because Python is the most benchmarked language for code generation, and because Python and PyTorch play an important role in the AI community – we believe a specialized model provides additional utility.
Code Llama – Instruct is an instruction fine-tuned and aligned variation of Code Llama. Instruction tuning continues the training process, but with a different objective. The model is fed a natural language instruction input and the expected output. This makes it better at understanding what people expect out of their prompts. We recommend using Code Llama – Instruct variants whenever using Code Llama for code generation since Code Llama – Instruct has been fine-tuned to generate helpful and safe answers in natural language.
Learn more about Code Llama on our AI blog or download the Code Llama model.