CodeLlama
Meta · August 2023
● activeOpen Weightdecoder onlycode
Parameters7B - 70B
Context Window16K tokens
Variants7B, 13B, 34B, 70B, Python, Instruct
Why It Matters
Meta's proof that specialized code models derived from general-purpose LLMs could outperform dedicated coding models.
Description
Meta's code-specialized version of LLaMA 2, further trained on code-heavy datasets. Available in sizes from 7B to 70B parameters, with specialized variants for Python and instruction-following. Proved that taking a strong general-purpose model and continuing to train it on code could outperform models built from scratch for coding.
Key Innovations
Code Gen
Code GenAbility to write, debug, and understand programming code across multiple languages.
Instruction Tuning
Instruction TuningFine-tuning a model on instruction-response pairs so it follows user commands more reliably.
Related Research (1)
LLaMA 2Scaling
2023 · Meta AI
Provided the most detailed public documentation of how to train, fine-tune, and safety-align a large language model, including their full RLHF methodo…