Methodology & Standards
The LLM Tree of Life is a curated taxonomy mapping the structural evolution and lineage of large language models. Here is how we verify, structure, and benchmark this data.
1Data Collection & Lineage Mapping
Our database is populated from primary academic literature, official corporate announcements, and verified repository releases. We compile new models weekly to keep the timeline and lineage trees up to date.
Lineage connections (parents) are established based on:
- Direct DescendantsModel initialized directly from the weights of another model (e.g., fine-tuning, instruction alignment, or model merges).
- Architectural ContinuitySuccessive iterations within a model family using the same design principles but updated datasets or training runs (e.g., Llama 2 to Llama 3).
- Conceptual InfluenceCross-family links (using
influenceIds) to signify major conceptual breakthroughs, such as adopting a new attention type or alignment method first published elsewhere. - DistillationLarge models generating training data to supervise smaller target student models (e.g., Phi, Gemma, or Qwen distilled variants).
2Parameter Definitions
Parameter Count
The total number of floating-point weights configured inside the network. For Mixture-of-Experts (MoE) models, we attempt to specify both active parameters per token and total parameters (e.g., Mixture of Experts (~45B total / 12B active)).
Context Window
The absolute limit of input + output tokens that the model can process in a single execution step. Evaluated under native training parameters without context-extension modifications.
Openness Classification
We group models into three distinct licensing and structural categories:
Weights, training code, and complete dataset disclosures are released publicly under open licenses (e.g., Pythia, OLMo).
Model weights are downloadable for local hosting/use, but exact dataset receipts and training recipes remain proprietary (e.g., Llama, Mistral).
The weights and design are entirely proprietary, accessed exclusively through APIs or conversational interfaces (e.g., GPT-4o, Claude 3.5 Sonnet).
3Standardized Benchmark Suite
To compare model performance objectively across timelines, we track the following academic and crowdsourced evaluation metrics. Where multiple evaluations exist, we favor officially disclosed scores from peer-reviewed papers or standardized eval harnesses.
Massive Multitask Language Understanding: Tests world knowledge, humanities, sciences, and professional tasks across 57 distinct subjects. Expressed as 5-shot or 0-shot accuracy (0-100%).
Graduate-Level Google-Proof Q&A: A highly challenging benchmark of graduate-level chemistry, physics, and biology questions designed by experts. Highly resistant to search engines and simple memorization.
Python Coding Proficiency: Measures programming capability using 164 hand-written coding problems. Evaluated using the pass@1 metric, testing if generated code executes successfully.
Multi-Step Math: 12,500 challenging high-school competition math problems. Evaluated using exact answer matching, assessing logical chain-of-thought and numeric execution.
Multi-Turn Benchmark: A set of 80 multi-turn conversational questions evaluated by GPT-4 acting as a judge. Assesses dialog flow, instruction-following, and safety limits (0 to 10 scale).
LMSYS Chatbot Arena: A crowd-sourced benchmark of human side-by-side preference ratings. Thousands of blind battles are converted using Bradley-Terry algorithms into an active ELO rating.
4Open-Source Contribution Guidelines
We welcome community contributions to expand and refine our dataset! To propose a change (e.g., correcting a parameter count, linking a new paper, or adding a model):
How to update the database:
1. Fork our GitHub repository.
2. Locate the static data configuration files under lib/data/:
models.ts— Primary database containing model nodes.papers.ts— Foundational academic research papers.hardware.ts— Hardware accelerator hardware components.
3. Append your new model or modify existing fields using the strict TypeScript interfaces in lib/types.ts.
4. Verify compile safety by running: npx tsc --noEmit
5. Open a Pull Request referencing official citations for any numerical claims.