By Model VS · Guide text updated . Category averages are computed at build time from the saved LiveBench snapshot with the same rule as the interactive table: a category is the unweighted mean of its task columns, and Overall exists only when every category is complete. A recorded zero would count. A blank task would remove the category. Report a correction if a task column is mis-assigned.
Overall treats every category as equal.
LiveBench publishes many task columns and groups them into categories. Model VS does not invent a second weighting. Overall is the unweighted mean of those category averages, and only for a row that has all of them. That is a convenient sort key for a table. It is a poor description of a product. If instruction following is the thing your users will notice, a strong math average can numerically hide a weak instruction average. The mean does not know which category pays your invoice.
The release name is part of the result. Scores from two releases are not a single ladder, even when the category titles look familiar. Our file records the release string and the time we saved it. The homepage shows that release when the LiveBench channel is selected. Quote both when you copy a number into a decision record. “LiveBench coding” without a release is not a citation.
The public project describes its own tasks at livebench.ai. This page does not restate their paper. It says how to use the copy we publish, and where our display stops. We do not convert a category average into a productivity percentage, a dollar saving, or a probability that the next pull request will pass review.
Read the task columns before you trust the category name.
Category names are short. The columns under them are the actual exams. Coding on this release is not “software engineering.” It is the mean of the coding tasks listed in the snapshot. Agentic coding is a different set of columns, closer to writing code in a language with a tool loop, and it still is not a SWE-bench submission. Instruction following is the IF group, not a general promise that the model will obey your company policy.
The table is generated from the category object in the snapshot. If a future release adds or drops a task, the table should change with the file. Do not memorize this list from a screenshot that has no release date.
| Category | Task columns averaged into it |
|---|---|
| Reasoning | theory_of_mind, zebra_puzzle, spatial, logic_with_navigation |
| Coding | code_generation, code_completion |
| Agentic Coding | javascript, typescript, python |
| Mathematics | AMPS_Hard, integrals_with_game, math_comp, olympiad |
| Data Analysis | consecutive_events, tablejoin, tablereformat |
| Language | connections, plot_unscrambling, typos |
| IF | paraphrase, simplify, story_generation, summarize |
A practical mapping, which is our editorial guidance and not a LiveBench score:
- If you will accept or reject a patch with existing tests, start with Coding, then read Agentic coding if the assistant is expected to use tools. Neither number is a SWE-bench resolved rate. That evidence lives on the specialist page, with its own flag rules.
- If the work is a calculation with a checkable answer, start with Mathematics. Fluent prose around a wrong number is a failure. Keep the derivation in your own pilot.
- If the work is joins, reformatting, or reading a table, start with Data analysis. Give every candidate the same file.
- If the work is rewriting, unscrambling, or catching typos in running text, start with Language. Your audience may still differ from the benchmark’s audience.
- If the work is “follow this schema, and refuse when the instruction conflicts,” start with IF. A high Language score does not imply this.
- If the work is a multi-step puzzle with a known solution, start with Reasoning. Keep plausible wrong answers in the failure pile.
Write the acceptance sentence before you open the table. Then open one category. If you open Overall first, you will remember the wrong name.
The build picks a row Overall flatters.
Among the ten complete rows with the highest Overall, the page selects the one whose Coding rank falls furthest behind its Overall rank. The selection is arithmetic on this snapshot. It is not an editorial grudge, and it is not a claim that the row is bad at coding in your repository. It is a demonstration that the mean and the category answer different questions. If a later snapshot changes the identity of that row, the paragraph still applies to whoever the rule selects. Re-read the table. Do not quote a model id from memory after the file has moved.
| Source model id | Overall rank | Overall | Coding rank | Coding | Agentic coding rank | Agentic coding |
|---|---|---|---|---|---|---|
| gpt-6-astra-max | 4 | 82.16 | 20 | 80.36 | 21 | 57.32 |
| Category | Average | Task columns in this release |
|---|---|---|
| Reasoning | 92.65 | theory_of_mind, zebra_puzzle, spatial, logic_with_navigation |
| Coding | 80.36 | code_generation, code_completion |
| Agentic Coding | 57.32 | javascript, typescript, python |
| Mathematics | 96.81 | AMPS_Hard, integrals_with_game, math_comp, olympiad |
| Data Analysis | 82.97 | consecutive_events, tablejoin, tablereformat |
| Language | 89.43 | connections, plot_unscrambling, typos |
| IF | 75.58 | paraphrase, simplify, story_generation, summarize |
Inside the top ten by Overall, this is the saved row whose Coding rank falls furthest behind its Overall rank. The gap is 16 places on this snapshot. That is a reason to open the category, not a verdict that the model fails coding, and not a percentage of engineering time saved.
Use the category table to see the shape, not only the two coding columns. A row can be excellent at mathematics and ordinary at agentic coding. If your pilot is “edit a function in place,” the mathematics average is almost a distraction. If your pilot is a contest problem, the coding rank is the distraction. The compare tool will show signed category differences for two ids on this same release. A category stays blank in that tool when a required task is missing. Do not treat a blank as a zero.
Freeze the configuration the id already names.
LiveBench ids in this file often include an effort or thinking suffix: high, xhigh, max, and similar tokens. That suffix is the configuration the score belongs to. A cheaper or faster setting from the same family is a different experiment. Copy the full id into the shortlist. When you call an API, record the endpoint, the date, the reasoning budget, the temperature, and the tool list. If you cannot match the suffix, say so. An unmatched suffix means you do not have this score.
Then leave the leaderboard. Three to five tasks from your own backlog are enough to notice a broken format, a missing tool, or a systematic refusal. They are not enough to estimate a success rate for a whole company. Keep the failures in the denominator. How we rate walks through that protocol, including the legitimate outcome that the pilot is inconclusive.
Cost is a separate column. A category score does not include the invoice, the retries, or the minutes a person spends correcting the patch. Write those down beside the score. A slightly lower category average can be the better operational choice when the higher row needs a large thinking budget you will not pay for on every request.
If Arena and LiveBench disagree, write the disagreement down.
They often will. Arena is a preference vote. LiveBench is a scored task on a release. A model people like to read can miss a checkable answer. A model that wins a category can sound blunt or verbose in a side-by-side chat. Averaging the two numbers deletes the disagreement and pretends you have a universal rank. You do not.
Pick the source that matches the decision. For a public assistant where people choose which reply they prefer, start with Arena intervals and then test tone on your prompts. For a pipeline with tests, start with the LiveBench category and ignore the chat ranking until the tests pass. If both matter, keep two shortlists. A model can be on one and off the other.
There is no honest “combined rank” of an Arena place and a LiveBench category. Publishing one would be a new benchmark we did not run.
A category conclusion expires when the release or the row changes.
Our October note compares the current LiveBench file with the archive saved on 24 September 2026. The release string can stay the same while the roster changes. A new id is not a new score for an old id. A removed id is not a quiet failure of that model on the tasks; it is an absence from this file. Read the changelog before you assume yesterday’s category winner is still in the table.
If synchronization fails, the site is built to keep the last validated file rather than invent a row. That means a page can be internally consistent and still older than LiveBench’s website. The snapshot time is the time to cite. If you need the newest public board, open the source and write down its release, then come back and see whether our file matches.