Benchmarks · /benchmarks/scale-hil-bench

HiL-Bench

Name: HiL-Bench
Creator: Scale Labs

Scale human-in-the-loop benchmark spanning software engineering and tool-assisted workflows.

Source · Scale Labs
Version · scale-labs snapshot 2026-10-03
Scores · 17

Test details

Visible tradeoffsThis is a rubric-judged signal, so it is more structured than arena taste but still depends on the scoring rubric.

source

Scale Labs

metric

Success rate (%)

judge

Rubric

direction

higher better

group id

scale_hil_current

domain

Coding

What it measures vs what it misses

✓ Measures

Recovery and execution quality on coding-heavy tasks with intervention paths.

✗ Misses

Pure chat fluency. Standalone latency metrics.

Why this countsIt tells you whether the model can generate, repair, and reason over code under evaluator pressure rather than marketing examples.Same-test ruleThis percentile only compares models inside the exact benchmark/version group shown here. It is not a universal score.What it missesIt does not fully capture repo-scale iteration, IDE ergonomics, or long debugging loops.

Leaderboard · this benchmark version

#1 · Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)

SL · Oct 3, 2026

Source label: Claude Fable 5.1

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 100%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Claude Fable 5.1. Collapse policy: highest reported score per canonical model.

61.5%

#2 · Claude Opus 5 (Adaptive Reasoning, Max Effort)

SL · Oct 3, 2026

Source label: Claude Opus 5

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 93.8%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Claude Opus 5. Collapse policy: highest reported score per canonical model.

57%

#3 · Claude Fable 5

SL · Oct 3, 2026

Source label: Claude Fable 5

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 87.5%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Claude Fable 5. Collapse policy: highest reported score per canonical model.

56.3%

#4 · glm-5.2-max

SL · Oct 3, 2026

Source label: GLM 5.2

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 81.3%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: GLM 5.2. Collapse policy: highest reported score per canonical model.

43.7%

#5 · Claude Opus 4.7

SL · Oct 3, 2026

Source label: Claude Opus 4.7

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 75%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Claude Opus 4.7. Collapse policy: highest reported score per canonical model.

41.7%

#6 · gemini-3.8-flash-high

SL · Oct 3, 2026

Source label: Gemini 3.8 Flash

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 68.8%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Gemini 3.8 Flash. Collapse policy: highest reported score per canonical model.

41.5%

#7 · GPT-5.5

SL · Oct 3, 2026

Source label: GPT-5.5

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 62.5%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: GPT-5.5. Collapse policy: highest reported score per canonical model.

39.7%

#8 · Claude Opus 4.6

SL · Oct 3, 2026

Source label: Claude Opus 4.6

verified runtimeexact aliasBackground only

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 56.3%
Last updated: recent
Eligibility: historical_model
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Claude Opus 4.6. Collapse policy: highest reported score per canonical model.

38.3%

#9 · Claude Opus 4.8

SL · Oct 3, 2026

Source label: Claude Opus 4.8

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 50%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Claude Opus 4.8. Collapse policy: highest reported score per canonical model.

35.3%

#10 · Gemini 3.1 Pro

SL · Oct 3, 2026

Source label: Gemini 3.1 Pro

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 50%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Gemini 3.1 Pro. Collapse policy: highest reported score per canonical model.

35.3%

#11 · GPT-5.6 Sol

SL · Oct 3, 2026

Source label: GPT 5.6 Sol

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 37.5%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: GPT 5.6 Sol. Collapse policy: highest reported score per canonical model.

32.3%

#12 · Gemini 3.5 Flash

SL · Oct 3, 2026

Source label: Gemini 3.5 Flash

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 31.3%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Gemini 3.5 Flash. Collapse policy: highest reported score per canonical model.

27.7%

#13 · Grok 4.20

SL · Oct 3, 2026

Source label: Grok-4.20

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 25%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Grok-4.20. Collapse policy: highest reported score per canonical model.

20%

#14 · kimi-k2.6

SL · Oct 3, 2026

Source label: Kimi-k2.6

verified runtimeexact aliasBackground only

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 18.8%
Last updated: recent
Eligibility: benchmark_derived_model
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Kimi-k2.6. Collapse policy: highest reported score per canonical model.

18.7%

#15 · GPT-5.4

SL · Oct 3, 2026

Source label: GPT-5.4

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 12.5%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: GPT-5.4. Collapse policy: highest reported score per canonical model.

9.7%

#16 · minimax-m2.5

SL · Oct 3, 2026

Source label: Minimax-M2.5

verified runtimeexact aliasBackground only

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 6.3%
Last updated: recent
Eligibility: benchmark_derived_model
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Minimax-M2.5. Collapse policy: highest reported score per canonical model.

6.3%

#17 · GPT-5.3 Codex

SL · Oct 3, 2026

Source label: GPT-5.3-codex

verified runtimeexact aliasBackground only

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 0%
Last updated: recent
Eligibility: specialized_model
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: GPT-5.3-codex. Collapse policy: highest reported score per canonical model.

4.3%

Benchmarks · /benchmarks/scale-hil-bench

HiL-Bench

Scale human-in-the-loop benchmark spanning software engineering and tool-assisted workflows.

Source · Scale Labs
Version · scale-labs snapshot 2026-10-03
Scores · 17

Test details

Visible tradeoffsThis is a rubric-judged signal, so it is more structured than arena taste but still depends on the scoring rubric.

source

Scale Labs

metric

Success rate (%)

judge

Rubric

direction

higher better

group id

scale_hil_current

domain

Coding

What it measures vs what it misses

✓ Measures

Recovery and execution quality on coding-heavy tasks with intervention paths.

✗ Misses

Pure chat fluency. Standalone latency metrics.

Leaderboard · this benchmark version

#1 · Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)

SL · Oct 3, 2026

Source label: Claude Fable 5.1

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 100%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Claude Fable 5.1. Collapse policy: highest reported score per canonical model.

61.5%

#2 · Claude Opus 5 (Adaptive Reasoning, Max Effort)

SL · Oct 3, 2026

Source label: Claude Opus 5

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 93.8%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Claude Opus 5. Collapse policy: highest reported score per canonical model.

57%

#3 · Claude Fable 5

SL · Oct 3, 2026

Source label: Claude Fable 5

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 87.5%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Claude Fable 5. Collapse policy: highest reported score per canonical model.

56.3%

#4 · glm-5.2-max

SL · Oct 3, 2026

Source label: GLM 5.2

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 81.3%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: GLM 5.2. Collapse policy: highest reported score per canonical model.

43.7%

#5 · Claude Opus 4.7

SL · Oct 3, 2026

Source label: Claude Opus 4.7

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 75%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Claude Opus 4.7. Collapse policy: highest reported score per canonical model.

41.7%

#6 · gemini-3.8-flash-high

SL · Oct 3, 2026

Source label: Gemini 3.8 Flash

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 68.8%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Gemini 3.8 Flash. Collapse policy: highest reported score per canonical model.

41.5%

#7 · GPT-5.5

SL · Oct 3, 2026

Source label: GPT-5.5

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 62.5%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: GPT-5.5. Collapse policy: highest reported score per canonical model.

39.7%

#8 · Claude Opus 4.6

SL · Oct 3, 2026

Source label: Claude Opus 4.6

verified runtimeexact aliasBackground only

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 56.3%
Last updated: recent
Eligibility: historical_model
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Claude Opus 4.6. Collapse policy: highest reported score per canonical model.

38.3%

#9 · Claude Opus 4.8

SL · Oct 3, 2026

Source label: Claude Opus 4.8

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 50%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Claude Opus 4.8. Collapse policy: highest reported score per canonical model.

35.3%

#10 · Gemini 3.1 Pro

SL · Oct 3, 2026

Source label: Gemini 3.1 Pro

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 50%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Gemini 3.1 Pro. Collapse policy: highest reported score per canonical model.

35.3%

#11 · GPT-5.6 Sol

SL · Oct 3, 2026

Source label: GPT 5.6 Sol

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 37.5%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: GPT 5.6 Sol. Collapse policy: highest reported score per canonical model.

32.3%

#12 · Gemini 3.5 Flash

SL · Oct 3, 2026

Source label: Gemini 3.5 Flash

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 31.3%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Gemini 3.5 Flash. Collapse policy: highest reported score per canonical model.

27.7%

#13 · Grok 4.20

SL · Oct 3, 2026

Source label: Grok-4.20

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 25%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Grok-4.20. Collapse policy: highest reported score per canonical model.

20%

#14 · kimi-k2.6

SL · Oct 3, 2026

Source label: Kimi-k2.6

verified runtimeexact aliasBackground only

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 18.8%
Last updated: recent
Eligibility: benchmark_derived_model
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Kimi-k2.6. Collapse policy: highest reported score per canonical model.

18.7%

#15 · GPT-5.4

SL · Oct 3, 2026

Source label: GPT-5.4

verified runtimeexact alias

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 12.5%
Last updated: recent
Eligibility: headline eligible
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: GPT-5.4. Collapse policy: highest reported score per canonical model.

9.7%

#16 · minimax-m2.5

SL · Oct 3, 2026

Source label: Minimax-M2.5

verified runtimeexact aliasBackground only

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 6.3%
Last updated: recent
Eligibility: benchmark_derived_model
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: Minimax-M2.5. Collapse policy: highest reported score per canonical model.

6.3%

#17 · GPT-5.3 Codex

SL · Oct 3, 2026

Source label: GPT-5.3-codex

verified runtimeexact aliasBackground only

Raw row drilldownsource, percentile, eligibility

Source URL: https://labs.scale.com/leaderboard/hil
Percentile: 0%
Last updated: recent
Eligibility: specialized_model
Identity: provider alias (0.92)

Parsed from the public Scale Labs page for scale-hil-bench. Reported model configuration: GPT-5.3-codex. Collapse policy: highest reported score per canonical model.

4.3%

HiL-Bench

Test details

What it measures vs what it misses

✓ Measures

✗ Misses

Leaderboard · this benchmark version

Loading benchmark evidence.

HiL-Bench

Test details

What it measures vs what it misses

✓ Measures

✗ Misses

Leaderboard · this benchmark version