Skip to content
All Benchmarks
BenchmarkSeptember 2026

Assistant Reliability Bench

AgentsTool UseLong-HorizonReliability

Assistant Reliability Bench evaluates persistent, tool-driven agent workflows across 28 tasks and 10 categories, combining deterministic checks, rubric assessment, and repeated-run analysis.

1. Introduction

Long-horizon software agents must execute workflows that extend far beyond text generation. In real-world settings, an agent must inspect a persistent workspace, select and use multiple tools, combine evidence from different sources, create or update durable artifacts, and leave the environment in a verifiable final state. Recent benchmarks have moved beyond static question answering. Gaia2 [1] targets dynamic and asynchronous environments. WebArena-Verified [2] offers a curated, version-controlled release with deterministic evaluation. OSWorld 2.0 [3] covers long-horizon computer use in realistic desktop environments. SWE-bench-Live [4] continuously updates multilingual, multi-OS software-engineering tasks. τ²-Bench-Verified [5] provides a corrected, human-verified release of the original dual-control conversational-agent benchmark. Each improves reproducibility, task validity, or environment coverage. Yet they remain largely domain-specific and do not provide a unified view of cross-domain execution, artifact correctness, cross-tool consistency, and repeatability. Building on evidence-centered benchmark design [6], Assistant Reliability Bench evaluates 28 long-horizon tasks across 10 workflow categories. Every task pairs a structured workspace with multi-step tool use and an explicit evaluation contract.

This work makes three contributions. First, it introduces a task suite for persistent, tool-driven workflows that complements existing benchmarks for web use, computer use, software engineering, and tool interaction [2–5]. Second, it combines deterministic pytest checks with rubric-based assessment of semantic quality and process completeness, while retaining trajectories and artifacts for audit. Third, it evaluates the measurement pipeline itself, including cases in which framework metadata contaminates deterministic tests or rubric judges compensate for failures already captured by pytest. Together, these analyses support a more rigorous comparison of agent capabilities and a stronger foundation for future benchmark design.

2. Leaderboard

RankModelTotal / 100PytestRubricTask SD
01Claude Fable 5.186.4548.2938.17—
02GPT-6 Astra85.3847.0438.353.91
03Kimi K378.2344.3533.899.28
04DeepSeek V4.1 Flash76.6943.6333.064.50
05GLM-5.375.3542.9632.3813.22
06Claude Opus 574.0042.9231.0914.73
07GPT-5.6-sol72.1037.8934.218.89
08DeepSeek V4 Pro61.3934.7426.6511.26
Deterministic checksRubric assessment

Lower Task SD indicates more stable results. An em dash indicates insufficient repeated-run data.

3. Benchmark Design

Assistant Reliability Bench comprises 28 long-horizon tasks across 10 workflow categories. The suite spans multiple applications and reflects the kinds of agent-driven workflows found in real work.

Task coverage
Task categoryTasksWhat it tests
Code intelligence3Modify, debug, test, and verify code changes
Communication and messaging3Handle support requests, customer communication, and business delivery
Creative synthesis2Integrate heterogeneous evidence into visual or business-oriented artifacts
Memory and personalization4Maintain personal context, plans, research notes, and persistent memory
Proactive assistance2Monitor external signals and review the current software or project state
Productivity workflows3Document handoffs, opportunity tracking, and financial reconciliation
Safety alignment1Complete a structured security review before release
Scheduling and long-running workflows3Create recurring tasks, monitor execution, and manage follow-up commitments
Search and retrieval3Search, filter, cross-reference, and reconcile local information sources
Skill use and orchestration4Orchestrate multiple specialized tools under scope, safety, and state constraints

Each task includes four components: a natural-language request, a preconfigured workspace, local applications or services, and a defined outcome. A workspace may contain documents, source code, spreadsheets, images, messages, business records, or existing artifacts. The local applications emulate common work environments, including email, messaging, file storage, office tools, project boards, CRM systems, calendars, monitoring dashboards, research repositories, and security registries. Some tasks use a fixed reference date. Others depend on the live state of the runtime environment. This design supports both reproducible state reasoning and time-sensitive workflow execution.

Success requires completed work, not a plausible final response. Depending on the scenario, an agent may need to update records, modify a code repository, produce a report or dashboard, send messages within a defined scope, create recurring tasks, or maintain an auditable chain from evidence to action to final artifact. The benchmark assesses both the quality of the result and whether the required intermediate and persistent states were completed.

Agents operate in isolated local environments. Every run begins from the controlled workspace defined by the task and proceeds through the applications exposed by the benchmark. The system records the interaction trajectory, tool calls, generated artifacts, and final environment state. Evaluation checks the final response, the delivered artifacts, and the resulting state, then uses the trajectory to explain why the task succeeded or failed.

Evaluation combines deterministic checks with rubric-based assessment. Deterministic checks verify observable outcomes, such as the presence of required files, records, messages, or state transitions, as well as the absence of prohibited side effects. The rubric covers qualities that resist binary testing, including evidence quality, semantic completeness, scope control, and end-to-end consistency. Reporting the two components separately, alongside a normalized 100-point total, distinguishes execution correctness from higher-level quality.

4. Task Definition

Each task is a controlled, stateful workflow rather than an isolated question-answering prompt. The environment contains a preconfigured workspace, local applications or services, initial records, and task-specific constraints. Depending on the scenario, it may include repositories, spreadsheets, documents, messages, images, research notes, project records, or existing artifacts. Every run starts from a known state and is isolated from uncontrolled external changes, so the outcome can be attributed to the agent’s decisions and actions.

The agent receives a natural-language request and the prepared environment. The request defines the user’s objective and may impose operational constraints, such as preserving source records, contacting only authorized recipients, using current evidence, avoiding external side effects, or leaving an auditable handoff. No fixed procedure is provided. The agent must identify relevant information, select appropriate tools, resolve ambiguous or conflicting records, and determine when the work is complete. This reflects real operational work, where success depends on coordinating state changes within scope and safety constraints, not merely generating an answer [6].

A task is defined by observable, verifiable end conditions rather than a single canonical response. Depending on the task, these conditions may include a corrected repository, a report or visualization, updated business records, sent messages, completed recurring operations, reconciled datasets, or a complete evidence package. A valid result must satisfy the requirements, preserve protected inputs, and avoid prohibited changes. Different execution paths are acceptable when they produce an equivalent verified state within the same safety boundaries.

Each task includes a task-specific reference specification that establishes that the task is well defined and solvable [9]. It may specify an oracle, pre-seeded records, expected entity sets, required state transitions, protected objects, and prohibited side effects. Deterministic tests cover directly observable conditions. The rubric captures semantic and process requirements that cannot be reduced to binary checks. Together, they define the acceptance boundary: what must exist, what must remain unchanged, and what lies outside the permitted scope.

Agents do not need to reproduce a single reference trajectory. Instead, a reference workflow identifies the key checkpoints that demonstrate solvability: inspect the relevant evidence, determine the in-scope objects, make the required changes, verify the resulting artifacts or records, and complete delivery or audit steps. Tool order and intermediate strategy may vary as long as the agent reaches the same verified state. The benchmark therefore measures problem solving rather than imitation of a fixed script.

Task validity rests on four safeguards. First, the workspace and local services provide the evidence and capabilities required for completion. Second, the reference specification and evaluation contract define the target state, protected boundaries, and prohibited behavior. Third, reference executions or successful model runs demonstrate that the terminal state is reachable. Failure is attributed to the task or evaluator only when the evidence identifies a defect in either one. Finally, every task undergoes manual review for sound design and end-to-end completeness.

5. Evaluation Protocol

The evaluation pipeline has six stages that separate environment quality, agent behavior, artifact quality, and evaluator judgment.

  1. 1.
    Initialize the environment.

    Before each run, the system loads a preconfigured workspace, initial records, local services, reference time, and task constraints. Every environment starts from a known state and runs in isolation. Fixed-clock tasks use a reproducible reference time. Runtime-clock tasks preserve live timing behavior. A preflight check distinguishes configuration and startup failures from agent execution failures.

  2. 2.
    Provide the task.

    The agent receives the task request and access to the workspace and local tools. No fixed procedure is provided. The agent must identify relevant records and tools, interpret the scope and constraints, and plan the complete workflow.

  3. 3.
    Record the agent’s execution.

    During execution, the system records messages, tool calls and results, intermediate observations, errors, timestamps, generated artifacts, runtime state, and the final response. This record helps distinguish failures in reasoning or tool use from premature termination and service outages.

  4. 4.
    Verify artifacts and final state.

    After the run, the evaluator compares the final workspace and service state with the initial state and the task requirements. It checks files, records, messages, scheduled tasks, database entries, evidence packages, delivery status, protected objects, and prohibited side effects. Because evaluation is based on the resulting state, multiple execution paths can be valid.

  5. 5.
    Run deterministic and rubric evaluations.

    After task execution, the evaluator examines the results at two levels.

    Deterministic checks run automatically and verify unambiguous conditions: required files and records, content structure, field values, completed state transitions, authorized recipients, resolvable references, and the absence of prohibited changes. The output identifies every passed and failed check with a specific diagnostic.

    A read-only rubric judge evaluates qualities that cannot be fully encoded as binary tests, including evidence completeness, semantic accuracy, scope control, delivery completeness, and alignment with the task objective. The judge can inspect the task, trajectory, artifacts, and deterministic results but cannot alter the environment.

    Human review is reserved for evaluator audits, disputed samples, and representative failures. It is not part of the default scoring pipeline unless the report defines its scope and scoring rules.

    Deterministic and rubric scores remain separate, preventing a polished narrative from masking missing artifacts, incomplete state transitions, or prohibited side effects.

  6. 6.
    Aggregate the results.

    A run is one model execution on one task. Runs completed under the same configuration and evaluation protocol form a batch. Incomplete runs remain visible in coverage reporting rather than being assigned an artificial score of zero.

    Across repeated runs, the system averages each task first and then macro-averages across tasks. Variation is reported separately so that average performance and consistency remain distinct.

6. Scoring and Results

6.1 Scoring Framework

Each task uses a hybrid evaluation that combines deterministic tests with a rubric-based judge. Deterministic tests verify the expected state changes and artifacts, including files, records, messages, structured data, protected state, and prohibited side effects. The rubric judge assesses qualities that are difficult to formalize completely, such as evidence completeness, interpretation, delivery quality, and consistency across artifacts.

For a run r of task t, the final score is expressed as:

St,r = Dt,r + Rt,r

Here, D is the deterministic (pytest) score and R is the rubric score. The point allocation varies by task, but the maximum score is always 100.

Evaluation covers three dimensions:

  • Task completion: whether the model achieves the task's core business objective
  • Artifact correctness: whether the files, records, messages, reports, or evidence packages generated by the model are correct
  • Instruction following: whether the model respects constraints such as task scope, time boundaries, target recipients, protected states, and forbidden side effects

These dimensions describe the deterministic and rubric results for a single run. Robustness is measured separately across repeated runs and does not contribute additional points to an individual task score.

Unless stated otherwise, the rubric judge produces the rubric score. Human review is reserved for evaluator audits, disputed samples, and representative failures. Human scores are not combined with automated judge scores.

6.2 Repeated-Run Protocol

Each model is evaluated repeatedly under the same task state, model configuration, agent interface, and evaluation protocol. Repeated runs reveal performance variance that a single run cannot capture.

Detailed execution analysis covers four models:

  • Claude Opus 5
  • GPT-5.6-sol
  • Kimi K3
  • DeepSeek V4 Pro.

For each model, scores are averaged within each task and then macro-averaged across all 28 tasks. Missing tasks, execution failures, and evaluation failures remain visible in coverage and completion metrics rather than being silently converted to zero.

6.3 Main Results

Within the original four-model cohort, Kimi K3 has the highest task-level macro-average, followed by Claude Opus 5 and GPT-5.6-sol, with DeepSeek V4 Pro scoring lowest. These results reflect the evaluated configuration, including the agent scaffold, reasoning settings, runtime, upstream services, and execution environment. They should not be read as an absolute ranking of general model capability.

The creative-synthesis-techstack-heatmap task illustrates the benchmark’s demands. The agent must use local portfolio evidence to build a self-contained, offline heatmap of eight projects across eight technology stacks. It must preserve distinctions such as current ownership, lifecycle stage, limited rollout, rollback status, and the absence of a current relationship. The artifact also needs a clear taxonomy, an ordinal legend, and accessible evidence for every cell. This single task tests evidence grounding, state interpretation, artifact production, and interaction design. Scores range from 32.0 for DeepSeek V4 Pro to 94.0 for Claude Opus 5, a 62-point spread. The difference reflects more than writing quality. Strong performance requires thorough workspace exploration, complete candidate selection within the relevant time window, reliable links among objects, identifiers, and evidence, completion of state transitions such as registration or delivery, and final verification. Trajectory analysis reveals failures at different stages. Some models make sound business judgments but do not complete delivery. Others produce well-structured audit artifacts with incorrect classifications, identifiers, or evidence links. Some deliver invalid files or stop before the workflow is complete. A credible final response is therefore insufficient. Evaluation must verify the required artifacts and final state.

6.4 Aggregation and Reporting

Model-level scores use task-level macro-averaging rather than pooling trajectories or individual assertions. The calculation proceeds as follows:

  1. 1.
    Compute each task's average total score across repeated runs
  2. 2.
    Compute each task's average pytest score
  3. 3.
    Compute each task's average rubric score
  4. 4.
    Average the task-level results over the 28 tasks to obtain the model-level scores.

The report presents:

  • mean total score
  • mean pytest score
  • mean rubric score
  • task coverage
  • execution success rate
  • rubric evaluation completion rate.

Task-level results are reported as mean ± standard deviation, while the heatmap shows how performance varies across the 28 tasks. Together, these views distinguish low average performance, strong but uneven performance, and localized instability that a single aggregate score would hide.

Task-level score heatmap. 28 tasks × 8 models, grouped by workflow category.

6.5 Robustness and Failure Analysis

Robustness is reported separately across repeated runs and does not add points to task scores. Models with similar averages can differ substantially in consistency.

Robustness is assessed through:

  1. 1.
    the standard deviation of the same task across repeated runs
  2. 2.
    execution success rate across runs
  3. 3.
    the proportion of tasks consistently completed across runs
  4. 4.
    the proportion of evaluations in which both pytest and rubric completed successfully
  5. 5.
    whether failures concentrate in a few tasks, tool calls, or environment components.

Failure analysis examines trajectories, pytest results, and rubric evidence in addition to the final response. Recurring failure modes include:

  • incomplete coverage of candidate sets or scenarios
  • incorrect tool-call order
  • artifacts generated but registration or delivery not closed
  • failure to retry or verify based on error messages
  • use of the wrong time, data source, or task scope
  • correct business outcomes paired with incomplete evidence or audit artifacts
  • material inconsistencies between rubric judgments and deterministic results.

Selected representative cases illustrate these recurring failure modes.

Performance and stability. Mean task score against mean task-level standard deviation. Lower SD indicates more stable runs.

7. Error Analysis

7.1 Analysis Scope

This section analyzes execution traces from repeated runs of Claude Opus 5, GPT-5.6-sol, Kimi K3, and DeepSeek V4 Pro.

The analysis draws on:

  • task instructions
  • execution logs
  • final responses
  • artifacts generated in the workspace
  • deterministic test results, including pytest and JUnit output
  • rubric judgments.

The analysis asks whether the model completed the required selection, state changes, evidence links, artifact delivery, and final verification—not merely whether its answer sounded correct.

7.2 Model Error Taxonomy

The following failure patterns recur across model trajectories.

Recurring model error patterns
Error patternSpecific manifestation
Insufficient environment groundingActed before confirming the active profile, workbook, build, list, or controlling state
Tool and API errorsUsed incorrect commands, paths, parameters, enumerations, identifiers, or tool entry points
Incomplete candidate-set coverageOmitted required objects or included inactive, historical, or irrelevant ones
State binding errorsBound results to the wrong run, profile, date, revision, list, or package
Broken identifiers and evidence linksArtifacts, decisions, evidence, versions, and packages did not resolve consistently
Incomplete deliveryCreated files without registering them, drafted messages without sending them, or produced reviews without attaching them to the target object
Domain-specific errorsIncorrect judgments of item classification, quantity, unit, amount, date, ownership, or security level
Weak recovery from tool errorsFailed to re-check state or change strategy after a tool error
State drift over long workflowsLost previously established constraints, repeated work, or stopped prematurely
Insufficient final verificationClaimed completion without verifying files, delivery status, hashes, references, or audit logs
Invalid artifact formatsProduced YAML, JSON, spreadsheets, or other deliverables that could not be parsed
Safety-boundary violationsModified protected sources, exceeded the permitted scope, contacted the wrong target, or performed prohibited external actions

7.3 Representative Model Failures

01November Retail Sales CloseConfusing a Summary with Completed Delivery

The task requires the model to close the November 2011 sales period, generate the supporting detail, send the package to the sales-team list, and verify successful delivery.

Representative run:

runs/batches/20260813T130048Z-ad186f/
november-2011-retail-sales-close/
20260813T130710Z-b61723/

The model's final reply:

Delivered to the active Retail Sales team list.

Pytest score: 7/74. Failed checks included:

  • canonical November population
  • original units
  • summary anchors
  • artifact join
  • final package identity
  • authorized send transition
  • terminal delivery reference
  • forbidden extras.

The run state also shows that the registered file’s content_hash differs from the actual file hash and that the email state contains no valid transition to sent.

The failure was not computational. The model did not complete the steps required for a valid delivery:

  1. 1.
    did not verify the canonical source population
  2. 2.
    did not confirm that the final workbook matches the registered object
  3. 3.
    did not confirm that the email moved from draft to sent
  4. 4.
    did not confirm that the delivery reference traces back to the final package
  5. 5.
    declared completion before the state was closed.

This is a clear example of a plausible result without a completed delivery state.

02Receipt ImportIncorrect Core Item Classification

Representative run:

runs/batches/20260813T130048Z-ad186f/
receipt-import-into-the-active-household-grocery-list/
20260813T134009Z-afb77f/

Pytest score: 18/36. In line-accounting.json, line-011 was recorded as:

{
  "canonical_name": "AA batteries",
  "catalog_class": "household_supply",
  "status": "included_grocery_item"
}

The model imported a household supply as a grocery item. That error then propagated through the list delta, import ledger, and notification audit.

Four errors drove the result:

  • the boundary judgment between active grocery items and household supplies
  • semantic classification of receipt lines
  • exclusion of non-grocery items
  • correct updates to the list delta and ledger.

This case matters because the audit artifacts were structurally complete and well referenced, yet the underlying business decision was wrong. Strong documentation cannot compensate for an incorrect classification.

03Recurring Safety ScanBreaking the End-to-End State Chain

Representative run:

runs/batches/20260813T130048Z-ad186f/
initialize-recurring-local-safety-scan/
20260813T134505Z-bec5fc/

Pytest score: 8/54. Failed checks covered:

  • selected alert place resolution
  • source scan run ID
  • active-scope non-alert disposition
  • selected alert classification
  • inactive-scope preservation
  • baseline traceability
  • notification commitment
  • missing-location recovery
  • evidence package traceability
  • summary references
  • cross-tool operation history.

The failure extended beyond a single cron field. The model did not maintain the scan as a coherent state chain:

  • did not fully distinguish active and inactive scopes
  • did not bind alerts to the correct source scan run
  • did not connect baseline, notification, and evidence package
  • did not write explicit dispositions for objects without triggered alerts
  • final-summary references could not be traced to committed state rows.

This is a long-horizon state-management failure: the model did not preserve the objects and constraints it had already resolved.

04Static Skill ReviewCorrect Conclusion, Incomplete Audit Evidence

Representative run:

runs/batches/20260813T130048Z-ad186f/
static-pre-publication-review-for-pinned-community-skill/
20260814T014515Z-fa44e2/

The model identified three main risks:

  1. 1.
    plaintext remote transfer
  2. 2.
    process-visible key material
  3. 3.
    restore-path activation.

Pytest score: 31/47. Failed checks included:

  • test_finding_membership_integrity
  • test_report_registry_finding_join
  • test_remediation_lifecycle.

The artifact contains several structural defects:

  • the same evidence ID is reused across multiple findings
  • the review report is not correctly bound to the Registry object
  • remediation items do not cover all blocking effects
  • the findings, report, registry, and remediation records do not form a complete audit chain.

The model reached the correct high-level decision, remediation_hold, but failed to organize the supporting evidence and remediation state accurately. A correct security conclusion does not, by itself, constitute a complete audit.

05PR Visual ReviewStrong Analysis, Invalid Deliverable

Representative run:

runs/batches/20260813T130048Z-ad186f/
review-pull-request-visual-build-evidence/
20260813T132137Z-f4999e/

Pytest score: 7/26. The review demonstrated several strengths:

  • distinguishing the StaffTrials and build-page visual families
  • recognizing uncertainty around retryFailure
  • linking the review to the current PR head and build lineage
  • establishing baselines and conditions for follow-up reviews.

The submitted YAML failed to parse at line 87:

yaml.scanner.ScannerError:
mapping values are not allowed here

The model performed strong visual and causal analysis but failed to validate the deliverable’s format.

The case separates two distinct capabilities:

  • visual evidence understanding and causal interpretation
  • formal artifact generation, serialization, and pre-submission validation.

Because the model did not re-open or parse the YAML before submission, sound reasoning never became a valid deliverable.

06Account MFACorrect Security Reasoning, Incorrect State Encoding

Representative run:

runs/batches/20260813T130048Z-ad186f/
account-security-console-mfa-setup/
20260813T134224Z-f7b4ab/

Pytest score: 20/43. Failed checks included:

  • out_of_population should be rejected_out_of_scope
  • the reason ref uses the wrong prefix
  • the selected setup path does not match the current state
  • the cloud and repository decision IDs are incorrect
  • the recovery decision does not join correctly.

The model understood the security concepts—MFA, active profiles, targets, and exceptions—but encoded them with incorrect enumerations, identifiers, and references. The failure lay in translating a sound conceptual judgment into the required state representation.

07Marriott HandoffCorrect Sessions, Incomplete Audit Package

Representative run:

runs/batches/20260813T130048Z-ad186f/
current-marriott-sxsw-session-handoff/
20260813T133949Z-0aa3b5/

The model identified all three sessions but still failed four requirements:

  • result artifact schema
  • reconciled signal classification
  • evidence references
  • single active result cardinality.

Although the session selection was correct, the model did not complete the evidence and delivery workflow:

  • linking sessions with the evidence chain
  • complete recording of exclusions
  • creation of a single active result
  • generation of an auditable handoff artifact.

The selected entities were correct, but the final handoff state was incomplete.

08Nightly Flaky CICorrect Test Selection, Incorrect Commitment Lineage

Several models selected similar flaky tests and quality gates, but their runs repeatedly exposed the following state errors:

commitment lineage is not the active local lineage
commitment must join to the current resolved gate
basis_refs must be resolvable evidence_ref values

Recurring failures included:

  • failure to bind the active context
  • broken links between gate decisions and PR commitments
  • incomplete representation of selected and excluded tests
  • unresolvable audit references
  • incorrectly scoped stale-state cleanup
  • handoff payloads with incomplete lineage.

The models often choose the right tests and quality gates, then lose the relationships among them when recording and delivering the result.

09Reconcile WorkbookWrong Tool Entry Point and Cross-Sheet Inconsistencies

Representative Claude Opus 5 run:

runs/batches/20260813T022455Z-405d70/
reconcile-household-net-worth-workbook/
20260813T025952Z-954d2b/

The model first invoked the wrong Office tool, then asked whether the workbook existed even though it was already present in the task directory.

In another GPT-5.6-sol run, the model found and modified the workbook but still failed checks covering:

  • economic framework
  • ownership
  • cross-context invariant
  • dated snapshot
  • forbidden side-effect checks.

This task exposed two distinct failures:

  1. 1.
    incorrect initial choice of tool and file location
  2. 2.
    failure to verify economic and temporal consistency across sheets after reading and writing.
10Local Job Search and PlaywrightWeak Error Recovery and Verification

Failed Local Job Search runs contained two unresolved state errors:

binding_ref_mismatch
unresolved_refs

The model continued to draw conclusions while key bindings remained unresolved. It failed to:

  • return to the active profile
  • re-confirm the snapshot
  • verify approval status and package state
  • fix unresolved references
  • adjust the final answer based on tool results.

Successful Playwright runs complete the full workflow. Failed runs commonly show:

  • use of the wrong test entry point
  • running an unnecessary npm install
  • modifying execution paths in an existing workspace
  • not re-reading the final UI state after page operations
  • stopping early after completing only some form steps.

These failures reflect weak tool selection, process control, and final verification.

7.4 Model Error Profiles

Claude Opus 5

Claude Opus 5 usually produces fluent, well-structured final responses, but long workflows expose recurring execution gaps:

  • premature claims of completion
  • missing final read-back
  • incorrect tool selection
  • loss of previously established governing state
  • artifacts produced without completing registration, delivery, or audit steps.

The November Retail Sales Close, Reconcile Workbook, and Recurring Safety Scan cases show a consistent gap between communication quality and execution completeness.

GPT-5.6-sol

GPT-5.6-sol performs well in visual analysis, causal reasoning, and evidence organization, but still exhibits:

  • invalid YAML, JSON, or other artifact formats
  • inconsistent evidence references and identifiers
  • incomplete state updates across worksheets
  • final responses issued before the state was complete.

The PR Visual case shows that strong reasoning and reliable delivery are distinct capabilities.

Kimi K3

Kimi K3 leads the original cohort on average and completes more deterministic state transitions on many tasks. Its remaining weaknesses include:

  • fine-grained evidence-linking errors
  • quantity and unit errors
  • confusion between active and inactive scope
  • inconsistent decision IDs
  • missing final-state verification.

Kimi K3’s advantage comes primarily from stronger process completion and fewer premature stops, although some state representations remain fragile.

DeepSeek V4 Pro

DeepSeek V4 Pro has the lowest average in the original cohort. Its main failure modes include:

  • unstable tool sequencing
  • weak alignment between workspace data and active context
  • insufficient candidate-set coverage
  • incomplete links among schemas, identifiers, and evidence
  • missing final verification after artifact generation.

DeepSeek V4 Pro nevertheless passes every deterministic check on some tasks, indicating weaknesses in interpretation, causal analysis, and handoff quality as well as execution.

8. References

  1. [1]
    Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments. arXiv:2602.11964, 2026.https://arxiv.org/abs/2602.11964
  2. [2]
    ServiceNow. WebArena-Verified: A Verified Release of the WebArena Benchmark. Technical release, 2025–2026.https://github.com/ServiceNow/webarena-verified
  3. [3]
    OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks. arXiv:2606.29537, 2026.https://arxiv.org/abs/2606.29537
  4. [4]
    Zhang et al. SWE-bench Goes Live! arXiv:2505.23419, 2025.https://arxiv.org/abs/2505.23419Current release: https://github.com/microsoft/SWE-bench-Live
  5. [5]
    Amazon AGI. τ²-Bench-Verified: Evaluating Conversational Agents in a Dual-Control Environment. arXiv:2512.07850, 2025.https://arxiv.org/abs/2512.07850Release: https://github.com/amazon-agi/tau2-bench-verified
  6. [6]
    WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation. arXiv:2605.10912v1, 2026.
  7. [7]
    Yao et al. ReAct: Synergizing Reasoning and Acting in Language Models. 2023.
  8. [8]
    Trivedi, H. et al. AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents. ACL 2024. arXiv:2407.18901.
  9. [9]
    Liu, X. et al. AgentBench: Evaluating LLMs as Agents. ICLR 2024. arXiv:2308.03688.