Finding Where the Buck Stops: An Automated Failure Attribution-Based Reflection Framework for Multi-Agent Collaboration · CiteArk

Recent activity

—Starting the worker
—Existing research plan reused
—Orchestrating training, evaluation and analysis
—Analysis environment started
—Execution needs attention

Credit estimate for this reproduction

Expected usage

549

Maximum reservation

1683

Estimated compute time

About 210 min

Actual usage was 0 credits; 0 credits were charged. Compute ran for about 0 minutes and the model used 0 tokens.

Failed attempts also used about 14 minutes of compute and 7142207 model tokens. They produced no importable result and were not billed.

Finding Where the Buck Stops: An Automated Failure Attribution-Based Reflection Framework for Multi-Agent Collaboration

Public

Authors:Xiaoqing Wang, Keman Huang, Bin Liang, Hongyu Li, Xiaoyong Du, Wuqiong Pan

EMNLP 2026 arXiv 2026 Artificial Intelligence Computation and Language Machine Learning (stat.ML)

Reproduction results

All claims ↗

No reproduction results yet

The paper’s claims are available in Research claims.

Full research claim catalog

0 / 4 claims verified

the rest still being verified

Key metricsPaper-reported → reproduced · click a row to expand

Across five trials, DoCtOR improves initial task success rates by 22% on HotPotQA, 26% on ChartQAPro, and 27% on Mind2Web, and outperforms the reported reflection baselines.Whole claim: Awaiting reproduction0/3 measurements supported · 0 assessed3measurementsExplore results

Experiment plan ready; no runs yet.

Measurement results

0/3 assessed

0 supported0 disputed or conflicting3 not assessed

The complete result group stays with its claim. Assessed rows come first; their judgments do not replace the whole-claim conclusion. Expand a row for conditions, sources, and evidence history.

Each row is a measurement with its reported value, latest available observation, and cumulative evidence state.
Metric and conditions	Reported	Latest observation	Evidence judgment
trials: 5 · dataset: HotPotQA · metric_scope: task success rate · test_tasks_per_trial: 100	22 percentage_points	—	Not assessed
trials: 5 · dataset: ChartQAPro · metric_scope: task success rate · test_tasks_per_trial: 100	26 percentage_points	—	Not assessed
trials: 5 · dataset: Mind2Web · metric_scope: task success rate · test_tasks_per_trial: 100	27 percentage_points	—	Not assessed

Platform or execution failure · no scientific conclusion

Required input unavailable · Execution will continue after the required code, data, or input is supplied.

Open claim details

ProFA achieves the strongest reported agent-level and step-level failure-attribution accuracies among the listed methods on the held-in HotPotQA, ChartQAPro, and Mind2Web datasets and on the held-out Algorithm-Generated and Hand-Crafted subsets.Whole claim: Awaiting reproduction0/3 measurements supported · 0 assessed3measurementsExplore results

Experiment plan ready; no runs yet.

Measurement results

0/3 assessed

0 supported0 disputed or conflicting3 not assessed

The complete result group stays with its claim. Assessed rows come first; their judgments do not replace the whole-claim conclusion. Expand a row for conditions, sources, and evidence history.

Each row is a measurement with its reported value, latest available observation, and cumulative evidence state.
Metric and conditions	Reported	Latest observation	Evidence judgment
split: held-in · method: ProFA · dataset: HotPotQA · value_scope: test set of constructed Who & When Pro dataset · accuracy_level: agent	0.85 fraction	—	Not assessed
split: held-out Algorithm-Generated · method: ProFA · dataset: Who & When · failure_logs: 184 · accuracy_level: step	0.53 fraction	—	Not assessed
split: held-out Hand-Crafted · method: ProFA · dataset: Who & When · failure_logs: 184 · accuracy_level: step	0.2 fraction	—	Not assessed

Platform or execution failure · no scientific conclusion

Experiment execution error · No automatic retry; a person must decide what to do next.

Open claim details

Qualitative findings

Using only reasoning steps after the decisive error step produces reflection quality comparable to using the complete failure trajectory, while diagnosis, correction, and PPO each contribute to performance.Experiment blocked; see the specific reason

The claim is supported by plotted curves without machine-readable scalar values, and direct evaluation requires the unavailable datasets, model assets, external agent services, and exact implementation details.

Open claim details

Other claims1

The evaluation is limited to English-language task-oriented collaborations with relatively small multi-agent teams and leaves open performance on open-ended or culturally diverse tasks and larger teams.Stated by the authors

This is an explicit limitation note, not an independent empirical measurement requiring reproduction.

Open claim details