Diagnosing the Fact-Grounding Gap in Multi-Hop Question Answering · CiteArk

Recent activity

—Analysis made new progress
—Analysis environment started
—Analysis made new progress
—Research inventory and experiment plan prepared
—Waiting for experiment selection

ArkGraph

Explore ArkGraph and select the steps to run.

Diagnosing the Fact-Grounding Gap in Multi-Hop Question Answering

Public

Authors:Kevin Mo, Nathan Mo, Richard Zhu

arXiv 2026 EMNLP 2026 Computation and Language Artificial Intelligence Information Retrieval

Reproduction results

All claims ↗

No reproduction results yet

The paper’s claims are available in Research claims.

Full research claim catalog

0 / 34 claims verified

the rest still being verified

Key metricsPaper-reported → reproduced · click a row to expand

The paper reports strong agreement between GPT-4.1-mini judging and human annotation, while noting a conservative error skew for fact-presence decisions.Whole claim: Awaiting reproduction0/5 measurements supported · 0 assessed5measurementsExplore results

Experiment plan ready; no runs yet.

Measurement results

0/5 assessed

0 supported0 disputed or conflicting5 not assessed

The complete result group stays with its claim. Assessed rows come first; their judgments do not replace the whole-claim conclusion. Expand a row for conditions, sources, and evidence history.

Each row is a measurement with its reported value, latest available observation, and cumulative evidence state.
Reported	Latest observation	Evidence judgment
92%	—	Not assessed
0.84 coefficient	—	Not assessed
200 examples	—	Not assessed
12 cases	—	Not assessed
16 cases	—	Not assessed

Metric and conditions	Reported	Latest observation	Evidence judgment
dataset: MuSiQue · flaggedHops: 0	68.7%	—	Not assessed
dataset: MuSiQue · flaggedHops: 0	69.1%	—	Not assessed
dataset: MuSiQue · flaggedHops: 1	52.9%	—	Not assessed
dataset: MuSiQue · flaggedHops: 1	56.6%	—	Not assessed
dataset: MuSiQue · flaggedHops: 2	44.9%	—	Not assessed
dataset: MuSiQue · flaggedHops: 2	49.6%	—	Not assessed
dataset: MuSiQue · flaggedHops: 3	38.8%	—	Not assessed
dataset: MuSiQue · flaggedHops: 3	44.7%	—	Not assessed
dataset: MuSiQue · flaggedHops: 4	8.5%	—	Not assessed
dataset: MuSiQue · flaggedHops: 4	25.5%	—	Not assessed
dataset: MuSiQue · flaggedHops: 0	0.4 percentage points	—	Not assessed
dataset: MuSiQue · flaggedHops: 4	17 percentage points	—	Not assessed

Metric and conditions	Reported	Latest observation	Evidence judgment
condition: Baseline	78.8%	—	Not assessed
condition: Baseline	0%	—	Not assessed
condition: Baseline	0 calls/question	—	Not assessed
condition: Always	78.7%	—	Not assessed
condition: Always	100%	—	Not assessed
condition: Always	2.15 calls/question	—	Not assessed
condition: DeBERTa targeted	78.9%	—	Not assessed
condition: DeBERTa targeted	54.1%	—	Not assessed
condition: DeBERTa targeted	1.16 calls/question	—	Not assessed

Metric and conditions	Reported	Latest observation	Evidence judgment
condition: Baseline	42.2%	—	Not assessed
condition: Baseline	0%	—	Not assessed
condition: Baseline	0 calls/question	—	Not assessed
condition: DeBERTa targeted	52.2%	—	Not assessed
condition: DeBERTa targeted	53.4%	—	Not assessed
condition: DeBERTa targeted	1.07 calls/question	—	Not assessed
condition: Oracle	52.3%	—	Not assessed
condition: Oracle	54.1%	—	Not assessed
condition: Oracle	1.08 calls/question	—	Not assessed
	10 percentage points	—	Not assessed

Metric and conditions	Reported	Latest observation	Evidence judgment
model: BERT-base	110 millions	—	Not assessed
model: BERT-base	0.753 score	—	Not assessed
model: BERT-base	0.737 score	—	Not assessed
model: BERT-base	0.745 score	—	Not assessed
model: RoBERTa-large	355 millions	—	Not assessed
model: RoBERTa-large	0.84 score	—	Not assessed
model: RoBERTa-large	0.683 score	—	Not assessed
model: RoBERTa-large	0.754 score	—	Not assessed
model: DeBERTa-v3-large	435 millions	—	Not assessed
model: DeBERTa-v3-large	0.805 score	—	Not assessed
model: DeBERTa-v3-large	0.765 score	—	Not assessed
model: DeBERTa-v3-large	0.785 score	—	Not assessed

Metric and conditions	Reported	Latest observation	Evidence judgment
flaggedHops: 0	556 questions	—	Not assessed
flaggedHops: 0	68.7%	—	Not assessed
flaggedHops: 1	868 questions	—	Not assessed
flaggedHops: 1	52.9%	—	Not assessed
flaggedHops: 2	690 questions	—	Not assessed
flaggedHops: 2	44.9%	—	Not assessed
flaggedHops: 3	255 questions	—	Not assessed
flaggedHops: 3	38.8%	—	Not assessed
flaggedHops: 4	47 questions	—	Not assessed
flaggedHops: 4	8.5%	—	Not assessed

Metric and conditions	Reported	Latest observation	Evidence judgment
intervention: augmentation	8.6 percentage points	—	Not assessed
intervention: reranking	25.3 percentage points	—	Not assessed
intervention: re-retrieval; intervention subtype not separately reported for this value	-0.5 percentage points	—	Not assessed
dataset: MuSiQue	97.7%	—	Not assessed
dataset: HotpotQA	100%	—	Not assessed
dataset: 2WikiMultihopQA	97.6%	—	Not assessed

Metric and conditions	Reported	Latest observation	Evidence judgment
questionType: comparison	2.7%	—	Not assessed
questionType: multi-step	16.6%	—	Not assessed
	97.4%	—	Not assessed
	0.983 score	—	Not assessed
	0%	—	Not assessed
predictor: 2WikiMultihopQA-retrained	78.9%	—	Not assessed
	78.8%	—	Not assessed
	0 probability	—	Not assessed
	0.999 probability	—	Not assessed

Metric and conditions	Reported	Latest observation	Evidence judgment
condition: Baseline	51.8%	—	Not assessed
condition: Baseline	0%	—	Not assessed
condition: Baseline	0 calls/question	—	Not assessed
condition: Always	60.9%	—	Not assessed
condition: Always	100%	—	Not assessed
condition: Always	2.65 calls/question	—	Not assessed
condition: DeBERTa targeted	60.3%	—	Not assessed
condition: DeBERTa targeted	50%	—	Not assessed
condition: DeBERTa targeted	1.33 calls/question	—	Not assessed
	8.5 percentage points	—	Not assessed

Metric and conditions	Reported	Latest observation	Evidence judgment
questionHops: 3	-2.1 percentage points	—	Not assessed
questionHops: 3	2.1 percentage points	—	Not assessed
questionHops: 3	4.2 percentage points	—	Not assessed

Metric and conditions	Reported	Latest observation	Evidence judgment
	0.449 coefficient	—	Not assessed
	0.042 coefficient	—	Not assessed
	5,283 hops	—	Not assessed
method: Majority	50.9%	—	Not assessed
method: Majority	0 score	—	Not assessed
method: Hop number	64.9%	—	Not assessed
method: Hop number	0.596 score	—	Not assessed
method: Retrieval recall	68.1%	—	Not assessed
method: Retrieval recall	0.716 score	—	Not assessed
method: String match · usesGold: true	80.8%	—	Not assessed
method: String match · usesGold: true	0.823 score	—	Not assessed
method: Gold paragraph present · usesGold: true	83.4%	—	Not assessed

Metric and conditions	Reported	Latest observation	Evidence judgment
hop: 1	68.5%	—	Not assessed
hop: 1	2,417 hops	—	Not assessed
hop: 2	43.7%	—	Not assessed
hop: 2	2,417 hops	—	Not assessed
hop: 3	30.7%	—	Not assessed
hop: 3	1,165 hops	—	Not assessed
hop: 4	18.5%	—	Not assessed
hop: 4	405 hops	—	Not assessed
hop: all	49.1%	—	Not assessed
hop: all	6,404 hops	—	Not assessed

Reported	Latest observation	Evidence judgment
1,962 hops	—	Not assessed
30.7%	—	Not assessed
1,749 hops	—	Not assessed
27.3%	—	Not assessed
2,690 hops	—	Not assessed
42%	—	Not assessed
47%	—	Not assessed
9%	—	Not assessed
3 hops	—	Not assessed

Metric and conditions	Reported	Latest observation	Evidence judgment
condition: Baseline	48.1%	—	Not assessed
condition: Baseline	0%	—	Not assessed
condition: Baseline	0 calls/question	—	Not assessed
condition: Always	57.1%	—	Not assessed
condition: Always	100%	—	Not assessed
condition: Always	2 calls/question	—	Not assessed
condition: DeBERTa targeted	53.2%	—	Not assessed
condition: DeBERTa targeted	30.7%	—	Not assessed
condition: DeBERTa targeted	0.61 calls/question	—	Not assessed
condition: Oracle	55.5%	—	Not assessed
condition: Oracle	46.9%	—	Not assessed
condition: Oracle	0.94 calls/question	—	Not assessed

Reported	Latest observation	Evidence judgment
81.7%	—	Not assessed
2,677 hops	—	Not assessed
18.3%	—	Not assessed
15%	—	Not assessed
468 hops	—	Not assessed

Metric and conditions	Reported	Latest observation	Evidence judgment
labels: string match · passages: excluded	54.7%	—	Not assessed
labels: string match · passages: excluded	0.547 score	—	Not assessed
labels: LLM judge · passages: excluded	63.6%	—	Not assessed
labels: LLM judge · passages: excluded	0.615 score	—	Not assessed
labels: LLM judge · passages: included	79.4%	—	Not assessed
labels: LLM judge · passages: included	0.785 score	—	Not assessed
	8.9 percentage points	—	Not assessed
	15.8 percentage points	—	Not assessed

Reported	Latest observation	Evidence judgment
6,404 sub-questions	—	Not assessed
4,100 sub-questions	—	Not assessed
2,304 sub-questions	—	Not assessed
1,315 patterns	—	Not assessed

Metric and conditions	Reported	Latest observation	Evidence judgment
condition: Baseline	51.9%	—	Not assessed
condition: Baseline	0%	—	Not assessed
condition: Baseline	0 calls/question	—	Not assessed
condition: Always	54.6%	—	Not assessed
condition: Always	100%	—	Not assessed
condition: Always	2.65 calls/question	—	Not assessed
condition: DeBERTa targeted	55.6%	—	Not assessed
condition: DeBERTa targeted	50%	—	Not assessed
condition: DeBERTa targeted	1.32 calls/question	—	Not assessed
condition: Oracle	54.9%	—	Not assessed
condition: Oracle	48.9%	—	Not assessed
condition: Oracle	1.29 calls/question	—	Not assessed

Metric and conditions	Reported	Latest observation	Evidence judgment
dataset: MuSiQue train	46,610 hops	—	Not assessed
	53.9%	—	Not assessed
	46.1%	—	Not assessed
dataset: MuSiQue dev	6,404 hops	—	Not assessed
	3 epoch	—	Not assessed

Metric and conditions	Reported	Latest observation	Evidence judgment
dataset: MuSiQue	139,416 passages	—	Not assessed
dataset: HotpotQA	507,494 passages	—	Not assessed
dataset: 2WikiMultihopQA	136,009 passages	—	Not assessed

Metric and conditions	Reported	Latest observation	Evidence judgment
condition: Baseline	51.2%	—	Not assessed
condition: Baseline	0%	—	Not assessed
condition: Baseline	0 calls/question	—	Not assessed
condition: DeBERTa targeted	54.7%	—	Not assessed
condition: DeBERTa targeted	45.2%	—	Not assessed
condition: DeBERTa targeted	1.2 calls/question	—	Not assessed
condition: Oracle	54.8%	—	Not assessed
condition: Oracle	50.9%	—	Not assessed
condition: Oracle	1.35 calls/question	—	Not assessed
	3.5 percentage points	—	Not assessed

ArkGraph

Diagnosing the Fact-Grounding Gap in Multi-Hop Question Answering

Research claims

No reproduction results yet

Key metricsPaper-reported → reproduced · click a row to expand

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Qualitative findings

Reported	Latest observation	Evidence judgment
654 traces	—	Not assessed
48.1%	—	Not assessed
51.6%	—	Not assessed
3.5 percentage points	—	Not assessed
2,417 questions	—	Not assessed
3.7 percentage points	—	Not assessed

Reported	Latest observation	Evidence judgment
3,201 hops	—	Not assessed
50%	—	Not assessed
50.9%	—	Not assessed

Reported	Latest observation	Evidence judgment
59.8%	—	Not assessed
49.1%	—	Not assessed
10.7 percentage points	—	Not assessed

Metric and conditions	Reported	Latest observation	Evidence judgment
dataset: MuSiQue · predictor: MuSiQue-trained · retriever: BM25 · intervention: Reranking	8.5 percentage points	—	Not assessed
dataset: MuSiQue · predictor: MuSiQue-trained · retriever: BM25 · intervention: Reranking	0 probability	—	Not assessed
dataset: MuSiQue · predictor: MuSiQue-trained · retriever: BM25 · intervention: Reranking	6.8 percentage points	—	Not assessed
dataset: MuSiQue · predictor: MuSiQue-trained · retriever: BM25 · intervention: Reranking	10.2 percentage points	—	Not assessed
dataset: MuSiQue · predictor: MuSiQue-trained · retriever: BM25 · intervention: Augmentation	3.7 percentage points	—	Not assessed
dataset: MuSiQue · predictor: MuSiQue-trained · retriever: BM25 · intervention: Augmentation	0 probability	—	Not assessed
dataset: MuSiQue · predictor: MuSiQue-trained · retriever: BM25 · intervention: Augmentation	2.2 percentage points	—	Not assessed
dataset: MuSiQue · predictor: MuSiQue-trained · retriever: BM25 · intervention: Augmentation	5.2 percentage points	—	Not assessed
dataset: MuSiQue · predictor: MuSiQue-trained · retriever: Contriever · intervention: Augmentation	3.5 percentage points	—	Not assessed
dataset: MuSiQue · predictor: MuSiQue-trained · retriever: Contriever · intervention: Augmentation	0 probability	—	Not assessed
dataset: MuSiQue · predictor: MuSiQue-trained · retriever: Contriever · intervention: Augmentation	2.1 percentage points	—	Not assessed
dataset: MuSiQue · predictor: MuSiQue-trained · retriever: Contriever · intervention: Augmentation	4.8 percentage points	—	Not assessed

Metric and conditions	Reported	Latest observation	Evidence judgment
predictor: Random	-0.066 coefficient	—	Not assessed
predictor: Random	0.0011 probability	—	Not assessed
predictor: DeBERTa	-0.216 coefficient	—	Not assessed
predictor: DeBERTa	0 probability	—	Not assessed
predictor: BM25 threshold	-0.254 coefficient	—	Not assessed
predictor: BM25 threshold	0 probability	—	Not assessed
predictor: DeBERTa intersection BM25 threshold	-0.282 coefficient	—	Not assessed
predictor: DeBERTa intersection BM25 threshold	0 probability	—	Not assessed
	42.9%	—	Not assessed
flagCategory: Both flag	1,971 hops	—	Not assessed
flagCategory: Both flag	30.8%	—	Not assessed
flagCategory: DeBERTa only	1,230 hops	—	Not assessed

Metric and conditions	Reported	Latest observation	Evidence judgment
dataset: MuSiQue dev · retriever: BM25	598 hops	—	Not assessed
dataset: MuSiQue dev · retriever: Contriever	725 hops	—	Not assessed
	77%	—	Not assessed
dataset: MuSiQue · retriever: Contriever	51.2%	—	Not assessed
dataset: MuSiQue · retriever: BM25	51.9%	—	Not assessed
dataset: HotpotQA · retriever: BM25	3.1%	—	Not assessed
dataset: HotpotQA · retriever: Contriever	4.5%	—	Not assessed