When and How Should an Agent Clarify? CIGAsk: Teaching LLMs to Clarify via Counterfactual Information Gain

Sensitivity to the ambiguity-bonus weight is non-monotonic on PACIFIC: among rho values 0.1, 0.3, and 0.5, rho=0.3 has the highest overall F1, ambiguous-query clarification rate, and reported selectivity. Raising rho to 0.5 reduces F1 and recall and does not improve selectivity over the main setting.

On unseen closed-book TriviaQA and NQ Open subsets, forced-direct CIGAsk 7B slightly exceeds the Qwen2.5-7B base F1, while under a multi-action prompt it clarifies only 5.6% and 9.4% of queries, respectively.

Replacing GPT-4o with Qwen2.5-7B-Instruct as the fixed latent-intent-aware simulator leaves PACIFIC performance close to the main setting and slightly raises selectivity, whereas Qwen2.5-3B-Instruct substantially lowers post-clarification F1 and ambiguous-query recall.

In the PACIFIC deferred-tax-asset case, a generic prompted clarification elicits only a restatement and produces the incorrect aggregate 131,028. CIGAsk asks which period is intended, receives 'as of December 31, 2019,' and returns the gold answer 73,260 in one clarification turn.

On PACIFIC fullval, CIGAsk 7B reports the best overall F1, post-clarification F1, and ambiguous-query clarification rate in Table 2; CIGAsk 3B also exceeds all prompting and SFT baselines. Scaling CIGAsk from 3B to 7B raises F1, post-clarification F1, and ambiguous-query recall while also raising the clear-query false-positive rate from 0.179 to 0.260. Prompting and SFT expose negative behaviors: Direct and SFT never clarify, FATA clarifies every input and has low F1, and ReAct clarifies more often on clear than ambiguous inputs.

The PACIFIC 7B component ablation separates the reward roles: retaining CIG but removing the asymmetric bonus yields useful post-clarification answers but low clarification recall; retaining the asymmetric bonus but removing CIG largely preserves recall while reducing post-clarification F1. Removing both performs worst overall among the four variants.

Replacing the frozen CIG reference with the current actor while holding other settings fixed substantially reduces overall F1, post-clarification F1, ambiguous-query clarification recall, and selectivity on PACIFIC. The current-actor variant's mean training reward changes only modestly from early to late training.

When the same recipe is retrained separately on passage-grounded AbgCoQA without per-dataset hyperparameter search, CIGAsk 7B reaches F1 0.724, above the 3B variant and all reported prompting and SFT baselines in Table 2.

On the Zephyr-7B backbone, CIGAsk reports higher overall and post-clarification F1 than ACT and additionally reports a 0.712 ambiguous-query clarification rate with a 0.044 clear-query clarification rate.

Across late-training checkpoints, PACIFIC items that the policy chose to clarify had higher F1 than items answered directly, with bootstrap 95% confidence intervals for the difference above zero throughout.

On the small OOD subsets where CIGAsk chooses to clarify, the appendix describes targeted questions about lexical ambiguity, year, temporal scope, or episode identity rather than generic requests for context, indicating qualitative transfer from table-grounded to text-grounded QA without retraining.

In a matched-seed 3B comparison, CIGAsk and outcome-only GRPO learned to clarify at similar rates, but only the CIG-equipped model consistently converted those turns into improved post-clarification F1; the outcome-only variant fell to near zero.

The three clarification benchmarks differ substantially in size and ambiguity prevalence: PACIFIC is table-grounded financial QA, AbgCoQA is passage-grounded conversational QA, and AmbigNQ is ambiguity-heavy open-domain QA.

In a manual audit of eight ambiguous PACIFIC cases, Qwen2.5-7B-Instruct returned an intent-consistent response in every case, whereas Qwen2.5-3B-Instruct did so in only one case; all three examples printed in Table 11 show the 7B reply matching the latent intent and the 3B reply contradicting it.

For CIGAsk 7B, both the mean CIG on clarification turns and the batch-weighted CIG rise over training; the paper reports that this rise coincides with the emergence of the ambiguous-recall versus clear-query false-positive-rate gap.

On the Gemma-2-9B backbone, CIGAsk reports higher F1 than deployable SGP and much higher ambiguous-query clarification with lower clear-query over-clarification, while its F1 remains below the gold-label SGP-Oracle upper bound.

On the ambiguity-heavy AmbigNQ subset, CIGAsk 7B reports EM 0.511, substantially above the prompting and SFT baselines and the CIGAsk 3B variant; it also exceeds the contextual SGP and SGP-Oracle AmbigQA values placed in this column by the paper.

Removing the asymmetric bonus roughly doubles the failure rate among generated clarification turns. The dominant failures are self-completion, where the policy supplies a question and plausible answer without yielding to the simulator, and declarative reformulation, which does not ask an actionable question.