HalluHard: A Hard Multi-Turn Hallucination Benchmark
Hard Multi-TurResearchers introduce HalluHard, a benchmark for evaluating multi-turn hallucination in large language models. It includes 950 seed questions across legal, research, medical, and coding domains. The benchmark requires inline citations for factual claims and uses a judging pipeline with web search to assess content grounding. Results show significant hallucination rates even with web search, highlighting the challenge of ensuring factual accuracy. The study identifies factors like model capacity and knowledge type that influence hallucination behavior.
Show original excerpt (English · first 3 paragraphs)
View PDF
HTML (experimental)
Abstract:Large language models (LLMs) still produce plausible-sounding but ungrounded factual claims, a problem that worsens in multi-turn dialogue as context grows and early errors cascade. We introduce $\textbf{HalluHard}$, a challenging multi-turn hallucination benchmark with 950 seed questions spanning four high-stakes domains: legal cases, research questions, medical guidelines, and coding. We operationalize groundedness by requiring inline citations for factual assertions. To support reliable evaluation in open-ended settings, we propose a judging pipeline that iteratively retrieves evidence via web search. It can fetch, filter, and parse full-text sources (including PDFs) to assess whether cited material actually supports the generated content. Across a diverse set of frontier proprietary and open-weight models, hallucinations remain substantial even with web search ($\approx 30\%$ for the strongest configuration, Opus-4.5 with web search), with content-grounding errors persisting at high rates. Finally, we show that hallucination behavior is shaped by model capacity, turn position, effective reasoning, and the type of knowledge required.
* For copyright reasons we quote only the first 3 paragraphs. Read the full article at the source.