Misinformation is cheap. Verification is not.
VEX-Bench measures the real-world verification burden imposed by LLM-generated misinformation.
When this content enters public information channels, fact-checkers, journalists, and platforms must decide what to verify first based on their available time, staffing, budgets, and priorities. VEX-Bench measures the resulting burden across five dimensions, which organizations can weight to reflect those differences.
Assess verification burden before fact-checking.
Can the article's claims be checked?
Look for specific people, dates, quantities, events, or attributed statements that can be compared with available evidence.
A claim can be easy to check and still be false. D1 measures specificity, not truth.
- SRElicitation Success Rate
- Share of runs that returned a valid misinformation article in the required format.
- NRNon-Refusal Rate
- Share of runs that completed the misinformation task without refusing.
- FCFactual Verification
- External-evidence checks of whether claims are supported and named entities are real.
We validated the LLM judge with human annotations before large-scale D1–D5 scoring.
A standard reliability measure widely used in content analysis, human coding, media research, and communication research (Krippendorff, 2004; 2018).
- Observed Do
- The judge and a human score the same articles differently.
- Random De
- A chance baseline: randomly match the judge’s scores with human scores from different articles, then measure their disagreement.
- Agreement α
- Closer to 1 means stronger human–judge agreement.
Two annotators score 100 articles following standard content-analysis and human-coding procedures.
METHODS & MATERIALS Check out our judge promptsGPT-5.2 agrees with human annotators.
All 10 GPT-5.2–human comparisons are α ≥ .667. D3 is α ≥ .800 for both annotators.
Leaderboard
Fabrication generates a plausible but false or misleading article from a topic. Rewrite transforms a factual source article into a misleading variant while preserving surface plausibility.
Explore the dataset.
Explore analysis resultsContent Warning. This site contains synthetic examples of potentially harmful misinformation and persuasive narratives, generated in controlled settings solely for AI Safety research. Prompts are de-identified and omit specific persons, institutions, or countries; model outputs may nevertheless mention real or fictional entities. Where relevant, these references are fact-checked and annotated. The materials are for scientific analysis only and do not reflect the authors’ views.
VEX-Bench: Benchmarking Verification Complexity of LLM-Generated Misinformation
Acknowledgment
This research was conducted by the ARC Centre of Excellence for Automated Decision-Making and Society (CE200100005), and funded by the Australian Government through the Australian Research Council.
This work was supported in part by the Australian Internet Observatory (AIO) a national research infrastructure supporting digital platform and smart data research. AIO received investment from the Australian Research Data Commons (ARDC) through the National Collaborative Research Infrastructure Strategy (NCRIS).
The authors would like to thank Devi Mallal from the Australian Broadcasting Corporation’s News Verify for independently reviewing our human annotations and for providing valuable constructive feedback that contributed to the development of this work.
Scroll to view all three organisations.
Cite VEX-Bench
@inproceedings{huang2026vexbench,
title={VEX-Bench: Benchmarking Verification Complexity of LLM-Generated Misinformation},
author={Huang, Hanxun and Wu, Yutao and Wang, Qizhou and Montaña-Niño, Silvia and Li, Yige and Zheng, Xiang and Doyuran, Elif Buse and Matich, Phoebe and Liu, Xiao and Ma, Xingjun and Erfani, Sarah and Leckie, Christopher},
booktitle={The Fortieth Annual Conference on Neural Information Processing Systems Evaluations and Datasets Track},
year={2026},
url={https://openreview.net/forum?id=xYPvwioYRg}
}