VL-DocIR: A Benchmark for Vision-Based
Long Document Information Retrieval

Abstract

Vision-based document retrieval has improved rapidly, but current evaluation settings still emphasize short documents and single relevant pages. This leaves an important gap between leaderboard performance and realistic long-document retrieval, where the evidence needed to answer a query may span multiple pages or even multiple documents. We present VL-DocIR, a page-level benchmark for evidence-complete visual long-document retrieval. VL-DocIR is built from 29,641 HTML-born documents rendered into 388,548 page images from Wikipedia, arXiv, PubMed, and SEC proxy statements. It contains 271,760 queries across 23 domains and six evidence structures, covering single-page, within-document multi-page, and cross-document evidence configurations. Evidence annotations are grounded to rendered pages and HTML element identifiers, and queries are filtered with a cleaning pipeline targeting clarity, correctness, closedness, and the absence of explicit layout references. We evaluate leading single-vector and multi-vector visual retrievers together with diagnostic retrieval settings such as document-context scoring, query splitting, reranking, and round-robin ensembles. The strongest evaluated configuration reaches 69.3 nDCGAll@10 and 80.6 RecallAll@10 under structure-balanced Macro evaluation, but source- and evidence-structure breakdowns show that cross-document retrieval remains substantially harder than single-page retrieval, and that document-context scoring is especially helpful for within-document multi-page queries. VL-DocIR exposes these failure modes and provides a more demanding target for visual long-document retrieval.

Evidence-Complete Evaluation

At evaluation time, a retriever receives only the question and ranks pages from the full corpus. For a query q, let E(q) be its ground-truth evidence-page set and R_k(q) the retrieved top-k page set. Completion requires every page in E(q) to appear in R_k(q).

nDCG uses binary relevance for evidence pages. We use structure-balanced Macro for the main comparison: we average query scores within each evidence structure, then give the six means equal weight. Micro is the mean over all queries.

Annotation Validation

We manually reviewed 800 entries across all 24 source × evidence-structure combinations, with 50 Cross-Doc entries and 30 entries from each other structure per source. 746/800 (93.25%) were fully correct (nominal 95% Wilson interval: 91.3–94.8%). Cross-Doc correctness was 88.5%.

The 54 entries with issues include 13 query-formulation errors, 20 incorrect evidence sets, 17 sets with an unnecessary evidence page, and four queries without directly retrievable positive evidence. These rates are measured before adjudication. The full audit is available in the dataset card.

@inproceedings{ott2026vldocir,
  title     = {VL-DocIR: A Benchmark for Vision-Based Long Document Information Retrieval},
  author    = {Ott, Simon and Chen, Yufan and Liu, Ruiping and Zheng, Junwei and
               Wei, Jiale and Wen, Di and Peng, Kunyu and Zhang, Jiaming and
               Stiefelhagen, Rainer},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  eprint    = {2610.17057},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url       = {https://arxiv.org/abs/2610.17057}
}

Download citation · Paper on arXiv · OpenReview