Description

Understanding visually-rich business documents to extract structured data and automate business workflows has been receiving attention both in academia and industry. Although recent multimodal language models have achieved impressive results, we find that existing benchmarks do not reflect the complexity of real documents seen in industry. We identify the desiderata for a more comprehensive benchmark and propose one we call Visually Rich Document Understanding (VRDU).

VRDU contains two datasets that represent several challenges: rich schema including diverse data types as well as nested entities, complex templates including tables and multi-column layouts, and diversity of different layouts (templates) within a single document type. We design few-shot and conventional experiment settings along with a carefully designed matching algorithm to evaluate extraction results. We hope this helps the community make progress on these challenging tasks in extracting structured data from visually rich documents.