Results - Document Visual Question Answering

method: Human Performance2020-06-13

Authors: Task1 Organizers

Affiliation: CVIT, IIIT Hyderabad, CVC-UAB, Amazon

Description: Human performance on the test set.
A small group of volunteers were asked to enter an answer for the given question and the image.

@InProceedings{docvqa_wacv, author = {Mathew, Minesh and Karatzas, Dimosthenis and Jawahar, C.V.}, title = {DocVQA: A Dataset for VQA on Document Images}, booktitle = {WACV}, year = {2021}, pages = {2200-2209} }

method: ERNIE-Layout-v22022-01-13

Authors: ERNIE-Layout

Description: 1. A knowledge enhanced pre-training approach for visual document understanding.
2. Ensemble with 30 models.

method: Mybank-DocReader2021-11-26

Authors: Members

Affiliation: Mybank

Description: 1. We train a OCR model to optimize blurred images and handwriting. we use LayoutReader to reorder bounding box.
2. We pretrain a discrete 2d-position embedding model with question generation and span mask, and finetune it to predict the start and end positions of certain questions.

Ranking Table

Description Paper Source Code

Date	Method	Score	Figure/Diagram	Form	Table/List	Layout	Free_text	Image/Photo	Handwritten	Yes/No	Others
2020-06-13	Human Performance	0.9811	0.9756	0.9825	0.9780	0.9845	0.9839	0.9740	0.9717	0.9974	0.9828
2022-01-13	ERNIE-Layout-v2	0.8841	0.6434	0.9177	0.8996	0.8899	0.9010	0.6223	0.7836	0.6124	0.8118
2021-11-26	Mybank-DocReader	0.8755	0.6682	0.9233	0.8763	0.8896	0.8713	0.6290	0.8047	0.5805	0.7804
2021-09-06	ERNIE-Layout	0.8753	0.6586	0.8972	0.8864	0.8902	0.8943	0.6392	0.7331	0.5434	0.8115
2021-02-12	Applica.ai TILT	0.8705	0.6082	0.9459	0.8980	0.8592	0.8581	0.5508	0.8139	0.6897	0.7788
2020-12-22	LayoutLM 2.0 (single model)	0.8672	0.6574	0.8953	0.8769	0.8791	0.8707	0.7287	0.6729	0.5517	0.8103
2020-08-16	Alibaba DAMO NLP	0.8506	0.6650	0.8809	0.8552	0.8733	0.8397	0.6758	0.7691	0.5492	0.7526
2020-05-16	PingAn-OneConnect-Gammalab-DQA	0.8484	0.6059	0.9021	0.8463	0.8730	0.8337	0.5812	0.7692	0.5172	0.7289
2022-01-07	LayoutLMV2-large on Textract	0.7873	0.4924	0.8771	0.8218	0.7726	0.7661	0.4820	0.7276	0.3793	0.6983
2020-05-14	Structural LM-v2	0.7674	0.4931	0.8381	0.7621	0.7924	0.7596	0.4756	0.6282	0.5517	0.6549
2020-05-15	QA_Base_MRC_2	0.7415	0.4854	0.8015	0.6738	0.7943	0.8136	0.5740	0.5831	0.5287	0.7161
2020-05-15	QA_Base_MRC_1	0.7407	0.4890	0.7984	0.6675	0.7936	0.8131	0.5854	0.6099	0.4943	0.7384
2020-05-15	QA_Base_MRC_4	0.7348	0.4735	0.8040	0.6647	0.7838	0.8043	0.5618	0.5810	0.4598	0.7332
2020-05-15	QA_Base_MRC_3	0.7322	0.4852	0.7958	0.6562	0.7842	0.8044	0.5679	0.5730	0.4511	0.7171
2020-05-15	QA_Base_MRC_5	0.7274	0.4858	0.7877	0.6550	0.7754	0.8047	0.5405	0.5619	0.4598	0.7084
2021-12-05	Electra Large Squad	0.6961	0.4485	0.7703	0.6348	0.7364	0.7644	0.4594	0.5438	0.5172	0.6470
2020-05-16	HyperDQA_V4	0.6893	0.3874	0.7792	0.6309	0.7478	0.7187	0.4867	0.5630	0.4138	0.5685
2020-05-16	HyperDQA_V3	0.6769	0.3876	0.7774	0.6167	0.7332	0.6961	0.4296	0.5373	0.4138	0.5650
2020-05-16	HyperDQA_V2	0.6734	0.3818	0.7666	0.6110	0.7332	0.6867	0.4834	0.5560	0.3793	0.5902
2020-05-09	HyperDQA_V1	0.6717	0.4013	0.7693	0.6197	0.7167	0.6922	0.3598	0.5596	0.4138	0.5504
2021-12-04	Bert Large	0.6447	0.3502	0.7535	0.5488	0.6920	0.7266	0.4171	0.5254	0.5517	0.6076
2020-05-09	bert fulldata fintuned	0.5900	0.4169	0.6870	0.4269	0.6710	0.7315	0.5124	0.4900	0.4483	0.5907
2020-05-01	bert finetuned	0.5872	0.2986	0.7011	0.4849	0.6359	0.6933	0.4622	0.4751	0.4483	0.4895
2020-04-30	HyperDQA_V0	0.5715	0.3131	0.6780	0.4732	0.6630	0.5716	0.3623	0.4351	0.3793	0.4941
2020-04-27	bert	0.4557	0.2233	0.5259	0.2633	0.5113	0.7775	0.4859	0.3565	0.0345	0.5778
2020-05-16	UGLIFT v0.1 (Clova OCR)	0.4417	0.1766	0.5600	0.3178	0.5340	0.4520	0.2253	0.3573	0.4483	0.3356
2020-05-14	Plain BERT QA	0.3524	0.1687	0.4489	0.2029	0.4321	0.4812	0.3517	0.3096	0.0345	0.3747
2020-05-16	Clova OCR V0	0.3489	0.0977	0.4855	0.2670	0.3811	0.3958	0.2489	0.2875	0.0345	0.3062
2020-05-01	HDNet	0.3401	0.2040	0.4688	0.2181	0.4710	0.1916	0.2488	0.2736	0.1379	0.2458
2020-05-16	CLOVA OCR	0.3296	0.1246	0.4612	0.2455	0.3622	0.3746	0.1692	0.2736	0.0690	0.3205
2020-04-29	docVQAQV_V0.1	0.3016	0.2010	0.3898	0.3810	0.2933	0.0664	0.1842	0.2736	0.1586	0.1695
2020-04-26	docVQAQV_V0	0.2342	0.1646	0.3133	0.2623	0.2483	0.0549	0.2277	0.1856	0.1034	0.1635
2021-02-08	seq2seq	0.1081	0.0758	0.1283	0.0829	0.1332	0.0822	0.0786	0.0779	0.4828	0.1052
2020-06-16	Test Submission	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000

Inactive evaluations

method: Human Performance2020-06-13

method: ERNIE-Layout-v22022-01-13

method: Mybank-DocReader2021-11-26

Ranking Table

Ranking Graphic

Ranking Graphic