Results - Document Visual Question Answering

method: HUMAN2020-06-13

Authors: Task1 Organizers

Affiliation: CVIT, IIIT Hyderabad, CVC-UAB, Amazon

Email: minesh.mathew@research.iiit.ac.in

Description: Human performance on the test set.
A small group of volunteers were asked to enter an answer for the given question and the image.

@misc{mathew2020docvqa, title={DocVQA: A Dataset for VQA on Document Images}, author={Minesh Mathew and Dimosthenis Karatzas and R. Manmatha and C. V. Jawahar}, year={2020}, eprint={2007.00398}, archivePrefix={arXiv}, primaryClass={cs.CV} }

method: PingAn-OneConnect-Gammalab-DQA2020-05-16

Authors: Han Qiu, Guoqiang Xu, Chenjie Cao, Chao Gao, Dexun Wang, Fengxin Yang, Xiao Xie, Yu Qiu, Ziqi Zheng

Affiliation: PingAn OneConnect GammaLab

Description: 1. We train a DB model to detect word-level bounding boxes and then use line-level boxes to sort them.
2. We train a text recognition model with four stages(TPS-ResNet-BiLSTM-Attention).
3. We pretrain a discrete 2d-position embedding model (DEModel-large) with question generation and span mask, and finetune it to predict the start and end positions of certain questions.
4. Our cycled splitting and merging algorithm with K-means filter out the irrelevant answer boxes effectively. Besides, we use post-processing (spell check) and data augmentation to further improve the performance.

method: Structural LM-v22020-05-14

Authors: Structural LM Team

Description: 1. Pre-train Structural LM based on StructBERT
2. Use the official OCR recognition result directly

Ranking Table

Description Paper Source Code

Date	Method	Score	Figure/Diagram	Form	Table/List	Layout	Free_text	Image/Photo	Handwritten	Yes/No	Others
2020-06-13	HUMAN	0.9811	0.9756	0.9825	0.9780	0.9845	0.9839	0.9740	0.9717	0.9974	0.9828
2020-05-16	PingAn-OneConnect-Gammalab-DQA	0.8484	0.6059	0.9021	0.8463	0.8730	0.8337	0.5812	0.7692	0.5172	0.7289
2020-05-14	Structural LM-v2	0.7674	0.4931	0.8381	0.7621	0.7924	0.7596	0.4756	0.6282	0.5517	0.6549
2020-05-15	QA_Base_MRC_2	0.7415	0.4854	0.8015	0.6738	0.7943	0.8136	0.5740	0.5831	0.5287	0.7161
2020-05-15	QA_Base_MRC_1	0.7407	0.4890	0.7984	0.6675	0.7936	0.8131	0.5854	0.6099	0.4943	0.7384
2020-05-15	QA_Base_MRC_4	0.7348	0.4735	0.8040	0.6647	0.7838	0.8043	0.5618	0.5810	0.4598	0.7332
2020-05-15	QA_Base_MRC_3	0.7322	0.4852	0.7958	0.6562	0.7842	0.8044	0.5679	0.5730	0.4511	0.7171
2020-05-15	QA_Base_MRC_5	0.7274	0.4858	0.7877	0.6550	0.7754	0.8047	0.5405	0.5619	0.4598	0.7084
2020-05-16	HyperDQA_V4	0.6893	0.3874	0.7792	0.6309	0.7478	0.7187	0.4867	0.5630	0.4138	0.5685
2020-05-16	HyperDQA_V3	0.6769	0.3876	0.7774	0.6167	0.7332	0.6961	0.4296	0.5373	0.4138	0.5650
2020-05-16	HyperDQA_V2	0.6734	0.3818	0.7666	0.6110	0.7332	0.6867	0.4834	0.5560	0.3793	0.5902
2020-05-09	HyperDQA_V1	0.6717	0.4013	0.7693	0.6197	0.7167	0.6922	0.3598	0.5596	0.4138	0.5504
2020-05-09	bert fulldata fintuned	0.5900	0.4169	0.6870	0.4269	0.6710	0.7315	0.5124	0.4900	0.4483	0.5907
2020-05-01	bert finetuned	0.5872	0.2986	0.7011	0.4849	0.6359	0.6933	0.4622	0.4751	0.4483	0.4895
2020-04-30	HyperDQA_V0	0.5715	0.3131	0.6780	0.4732	0.6630	0.5716	0.3623	0.4351	0.3793	0.4941
2020-04-27	bert	0.4557	0.2233	0.5259	0.2633	0.5113	0.7775	0.4859	0.3565	0.0345	0.5778
2020-05-16	UGLIFT v0.1 (Clova OCR)	0.4417	0.1766	0.5600	0.3178	0.5340	0.4520	0.2253	0.3573	0.4483	0.3356
2020-05-14	Plain BERT QA	0.3524	0.1687	0.4489	0.2029	0.4321	0.4812	0.3517	0.3096	0.0345	0.3747
2020-05-16	Clova OCR V0	0.3489	0.0977	0.4855	0.2670	0.3811	0.3958	0.2489	0.2875	0.0345	0.3062
2020-05-01	HDNet	0.3401	0.2040	0.4688	0.2181	0.4710	0.1916	0.2488	0.2736	0.1379	0.2458
2020-05-16	CLOVA OCR	0.3296	0.1246	0.4612	0.2455	0.3622	0.3746	0.1692	0.2736	0.0690	0.3205
2020-04-29	docVQAQV_V0.1	0.3016	0.2010	0.3898	0.3810	0.2933	0.0664	0.1842	0.2736	0.1586	0.1695
2020-04-26	docVQAQV_V0	0.2342	0.1646	0.3133	0.2623	0.2483	0.0549	0.2277	0.1856	0.1034	0.1635
2020-06-16	Test Submission	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000

Inactive evaluations

method: HUMAN2020-06-13

method: PingAn-OneConnect-Gammalab-DQA2020-05-16

method: Structural LM-v22020-05-14

Ranking Table

Ranking Graphic