Results - Document Visual Question Answering

method: InternVL2-Pro (generalist)2024-06-30

Authors: InternVL team

Affiliation: Shanghai AI Laboratory & Sensetime & Tsinghua University

Description: InternVL Family: Closing the Gap to Commercial Multimodal Models with Open-Source Suites —— A Pioneering Open-Source Alternative to GPT-4V

Demo: https://internvl.opengvlab.com/
Code: https://github.com/OpenGVLab/InternVL
Model: https://huggingface.co/OpenGVLab/InternVL-Chat-V1-5

@article{chen2023internvl, title={Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks}, author={Chen, Zhe and Wu, Jiannan and Wang, Wenhai and Su, Weijie and Chen, Guo and Xing, Sen and Muyan, Zhong and Zhang, Qinglong and Zhu, Xizhou and Lu, Lewei and others}, journal={arXiv preprint arXiv:2312.14238}, year={2023} }

@article{chen2024far, title={How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites}, author={Chen, Zhe and Wang, Weiyun and Tian, Hao and Ye, Shenglong and Gao, Zhangwei and Cui, Erfei and Tong, Wenwen and Hu, Kongzhi and Luo, Jiapeng and Ma, Zheng and others}, journal={arXiv preprint arXiv:2404.16821}, year={2024} }

Source code

method: Molmo-72B2024-09-25

Authors: Molmo Team

Affiliation: Allen Institute for Artificial Intelligence

Description: The 72B member of the Molmo family of open vision-language models developed by the Allen Institute for AI. Molmo models are trained on PixMo, a dataset of 1 million, highly-curated image-text pairs and have open source weights, training data, and training recipe.

@article{Deitke2024molmo, title={Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models}, author={Matt Deitke and Christopher Clark and Sangho Lee and Rohun Tripathi and Yue Yang and James Park and Reza Salehi and Niklas Muennighoff and Kyle Lo and Luca Soldaini and Jiasen Lu and Taira Anderson and Erin Bransom and Kiana Ehsani and Huong Ngo and YenSung Chen and Ajay Patel and Mark Yatskar and Chris Callison-Burch and Andrew Head and Rose Hendrix and Favyen Bastani and Eli VanderBilt and Nathan Lambert and Yvonne Chou and Arnavi Chheda and Jenna Sparks and Sam Skjonsberg and Michael Schmitz and Aaron Sarnat and Byron Bischoff and Pete Walsh and Chris Newell and Piper Wolters and Tanmay Gupta and Jon Borchardt and Dirk Groeneveld and Jen Dumas and Crystal Nam and Sophie Lebrecht and Caitlin Wittlif and Carissa Schoenick and Oscar Michel and Ranjay Krishna and Luca Weihs and Noah Smith and Hannaneh Hajishirzi and Ross Girshick and Ali Farhadi and Aniruddha Kembhavi}, year={2024}, url={https://molmo.allenai.org/paper.pdf} }

Source code

method: InternVL-1.5-Plus (generalist)2024-04-27

Authors: InternVL team

Affiliation: Shanghai AI Laboratory & Sensetime & Tsinghua University

Email: czcz94cz@gmail.com

Source code

Ranking Table

Description Paper Source Code

			Answer type				Evidence					Operation
Date	Method	Score	Image span	Question span	Multiple spans	Non span	Table/List	Textual	Visual object	Figure	Map	Comparison	Arithmetic	Counting
2024-06-30	InternVL2-Pro (generalist)	0.8334	0.8681	0.8929	0.7350	0.6969	0.8335	0.9260	0.7757	0.8093	0.7186	0.7301	0.8584	0.5368
2024-09-25	Molmo-72B	0.8186	0.8513	0.8827	0.6821	0.7041	0.8184	0.9136	0.8062	0.7945	0.6960	0.7054	0.8188	0.5930
2024-04-27	InternVL-1.5-Plus (generalist)	0.7574	0.7989	0.8124	0.6425	0.5987	0.7544	0.8733	0.7306	0.7234	0.6216	0.6065	0.7386	0.4623
2024-01-24	qwenvl-max (single generalist model)	0.7341	0.7756	0.8083	0.6035	0.5717	0.7291	0.8856	0.6708	0.6892	0.5967	0.6009	0.7152	0.4388
2024-05-31	GPT-4 Vision Turbo + Amazon Textract OCR	0.7191	0.7575	0.7795	0.6591	0.5553	0.7183	0.8201	0.6696	0.6904	0.6926	0.5815	0.6759	0.4281
2024-11-01	MLCD-Embodied-7B: Multi-label Cluster Discrimination for Visual Representation Learning	0.6998	0.7330	0.7930	0.5955	0.5564	0.6951	0.8271	0.6654	0.6614	0.5495	0.5523	0.6350	0.4905
2024-04-02	InternLM-XComposer2-4KHD-7B	0.6855	0.7336	0.7570	0.5151	0.5124	0.6643	0.8240	0.6598	0.6471	0.5241	0.5120	0.6636	0.3610
2023-11-15	SMoLA-PaLI-X Specialist Model	0.6621	0.7166	0.7252	0.5838	0.4292	0.6448	0.8261	0.6714	0.6110	0.5065	0.5238	0.5054	0.3506
2024-02-10	ScreenAI 5B	0.6590	0.7162	0.7247	0.5734	0.4140	0.6525	0.8315	0.5968	0.6020	0.4467	0.4815	0.5303	0.3000
2021-04-11	Applica.ai TILT	0.6120	0.6765	0.6419	0.4391	0.3832	0.5917	0.7916	0.4545	0.5654	0.4480	0.4801	0.4958	0.2652
2024-07-22	Snowflake Arctic-TILT 0.8B	0.5695	0.6274	0.6074	0.4123	0.3653	0.5478	0.7530	0.4204	0.5109	0.4410	0.4350	0.5042	0.2238
2023-08-20	PaLI-X (Google Research, Single Generative Model)	0.5477	0.5940	0.6950	0.4122	0.3534	0.5145	0.6891	0.6373	0.5040	0.4013	0.4290	0.4053	0.3091
2021-04-09	IG-BERT (single model)	0.3854	0.4181	0.4481	0.2197	0.2849	0.3373	0.5016	0.3013	0.3706	0.3347	0.2939	0.3564	0.2000
2022-09-18	pix2struct-base	0.3820	0.4145	0.4381	0.1655	0.3014	0.3351	0.4971	0.2380	0.3632	0.3257	0.2344	0.4036	0.1888
2021-04-11	NAVER CLOVA	0.3219	0.3996	0.2317	0.1064	0.1068	0.2653	0.4488	0.1878	0.3095	0.3231	0.2020	0.1480	0.0695
2021-04-10	Ensemble LM and VLM	0.2853	0.3337	0.4181	0.0748	0.1169	0.2439	0.3649	0.2331	0.2645	0.2845	0.2580	0.1628	0.0647
2022-03-03	InfographicVQA paper model	0.2720	0.3278	0.2386	0.0450	0.1371	0.2400	0.3626	0.1705	0.2551	0.2205	0.1836	0.1559	0.1140
2021-04-05	BERT fuzzy search	0.2078	0.2625	0.2333	0.0739	0.0259	0.1852	0.2995	0.0896	0.1942	0.1709	0.1805	0.0160	0.0436
2021-04-10	BERT	0.1678	0.2149	0.2117	0.0126	0.0152	0.1479	0.2450	0.1054	0.1505	0.1768	0.1578	0.0158	0.0185

Inactive evaluations

method: InternVL2-Pro (generalist)2024-06-30

method: Molmo-72B2024-09-25

method: InternVL-1.5-Plus (generalist)2024-04-27

Ranking Table

Ranking Graphic