Authors: Applica.ai research team

Affiliation: Applica.ai

Description: Following the same evaluation rules as others, the OCR mismatch errors are excluded in the submission.

Additionally, we have manually fixed annotation discrepancies in "total" entity in the test set.

Note:
1. We submitted the best solution out of 100 fine-tuned models
2. In this task there is an annotation discrepancy in "total" entity which caused unfair comparison between models (In train/test sets "total" entity was randomly prefixed by "RM"). Number of errors in the top solutions caused by this kind of annotation error:

Applica.ai Lambert 2.0 + Excluding OCR Errors + Fixing total entity = 0
LayoutLM 2.0 (single model) = 3 (example: 275)
Applica.ai Lambert 2.0 + Excluding OCR Mismatch = 8 (example: 77)
Tencent Youtu = 8 (example: 120)
VIE = 0
HIK_OCR_Exclude_ocr_mismatch = 0
LayoutLM + Excluding OCR Mismatch = 9 (example: 121)

Authors: Applica.ai Research Team

Affiliation: Applica.ai

Description: TILT neural network architecture which simultaneously learns layout information, visual features, and textual semantics. Contrary to previous approaches, we rely on a encoder-decoder architecture. We submitted results from a single model. Following the same evaluation procedure as others we removed OCR mismatches and fixed discrepancies of total amount randomly prefixed by "RM".

method: LayoutLM 2.0 (single model)2020-12-24

Authors: LayoutLM Team

Affiliation: LayoutLM Team

Description: Multi-modal Pre-training for Visually-Rich Document Understanding

Ranking Table

Description Paper Source Code
DateMethodRecallPrecisionHmean
2021-01-02Applica.ai Lambert 2.0 + Excluding OCR Errors + Fixing total entity96.83%99.56%98.17%
2021-02-16Applica.ai TILT + Excluding OCR Errors + Fixing total entity96.83%99.41%98.10%
2020-12-24LayoutLM 2.0 (single model)96.61%99.04%97.81%
2021-01-01Applica.ai Lambert 2.0 + Excluding OCR Mismatch96.40%99.11%97.74%
2020-04-15PICK-PAPCIC & XZMU95.46%96.79%96.12%
2021-02-21RoBERTa-base92.22%92.55%92.39%
2019-05-04H&H Lab89.63%89.63%89.63%
2019-05-02CLOVA OCR89.05%89.05%89.05%
2019-05-06BOE_IOT_AIBD82.71%82.71%82.71%
2019-04-28A Simple Method for Key Information Extraction as Character-wise Classification with LSTM75.58%75.58%75.58%
2019-05-02With receipt framing63.04%63.54%63.29%

Ranking Graphic