method: IE2021-04-19
Authors: liming
Description: An effective methods for IE
method: Linklogis_BeeAI2021-07-20
Authors: Juntao Zhang, Donglai Chen, Yonghong Chen，Fangchao Ji, Jiayu Chen，Jinyuan He，Zhiqi zhu, Yan Xu
Affiliation: Linklogis_BeeAI Team
Description: Our model is developed based on a multimodal framework that is capable of extracting both layout information and textual semantics from scanned receipts. Following the same evaluation protocol as others, we remove any OCR mismatches and fix any discrepancies found in total amount randomly prefixed by *RM.
method: Applica.ai Lambert 2.0 + Excluding OCR Errors + Fixing total entity2021-01-02
Authors: Applica.ai research team
Affiliation: Applica.ai
Description: Following the same evaluation rules as others, the OCR mismatch errors are excluded in the submission.
Additionally, we have manually fixed annotation discrepancies in "total" entity in the test set.
Note:
1. We submitted the best solution out of 100 fine-tuned models
2. In this task there is an annotation discrepancy in "total" entity which caused unfair comparison between models (In train/test sets "total" entity was randomly prefixed by "RM"). Number of errors in the top solutions caused by this kind of annotation error:
Applica.ai Lambert 2.0 + Excluding OCR Errors + Fixing total entity = 0
LayoutLM 2.0 (single model) = 3 (example: 275)
Applica.ai Lambert 2.0 + Excluding OCR Mismatch = 8 (example: 77)
Tencent Youtu = 8 (example: 120)
VIE = 0
HIK_OCR_Exclude_ocr_mismatch = 0
LayoutLM + Excluding OCR Mismatch = 9 (example: 121)
| Date | Method | Recall | Precision | Hmean | |||
|---|---|---|---|---|---|---|---|
| 2021-04-19 | IE | 97.05% | 99.56% | 98.29% | |||
| 2021-07-20 | Linklogis_BeeAI | 97.05% | 99.34% | 98.18% | |||
| 2021-01-02 | Applica.ai Lambert 2.0 + Excluding OCR Errors + Fixing total entity | 96.83% | 99.56% | 98.17% | |||
| 2021-06-02 | Multimodal Transformer for Information Extraction | 96.76% | 99.56% | 98.14% | |||
| 2021-02-16 | Applica.ai TILT + Excluding OCR Errors + Fixing total entity | 96.83% | 99.41% | 98.10% | |||
| 2020-12-24 | LayoutLM 2.0 (single model) | 96.61% | 99.04% | 97.81% | |||
| 2021-01-01 | Applica.ai Lambert 2.0 + Excluding OCR Mismatch | 96.40% | 99.11% | 97.74% | |||
| 2020-12-07 | Tencent Youtu | 96.47% | 98.89% | 97.67% | |||
| 2020-12-28 | IE method | 96.33% | 98.53% | 97.41% | |||
| 2020-05-07 | HIK_OCR_Exclude_ocr_mismatch | 96.33% | 98.38% | 97.34% | |||
| 2020-04-18 | LayoutLM + Excluding OCR Mismatch | 96.04% | 98.16% | 97.09% | |||
| 2020-04-15 | PICK-PAPCIC & XZMU | 95.46% | 96.79% | 96.12% | |||
| 2020-04-16 | LayoutLM | 96.04% | 96.04% | 96.04% | |||
| 2020-03-26 | Applica.ai roberta-base-2D | 95.39% | 95.80% | 95.60% | |||
| 2020-06-05 | great | 94.24% | 94.24% | 94.24% | |||
| 2019-08-14 | PATech_AICenter | 94.02% | 94.02% | 94.02% | |||
| 2021-06-11 | IE + GCN+ OCR | 92.44% | 94.90% | 93.65% | |||
| 2021-02-21 | RoBERTa-base finetuned on business documents | 92.80% | 93.27% | 93.03% | |||
| 2021-06-13 | GEM AI -OCR TEAM | 92.72% | 92.72% | 92.72% | |||
| 2021-02-21 | RoBERTa-base | 92.22% | 92.55% | 92.39% | |||
| 2020-05-23 | GIE | 91.21% | 93.43% | 92.31% | |||
| 2020-07-07 | Taikang Insurance Group Research Institute | 91.79% | 91.99% | 91.89% | |||
| 2021-06-30 | N | 91.79% | 91.79% | 91.79% | |||
| 2019-08-05 | PATECH_CHENGDU_OCR_V2 | 91.21% | 91.21% | 91.21% | |||
| 2020-02-20 | Character & Word BiLSTM Encoder | 90.85% | 90.85% | 90.85% | |||
| 2019-05-05 | Ping An Property & Casualty Insurance Company | 90.49% | 90.49% | 90.49% | |||
| 2019-04-29 | Enetity detection | 89.70% | 89.70% | 89.70% | |||
| 2019-05-04 | H&H Lab | 89.63% | 89.63% | 89.63% | |||
| 2019-05-02 | CLOVA OCR | 89.05% | 89.05% | 89.05% | |||
| 2021-03-02 | layoutLM with ocr gt | 85.81% | 87.38% | 86.59% | |||
| 2020-12-29 | coldog | 86.17% | 86.17% | 86.17% | |||
| 2019-09-23 | ASTRI-CCT-MSA | 85.45% | 85.45% | 85.45% | |||
| 2019-05-05 | GraphLayout | 85.09% | 85.09% | 85.09% | |||
| 2021-03-02 | Qubitrics | 82.06% | 86.75% | 84.34% | |||
| 2020-06-15 | End-to-end learning with PGN | 83.86% | 83.86% | 83.86% | |||
| 2019-05-04 | HeReceipt-withoutRM | 83.00% | 83.24% | 83.12% | |||
| 2020-06-17 | Graph Neural Net with Bert Embeddings | 82.78% | 82.78% | 82.78% | |||
| 2019-05-06 | BOE_IOT_AIBD | 82.71% | 82.71% | 82.71% | |||
| 2019-05-05 | PATECH_CHENGDU_OCR | 81.70% | 82.29% | 82.00% | |||
| 2020-05-28 | SROIE LSTM - Axel Alejandro Ramos García | 81.99% | 81.99% | 81.99% | |||
| 2020-04-28 | BERT-MRC | 81.05% | 81.05% | 81.05% | |||
| 2021-03-09 | Character Level BiLSTM | 79.25% | 79.25% | 79.25% | |||
| 2020-05-29 | Cool Method Remix | 79.03% | 79.03% | 79.03% | |||
| 2019-04-30 | NER with spaCy model | 78.96% | 79.02% | 78.99% | |||
| 2021-07-05 | test2 | 78.89% | 78.89% | 78.89% | |||
| 2021-05-24 | test | 78.89% | 78.89% | 78.89% | |||
| 2020-12-28 | Custom Named Entity Recognition | 77.59% | 77.59% | 77.59% | |||
| 2019-05-05 | CITlab Argus Information Extraction (positional & line features, enhanced gt) | 77.38% | 77.38% | 77.38% | |||
| 2021-01-02 | lstm deep | 77.38% | 77.38% | 77.38% | |||
| 2021-01-02 | lstm standard method trained 100 epochs constant learning rate | 76.15% | 76.15% | 76.15% | |||
| 2019-04-28 | A Simple Method for Key Information Extraction as Character-wise Classification with LSTM | 75.58% | 75.58% | 75.58% | |||
| 2019-04-30 | Bi-directional LSTM-CNNs-CRF (version2) | 74.86% | 74.86% | 74.86% | |||
| 2019-05-05 | Location-aware BERT model for Text Information Extraction | 74.42% | 74.42% | 74.42% | |||
| 2020-05-23 | test | 73.63% | 73.63% | 73.63% | |||
| 2021-04-07 | Token level multi modal bilstm | 70.03% | 70.03% | 70.03% | |||
| 2021-04-07 | Token level bert embed + bilstm | 66.43% | 66.43% | 66.43% | |||
| 2019-04-30 | BERT with Multi-task Confidence Prediction | 66.14% | 66.14% | 66.14% | |||
| 2019-05-02 | With receipt framing | 63.04% | 63.54% | 63.29% | |||
| 2019-05-05 | IFLYTEK-textNLP_v2 | 61.24% | 61.24% | 61.24% | |||
| 2019-05-05 | SituTech_OCR | 59.01% | 62.38% | 60.64% | |||
| 2019-04-30 | Key Information Extraction from Scanned Receipts | 28.75% | 36.31% | 32.09% |