Authors: Yu Hao, Chuhui Xue, Wenqing Zhang, Song Bai

Affiliation: ByteDance Inc.

Email: jinyu121@gmail.com

Description: The method we use is TransDETR[1]. First, we get the weights pre-trained on the ICDAR2015 video, then use the RoadText3K and BOVText to fine-tune the network for 20 epochs. Finally, we use the RoadText to fine-tune the network for 20 epoch.

[1] End-to-end Video Text Spotting with Transformer
[2] Read while you drive - multilingual text tracking on the road

Authors: Anthony Sherbondy, Renshen Wang

Affiliation: Google

Email: tonysherbondy@google.com

Description: ClusterFlow is especially designed to address the problem of extracting text from videos as presented in the RoadText1k dataset. The main motivation is to demonstrate the utility of combining commodity algorithms for OCR, optical flow, clustering and classification with decision trees.

First, we use a public cloud API for extracting OCR results at line-level granularity on every image frame (~300) of each video. Next, we use a modern RAFT implementation to calculate a dense optical flow field at the pixel level for every image. The optical flow field is then used to extrude the OCR line results temporally to create tubes or tracklets of lines. Next, an unsupervised clustering algorithm is used to group the line text tracklets into clusters across the entire video. The distance metric between tracklets, clustering algorithm and hyperparameters for the clustering algorithm is searched on the training dataset.

Given the clustered tracklets, the algorithm then selects geometry and text from the tracklet to create tracked lines that have at most a single appearance within any video frame. To do this a set of features are generated from each line appearance, tracklet and cluster and input into a classification algorithm. The classification algorithm is trained to select the appearances of the cluster that would match with groundtruth in the training set. At inference the classification probabilities are used to select amongst possible line text appearances within a cluster at any video frame.

Authors: Gang Yao*, Ning Ding*, Kemeng Zhao, Huan Yu, Pei Tang, Haodong Shi, Liangrui Peng [*equal contribution]

Affiliation: Tsinghua University

Email: dn22@mails.tsinghua.edu.cn

Description: The TH-DL method provides an integrated scheme for text detection, recognition, and tracking in driving videos. For text detection and recognition, TESTR[1] based on Transformer is adopted. The pre-trained TESTR model is finetuned on the training set of the Roadtext Challenge. For multi-object tracking, ByteTrack[2] is employed which uses the similarities with tracklets to recover true objects from low score detection boxes. Post-processing module is added to filter duplicate instances of text detection and recognition.

[1] Zhang X, Su Y, Tripathi S, et al. Text spotting transformers. CVPR, 2022: 9519-9528.

[2] Zhang Y, Sun P, Jiang Y, et al. ByteTrack: Multi-object tracking by associating every detection box. ECCV, 2022, LNCS, vol 13682: 1-21.