ResneSt-Transformer: Joint attention segmentation-free for end-to-end handwriting paragraph recognition model

Hamdan, Mohammed et Cheriet, Mohamed. 2023. « ResneSt-Transformer: Joint attention segmentation-free for end-to-end handwriting paragraph recognition model ». Array, vol. 19.
Compte des citations dans Scopus : 1.

Prévisualisation

PDF Cheriet-M-2023-27092.pdf - Version publiée Licence d'utilisation : Creative Commons CC BY-NC-ND. Télécharger (1MB) | Prévisualisation

Résumé

Offline handwritten text recognition (HTR) typically relies on segmented text-line images for training and transcription. However, acquiring line-level position and transcript information can be challenging and time-consuming, while automatic line segmentation algorithms are prone to errors that impede the recognition phase. To address these issues, we introduce a state-of-the-art solution that integrates vision and language models using efficient split and multi-head attention neural networks, referred to as joint attention (ResneSt-Transformer), for end-to-end recognition of handwritten paragraphs. Our proposed novel one-stage, segmentation-free pipeline employs joint attention mechanisms to process paragraph images in an end-to-end trainable manner. This pipeline comprises three modules, with the output of one serving as the input for the next. Initially, a feature extraction module employing a CNN with a split attention mechanism (ResneSt50) is utilized. Subsequently, we develop an encoder module containing four transformer layers to generate robust representations of the entire paragraph image. Lastly, we designed a decoder module with six transformer layers to construct weighted masks. The encoder and decoder modules incorporate a multi-head self-attention mechanism and positional encoding, enabling the model to concentrate on specific feature maps at the current time step. By leveraging joint attention and a segmentation-free approach, our neural network calculates split attention weights on the visual representation, facilitating implicit line segmentation. This strategy signifies a substantial advancement toward achieving end-to-end transcription of entire paragraphs. Experiments conducted on paragraph-level benchmark datasets, including RIMES, IAM, and READ 2016 test datasets, demonstrate competitive results compared to recent paragraph-level models while maintaining reduced complexity. The code and pre-trained models are available on our GitHub repository here: HTTPSlink.

Type de document:	Article publié dans une revue, révisé par les pairs
Professeur:	Professeur Cheriet, Mohamed
Affiliation:	Génie des systèmes
Date de dépôt:	26 juill. 2023 17:54
Dernière modification:	16 oct. 2023 16:16
URI:	https://espace2.etsmtl.ca/id/eprint/27092

Actions (Authentification requise)

Dernière vérification avant le dépôt