Hamdan, Mohammed et Cheriet, Mohamed.
2023.
« ResneSt-Transformer: Joint attention segmentation-free for end-to-end handwriting paragraph recognition model ».
Array, vol. 19.
Compte des citations dans Scopus : 3.
Prévisualisation |
PDF
Cheriet-M-2023-27092.pdf - Version publiée Licence d'utilisation : Creative Commons CC BY-NC-ND. Télécharger (1MB) | Prévisualisation |
Résumé
Offline handwritten text recognition (HTR) typically relies on segmented text-line images for training and transcription. However, acquiring line-level position and transcript information can be challenging and time-consuming, while automatic line segmentation algorithms are prone to errors that impede the recognition phase. To address these issues, we introduce a state-of-the-art solution that integrates vision and language models using efficient split and multi-head attention neural networks, referred to as joint attention (ResneSt-Transformer), for end-to-end recognition of handwritten paragraphs. Our proposed novel one-stage, segmentation-free pipeline employs joint attention mechanisms to process paragraph images in an end-to-end trainable manner. This pipeline comprises three modules, with the output of one serving as the input for the next. Initially, a feature extraction module employing a CNN with a split attention mechanism (ResneSt50) is utilized. Subsequently, we develop an encoder module containing four transformer layers to generate robust representations of the entire paragraph image. Lastly, we designed a decoder module with six transformer layers to construct weighted masks. The encoder and decoder modules incorporate a multi-head self-attention mechanism and positional encoding, enabling the model to concentrate on specific feature maps at the current time step. By leveraging joint attention and a segmentation-free approach, our neural network calculates split attention weights on the visual representation, facilitating implicit line segmentation. This strategy signifies a substantial advancement toward achieving end-to-end transcription of entire paragraphs. Experiments conducted on paragraph-level benchmark datasets, including RIMES, IAM, and READ 2016 test datasets, demonstrate competitive results compared to recent paragraph-level models while maintaining reduced complexity. The code and pre-trained models are available on our GitHub repository here: HTTPSlink.
| Type de document: | Article publié dans une revue, révisé par les pairs |
|---|---|
| Professeur: | Professeur Cheriet, Mohamed |
| Affiliation: | Génie des systèmes |
| Date de dépôt: | 26 juill. 2023 17:54 |
| Dernière modification: | 16 oct. 2023 16:16 |
| URI: | https://espace2.etsmtl.ca/id/eprint/27092 |
Actions (Authentification requise)
 |
Dernière vérification avant le dépôt |