A showcase of ÉTS researchers’ publications and other contributions

ResneSt-Transformer: Joint attention segmentation-free for end-to-end handwriting paragraph recognition model


Downloads per month over past year

Hamdan, Mohammed and Cheriet, Mohamed. 2023. « ResneSt-Transformer: Joint attention segmentation-free for end-to-end handwriting paragraph recognition model ». Array, vol. 19.
Compte des citations dans Scopus : 1.

[thumbnail of Cheriet-M-2023-27092.pdf]
Cheriet-M-2023-27092.pdf - Published Version
Use licence: Creative Commons CC BY-NC-ND.

Download (1MB) | Preview


Offline handwritten text recognition (HTR) typically relies on segmented text-line images for training and transcription. However, acquiring line-level position and transcript information can be challenging and time-consuming, while automatic line segmentation algorithms are prone to errors that impede the recognition phase. To address these issues, we introduce a state-of-the-art solution that integrates vision and language models using efficient split and multi-head attention neural networks, referred to as joint attention (ResneSt-Transformer), for end-to-end recognition of handwritten paragraphs. Our proposed novel one-stage, segmentation-free pipeline employs joint attention mechanisms to process paragraph images in an end-to-end trainable manner. This pipeline comprises three modules, with the output of one serving as the input for the next. Initially, a feature extraction module employing a CNN with a split attention mechanism (ResneSt50) is utilized. Subsequently, we develop an encoder module containing four transformer layers to generate robust representations of the entire paragraph image. Lastly, we designed a decoder module with six transformer layers to construct weighted masks. The encoder and decoder modules incorporate a multi-head self-attention mechanism and positional encoding, enabling the model to concentrate on specific feature maps at the current time step. By leveraging joint attention and a segmentation-free approach, our neural network calculates split attention weights on the visual representation, facilitating implicit line segmentation. This strategy signifies a substantial advancement toward achieving end-to-end transcription of entire paragraphs. Experiments conducted on paragraph-level benchmark datasets, including RIMES, IAM, and READ 2016 test datasets, demonstrate competitive results compared to recent paragraph-level models while maintaining reduced complexity. The code and pre-trained models are available on our GitHub repository here: HTTPSlink.

Item Type: Peer reviewed article published in a journal
Cheriet, Mohamed
Affiliation: Génie des systèmes
Date Deposited: 26 Jul 2023 17:54
Last Modified: 16 Oct 2023 16:16

Actions (login required)

View Item View Item