3.1 Encoder–decoder models

The encoder–decoder (Bahdanau et al., Reference Bahdanau, Cho and Bengio2014; Laitonjam and Singh Reference Laitonjam and Singh2022) models work in a sequence-to-sequence manner. The input sequence is compressed into a context vector by the encoder, and the decoder utilizes this context vector to reconstruct the target sequence.

In the encoder, the hidden state $h_t$ is computed using the current input vector $x_t$ and the previous encoder hidden state $h_{t-1}$ . This is represented as $h_t = E_A(h_{t-1}, x_t)$ (Cho et al. Reference Cho, Merrienboer, Bahdanau and Bengio2014), where $E_A$ is an activation function.

In the decoder, the hidden state $s_t$ is computed using the previous decoder hidden state $s_{t-1}$ , the previous decoder output $y_{t-1}$ , and the context vector $c$ . It is represented as $s_t = E_A(s_{t-1}, y_{t-1}, c)$ (Phan-Vu et al. Reference Phan-Vu, Tran, Nguyen, Dang and Do2019), where $E_A$ is an activation function.

Understanding the specific architectures like LSTM, gated recurrent unit (GRU), and transformer necessitates a thorough grasp of the explanations of the encoder and decoder components.

3.1.1 Long short-term memory (LSTM)

The LSTM network, as detailed by Hochreiter and Schmidhuber (Reference Hochreiter and Schmidhuber1997), represents an evolution of the conventional RNN design. It overcomes the limiting factor of vanishing gradients inherent in standard RNNs, thereby enhancing the model’s ability to learn and sustain long-term dependencies. The LSTM cell has several gates that control the flow of information, including the input gate ( $i$ ), forget gate ( $f$ ), and output gate ( $o$ ). The formula for the LSTM cell’s hidden state ( $h_t$ ) is determined as follows:

\begin{equation*} h_t = o \cdot \tanh (c_t) \end{equation*}

where

- $h_t$ represents the hidden state at time step $t$ .

- $o$ is the output gate’s activation, which controls the amount of information to be passed to the output.

- $c_t$ is the cell state at time step $t$ , computed using the input gate ( $i$ ), forget gate ( $f$ ), and a combination of the current input ( $x_t$ ) and the previous hidden state ( $h_{t-1}$ ).

The LSTM architecture incorporates a gating mechanism that empowers the model to selectively retain or discard information. This capability enables the LSTM to effectively handle long sequences and maintain contextual dependencies during the translation process. Figure 1 illustrates the structure of the LSTM model.

Figure 1. Long short-term memory(LSTM) architecture (Hochreiter and Schmidhuber, Reference Hochreiter and Schmidhuber1997).

3.1.2 Gated recurrent units (GRUs)

GRUs are a type of RNN architecture that were introduced in 2014 by Cho et al. (Reference Cho, Merrienboer, Bahdanau and Bengio2014) as a simpler alternative to LSTM units. GRUs consist of two main components, namely the update and reset gates. The purpose of the update gate is to regulate the quantity of prior information that is to be retained, whereas the reset gate’s role is to dictate the extent of forgetting or ignoring the previous hidden state.

The formula for a GRU unit can be written as follows:

• • Reset gate: $r_t = \sigma (W_r x_t + U_r h_{t-1} + b_r)$

• • Update gate: $z_t = \sigma (W_z x_t + U_z h_{t-1} + b_z)$

• • Candidate activation: $\tilde{h}t = \tanh (W x_t + r_t \circ (U h{t-1}) + b)$

• • Hidden state: $h_t = (1 - z_t) \circ h_{t-1} + z_t \circ \tilde{h}_t$

Here, $x_t$ is the input at time step $t$ , $h_{t-1}$ is the hidden state at the previous time step, $r_t$ and $z_t$ are the reset and update gate activations, respectively, and $\tilde{h}_t$ is the candidate activation. $\circ$ signifies the operation of element-wise multiplication, while $\sigma$ represents the sigmoid activation function and $\tanh$ corresponds to the hyperbolic tangent activation function. $W$ , $U$ , and $b$ are the learnable weights and biases of the network. The architecture of GRU is depicted in Figure 2.

Figure 2. Gated recurrent unit (GRU) architecture (Cho et al. Reference Cho, Merrienboer, Bahdanau and Bengio2014).

3.1.3 Transformer

The architectural design known as the transformer was first proposed by Vaswani et al. (Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017), revolutionized sequence-to-sequence tasks by replacing RNNs with self-attention mechanisms. The transformer architecture is comprised of two major parts: an encoder and a decoder, both of which are constructed using multiple layers. In contrast to conventional recurrent neural networks, transformers process the input sequence in parallel, making them highly efficient for long sequences. In the encoder, self-attention is used to compute attention scores between query ( $Q$ ), key ( $K$ ), and value ( $V$ ) embeddings of each word. These scores are then used to weight the value embeddings, producing the encoded representation of the input sequence.

In the decoder, in addition to self-attention, the transformer that employs encoder–decoder attention emphasizes on various elements of the input’s encoding sequence while generating the output sequence. The transformer also incorporates positional encodings to account for the sequential order of the words. Figure 3 illustrates the transformer architecture. Table 1 summarizes the technical differences between LSTM, GRU, and transformer models for sequence-to-sequence tasks (Bahdanau et al., Reference Bahdanau, Cho and Bengio2014; Cho et al., Reference Cho, Merrienboer, Bahdanau and Bengio2014; Vaswani et al., Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017).

Figure 3. Transformer architecture (Vaswani et al. Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017).

Table 1. Comparison of LSTM, GRU, and transformer

Figure 4. Working of BPE solving OOV issue.

3.2 Byte-pair encoding (BPE)

In information theory, BPE is a compression technique that iteratively replaces the most frequent consecutive byte pairs (strings) in the data with codes that are not present in the original data. In NLP, BPE is used as a subword tokenization technique. A variant of this approach finds its application in models for natural language interpretation, such as LSTM, GRU, transformers, and GPT-2 for tokenizing word sequences. BPE is used during the preprocessing of data. In BPE, a word is segmented into individual characters. BPE maintains a counter for all the unique words in the vocabulary. For example, there is a vocabulary of words such as smart, smarter, oldest, and wildest, with frequencies of 3, 4, 6, and 3, respectively. In BPE, these words are split into characters, and the vocabulary is formed along with their frequency (Ramesh et al. Reference Ramesh, Doddapaneni, Bheemaraj, Jobanputra, Ak, Sharma, Sahoo, Diddle, Kumar, Pradeep, Nagaraj, Deepak, Raghavan, Kunchukuttan, Kumar and Khapra2022). The whole process is shown in Figure 4 for a better understanding:

\begin{equation*}s m a r t \lt/w\gt\,:\, 3, s m a r t e r \lt/w\gt \,:\, 4, o l d e s t \lt/w\gt \,:\, 6, w i l d e s t \lt/w\gt \,:\, 3\end{equation*}

A fixed number of iterations is run, and at every iteration, the frequency of each consecutive byte pair is counted. The pair with the highest frequency is combined as a single token. In this example, it is observed that the words “oldest” and “wildest” have occurred six and three times, respectively. After all the words are tokenized as individual characters, the first iteration of the process of generating byte-pair tokens is started. When the iteration begins, consecutive pairs of characters are checked to compute the pair that has the highest frequency. It is seen that the consecutive pair of characters “e” and “s” is the pair with the highest occurrence (6 + 3 = 9), which comes from the words “oldest” and “wildest”. Hence, the characters “e” and “s” are combined as a single token, “es”. The process of combining the most frequent pairs continues until a maximum number of iterations is reached.

One of the advantages of BPE is that it solves OOV problems to an extent because the vocabulary prepared by BPE consists of subword tokens instead of whole words. Unknown words are split using the subword tokens in the vocabulary, hence eliminating the OOV issue.

4.1 Translation model

The NMT model utilized in this research adopts an encoder–decoder architecture, incorporating a hybrid attention mechanism (Nath, Sarkar, and Das Reference Nath, Sarkar and Das2022). The encoder and decoder components consist of unidirectional, single-layered GRUs with 1024 hidden units within each layer. A custom bilingual corpus of sentences from the target and source languages is used for training the model. Additionally, the decoder is composed of 1024 units in the final output layer.

This research uses a hybrid attention mechanism and an encoder–decoder architecture to create both translation and transliteration model (Nath et al. Reference Nath, Sarkar and Das2022). Hybrid attention selects the attention mechanism among additive and multiplicative processes for a Bilingual corpus. During the initial learning phase of the NMT model, both additive and multiplicative attention processes are used to provide prediction output. Hybrid attention mechanism’s main idea is to find out the average loss produced by additive and multiplicative attention mechanisms individually. Then the minimum loss obtained from the average losses of additive and multiplicative attention mechanisms is used to update the weights of the model’s network. The average loss is calculated by taking the mean (average) of the losses obtained from two different attention mechanisms.

4.2 Model network for transliteration

The transliteration model Shao and Nivre (Reference Shao and Nivre2016); Younes et al. (Reference Younes, Achour, Souissi and Ferchichi2020) comprises a unidirectional single-layer encoder and decoder with GRUs with 512 hidden units per layer. Figure 5 illustrates the encoder–decoder architecture employed in the transliteration model. The model is trained on a bilingual corpus. The bilingual corpus of words is first converted to parallel subword tokens. The vocabulary is prepared from the corpus, and the embedding vectors are calculated accordingly. The Softmax layer in the decoder consists of 512 units with a sparse categorical cross-entropy loss function. BPE is used to attain subword tokens from the words of the corpus. In a machine transliteration model, to learn the relationship between parallel words, individual character-to-character mapping is learned. Hence, the atomic units considered here are characters in the words. Similarly, the atomic units of mapping are taken as subword tokens to get a more accurate translation for named entity words. The details of subword tokens are given in Table 2.

The steps for the transliteration model are illustrated with an example and are given below:

Table 2. Subword tokens

Figure 5. Encoder–decoder architecture for transliteration model.

1. 1. An input word is taken, such as “Parker.”

2. 2. The word is split into characters or pairs based on corresponding phonetic representation, such as “P a r k e r”.

3. 3. The characters are then tokenized using BPE.

4. 4. The input “Parker” is fed into the encoder component and converts it into a vector [0.3, 0.1, 0.2,……].

5. 5. The decoder receives the numerical representation and converts it into the corresponding word in the target language. “”.

6. 6. The output is then displayed, alike “”.

4.3 Translation model integrated with transliteration

The integrated MT model consists of both a translation and a transliteration model. The model is trained independently in two distinct phases. While the training of the models was done independently in two phases, the models shared the same session during the training process. This allows seamless integration of the transliteration model into the inference pipeline of the NMT model. During evaluation, the transliteration model is invoked when named entities are detected in the input text. We employ subword tokenization in the transliteration model to handle OOV words like proper nouns that are not explicitly part of the model’s training vocabulary. This enables the transliteration of unseen words. The translation model executes the task of translating, while the transliteration model is specifically tasked with transliterating named entities. Figure 6 depicts the proposed integrated translation–transliteration system architecture. A nonlinear alignment between the positions of source words and corresponding target words poses a significant challenge for the precise placement of translated named entities within the target sequence.

In order to solve this problem, we used the attention weights provided by the attention layer in the translation model. This layer is used by the decoder to attain the context vector, which is essential for predicting the output sequence. The attention mechanism relies on encoder output and encoder hidden states for its computation of attention weights, which in turn facilitate the formation of the context vector.

The attention weights act as indicators of the relevance of a particular word within the source sequence during the process of translation. As a result, the utilization of attention weights allows the decoder to focus on specific words at specific instances of time, leading to a more precise translation.

We leveraged the attention layer’s functionality within the translation model via the utilization of its attention weights or attention scores. These scores are computed using the previous decoder hidden state (h_t) and each encoder hidden state (h_s) and this is represented as:

\begin{equation*} \alpha _{ts} = \frac {{\exp (\text {score}(h_t, \bar {h}_s))}}{{\sum _{s^{\prime}} \exp (\text {score}(h_t, \bar {h}_{s^{\prime}}))}} \end{equation*}

These alignment scores indicate how much attention or emphasis should be placed on each word in the source sequence during translation.

The attention mechanism then combines these alignment scores to create an alignment vector, which has the same length as the source sequence. Each value in this vector corresponds to the importance (or probability) of the corresponding word in the source sequence. This helps the decoder decide what to focus on at each time step. The attention weights help identify the most focused input token by finding the index of the maximum value in the attention weight vector. This index is used to retrieve the corresponding word from a pre-built data structure mapping integers to words.

We employ the NER tool from SpaCy to determine whether the identified word is a named entity. If the word is classified as a LOCATION or ORGANIZATION type entity by the NER model, we check if a standard Assamese translation exists in our custom databases of common country names and institution names (e.g. “Bharot” for India). If a translation is available, we use that standard translation in the output sequence.

However, not all LOCATION and ORGANIZATION entities have an existing standard Assamese translation. For named entities that do not have a standard translation available in our databases, as well as other types of named entities like PERSON, we send the word through our transliteration model to produce an appropriate Assamese transliteration in the output sequence.

For words not identified as named entities by the NER tool, we directly use the translated output from our translation model in the output sequence.

Figure 6. Proposed integrated translation–transliteration system architecture.

The algorithm is described in detail below. The following steps illustrate how an input sentence is translated by the integrated MT model.:

At each time instant t = i, where i ranges from 0 to the size of the source sentence:

1. 1. Padding is applied to the input sentence to align its length with the expected source sentence length. Subsequently, the input tokens within the source sentence are converted into their corresponding unique integers, as defined during the vocabulary preparation phase.

2. 2. The input sequence is then passed to the encoder component of the translation model. The encoder produces the encoder output and the encoder hidden state.

3. 3. The decoder component of the translation model incorporates an attention layer, which takes as input the encoder output and encoder hidden state. The attention layer with the help of it fine-tuned attention weights computes the context vector. The context vector is obtained by multiplying vectors of the same shape, namely the encoder output and the attention weight.

4. 4. The decoder receives the context vector and attention weights from the attention layer.

5. 5. As the attention weight signifies the relative importance of each source word at a particular instant, we determine the index of the maximum value from the attention weight vector. The attention weights are represented as:

   \begin{equation*} \begin {bmatrix} 0.12 & \quad 0.0085 & \quad \ldots & \quad 0.61 \\ 0.7 & \quad 0.003 & \quad \ldots & \quad 0.01 \\ \vdots & \quad \vdots & \quad \ddots & \quad \vdots \\ \end {bmatrix} \end{equation*}
   where each row represents a source token, and the number of columns corresponds to the encoder hidden state.

6. 6. This index is utilized to retrieve the corresponding source token from the input array. This particular source token represents the focal point of translation at instant t = 0.

7. 7. Subsequently, we pass on this source token to the NER checker tool. In the case where the input token is identified as a named entity, we further break it down into subwords. If these subwords exist within the vocabulary, the subword sequence is then forwarded to the transliteration model. The output subwords generated by the transliteration model are merged to form a single word. This resulting word is placed in the output sequence at the corresponding position for instant t.

8. 8. If the input token is not identified as a named entity, or if the subwords are absent from the vocabulary, we employ the decoder component of the translation model to compute the translated output using the context vector. The translated output token is subsequently positioned within the output sequence at the respective position for instant t.

9. 9. The maximum length of each iteration of these steps corresponds to the maximum length of the target sequence. Also when the “end of sentence” token (<eos>) is found out, the prediction ends.

5.1 Data preprocessing

The corpus used to train both the translation and the transliteration models is taken from TDIL (TDIL-DC 2006), and in addition to that, we have increased data by 11% using back translation as well as back transliteration where required. We manually generated our back transliteration and translation data using the Aksharantar (ai4bharat 2022) and Bing websites (MSFT 2020). Over 7% of our transliteration and 4% of our translation corpora were generated by back-transliterating and translating a selection of English words and sentences that weren’t included in the original corpus. The integrated model uses a corpus of parallel sentences which belong to various domains such as agriculture, entertainment, and history. All unwanted and irrelevant characters have been removed from the bilingual corpora. Both the corpora words have been converted to lowercase. Ninety-five percent of the corpus is used for training, while 5% is allocated for testing. The validation set is of the same size as the test set and is derived from an external dataset (ai4bharat 2022). All the words (from both the language pair) in the corpus are converted to a sequence of subword tokens. The bilingual corpus for the integrated translation–transliteration model undergoes padding for shorter sentences to make all the sentences of a particular language pair equal. This is done so that the model may be trained in batches. All the punctuations have been retained in this corpus. If there is no white space between a word and punctuation, then white is given in between them to consider the punctuation as an individual character. The corpus statistics for the translation and Tables 3 and 4 present the transliteration models.

Table 3. Corpus statistics for translation model

Table 4. Corpus statistics for transliteration model

We utilized 1200 merge operations in our models. This value was selected based on testing a range from 500 to 2000 merge ops. We used two metrics to evaluate the performance of the algorithm: accuracy and runtime. We found that the accuracy of the algorithm was maximized at 1200 merge operations. The runtime of the algorithm also decreased as the number of merge operations increased, but only up to a point. After 1200 merge operations, the runtime of the algorithm started to increase again. Based on these results, we determined that 1200 is the optimal number of merge operations for our specific dataset and algorithm.

5.2 Model training

The proposed model is a GRU-based translation and transliteration model, trained on a system with the following hardware configuration: 24 GB of RAM and single-core, hyper-threaded Xeon processors with a clock rate of 2.3 GHz. Additionally, the system features a Tesla K80 GPU instance with 2496 CUDA cores and 12 GB of VRAM.

We have also trained an LSTM-based translation and transliteration model with the same parameters as the GRU model, as detailed in Table 5. In addition to the GRU and LSTM models, further experiments were conducted by training a transformer model. The transformer model shares parameters with both the GRU and LSTM models, keeping consistency in our methodology. The detailed parameters for the transformer-based model are provided in Table 6. By adopting this methodology, we aim to facilitate a better comprehension of the model’s behavior and interpretation of the results.

Table 5. Parameters for LSTM and GRU-based translation and transliteration models

Table 6. Parameters for transformer-based translation model

5.3 Evaluation metrics

The evaluation metrics used are BLEU (bilingual evaluation understudy) (Papineni et al. Reference Papineni, Roukos, Ward and Zhu2002), SacreBLEU (Post, Vilar, and Rebele Reference Post, Vilar and Rebele2018), TER (translation edit rate) (Snover et al. Reference Snover, Dorr, Schwartz, Micciulla and Makhoul2006), METEOR (metric for evaluating translation with explicit ordering) (Lavie and Denkowski Reference Lavie and Denkowski2009), chrF (character F-score) (Popović, Reference Popović2015), and RIBES (reference independent BLEU estimation score) (Snover et al. Reference Snover, Dorr, Schwartz, Micciulla and Makhoul2006). In addition to these automated metrics, human evaluation was also conducted to assess translation quality. For TER, lower scores in these evaluation metrics indicate higher prediction accuracy.

5.4 Experimental results

In this study, the transliteration and translation models were developed and evaluated independently. Subsequently, the translation model was integrated with the transliteration model, and the combined system was tested on 4,077 sentences. In Table 7, the transliteration model’s performance is assessed using the accuracy metric introduced by Zhang et al. (Reference Zhang, Li, Kumaranz and Banchs2016), denoted as ACC (accuracy). ACC measures the correctness of the transliteration output, providing insights into the model’s ability to accurately capture transliteration nuances based on the criteria established by Zhang et al. The experimental results for the standalone transformer model, GRU-based translation model, GRU-based translation model with a hybrid attention mechanism, proposed integrated system along with the Bahdanau attention mechanism, hybrid attention mechanism, as well as the LSTM-based translation models are presented in Table 8.The graphical representation in Figure 7 illustrates the progress of our model’s training through the epoch vs loss graph. The presented visual representation illustrates the gradual decrease in loss metrics over multiple epochs, indicating the model’s iterative improvement in error minimization and optimal fitting of the training dataset. The decision to utilize GRU in our translation model is driven by its competitive performance and potential ease of training.

Table 7. Performance of transliteration model

Table 8. Performance of distinct models for translation English to Assamese language

The integration of a transliteration model with a GRU-based translation model, along with a hybrid attention mechanism, results in improved performance compared to the base translation model, as evidenced by various evaluation metrics. Specifically, the integration of the transliteration model leads to overall improvements in translation quality, particularly for named entities, resulting in improved scores across various evaluation metrics. The performance comparison graph is depicted in Figure 8, while Tables 9, 10, and 11 present output samples from a standalone transliteration model and translation samples from the integrated system, respectively.

Figure 7. Epoch vs loss graph – training progress and error minimization.

Figure 8. Bar chart for performance comparison I.

In Table 9, the ^*-marked words are OOV, and the transliteration model recognizes and transliterates them. Hence, the OOV issue is solved by the transliteration model with the help of subword tokenization.

Table 9. Sample outputs from standalone transliteration model

Table 10. Sample translation by integrated system without transliteration

Table 11. Sample translation by integrated system with transliteration

In this study, we compare the performance of our proposed integrated system with two existing MT approaches: Samanantar (Ramesh et al. Reference Ramesh, Doddapaneni, Bheemaraj, Jobanputra, Ak, Sharma, Sahoo, Diddle, Kumar, Pradeep, Nagaraj, Deepak, Raghavan, Kunchukuttan, Kumar and Khapra2022), which is based on transformer-based models trained using Fairseq, and the commercial Microsoft MT system (MSFT 2020). The Samanantar model has six layers of encoders and decoders, 1536 input embeddings with 16 attention heads, and a 4096 feed-forward dimension. It uses label smoothing of 0.1 and gradient clipping of 1.0 in the Adam optimizer and starts with a learning rate of 5e-4 and 4000 warm-up steps. We tested this model using the identical test corpus as our proposed model, and the findings are presented in Table 12. In the second comparison, the Microsoft Azure Cognitive Services Translation API were used to translate all the sentences in our test set. The experimental results of the proposed integrated system and the two existing approaches are compared in terms of the BLEU score, sacreBLEU, METEOR, and TER, and the results are presented in Table 12 and Figure 9, respectively.

Table 12. Performance comparison between integrated system and existing models

Table 13. Human evaluation scores for English to Assamese translations

Table 14. Sample output from MSFT system

Table 15. Sample output from Samanantar system

Figure 9. Bar chart for performance comparison II.

According to the experimental findings shown in Table 12, it is observed that the proposed integrated model for English to Assamese translation outperforms both MSFT and Samanantar’s IndicTrans translation system. Specifically, the proposed model achieves a BLEU score of 28.13, compared to 20.31 and 11.43 for MSFT and Samanantar’s Indic translation system, respectively. Additionally, in Tables 14, 15, and 16 we presented a few sample predicted sentences to further examine the translation quality of the integrated system, MSFT, and Samanantar’s IndicTrans translation system. From the predicted sentences in Tables 13, 14, 15 and 16 it is observed that the integrated system is capable of accurately translating named entities and performs better than the existing systems for English to Assamese translation. Furthermore, the proposed integrated system is able to transliterate unknown named entities, as demonstrated in Table 8. Overall, the translation quality of the proposed integrated model surpasses that of the existing systems. To thoroughly assess the translation quality of MT models, we conducted a human evaluation following the approach of Laskar et al. (Reference Laskar, Paul, Dadure and Manna2023), in addition to using automatic metrics. This is important because it can capture nuances that automatic metrics often miss. Human evaluation method focused on two key aspects: adequacy, which measures how accurately the translation conveys the meaning of the source sentence, and fluency, which assesses how natural and idiomatic the translation is. To obtain an overall score, we averaged the adequacy and fluency ratings. Three evaluators independently assessed translation quality of three models using a 1–5 scale for a randomly chosen set of 100 sample sentences. Table 13 summarizes the human evaluation scores for the three models, providing insights into English-to-Assamese translation performance.