2. Related work

In general, there has been scarcity of research work done in the field of NLP on Khasi language. It is only as recent as 2018 that works on POS tagging and machine translation started. Tham (Reference Tham2018) proposed a POS tagger using the Hidden Markov Model (HMM) on a corpus that consisted of 86,087 tokens and achieved an accuracy of 95.68 per cent. Other tools available in NLTK for tagging have also been discussed in the paper, with reported accuracy rates of 86.76 per cent for BaselineTagger, 88.23 per cent for NLTK Bigram Tagger, 88.64 per cent for NLTK Trigram Tagger and 89.7 per cent for Natural Language Toolkit (NLTK) Tagger. Warjri et al. (Reference Warjri, Pakray, Lyngdoh and Maji2019) developed a Khasi POS tagger based on the HMM model. For the task, a Khasi tagset consisting of fifty-four tags has been designed. A manually tagged Khasi lexicon of around 7,500 tokens has been used on the model, and a test accuracy of 76.70 per cent has been reported. Tham (Reference Tham2020) created a hybrid POS tagger by integrating Conditional Random Field (CRF) with a HMM Khasi POS tagger in order to reduce the HMM tagger’s tagging errors and reported a tagging accuracy of 95.29 per cent, a 1.9 per cent improvement over the HMM tagger. Warjri et al. (Reference Warjri, Pakray, Lyngdoh and Maji2021) proposed a Khasi POS tagger using the CRF method. As part of their work, a tagset has been designed, customarily, for the Khasi language, and a corresponding POS tagged corpus of 71,000 tokens has been built. The CRF model as designed yielded a test accuracy of 92.12 per cent and an F1-score of 0.91. Thabah et al. (Reference Thabah, Mitri, Saha, Maji and Purkayastha2022) used BERT, to get pretrained embeddings for Khasi, and they reported convergence after 500 epochs of training and a training accuracy of 0.08. Because the tokenisation technique used in this work creates a vocabulary that is comprised of all unique terms encountered in the corpus, one disadvantage is the potential for unknown tokens to appear if they are not already there in the vocabulary constructed.

Regarding POS tagging in other low resource Indian languages, a number of works have been reported in the literature. In their paper, Nongmeikapam and Bandyopadhyay (Reference Nongmeikapam and Bandyopadhyay2016) have used genetic algorithm for selecting the best feature combination for use in the task of POS tagging with CRF model. According to them feature selection is crucial to improving the performance of the tagger. They reported an accuracy of 80.00 per cent Recall, 90.43 per cent Precision and 84.90 per cent F1-score. Nunsanga et al. (Reference Nunsanga, Pakray, Lallawmsanga and Singh2021) employed CRF method in POS tagging of the Mizo Language. A Mizo tagset consisting of twenty-six tags has been presented, and the tagger was able to achieve an accuracy of 89.42 per cent on a corpus of 30,000 words. Odia is another Indian language that is low resource and an attempt to develop a POS tagger for it has been made by Dalai, Mishra and Sa (Reference Dalai, Mishra and Sa2022). They have experimented with statistical as well as deep learning based methods in their work and found that the deep learning Bidirectional Long Short-Term Memory (bi-LSTM) network with character sequence feature using a pretrained word vector yielded the best accuracy of 94.58 per cent. A comparative analysis of various POS taggers for the resource scarce Bengali language has been reported by Jahara et al. (Reference Jahara, Barua, Iqbal, Das, Sharif, Hoque and Sarker2021). They have employed eight stochastic methods and eight transformation-based methods on a corpus of 7,390 lines. They compared the performance of these methods on two tagsets—one consisting of eleven tags, while the other consisting of thirty tags. An accuracy of 91.83 per cent has been reported using Brill with CRF method on the former tagset and an accuracy of 84.5 per cent on the later. A problem that has been highlighted in the paper is that the language has a complex morphology rendering the POS tagging task challenging. A deep learning approach, as described by Pathak, Nandi and Sarmah (Reference Pathak, Nandi and Sarmah2022) in their paper, has been utilised for the POS tagging task of the Assamese language. Again, in this work, the bi-LSTM with CRF model has been used to build the tagger and an accuracy of 86.52 per cent has been reported. Boro and Sharma (Reference Boro and Sharma2020) discussed the POS tagging challenges encountered in Assamese language in their study on POS tagging for Assamese. The paper mentions that designing a tagger for Assamese is significantly more difficult because the language is regarded as having a free form word order and a rich morphology. Even though (subject-verb-object (SVO) is the most predominant form of word order, sentences can still be constructed using other word orders including SOV, VOS, VSO, OVS and OSV. Another issue that has been highlighted in their work is the tagging of ambiguous words where the same word may be associated with more than one tag.

In general, in most of the papers in the literature, where limitations have been discussed, the most pertinent issue in POS tagging that still needs to be addressed is the problem with ambiguous words. To resolve this, more information is required which may be morphological, syntactic, semantic, etc. However, such information is scarce for low resource languages. Hence, this issue may be considered a challenge till date.

4. Dataset

The tagset designed by Warjri et al. (Reference Warjri, Pakray, Lyngdoh and Maji2018) and the corresponding POS tagged corpus of about 71,000 tokens have been used as the baseline. To this, an additional 35,109 number of tokens that are tagged manually has been added, thus expanding the POS tagged corpus size to 106,109. The additional set has been taken from articles from the local daily, ‘U Rupang’.

Table 1. List of borrowed words extracted from the corpus

Figure 2. Aligning labels to tokens.

4.1. New and modified tags

In the original tagset defined by Warjri et al. (Reference Warjri, Pakray, Lyngdoh and Maji2018), foreign words that appear in the corpus, such as ‘Sports’, ‘Association’, ‘Hills’ etc., have been tagged as FR (foreign word tag). Some foreign words, however, have evolved to become part and parcel of the Khasi language with their spellings changed to meet the Khasi manner of pronunciation and articulation. Words like ‘sorkar’ and ‘pisa’ are words that have been adopted and adapted into Khasi but are derived or borrowed directly or indirectly from other contact languages like Hindi or Assamese. In the same vein, words like ‘kolshor’ and ‘politik’ for ‘culture’ and ‘politics’, respectively, come directly from English. In the original tagset, such words have been tagged as FR. In this paper we propose tagging them as BR (borrowed word tag), indicating that they have been borrowed and have become an intrinsic part of the Khasi language. Table 1 lists the borrowed words found in the corpus. Another modification that is proposed in this paper is in the way abbreviations have been tagged. Here, we propose tagging all abbreviations with PPN, the proper noun tag instead of FR as in the original dataset.

In line with the above, some of the FR tags in the base corpus have been manually changed to BR tags. Morphological break-up of certain words in the base corpus has been done to a certain extent. Therefore, as a part of preprocessing, this morphological break-up of certain words has to be done with the additional dataset to comply with the base corpus. At present there is no tool for morphology, so this task has to be done manually on a corpus of 53 MB and a lot of time has been spent in this painstaking task. For the experiments to follow, the dataset is split as in Table 2.

Table 2. Dataset for training, testing and validation

5. Experimental setup and implementation

The model has been implemented in the Pytorch framework developed by Paszke et al. (Reference Paszke, Gross, Massa, Lerer, Bradbury, Chanan, Killeen, Lin, Gimelshein, Antiga, Desmaison, Köpf, Yang, DeVito, Raison, Tejani, Chilamkurthy, Steiner, Fang, Bai and Chintala2019), with computational resources from Google Colaboratory. Implementation of RoPOS is outlined as a two-step process as follows:

Table 3. Hardware configuration

Table 4. Parameter and hyperparameter setting for pretraining RoBERTa

• For pretraining of the language model, the generic Khasi corpus described in Section 3(1) is used. Since RoBERTa uses MLM, we mask out 15 per cent of the input sequence dynamically after tokenisation. Table 3 gives the hardware configuration on which the language model has been trained on, and Table 4 specifies parameters and hyperparameters that have been used. As described earlier, two instances of RoBERTa have been trained for ten epochs each and the best model is achieved at an average loss of 0.02 for both the instances and training accuracy of 98 per cent. As in Table 4, the maximum sequence length is set to 512, while Thabah et al. (Reference Thabah, Mitri, Saha, Maji and Purkayastha2022) capped this length to forty. The models have been tested on the word prediction task using the transformers pipeline where words are masked out at random for the model to predict and it obtains comparable results as Thabah et al. (Reference Thabah, Mitri, Saha, Maji and Purkayastha2022). The aim of this pretraining step is to extract the model’s embeddings that it has learnt, in order to use them for the POS tagging task.

• Next is implementing the POS tagger, RoPOS, whose structure is shown in Fig. 3. A single linear layer is added that is used for prediction of the tag for each token in the input sequence. Within the model, the embeddings generated by the pretrained RoBERTa model are extracted and the embedding dimension, of size 768, is obtained from the pretrained model’s hidden_size attribute. The hyperparameter configuration of Table 5 is used for both instances of RoPOS. Before training RoPOS, the POS tagged dataset is prepared and preprocessed with label alignment as described in Section 3(4).

Figure 3. Basic structure of RoPOS model.

Table 5. RoPOS model parameter and hyperparameter setting

Figure 4. Accuracy and Loss of RoPOS.

The accuracy and loss plots of the RoPOS over time for fifty epochs using BPE tokenisation and that using Wordpiece tokenisation for twenty epochs are shown in Fig. 4.

A training and test accuracy of 85 per cent and 83 per cent, respectively, is achieved for RoPOS with BPE while that of 95 per cent and 92 per cent for RoPOS with Wordpiece tokenisation.

5.1. Different machine and deep learning based models

Various machine learning and deep learning models have been experimented and implemented for the same task. The models that have been considered include Logistic Regression (Pranckevičius and Marcinkevicius Reference Pranckevičius and Marcinkevicius2016), Naive Bayes (Bulusu and Sucharita Reference Bulusu and Sucharita2019), Random Forest (Jahara et al. Reference Jahara, Barua, Iqbal, Das, Sharif, Hoque and Sarker2021), Support Vector Machine (Ekbal and Bandyopadhyay Reference Ekbal and Bandyopadhyay2008), simple Recurrent Neural Network (RNN) (Jordan Reference Jordan1997), LSTM (Hochreiter and Schmidhuber Reference Hochreiter and Schmidhuber1997), bi-LSTM (Fei and Tan Reference Fei and Tan2018) and a Gated Recurrent Unit (GRU) (Cho et al. Reference Cho, Merrienboer, Bahdanau and Bengio2014). The approach to solve the problem of POS tagging in all of these models is one of classification with as many classes as there are tags in the tagset. The models have been implemented in Google Colab using the keras API (Chollet Reference Chollet2018) and Pytorch framework (Paszke et al. Reference Paszke, Gross, Massa, Lerer, Bradbury, Chanan, Killeen, Lin, Gimelshein, Antiga, Desmaison, Köpf, Yang, DeVito, Raison, Tejani, Chilamkurthy, Steiner, Fang, Bai and Chintala2019).

The configuration shown in Table 6 is the one that has been used for the deep learning models, viz., RNN (with sixty-four RNN units), LSTM (sixty-four LSTM cells), bi-LSTM (128 LSTM Cells) and GRU (sixty-four recurrent units). The tokeniser from Keras API has been used for all of the deep learning models. The vocabulary has 111,425 distinct words because tokenisation is done word by word, making it significantly larger than the subword tokenisation outlined in Section 1.1.2. The accuracy plots of these models trained for a varied number of epochs are shown in Fig. 5.

Table 6. Deep learning models’ configuration

Figure 5. Accuracy plots of deep learning models.

Overfitting has been handled by adding a dropout layer to all the deep learning models. For RoPOS, we add two dropout layers before the embeddings are fed to the final linear layer. The model improve with this configuration which can be seen in the train and validation accuracy plots. Cross validation during training helps us decide on early stopping of the training.

Table 7 shows the parameter values that have been chosen for the machine learning models. For Logistic Regression we find that the regularisation parameter, C, gives better accuracy and convergence when set to value 5 while for SVM C= 10, gives a better accuracy value.

8. Limitations and future scope

One aspect of RoPOS that may be considered as a drawback is the fact that the length of the input to the model is not only limited by the maximum sequence (sentence) length that we cap RoPOS with, but it is also further limited by the length of the tokens and/or subtokens that results from the process of tokenisation. This has been discussed in Section 3(4) where the tokenised input sequence length may be equal to or greater than the length of the original input sentence. Another limitation is that, more often, the model misclassifies certain types of nouns such as the common nouns, $\lt$ CMN $\gt$ , and proper nouns, $\lt$ PPN $\gt$ , and the foreign word tag, $\lt$ FR $\gt$ . The tagging of proper nouns is not always accurate, for example, it is common in Khasi to have Wonderful, Preacher, and English words of the like, as the name of a person. Such nouns have been tagged as $\lt$ FR $\gt$ , the foreign word tag and vice versa. The confusion matrices obtained after inferencing on articles taken from three different domains are shown in Figs. 6, 7 and 8. In all of the three confusion matrices, most of the misclassification is on the nouns and foreign words. Such ambiguity can be minimised if the embeddings could be pretrained on a larger corpus. Another possible way of minimising this is to incorporate Named Entity Recognition (NER) tags which may be a future work as there is no standard NER corpus built for Khasi. Since pretrained embeddings are the core behind RoPOS, it is beneficial to improve upon them by pretraining upon a larger corpus. Thus, corpus collection and cleaning is an important future work. Next is, increasing the size of the POS tagged dataset. This task will no longer be tedious as RoPOS can be used for preliminary tagging and then manually validating them for correctness. With the availability of a significant volume of POS tagged dataset, a future work extension would be designing a shallow parser on this specific dataset. As not much work has been done in this area other than POS tagging, such extension work is a requirement. Additionally, this will support subsequent NLP tasks such as text summarisation, NER, Question Answering and others. Another area that has room for improvement is the tagging of borrowed words with the $\lt$ BR $\gt$ tag. Based on the usage context, this tag may be further classified to specify the precise POS of the borrowed word.