2.1 Morphological analysis development landscape

Since morphological analysis serves as foundation for almost all higher-level NLP tasks, significant research is done in this field since the early eighties. Majority of the research in this area was focused only on a few resource-rich languages. The first computational model for the morphology was developed by Kimmo Koskenniemi (Koskenniemi Reference Koskenniemi1984) which had a support for word form recognition and generation. It is considered as the pioneer work in the analysis of morphologically complex languages. In Beesley and Karttunen (Reference Beesley and Karttunen1992), the authors developed current C version of two-level compiler, called TWOLC. Another popular methodology known as finite-state morphology was evolved inspired from the idea of two-level morphology. XFST—tthe tool for implementing finite-state morphology—was introduced by Beesley and Karttunen (Reference Beesley and Karttunen2003). For the recognition of the valid word form and generation of various grammatical features, finite-state automata and finite-state transducer were used, respectively. Notable research works in the field of finite-state morphology are Beesley (Reference Beesley1998, Reference Beesley2003), Megerdoomian (Reference Megerdoomian2004). In the early 2000s, the paradigm-based model for the development of the morph analyzer became very popular. It was first proposed by Akshar Bharati (Bharati et al. Reference Bharati, Chaitanya, Sangal and Gillon2002). A paradigm table captures rules for constructing all the possible word forms that can be derived from the given root form along with the grammatical features of each word form. Different paradigm tables are created, and inflected input word is mapped with the corresponding paradigm table to perform the analysis (Bapat, Gune, and Bhattacharyya Reference Bapat, Gune and Bhattacharyya2010). The paradigm of research in the field of computational morphology has shifted from the conventional methods to the machine learning-based methods in the last decade. The reason is the requirements of handcrafted rules, dictionaries, paradigm tables, and suffix tables in the standard rule-based and paradigm-based methods. In Anand Kumar et al. (Reference Anand Kumar, Dhanalakshmi, Soman and Rajendran2010), the authors treat the problem of morphological analysis as classification task. SVM architecture is used which captures the nonlinear relationship of the grammatical features. A statistical morphological analyzer is developed in the work (Malladi and Mannem Reference Malladi and Mannem2013; Srirampur, Chandibhamar, and Mamidi Reference Srirampur, Chandibhamar and Mamidi2015). Few unsupervised models have also been experimented for the morphological tagging task. The pioneer work in this field is done by Goldsmith (Reference Goldsmith2005). Another popular unsupervised model is the morfessor (Creutz Reference Creutz and Lagus2005). Ak and Yildiz (Reference Ak and Yildiz2012) and Shambhavi et al. (Reference Shambhavi, P, Srividya, Jyothi, Kundargi and Shastri2011), Narasimhan, Barzilay, and Jaakkola (Reference Narasimhan, Barzilay and Jaakkola2015) have also proposed unsupervised morphological analyzer using Trie-based approach and log-linear methods.

Morphology in general is classified into two types: inflectional and derivational. The inflectional morphology makes a different form of the same word such as plural or past tense. The derivational morphology forms a new word by changing the POS category of the word. The common morphology types are as follows:

• Affixation: Adding an affix ( suffix or prefix ) (go $\rightarrow$ goes)

• Compounding: Combining two or more roots in a single word (greenhouse)

• Internal change: Changing part of the root (sit $\rightarrow$ sat)

• Suppletion: Changing the root completely (good $\rightarrow$ better)

During the last few years, many deep neural network-based architectures have emerged and also given promising results for numerous challenging NLP tasks. For the computational language processing tasks, researchers have used deep learning models (Yu, Falenska, and Vu Reference Yu, Falenska and Vu2017; Malaviya, Gormley, and Neubig Reference Malaviya, Gormley and Neubig2018). In Liu (Reference Liu2021), the authors have done the brief survey on various neural models for the computational morphology related tasks. The paper gives good insights on different architectures and how to leverage them for the tasks such as lemmatization, morphological tagging, and joint learning of lemma and tags. The deep learning models require a huge amount of training data for accurate feature learning. The requirement of large data becomes a challenge for the low-resource languages. In the work (Chakrabarty, Chaturvedi, and Garain Reference Chakrabarty, Chaturvedi and Garain2016), a neural lemmatizer for the Bengali language is discussed. The proposed lemmatizer makes use of contextual information of the surface word to be lemmatized. The work (Heigold, Neumann, and van Genabith Reference Heigold, Neumann and van Genabith2016) proposes neural morphological tagger modules for morphologically rich languages. This work is extended in Heigold, Neumann, and van Genabith (Reference Heigold, Neumann and van Genabith2017) by obtaining results on fourteen different languages. In Chakrabarty, Pandit, and Garain (Reference Chakrabarty, Pandit and Garain2017), a deep neural model is described which identifies the correct word lemma transformation. RNN-based morpheme segmentation model is discussed in Premjith, Soman, and Kumar (Reference Premjith, Soman and Kumar2018) for the Malayalam language. In Tkachenko and Sirts (Reference Tkachenko and Sirts2018), various neural models for morphological tagging are discussed by keeping the same encoder discussed in Lample et al. (Reference Lample, Ballesteros, Subramanian, Kawakami and Dyer2016). In Gupta et al. (Reference Gupta, Krishna, Goyal and Hellwig2020), the results of various neural morphological taggers are compared for the Sanskrit language.

In the recent times, transformer-based approaches have become popular for almost all NLP tasks. The core of transformer-based models such as BERT are multilingual pre-trained models. Such models are trained on the large corpus, and then they can be fine-tuned for the specific task. For the task of morphological analysis and lemmatization, transformer-based approaches have been experimented (Kondratyuk Reference Kondratyuk2019; Singh, Rutten, and Lefever Reference Singh, Rutten and Lefever2021; Acikgoz et al. Reference Acikgoz, Chubakov, Kural, Şahin and Yuret2022).

2.2 Gujarati morphology

Gujarati belongs to the Indo-Aryan language family. The language is derived from Sanskrit—an ancient Indian language. The grammar of Gujarati is similar to other Indo-Aryan languages such as Hindi, Bengali, etc. Gujarati is an agglutinative language. Various suffixes are added to the root to create new word forms expressing grammatical relations. The word order for Gujarati is Subject-Object-Verb (SOV). There are three genders and two numbers (Chauhan and Shah Reference Chauhan and Shah2021). We now discuss Gujarati noun, verb, and adjective POS categories with their important grammatical features.

Gujarati nouns inflect for the gender, number,and case. Gujarati has three genders: male, female, and neutral; two numbers: singular and plural; and six cases. Table 1 shows various cases along with case markers.

Table 1. List of case markers for Gujarati noun

Gujarati verbs take inflections for gender, number, person, tense, aspect, and mood features. Tables 2 and 3 show the examples of verb features.

Table 2. Examples of moods in Gujarati verb

Table 3. Examples of Gujarati verb aspects

The features identified for adjectives are type, gender, and number. The type feature describes the nature of the adjectives. In Gujarati, some adjectives inflect with for the gender and number features and other adjectives do not inflect at all. Table 4 shows the examples.

Table 4. Gujarati adjective inflection

Below are some of the challenges in performing morphological processing for the Gujarati language:

• The standard dataset in which each word is mapped with the corresponding root form along with the grammatical features is not available for the Gujarati language. For the other resource-rich languages, such data can be found in the UD Treebank. The lack of dataset created hurdle in experimenting with modern deep neural network-based models.

• Due to the high degree of inflections, the standard rule-based and unsupervised systems cannot be generalized and often fail to handle the ambiguities in the word formation process.

• Due to the differences in the fundamental grammatical rules, the existing models for other languages of a similar family such as Hindi cannot be directly used. For example, Hindi has two genders, while Gujarati has three genders.

As discussed in the literature review section, the approaches for developing morph analyzer can be classified into rule-based and machine learning-based approaches. Due to rich morphology and high degree of ambiguities present in the word formation rules, it is not preferable to use the rule-based methods such as the paradigm approach and Finite State Transducer (FST). The work done in Baxi et al. (Reference Baxi, Patel and Bhatt2015) supports this argument. The standard machine learning approaches require manual feature engineering which is challenging because identifying word-level features from highly inflectional language is a difficult task. By studying various unsupervised models of morphology in the literature, we observe that even though attempts are made to develop morph analyzer using this approach, the best results for the inflectional morphology are obtained only through standard rule-based methods (Hammarström and Borin Reference Hammarström and Borin2011; Creutz Reference Creutz and Lagus2005; Goldsmith Reference Goldsmith2005).

2.3 Baseline system

We consider Gujarati morphological analyzer (Baxi and Bhatt Reference Baxi and Bhatt2021) as baseline system. Their model is also Bi-LSTM based, but the labels are represented using monolithic scheme. The system has following limitations:

• Due to the monolithic label representation, a number of output classes are very high which leads to inefficient training and low accuracy especially for verb POS category.

• We observe incorrect output when two words with similar structure belong to different POS categories. For example, consider the following sentences:

  Sentence 1: (Transliteration: Mārā nānī gāmaḍē rahē chē. English Translation: My grandmother lives in a village). Here, the word means grandmother, and the POS category is noun.

  Sentence 2: (Transliteration: Mēṁ nānī bacata yōjanāmāṁ rōkāṇa karyuṁ chē. English Translation: I have invested in a small saving scheme). Here, the word means small, and the POS category is adjective. Such ambiguities are not resolved using monolithic label representation.

3.1 Outline of the approach

In this section, we discuss the architecture and working of the proposed model used to create Gujarati morph analyzer. Our morph analyzer has two modules: morpheme boundary detection module which separates the root and inflection part from the word and grammatical feature prediction module that assigns morphological features to an inflected word. The morpheme boundary detection model is similar to Baxi and Bhatt (Reference Baxi and Bhatt2021) with the enhancement in the dataset. The authors observed that the primary limitation of the system in Baxi and Bhatt (Reference Baxi and Bhatt2021) is low accuracy for the morph feature prediction task especially for verb POS category due to the monolithic representation of the labels. We use the different label represent technique in the proposed approach and observe the improvements in the overall results.

We propose the change in the label representation technique through which the number of target classes is reduced and there is improvement in the results for the grammatical feature prediction task. Given any inflected word, the task of grammatical feature prediction module is to tag the word with corresponding grammatical feature values. As describes in the previous section, for the Gujarati language, we have identified various features for the noun, verb, and adjective categories. Each grammatical feature has multiple possible labels. Table 5 shows labels associated with each grammatical feature. A number of labels determine the number of output classes in case of multiclass classification model. In the monolithic label representation used in Baxi and Bhatt (Reference Baxi and Bhatt2021), each combination of labels was assigned a class label. For example, the combination male–singular–nominative is one class, the combination female–singular–nominative is another class, and so on. Due to many different possible combinations, a number of output classes are very large due to which the model is not trained properly and results into poor accuracy. The similar observations are also reported by Tkachenko and Sirts (Reference Tkachenko and Sirts2018) and Chakrabarty et al. (Reference Chakrabarty, Pandit and Garain2017). To overcome this limitation, we use different label representation technique. In the proposed technique, we represent each feature as separate label instead of combining them together. In this way, a number of classes are equal to number of possible labels for a feature. For example, consider noun POS category which has gender, number, and case feature. In the proposed model, the total number of output classes will be 11 (gender: male, female, and neutral; number: singular, plural; and case: nominative, dative, ergative, genitive, ablative, and locative). Each label can have binary value indicating whether it is present or not. Table 6 shows the reduction in number of classes compared to the previous monolithic representation.

Table 5. List of features along with corresponding labels

Table 6. Comparison of number of classes in baseline and proposed individual label representation technique

3.2 Model selection

For the task of morphological analysis, we consider the following neural architectures: RNN, Bi-LSTM, Convolutional Neural Network (CNN), and transformer. While CNNs have been successful in image processing tasks, their application in NLP has been relatively limited. Transformer-based approaches are based on large pre-trained models like BERT. Such models are fine-tuned for the specific task. For the Gujarati language, such robust models are not available. In the task of morphological analysis, each label may be dependent on the previous predicted label, so we treat it as sequence labeling problem. RNN and Bi-LSTM models are well-suited for the sequential tasks. Due to this, we select Bi-LSTM as our implementation model and also compare the results with RNN and unsupervised model.

3.3 Dataset

A morphological analyzer is a program which takes the inflected word as input and gives corresponding root word and grammatical feature information as an output. In order to implement the morphological analyzer model, we require a dataset to train the system. The dataset must contain inflected words for various POS categories along with their morpheme boundary segmentation and grammatical feature information. Most of the existing models discussed in the previous section uses either Universal Treebank or UniMorph datasets. Universal Dependencies (UD) is a framework for consistent annotation of grammar across different human languages. UD is an open community effort with over 300 contributors producing nearly 200 treebanks in over 100 languages including Hindi (Nivre et al. Reference Nivre, de Marneffe, Ginter, Goldberg, Hajič, Manning, McDonald, Petrov, Pyysalo, Silveira, Tsarfaty and Zeman2016, Reference Nivre, de Marneffe, Ginter, Hajič, Manning, Pyysalo, Schuster, Tyers and Zeman2020). The Universal Morphology (UniMorph) project (Kirov et al. Reference Kirov, Cotterell, Sylak-Glassman, Walther, Vylomova, Xia, Faruqui, Mielke, McCarthy, Kübler, Yarowsky, Eisner and Hulden2018; McCarthy et al. Reference McCarthy, Kirov, Grella, Nidhi, Xia, Gorman, Vylomova, Mielke, Nicolai, Silfverberg, Arkhangelskiy, Krizhanovsky, Krizhanovsky, Klyachko, Sorokin, Mansfield, Ernštreits, Pinter, Jacobs, Cotterell, Hulden and Yarowsky2020) is a collaborative effort to improve how NLP handles complex morphology in the world’s languages. The goal of UniMorph is to annotate morphological data in a universal schema.

Being a low-resource language, the dataset for the Gujarati language was not yet available in the UD treebank or UniMorph datasets so we have created our own dataset and annotated it as per the UniMorph schema. The dataset creation process is published in Baxi and bhatt (Reference Baxi and bhatt2022), Batsuren et al. (Reference Batsuren, Goldman, Khalifa, Habash, Kieraś, Bella, Leonard, Nicolai, Gorman, Ate, Ryskina, Mielke, Budianskaya, El-Khaissi, Pimentel, Gasser, Lane, Raj, Coler, Samame, Camaiteri, Rojas, Francis, Oncevay, Bautista, Villegas, Hennigen, Ek, Guriel, Dirix, Bernardy, Scherbakov, Bayyr-ool, Anastasopoulos, Zariquiey, Sheifer, Ganieva, Cruz, Karahóǧa, Markantonatou, Pavlidis, Plugaryov, Klyachko, Salehi, Angulo, Baxi, Krizhanovsky, Krizhanovskaya, Salesky, Vania, Ivanova, White, Maudslay, Valvoda, Zmigrod, Czarnowska, Nikkarinen, Salchak, bhatt, Straughn, Liu, Washington, Pinter, Ataman, Wolinski, Suhardijanto, Yablonskaya and Stoehr2022).

3.3.1 Summary of dataset creation steps

For the dataset creation, we first studied the morphology and word formation process of Gujarati. After that, we surveyed about various available corpus for Gujarati and selected TDIL-ILCI-II corpus which contains 30,000 POS tagged sentences. We prepared wordlist files for each POS category. After that, with the help of linguists, for each inflected word, we identified corresponding root morpheme and also associated grammatical feature lists such as gender, person, case, etc. We have also developed the annotation tool using which each word was annotated and the data was prepared in the standard UniMorph format.

Our dataset is divided into two parts: dataset for morpheme boundary segmentation and the dataset for grammatical feature tagging task. In the morpheme boundary segmentation dataset, each word is represented in its root and suffix part using binary encoding. The second part is the dataset for capturing various grammatical features of an inflected word. We included noun, verb, and adjective POS categories and identified various grammatical features associated with them. Table 7 shows the details about the features and number of the words in our dataset.

Table 7. Details about dataset

Our dataset contains 16,327 unique inflected words with their correct root word mappings. The proposed Gujarati dataset is included in the upcoming version of the UniMorph schema (Batsuren et al. Reference Batsuren, Goldman, Khalifa, Habash, Kieraś, Bella, Leonard, Nicolai, Gorman, Ate, Ryskina, Mielke, Budianskaya, El-Khaissi, Pimentel, Gasser, Lane, Raj, Coler, Samame, Camaiteri, Rojas, Francis, Oncevay, Bautista, Villegas, Hennigen, Ek, Guriel, Dirix, Bernardy, Scherbakov, Bayyr-ool, Anastasopoulos, Zariquiey, Sheifer, Ganieva, Cruz, Karahóǧa, Markantonatou, Pavlidis, Plugaryov, Klyachko, Salehi, Angulo, Baxi, Krizhanovsky, Krizhanovskaya, Salesky, Vania, Ivanova, White, Maudslay, Valvoda, Zmigrod, Czarnowska, Nikkarinen, Salchak, bhatt, Straughn, Liu, Washington, Pinter, Ataman, Wolinski, Suhardijanto, Yablonskaya and Stoehr2022). The dataset is publicly available.^{Footnote b}

4.1 Experiment setup

In our morphological analysis, we solve two distinct problems: morpheme boundary detection and grammatical feature tagging. We first discuss how both problems can be formulated as ML classification problems and provide details of data points and labels as described in Jung (Reference Jung2022a).

• Morpheme boundary detection: This problem involves breaking down words into individual characters, which are then represented as character embeddings. Each character serves as a data point, and the goal is to classify whether a given character represents the boundary between two morphemes or not. In other words, it’s a binary classification task where the labels are binary, indicating whether there is a morpheme split at a particular character position.

• Grammatical feature tagging: In this task, each word is treated as a data point, and the objective is to assign grammatical features, such as gender, case etc., to that word. This is a multiclass-multilabel problem, meaning that for each word, there can be multiple grammatical features(labels) to assign, and each of these features can have multiple possible classes.

Figure 1a and b shows the architecture of the system for both tasks.

Figure 1. System architecture for the morpheme segmentation and grammatical feature prediction task, respectively.

We use the same architecture used in Baxi and Bhatt (Reference Baxi and Bhatt2021) for the morpheme boundary detection task. The data is represented in the binary encoded format such that 1 in the string represents the morpheme splitting location. Fig. 2 shows the example of the data representation for the morpheme boundary detection task. Table 8 shows the results obtained for the morpheme boundary detection task. In this example, an inflected word (English meaning in the morning) is represented. The corresponding root word would be (English meaning morning).

Figure 2. Example of encoding for the identification of morpheme boundary.

Table 8. Morpheme boundary detection results—POS category wise

For the grammatical feature prediction task, the format of the training data is such that each word is associated with a binary string. Each value in the string corresponds to individual feature labels as mentioned in Table 5. Once the training data is prepared, we first apply character level tokenization and then apply padding and generate sequences for the input. We split the data for the training and testing and define the model as per the model architecture mentioned in the previous section.

We use the following model architecture:

• Layer 1: Character level embedding layer, dimensions—50.

• Layer 2: Bidirectional LSTM: Hidden layer size: 128. For the morpheme segmentation task, return _sequences argument is True.

• Layer 3: Dropout layer for the prevention of overfitting.

• Layer 4: Dense layer: activation sigmoid. Numbers of output neurons are equal to number of features for the grammatical feature prediction task and is 1 for the morpheme boundary detection task.

• Loss = binary cross entropy.

4.2 Working of the model

As highlighted in the previous sections, the morphological analyzer performs two tasks. The morpheme boundary detection task is treated as sequence labeling problem in which each character of the input word is assigned a label indication whether the character is split point or not. Once the split point is identified, the root and suffix part can be identified. Each character is converted into correspondingly character embeddings before the Bi-LSTM layer. The final dense layer assigns binary label 0 or 1 indicating the split point.

The second task of grammatical feature prediction is treated as multiclass-multilabel classification problem. Depending on the number of features for the particular POS, output classes are decided. Similar to the morpheme boundary detection task, the character embeddings are given as an input to the Bi-LSTM layer, and the dense layer predicts the output classes for each feature.

4.3 Evaluation

In this section, we discuss the results obtained through the proposed individual feature representation technique for the grammatical feature prediction task and compare the obtained results with the monolithic label representation. Table 9 shows the comparison of F1 scores and accuracy between monolithic and proposed individual feature representation technique. We have also performed the grammatical feature tagging experiments individually for each POS category—feature combination. The purpose of this analysis is to observe how each feature and POS category contributes to the overall results. The results are shown in Table 10. We observe from the results that for the noun and adjective POS categories, the proposed system gives good results for all the grammatical features. However, for the verb POS category, the system does not give good results specifically for gender and person features. In this way, we are able to identify the exact features which are not learned properly by the system. This may be due to ambiguities at the word formation level or insufficient examples related to the particular grammatical feature. In this table, accuracy denotes the percentage of examples for which all the features are classified correctly. Because the training data is slightly unbalanced, the precision, recall, and F1 measures are used. Figure 3 shows the visual representation of improvement in the result accuracy due to proposed system. We note that there is huge improvement in the results due to the proposed change in the label representation. From the machine learning perspective, we identify the following reasons for the performance improvement in the proposed approach compared to the baseline model:

Table 9. Comparison of accuracy F1 scores—monolithic and individual feature representation

Table 10. Results for morphological feature tagging task for individual features using proposed label representation technique—POS category and feature wise

Figure 3. Comparison of result accuracy between the monolithic representation and individual feature representation.

• Feature representation: As highlighted in Table 6, due to individual label representation, a number of output classes are reduced. This reduction makes the model more powerful and allows better feature interaction allowing the model for learning more complex patterns.

• Better generalization: Because of the proposed change in the label representation, the model is better generalized for the unseen data.

• Error analysis: We can diagnose the performance of the model by analyzing the training and validation errors, as outlined in the study by Jung (Reference Jung2022b). Table 11 presents a comparison of training and validation loss between the baseline model and the current system.

From the error analysis table, we make the following observations:

• The loss in the current approach has shown a substantial reduction compared to the baseline. It suggests that the current approach performs better than the baseline system.

• The difference between the training and validation loss remains less than 0.1. This minimal difference indicates that there is no overfitting.

Table 11. Analysis of training and validation errors for baseline and present system

The proposed model uses the Bi-LSTM model. It is important to compare the results of the proposed model with the other popular deep learning architectures. We have compared the results of the proposed model with vanilla RNN and simple LSTM models, and the results are shown in Table 12. It is clear from the comparison that the proposed model performs better than the other deep learning architectures. In Table 13, we compare the results of the proposed Bi-LSTM-based neural model (both monolithic and individual feature representation) with the unsupervised model. The results of morpheme boundary detection with the unsupervised model are taken from Baxi and Bhatt (Reference Baxi and Bhatt2021). The unsupervised model was implemented using the morfessor model (Creutz Reference Creutz and Lagus2005) and can be used through the Indic-NLP library (Kunchukuttan Reference Kunchukuttan2020). From the comparison, it is clearly inferred from the results that the neural model using proposed individual feature representation technique outperforms the unsupervised model by a large margin.

Table 12. Comparison of the results of proposed approach with other deep neural network architectures—RNN and LSTM

Table 13. Comparison of results obtained from neural model and unsupervised model

We observe that for the verb POS category, the system gives correct output for the examples where multiple suffixes attach to the root verb as shown in Table 14. We did the manual analysis of the incorrectly predicted words to understand the possible linguistic angle behind the incorrect output. We make note of the following points:

Table 14. System output prediction for the words containing multiple suffix attachments

• For the morpheme segmentation task, the rules to form valid root word are different for each POS category and they also depend on the grammatical features. The output obtained by the system may be a valid stem but may not be correct root word. In future, the neural method can be used with rule-based method in combination to address this issue. Table 15 highlights such examples.

• Ambiguities in the word formation rules and attachment of multiple suffixes lead to incorrect morpheme splitting output.

• The feature prediction model captures the pattern of inflections. In some cases, the grammatical features do not have any correlation with the attached suffixes. Due to these exceptions, some errors are present in the grammatical feature prediction output. For example, consider two adjectives: (Sārī) and (Ajñānī). Both have (Ī) suffix which indicates female gender, but the adjective (Ajñānī) does not demonstrate any gender; it can be used in the same form for all genders.

Table 15. Examples of incorrect root identification