2.1 Discourse relation realization types

As already mentioned, the PDTB mainly annotates explicitly and implicitly conveyed relations together with their arguments (Arg1, Arg2) and senses. This section overviews the discourse relation realization types in the PDTB 3.0 (Webber, Prasad, and Lee Reference Webber, Prasad, Alan and Joshi2019), which extends the earlier version keeping the rules and principles of the PDTB framework intact. The sense hierarchy is also revised. In the examples, discourse connectives, where available, are underlined for clarity.

Explicit DRs are those relations that are explicitly signaled through lexico-syntactic elements. They may be realized across or within sentences and involve coordinating conjunctions (and, so, but) subordinating conjunctions (because, after, while), adverbials (however, additionally, consequently), and prepositional phrases (in summary, on the contrary). These markers are referred to as explicit discourse connectives, as shown in example 1. In the PDTB 3.0, explicit discourse connectives have been extended to cover subordinators (i.e., prepositions such as because of, by, despite, for, from) to more fully annotate intra-sentential relations and are only annotated when they take clausal complements, as in example 2.

1. (1) The city’s Campaign Finance Board has refused to pay Mr. Dinkins $95,142 in matching funds because his campaign records are incomplete.

2. (2) Eliminate arbitrage and liquidity will decline instead of rising, creating more volatility instead of less.

Implicit DRs: In the absence of a connective supporting a discourse relation, readers can infer the meaning of the relation, and the annotators are asked to insert an explicit marker that best conveys the sense of the inferred relation. These are called implicit connectives. While the PDTB 2.0 only annotated implicit DRs that hold between sentences (example 3), the PDTB 3.0 annotated implicit relations within sentences as well (e.g., between VPs or clauses conjoined implicitly by punctuation, as in example 4) (Zhao and Webber Reference Zhao and Webber2021). The PDTB 3.0 also annotates implicit relations when they co-occur with explicit relations, as will be exemplified in examples 12 and 13 below.

1. (3) So much of the stuff poured into its Austin, Texas, offices that its mail rooms there simply stopped delivering it. (Implicit = so) Now, thousands of mailers, catalogs, and sales pitches go straight into the trash.

2. (4) Father McKenna moves through the house praying in Latin, (Implicit = and) urging the demon to split.

Alternative Lexicalization (AltLex): When an implicit discourse relation is inferred to hold between or within sentences but the insertion of an implicit connective in the relation is perceived redundant, the relation is referred to as being alternatively lexicalized. Expressions that are inferred to confirm the presence of a discourse relation are annotated as AltLex (example 5). The PDTB 3.0 extended AltLexes to cover AltLexC relations, that is, lexico-syntactic constructions that unambiguously signal discourse relations, such as example 6.

1. (5) After trading at an average discount of more than 20% in late 1987 and part of last year, country funds currently trade at an average premium of 6%. The Reason: Share prices of many of these funds this year have climbed much more sharply than the foreign stocks they hold.

2. (6) Crucial as these elections are for Greece, pressing issues of state are getting lost in the shuffle.

Entity Relations (EntRels): Where no discourse relation can be inferred between adjacent sentences and adjacent sentences form entity-based coherence with the same entity being realized in both sentences either directly or indirectly, the discourse relation is annotated as an EntRel (example 7).

1. (7) Pierre Vinken, 61 years old, will join the board as a non-executive director Nov. 29. Mr. Vinken is chairman of Elsevier N.V., the Dutch publishing group.

No Relations (NoRels): Where a semantic relation between adjacent sentences cannot be identified, the adjacent pair of sentences are annotated as NoRel (example 8).

1. (8) Mr. Rapanelli met in August with U.S. Assistant Treasury Secretary David Mulford. Argentine negotiator Carlos Carballo was in Washington and New York this week to meet with banks.

Hypophora: Where a question is asked and a meaningful response is provided, Hypophora is annotated. This is a new inter-sentential relation in the PDTB 3.0, and it stands for discourse relations involving dialogue acts, which cannot be instantiated through connectives (example 9).

1. (9) But can Mr. Hahn carry it off? In this instance, industry observers say, he is entering uncharted waters.

In summary, in the PDTB 3.0, not only explicit and AltLex discourse relations but also implicit discourse relations are annotated both across and within sentences and are assigned a sense tag from the revised hierarchy of senses. The PDTB 3.0 keeps the four Level-1 senses of earlier versions, namely Expansion, Comparison, Contingency, and Temporal. These are refined to more specific senses at the second level. For symmetric relations, the sense hierarchy stops at the second level; for asymmetric relations, the hierarchy provides third-level senses to define the semantic contribution of each argument.Footnote ^a (The full sense hierarchy is provided in Appendix A). EntRels, NoRels, and Hypophora are annotated as inter-sentential relations and not assigned a sense tag.

Multiple Relations: In addition to the DRs overviewed so far, the PDTB 3.0 annotates multiple relations in certain cases. Multiple relations are those relations that can hold between a pair of spans. For example, when there are co-occurring explicit connectives between the same spans, multiple relations are annotated by creating two tokens (examples 10, 11).

1. (10) Small businesses say a recent trend is like a dream come true: more-affordable rates for employee-health insurance, initially at least. But then they wake up to a nightmare.

2. (11) Small businesses say a recent trend is like a dream come true: more-affordable rates for employee-health insurance, initially at least. But then they wake up to a nightmare.

Also, in cases where annotators infer a sense separate from the one conveyed by an explicit connective or AltLex or AltLexC, separate implicit relations are annotated by inserting implicit connectives (examples 12, 13).

1. (12) We’ve got to get out of the Detroit mentality and be part of the world mentality, declares Charles M. Jordan, GM’s vice president for design…

2. (13) We’ve got to get out of the Detroit mentality and (Implicit=instead) be part of the world mentality, declares Charles M. Jordan, GM’s vice president for design…

2.2 Shallow discourse parsing sub-tasks

A typical shallow discourse parser consists of three main sub-tasks: (i) connective identification, (ii) argument extraction, and (iii) sense classification. In the current paper, we have adopted a different approach and merged and converted the first two tasks into the identification task of a discourse relation in a given text piece, referred to as DR realization type identification. That is, our approach does not perform distinct connective and argument extraction, but it is still able to identify the discourse relation, if there is any, in a given text piece and further disambiguates the discourse relations in terms of the senses. Yet, we would like to note that the model is trained on both inter- and intra-relations without the knowledge of distinctionFootnote ^b and is unable to detect multiple discourse relations simultaneously. If the given text piece has more than one different DRs, the model will be able to find only one of them.

DR realization type identification: This is one of the least studied aspects of discourse. Here, it is modeled as a six-way classification task which aims to identify the specific realization type of the discourse relation in the given text. Although the usage disambiguation of discourse connectives (distinguishing between the connectives’ discourse and non-discourse role) has been investigated for many languages as discussed in Section 3, and implicit relation identification has also been targeted, these have often been considered as tasks on their own. The multi-level identification of relation realization type is a challenging task, which to our knowledge has not been tackled before.

Sense classification of discourse relations: This is the most popular sub-task of discourse parsing, where the aim is to find the sense conveyed by a given explicit or implicit discourse relation. The implicit classification task is rather (in)famous for its challenging nature due to the lack of an explicit signal, that is, the discourse connective, and renders the task extremely challenging, where the models must encode the semantics of each argument correctly. Due to its challenging nature, the task is most commonly limited to only the Level-1 senses in the PDTB 3.0 sense hierarchy; hence, in the current work, it is modeled as a four-way classification task. To the best of our knowledge, the four-way sense classification of implicit discourse relation is the first attempt over Turkish data. The second and third-level senses are out of scope of the current work.

3.1 Attempts toward the development of discourse parsers

Prior to the work of Lin, Ng, and Kan (Reference Lin, Ng and Kan2014), research only targeted individual tasks involved in discourse parsing. For example, the task of automatically identifying explicit discourse connectives was shown to be doable with high accuracy with the help of linguistic features and information about the connective’s surrounding context. In an early paper, Pitler et al. (Reference Pitler, Raghupathy, Mehta, Nenkova, Lee and Joshi2008) developed decision trees (i) to distinguish between explicit and implicit discourse relations in the PDTB 2.0, (ii) to distinguish tokens of each Level-1 sense relation from all the others, (iii) to carry out four-way sense classification of all tokens and a separate four-way sense classification of tokens with explicit connectives. Their model reached a higher performance in the classification of explicit discourse connectives than implicit ones. One of the earliest studies on the identification of discourse versus non-discourse usage of explicit connectives has been carried out by Pitler and Nenkova (Reference Pitler and Nenkova2009) over the PDTB 2.0. Feeding syntactic features extracted from the arguments of discourse connectives into a maximum entropy classifier,Footnote ^c the authors reached an F-Score of 0.92 in explicit discourse connective disambiguation and 94% accuracy in the four-way sense classification of explicit discourse connectives.

Later work has reached highly successful results in domain-specific applications. Ramesh et al. (Reference Ramesh, Prasad, Miller, Harrington and Yu2012) used various supervised machine-learning-based algorithms for automatically identifying explicit discourse connectives in the BioDRB corpus and proposed a hybrid classifier based on a Conditional Random Fields-based classifier and a combination of instance pruning, feature augmentation, and domain adaptation techniques. Extracting syntactic features such as the part-of-speech (POS) tags of the tokens, the syntactic labels of the immediate parent of the token’s POS in the parse tree, and the POS tags of the left sibling (the token to the left of the current word inside the innermost constituent), an F-Score of 0.76 is reached. Gopalan and Devi (Reference Gopalan and Devi2016) used fewer linguistic features relevant to discourse and employed machine-learning models to automatically extract explicit discourse connectives, their senses, and arguments. They reported an F-Score of 0.85 in the classification of explicit discourse connectives’ senses with the Conditional Random Fields algorithm.

Work on non-English languages such as Arabic discovered features that contribute to the disambiguation of the discourse usage of explicit connectives. Alsaif and Markert (Reference Alsaif and Markert2011) worked on an Arabic corpus where explicit discourse connectives are marked. They used syntactic features such as the position of the potential connective (sentence-initial, sentence-medial, or sentence-final), lexical features of the surrounding words, the POS tags of the words, and the syntactic category of the parent of the potential connective. The authors also discovered the predictive role of the infinitive form of the verb in the second argument of prepositional connectives and achieved an F-Score of 0.78 in explicit discourse connective recognition with the best feature combination.

A recent work by Başıbüyük and Zeyrek (Reference Başıbüyük and Zeyrek2023) disambiguates the discourse usage of Turkish connectives with a rule-based and machine-learning approach using different sets of linguistic rules for discourse connectives belonging to different syntactic classes. The machine-learning approach achieves an F1-score of 84.44 for single/phrasal connectives with the Fast Forest Binary algorithm and an F1-score of 80 for suffixal connectives with the Averaged Perceptron Binary algorithm with the best feature combination.

While it could be said that explicit discourse relation recognition is largely a solved problem, especially for well-studied languages such as English, identification of implicit relations still remains a challenge. Marcu and Echihabi (Reference Marcu and Echihabi2002) have been the first to identify implicit relations by removing discourse connectives to cheaply gather large amounts of training data. They also discovered that word pairs are indicative of implicit relations (e.g., the pair embargo … legally was a good indicator of contrast relations) and used them in extracting large amounts of data. They reached high accuracies with this technique of obtaining artificial implicit relations. Particularly, two of their classifiers were successful in distinguishing between Cause-Explanation-Evidence versus Elaboration (93%) and Cause-Explanation-Evidence versus Contrast (87.3%).

A later paper by Pitler, Louis, and Nenkova (Reference Pitler, Louis and Nenkova2009) predicted the Level-1 senses of implicit relations in the PDTB 2.0 in a realistic setting taking advantage of linguistically informed features as well as lexical pairs from unannotated text. They achieved an F-Score of 0.76 with the best combination of their features (polarity+Inquirer tags+context). Lin, Kan, and Ng et al. (Reference Lin, Kan and Ng2009) took parser production rules as the main source of features and worked on the Level-2 senses of the PDTB 2.0 hierarchy. They showed that syntactic patterns could contribute to predicting the senses of implicit relations. In CoNLL-2015 Shared Task, the feature-based work of Rutherford and Xue (Reference Rutherford and Xue2014) presented statistical classifiers for identifying the senses of implicit relations. The authors introduced novel feature sets that exploit distributional similarity and coreference information. They showed that Brown cluster pairs (Brown et al. Reference Brown, DeSouza, Mercer, Pietra and Lai1992) work well in implicit relation recognition.

A breakthrough is seen in the field with the development of an end-to-end discourse parser (for English) by Lin et al. (Reference Lin, Ng and Kan2014). Working on the PDTB 2.0, the authors produced a PDTB-styled parser, constructing a thorough pipeline for parsing the text in seven sequential steps titled as connective classification, argument labeling, argument position classification, argument extraction, explicit relation classification, and sense recognition of explicit as well as non-explicit relations. The authors classified and labeled discourse relations and the attribution spans, where relevant. Their parser can parse any unrestricted English text into its discourse structure in the PDTB style. The best F1-Scores are reported as 0.87 for the explicit classifier, and 0.4 for the non-explicit classifier.

Lately, shallow discourse parsing has been attempted in a series of shared tasks. As in the work of Mukherjee et al. (Reference Mukherjee, Tiwari, Gupta and Singh2015), shallow discourse parsing is mostly conducted by using syntactic and semantic features for classification. It is also attempted in CoNLL Shared Task (2015), to which 16 teams participated by using a piece of newswire text as input and returning relations in the form of a discourse connective (either explicit or implicit) with two arguments. Each team developed an end-to-end system that could be regarded as variations of Lin et al. (Reference Lin, Ng and Kan2014), detecting and categorizing individual discourse relations and returning a set of relations contained in the text. The best system achieved an F1-Score of 0.24 on the blind test set, reflecting the serious error propagation problem in such a system (Xue et al. Reference Xue, Ng, Pradhan, Prasad, Bryant and Rutherford2015, 14).

The shared task DISRPT (2019) (DR Parsing and Treebanking) was held on discourse unit segmentation across formalisms, including shallow discourse parsing, aiming to promote the convergence of resources and a joint evaluation of discourse parsing approaches. The corpora included 15 datasets in 10 languages, 12 of which target elementary discourse unit segmentation, and three dedicated to explicit connective annotation (Zeldes et al. Reference Zeldes, Das, Maziero, Antonio and Iruskieta2019). In the overall evaluation, ToNy (Muller, Braud, and Morey Reference Muller, Braud and Morey2019) performed the best on most of the datasets reaching an F-Score of 0.9 in the average of all its tests. Turkish is also represented in the dataset with the TDB 1.0, where only explicit discourse connectives are annotated. On this data, ToNy (Muller et al. Reference Muller, Braud and Morey2019) obtained the best results in discourse connective detection on plain, unannotated data, reaching an F-Score of 0.85. In the DISRPT (2021) shared task event, the approach of the group DiscoDisco increased the F1-Score of Turkish explicit discourse connective detection sub-task to 0.94.

One of the latest noteworthy achievements has been reported by Liang, Zhao, and Webber (Reference Liang, Zhao and Webber2020). The authors worked on the PDTB 3.0, where implicit relations are annotated both at the inter-sentential and intra-sentential levels. In addition to these stand-alone implicits, implicit sense relations between the arguments of explicit relations are also annotated. In a series of experiments, the authors first recognized the location of implicits, then they recognized their senses, arguing that the data annotated in this way simplifies the difficult problem of sense-labeling of implicits.

To sum up, the supervised classification algorithms described above have been able to classify discourse relations, particularly explicit ones, very successfully in texts. But the results lack general impact, as less-studied, low-resource languages have hardly been targeted. Moreover, sense-labeling of implicit discourse relations still lags behind that of explicit relations. The next section deals with the impact of deep learning models in the field of discourse understanding as background to the experiments conducted in the current work.

3.2 Pre-trained language models: The paradigm shift in NLP domain

The advent of attention mechanism has been a breakthrough in NLP (Bahdanau, Kyunghyun, and Yoshua Reference Bahdanau, Kyunghyun and Yoshua2015; Luong, Pham, and Manning Reference Luong, Pham and Manning2015) which, consequently, gave rise to the transformer architecture (Vaswani et al. Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017). BERT is undoubtedly the most famous language model that is based on the transformer architecture (Devlin et al. Reference Devlin, Chang, Lee and Toutanova2019). BERT uses a masked language model which randomly masks 15% of the tokens from the input in order to predict the original vocabulary by feeding the final hidden vectors (corresponding to the masked tokens) into an output softmax over the vocabulary. Discourse relation classification benefited from BERT. For example, Nie, Bennett, and Goodman (Reference Nie, Bennett and Goodman2019) showed that BERT can outperform previous state-of-the-art models in implicit discourse relation classification, and Kishimoto, Murawaki, and Kurohashi (Reference Kishimoto, Murawaki and Kurohashi2020) adapt BERT to the task, managing to reach an F1-Score of 0.59.

3.3 Cross-lingual transfer learning

In recent years, cross-lingual transfer has become an active area of research in favor of low-resource languages. It is a fairly specific way of training models using the data available for languages with ample resources so that it can solve the same task in the target low-resource language(s). Cross-lingual transfer learning serves many research domains including the construction of bilingual dictionaries (Wang et al. Reference Wang, Zheng, Yu and Miao2019), zero-shot translation (Johnson et al. Reference Johnson, Schuster, Le, Krikun, Wu, Chen, Thorat, Viégas, Wattenberg, Corrado, Hughes and Dean2017), spoken language understanding (Yazdani and Henderson Reference Yazdani and Henderson2015), semantic utterance classification (Dauphin et al. Reference Dauphin, Tür, Tür and Heck2014), entity extraction from Web pages (Pasupat and Liang Reference Pasupat and Liang2014), fine-grained named entity typing (Ma, Cambria, and Gao Reference Ma, Cambria and Gao2016), cross-lingual document retrieval (Funaki and Nakayama Reference Funaki and Nakayama2015), relation extraction (Levy et al. Reference Levy, Seo, Choi and Zettlemoyer2017), multilingual task-oriented dialog (Schuster et al. Reference Schuster, Gupta, Shah and Lewis2019), and event detection (Caselli and Üstün Reference Caselli and Üstün2019).

A more specific case of transfer learning is known as zero-shot learning, where the classifier is able to classify examples it is never exposed to during the training. In the cross-lingual scenario, this often translates into training the classifier in one language and applying it to other languages with a minimal performance loss. The advantage of cross-lingual transfer learning, in general, is being able to leverage information from high-resource languages into the low-resource ones. In the case of discourse parsing, it is highly relevant as almost all of the non-English languages lack large manually annotated datasets.

4.1 Datasets

The datasets used in the paper involve the following: (i) The TDB 1.2, (ii) the PDTB 3.0, and (iii) the Turkish section of the TED-MDB.

4.1.1 Turkish Discourse Bank

The TDB 1.2 is chosen as the main data source. It is a 40,000-word discourse-level resource of Turkish created by manually annotating a corpus of modern Turkish texts written between 1990 and 2000 (Zeyrek and Webber Reference Zeyrek and Webber2008). The annotation principles of the TDB 1.2 and the annotated categories closely follow those of the PDTB; most notably, all discourse relation realization types described in Section 2.1 are spotted and annotated together with their binary arguments and senses, where relevant.

The basic principle in annotating explicitly marked relations is the PDTB’s minimality principle, that is, the annotators are asked to select the shortest text spans (e.g., clauses or sentences) that are necessary and sufficient to interpret a discourse relation encoded by a connective. The datasets over which the two corpora are built differ; while the PDTB is built over Wall Street Journal Corpus, the TDB is built over multiple genres (newspaper editorials, fiction, popular magazines). Other differences involve (i) the way different types of discourse connectives are annotated (e.g., suffixal connectives are annotated as a type of explicit connectives (Zeyrek and Basıbuyuk, Reference Zeyrek and Başıbüyük2019)) (ii) a small number of new Level-2 sense tags are spotted and annotated (Zeyrek and Kurfalı Reference Zeyrek and Kurfalı2017), and (iii) attribution and AltLexC relations have not been annotated in the TDB 1.2 so far.

In the annotation stage, any possible discourse relation is searched and annotated by going through the texts sentence by sentence, similar to the incremental processing of discourse. Annotations were performed by a team of trained graduate students, sustaining a minimum value of $\kappa$ = 0.7 for inter-annotator agreement (Zeyrek and Kurfalı Reference Zeyrek and Kurfalı2017). Even though this is below the normal threshold of 0.8, due to the ambiguity of coherence relations, the annotation task is very hard, forcing the team to take 0.7 as a satisfactory level as suggested by Spooren and Degand (Reference Spooren and Degand2010).

The TDB 1.2 contains 3987 discourse relations (Table 1), where discourse senses are annotated on the basis of the PDTB 3.0 sense tag-set (Appendix A) (Zeyrek and Er Reference Zeyrek and Er2022). Furthermore, explicit and implicit relations and AltLexes are annotated both at the inter-sentential and intra-sentential level. EntRels, NoRels, and Hypophora (13% of the corpus) are annotated only at the inter-sentential level.

Table 1. Distribution of DR types and Level-1 senses in the TDB 1.2 and the PDTB 3.0

4.2 Experimental setup

Each task is modeled as a multi-way classification task. As we do not assume a separate identification of arguments, each discourse relation is presented as one unit to BERT; hence, in practice, the models do not have any information regarding the boundaries of the discourse arguments but see them as one continuous text piece.

For each dataset, 10% of the data is allocated as the validation set and another 10% as the test set. The distribution of labels in each set is provided in Table 2. For each task, we fine-tune BERT following the standard practice suggested by Devlin et al. (Reference Devlin, Chang, Lee and Toutanova2019).

Table 2. Summary of the train, development, and test splits of the TDB 1.2 used in the experiments. The upper part presents the distribution of labels in the DR realization type classification experiments and the lower part presents the sense experiments over explicit and implicit DRs.

Following the previous work, we set the maximum sequence length to 128. We use AdamW optimizer (Loshchilov and Hutter Reference Loshchilov and Hutter2017) with the learning rate of $5e-5$ . We also apply a learning rate warm-up where the learning rate is linearly increased from 0 to $5e-5$ over the first 10% of iterations and then linearly decreases to 0. The models are fine-tuned for 50 epochs, with the batch sizes of 16. We apply early stopping with patienceFootnote ^d of 25 according to the performance on the development set. The models are evaluated four times in each epoch and the one with the best development performance is stored as the final model. As the target metric, we use Macro-F1. Yet, for each experiment, we report the performance of our models on the set via accuracy, recall, precision, Macro-F1, and Weighted F1,Footnote ^e calculated as:

(1a) $$ \begin{equation} Recall = \frac{TP}{TP+FN} \qquad Precision = \frac{TP}{TP+FP} \qquad F1 = \frac{2*Recall*Precision}{Precision + Recall} \end{equation}$$

(1b) $$ \begin{equation} \mbox{Macro-F1} = \frac{1}{N}\sum _i^N F1_i \quad \textit{(where N is the number of labels)} \end{equation}$$

(1c) $$ \begin{equation} \mbox{Weighted-F1} = \sum _i^N w_i \cdot F1_i \quad \textit{where} (w_i) \textit{is proportional to the frequency of class i} \end{equation}$$

All the training is performed on a single T4 GPU. The experiments for the development of a classification model gave rise to the Bert_MultiClass model, built based on the transformers.TFBertModel Footnote ^f class (More details about the classification model, the method of fine-tuning, and the tests are provided in Appendix B).