5.1 Study design

This time, we kept the same empirical approach with an incomplete factorial design, but we modified the number of items answered by each participant. We chose 6 different blocks of 5 complete tweets of different length to compose 9 different questionnaires that included 10 tweets to evaluate. Annex I provides one example of each category. The questionnaires were as follows (Table 1).

Table 1. Distribution of tweets among the nine questionnaires

The survey form presented the tweets one at a time to prevent participants from anticipating the existence of some systematic pattern. The questionnaire included textual and paratextual information: the original display of the tweet as seen in the Twitter app was recreated to facilitate the right contextualization of the text. We also kept a yes/no question, but we changed the formulation and focused on the evaluation of the naturalness without asking participants to judge if they were directly written in Galician or MT into Galician. The new question was: can this tweet or thread be considered a natural tweet or thread in Galician? Because of this formulation, an explanation of the concept of linguistic naturalness was given at the beginning of the questionnaire. The explanation written in Galician said that we understood naturalness as a usual use of the language, with common expressions and a fluid style, free of grammar mistakes and influences from other languages such as Spanish. As the participants spoke both Spanish and Galician (both are official languages in the region, but Spanish is the dominant language), the absence of influence from the major language was willingly emphasized as part of the definition of naturalness. Since the survey targeted all kinds of Galician speakers, we asked Galician associations of all types to promote it among their affiliates and obtained answers from 218 participants.

The questionnaire included five demographic questions and 10 tweet evaluation questions. In the demographic questions, we asked about age, level of Galician linguistic competence, regular use of Galician, level of Twitter use in Galician, and level of Galician education received. To build the tweets sections, we took into account different types of tweets as shown in the previous table, which were classified into two broad categories. First, they were classified according to their origin: original text if the text was written in Galician, and MT if the text was machine translated from Spanish into Galician using our NMT. Second, they were classified according to their length: short sentence, long sentence, paragraph composed of short sentences, paragraph composed of long sentences, and threads.

The end-user perception results do not allow us to conclude whether MT was accurate, so all machine-translated tweets were also subjected to human evaluation. We performed an error evaluation using the MQM-DQF framework (Lommel et al., Reference Lommel and Melby2018), which was carried out by three professional linguists with over ten years of experience using the online platform ContentQuo.Footnote f The goal of this study was to find a link between errors and a negative attitude toward the machine-translated tweet. We are not going to give in-depth details about this evaluation because it is not the objective of this paper, nevertheless, this was carried out this evaluation because we wanted to ensure that the translation perceived as natural does not have critical errors. Although some errors were found, the raw NMT tweets obtained a DQF rate of over 90% (see do Campo et al., Reference do Campo and Sánchez-Gijón2023).

5.2 The non-inferiority model

Our statistical approximation, the non-inferiority principle (Tunes da Silva et al., Reference Tunes da Silva, Logan and Klein2009), followed the same parameters as the pilot study: a generalized linear mixed model with a binomial distribution, tweets (ITEM), and participants (ID) as crossed random effects, and using the tweet characteristics as fixed factors. Here, $p$ is the proportion of success (positives), $\alpha$ is the independent term of the model, $\beta$ is the vector of coefficients associated with the explanatory variables, $X$ is the design matrix of the explanatory variables, $a_i$ is the random effect associated with the items, following $N\big(0, \sigma _{\text{ITEM}}^2\big)$ and $b_j$ is the random effect associated with the subjects, following $N\big(0, \sigma _{\text{ID}}^2\big)$ .

(1) \begin{equation} \begin{aligned} \log\!\left (\frac{p}{1-p}\right ) &= \alpha + \beta X + a_i + b_j \\[5pt] a_i &\sim N\big(0, \sigma _{\text{ITEM}}^2\big) \\[5pt] b_j &\sim N\big(0, \sigma _{\text{ID}}^2\big) \end{aligned} \end{equation}

In the final evaluation, we used a generalized linear mixed model with a binomial distribution and a logit link to compare the naturalness perception of the machine-translated tweets into Galician and the tweets directly written in Galician (Agresti, Reference Agresti2015; Zuur et al., Reference Zuur, Ieno, Walker, Saveliev and Smith2009). This model randomly crossed the tweets (Item) and participants (ID), using the tweet characteristics as fixed factors. We also tested the perception of naturalness according to age or social network usage. Because different tweets are evaluated by different subjects, the model included subject and tweet-specific random effects to account for the evaluation of different tweets by different subjects. Hence, $p$ is the success proportion (positive), $\alpha$ is the independent term of the model, $\beta$ is the coefficient associated with the variable Original, $\gamma _j$ is the coefficient associated with category $j$ of the variable Type of Tweet, $\beta \gamma _j$ is the coefficient associated with category $j$ of the interaction between Original and Type of Tweet, $a_i$ is the random effect associated with items, with variability $\sigma ^2_{\text{item}}$ , $b_j$ is the random effect associated with subjects, with variability $\sigma ^2_{\text{id}}$ .

(2) \begin{equation} \begin{aligned} N_{ij}[k] &\sim \text{Bin}(1, p_{ij}[k]) \\[5pt] \log\!\left (\frac{p_{ij}[k]}{1-p_{ij}[k]}\right ) &= \mu + \alpha _i^{Id} + \alpha _{j[k]}^{Tw} + \beta O_Y + \gamma _j \text{Type}_j + (\beta \gamma )_j (O_Y \times \text{Type}_j) \\[5pt] \alpha _i^{Id} &\sim N(0, \sigma _{(\alpha ^{Id})^2}) \quad \text{for } i = 1, \ldots, I \\[5pt] \alpha _{j[k]}^{Tw} &\sim N(0, \sigma _{(\alpha ^{Tw})^2}) \quad \text{for } j = 1, \ldots, J, \quad k = 1, \ldots, K_j \end{aligned} \end{equation}

Finally, a detailed comparison of the naturalness perception of machine-translated tweets into Galician and directly written tweets was made, considering a one-sided non-inferiority test with a non-inferiority limit of 10% points. The selection of this margin is critical and represents one of the main challenges in non-inferiority trials, as there is no universal value. Regulatory agencies provide guidance on how to choose this margin but do not give specific values. Molina Nadal (Reference Molina Nadal2020) states that in most of the cases, a 30% margin is decided, while other authors established a 5% margin (Althunian et al., Reference Althunian, de Boer, Groenwold and Klungel2017) or 10% (Senn, Reference Senn2000). So for the purpose of this study, we decided to reduce the non-inferiority margin used in the pilot study to 10%, to be as close as most pharmaceutical trials.

Table 2. Distribution of responses among the nine surveys

6.1 Participants’ description

From the 228 Twitter users’ responses, some were excluded. Two participants were disqualified because their response time exceeded 70 minutes. Four participants were disqualified because they did not recognize any of the tweets as being original. Four more participants were ruled out because they were under the age of 18. The texts used in the surveys covered topics familiar to adults, for example, bureaucracy, which would be unusual to young people. This meant that any answers given by minors were discarded. Finally, 218 people contributed to the analysis database.

In terms of participant age, we found a similar distribution among age groups except for people over 65 years old, which is likely due to this group’s lack of experience or usage of Twitter. We can then say that we have a true picture of the social media age engagement and is in line with our study as we want to evaluate end-user perceptions (see Table 3).

Table 3. Distribution of responses among age groups

In terms of Galician linguistic competence, all participants were native speakers. In terms of written competence, we considered their level of certified education. Only 3% of the sample population had no accredited education. 15.6% of participants were accredited with primary education. Participants with secondary education or a university degree predominated (61.9%), followed by participants with a specific language training such as professors, reviewers, etc. (19.3%). When asked about Galician in their daily lives, the majority of responders claimed that they almost always use Galician (40.8%). Then, there is another significant proportion of people who use primarily Spanish (22.5%), followed by those who use both languages interchangeably (17.9%) or primarily Galician (15.6%). Only a few people never use Galician (3.21%). These percentages are similar to those obtained when participants were asked in which language they write their tweets. Finally, when asked about their education in Galician, only 12.4% of the participants had not received any formal education in Galician. These were control questions used to determine the profile of our respondents, as well as to validate their answers based on their knowledge.

Table 4. Tweets judged natural

6.2 Description of the tweets and responses collected

Tweets originally written in Galician and tweets translated with our NMT engine were distributed equally but in different proportions across the 9 questionnaires. In Table 4, we can advance the direct responses collected regarding the naturalness of each of the tweets.

A priori, we might expect that tweets written in Galician (red column of Figure 1) would be perceived as more natural. However, it is noteworthy that they are never perceived as 100% natural and that the NMT tweets figures (blue columns) are close to the originally written texts, sometimes exceeding them.

Figure 1. Tweets judged natural.

Absolute figures only show a tendency, which has to be proven or disproved. In this study, the principle of non-inferiority is used as the statistical method of analysis for this purpose, and the results are presented in the following sections.

6.3 Bivariate analysis

We compared the tweet naturalness rates with the different variables (questionnaires, age group, language knowledge, language use, language use on Twitter, and education) to search for anomalies that could lead us to failures in the model. We only found significant differences in two variables: the different questionnaires and the group age. Some questionnaires gave a higher rate of acceptance than others, but the rates obtained were not very distant, which could be due to a better translation of the NMT engine or the lack of relevant errors. On the other hand, groups aged 18 to 25 (85.3%) and 35 to 45 (83.0%) were the most likely to accept a machine-translated tweet as the original. People aged 45 to 55, on the other hand, were the most demanding.

6.4 Non-inferiority analysis

The experiment design is reflected in this non-inferiority model. The model assumes that different questionnaires include shared tweets, so some tweets are evaluated in more than one questionnaire. Furthermore, each tweet falls into a unique category of all possible categories (mathematically represented as J). As we wanted to be as thorough as possible in our analysis by taking this into account, with this model, we can state the global non-inferiority of our engine based on a p-value of 0.024 (see Table 5).

Table 5. Global non-inferiority rate

Through the model used, we contrasted whether the proportion of MT tweets perceived as natural is non-inferior compared to the proportion of originally written tweets considered as natural. In other words, it consists of demonstrating that the proportion of natural tweets in machine-translated tweets is higher than the proportion of natural tweets in original tweets, minus a non-inferiority margin.Footnote g The unilateral p-value and the statistic Z confirms the non-inferiority.

The non-inferiority test is designed to show that the observed odds ratio is greater than the odds ratio that would occur under the null hypothesis. The statistic Z presented in the table is the score corresponding to this test of non-inferiority, as is the p-value shown as 0.024. If the p-value is below 0.05, the non-inferiority test refutes the hypothesis that the NMT is worse and supports the hypothesis that the NMT is non-inferior. To complete the interpretation, the odds ratio is shown. This is a measure of the differences between the percentages that result from the model used (see previous section). The odds ratio is also shown under the null hypothesis.

To gain a deeper understanding of our engine’s performance strengths and weaknesses, we replicated this analysis while accounting for the variable of tweet type. Taking a closer look at the different length of the tweets, we found a different behavior and two clear groups.

As shown in Table 6, NMT tweets consisting of short paragraphs and threads had the lowest probability of being considered not inferior to originally written tweets (short paragraphs: p-value of 0.952 and z-ratio of −1.669; threads: p-value of 0.589 and z-ratio of −0.226), as can also be seen in Figure 1. Indeed, this figure shows that originally written tweets were perceived as more natural than NMT tweets. One reason for explaining this in tweets formed by short paragraphs could be that those tweets had a more cryptic style, similar to a telegram, jeopardizing the naturalness and fluency, and that paratextual context cannot compensate. In terms of threads, we can expect that even if they were treated as a complete section in our preprocessing, our engine is not context-aware, so cohesion will be disregarded. In other words, providing a longer and more coherent source text that contextualizes each sentence does not necessarily result in a more coherent MT. Moreover, following the MQM framework, more minor errors of fluency, precision, and terminology were found in this category.

Table 6. Non-inferiority rate by type of tweet

In contrast, tweets formed by a long sentence or by a long paragraph were accepted as the most natural (long sentence: p-value of 0.015 and z-ratio 2.181; long paragraph: p-value of 0.001 and z-ratio of 3.268). Surprisingly, tweets formed by a long paragraph were accepted more than the same category of the tweets directly written in Galician, a result which was also found in the previous pilot study. One possible reason for such results could be that participants have more context and a complete text in long NMT sentences and paragraphs, whereas originally written texts could be shortened trying to adjust to the character restriction of Twitter. This would mean that, by translating sentences with a greater degree of syntactic complexity, the engine will achieve a more coherent translation. In other words, each phrase of the sentence is contextualized by the rest of the sentence, so that the result of MT allows for a better perception of naturalness. Tweets consisting of short sentences were closer to being considered non-inferior than tweets consisting of short paragraphs (p-value of 0.157 and z-ratio of 1.007). NMT short tweets were perceived as slightly more natural than originally written texts (see Table 4). The acceptance rate was higher than in the previous study, which could be a direct result of the NMT engine’s retraining. Nonetheless, this short-sentence weakness is a well-known drawback of MT systems, which is usually explained by a lack of context. There is no clear indication that the text is natural here, especially if these tweets are presented alone. As a matter of fact, we took a closer look at these tweets to see if we could draw any additional conclusions. These tweets are grammatically correct, but they appear to be closer to typical Spanish syntax than long texts because there are fewer unique expressions to distinguish them. Due to the close relationship between Spanish and Galician, this may be interpreted as less natural in Galician, hence, we can declare the significant non-inferiority only in two categories: long sentences and long paragraphs. For the other three categories, differences are not significant, so non-inferiority cannot be stated.

6.5 Variations to the model

We experimented with the way responses and the nature of the tweet were interpreted to gain a better understanding of our analysis model. We repeated our analysis without taking into account the random effects. First, we modified one parameter of our formula: each participant responds to a block of 10 tweets; however, the blocks of tweets are not identical across the nine questionnaires. So, the model still takes into account the participant effect but considers each tweet as unique. Here, $p$ is the proportion of success (positives, in our case the tweets that are perceived as natural), $\alpha$ is the independent term of the model, $\beta$ is the coefficient associated with the variable Original, $\gamma _j$ is the coefficient associated with category $j$ of the variable Type of Tweet, $\beta \gamma _j$ is the coefficient associated with category $j$ of the interaction between Original and Type of Tweet, and $a_i$ is the random effect associated with items, with variability $\sigma ^2_{\text{item}}$ . Although the p-values in this analysis were lower than in the previous one, the non-inferiority was achieved in the same tweet categories. This is the modified formula:

(3) \begin{equation} [H] \begin{aligned} N_{ij} &\sim \text{Bin}(1, p_{ij}) \\[5pt] \log\!\left (\frac{p_{ij}}{1-p_{ij}}\right ) &= \mu + \alpha _i^{Id} + \beta O_Y + \gamma _j \text{Type}_j + (\beta \gamma )_j (O_Y \times \text{Type}_j) \\[5pt] \alpha _i^{Id} &\sim N(0, \sigma _{(\alpha ^{Id})^2}) \quad \text{for } i = 1, \ldots, I \end{aligned} \end{equation}

Second, we went one step further and removed the participant effect from the formula. This way we were able to evaluate our responses using a model that assumes all answers and tweets are unique.

(4) \begin{equation} [H] \begin{aligned} N_{ij} &\sim \text{Bin}(1, p_{ij}) \\[5pt] \log\!\left (\frac{p_{ij}}{1-p_{ij}}\right ) &= \mu + \beta O_Y + \gamma _j \text{Type}_j + (\beta \gamma )_j (O_Y \times \text{Type}_j) \end{aligned} \end{equation}

In this case, the p-values were low enough to state non-inferiority in tweets composed of short sentences (see Table 7).

Table 7. Non-inferiority rate per type of tweet without random effects

With these changes, we hoped to demonstrate that our model took into account the nature of our design in order to interpret our responses consistently. Nonetheless, this example provides researchers with additional options for interpreting their database, particularly if they do not have a large sample of answers or tweets. The model can be adapted to the particular needs of their language combination and sample population.