1 Introduction

This study investigates the possible contribution of machine learning techniques to the coding of natural language transcripts from experiments. The aim is to evaluate whether simple tools from Natural Language Processing (NLP) and machine learning (ML) provide valid and economically viable assistance to the manual approach of coding even when complex concepts are coded.

In recent years, the analysis of communication has been an increasingly important element of many studies in economics. Communication transcripts are being consulted to understand behavior beyond what can be inferred from choice data and to obtain insights into team deliberation processes (e. g. Cooper and Kagel Reference Cooper and Kagel2005; Burchardi and Penczynski Reference Burchardi and Penczynski2014; Goeree and Yariv Reference Goeree and Yariv2011; Penczynski Reference Penczynski2016a). Computerized experiments make the collection of communication data very easy. And communication data are potentially very informative about reasoning processes. This strength, however, comes with the natural disadvantage that the coding of text—which is usually done manually—is time-intensive and based entirely on human judgment.Footnote ¹

Enabling the assistance of computers in the processing of natural language is the aim of the many different research fields of NLP, such as machine translation, question answering and speech recognition.Footnote ² A basic judgment of texts can be made with the help of simple statistics, such as message counts, word counts and word ranks. Moellers et al. (Reference Moellers, Normann and Snyder2017) fruitfully use those concepts when they experimentally investigate communication in vertical markets. More automated approaches like the Linguistic Inquiry and Word Count program (LIWC) group words in semantic classes such as positive or negative emotions, money, past tense etc. Abatayo et al. (Reference Abatayo, Lynham and Sherstyuk2017) analyse communication in cooperation experiments with the help of such software. This automation comes at the cost that “the semantic classes may or may not fit the theory being investigated” (Crowston et al. Reference Crowston, Allen and Heckman2012, p. 526). A closer fit with a specific economic theory and a higher level of automation can be achieved when statistical techniques such as ML use manually coded examples to build models of linguistic phenomena, an approach that I follow here.Footnote ³

Machine learning—or statistical learning—is a way of obtaining statistical models for prediction in large datasets. Due to the increasing importance of Big Data and variable selection, ML is making its way into the toolbox of econometricians and applied economists (Varian Reference Varian2014). For example, its strong out-of-sample prediction capabilities support causality studies by estimating policy implementation and counterfactuals (Mullainathan and Spiess Reference Mullainathan and Spiess2017). The computational handling of text data leads to datasets with many variables and makes these techniques appropriate.

Across the sciences, text analysis with the help of ML has become more popular in recent years. Physicians classify suicide notes and observe that the trained computer model outperforms experienced specialists in suicide predictions (Pestian et al. Reference Pestian, Nasrallah, Matykiewicz, Bennett and Leenaars2010). Linguists use ML to sift Twitter for useful information during mass emergencies (Verma et al. Reference Verma, Vieweg, Corvey, Palen, Martin, Palmer, Schram and Anderson2011). Based on large volumes of text such as party programs and speeches, political scientists use ML to locate politicians and parties in the political space, for example in the left-right spectrum (Benoit et al. Reference Benoit, Laver and Mikhaylov2009). Similarly, economists have used it to quantify the slant of media (Gentzkow and Shapiro Reference Gentzkow and Shapiro2010) or the consequences of transparency rules for central banks (Hansen et al. Reference Hansen, McMahon and Prat2017). To my knowledge, this is the first study to investigate this technique’s usefulness for experimental text data. A facilitating feature of experimental data is that the topic of the chat conversation is usually known which simplifies the machine learning analysis.Footnote ⁴

The communication transcripts studied here are obtained from implementations of Burchardi and Penczynski’s (Reference Burchardi and Penczynski2014) intra-team communication design in beauty contest, hide and seek, social learning and asymmetric-payoff coordination games. Among the applications in experimental work, the classification of reasoning in terms of the level-k model is certainly one of the more ambitious tasks.

Still, the results are clearly positive and show that the out-of-sample computer classification is able to replicate many results of the human classification. They suggest that in similar or easier classification tasks, computer classification can be a valid option to reduce the additional effort that comes with communication analyses, especially large ones. The following sections will introduce the data and the machine learning techniques that are used. Afterwards, results will be presented for three different applications. The technical appendix introduces the computational method based on an example code.

2 Data

All communication transcripts in this study are generated by the intra-team communication protocol that was introduced in Burchardi and Penczynski (Reference Burchardi and Penczynski2014). Teams of two subjects play as one entity and exchange arguments as follows. Both subjects individually make a suggested decision and write up a justifying message. Upon completion, this information is exchanged simultaneously and both subjects can enter individually a final decision. The computer draws randomly one final decision to be the team’s action in the game. The protocol has the advantage of recording the arguments of the individual player at the time of the decision making. Furthermore, the subject has incentives to convince his team partner of his reasoning as the partner determines the team action with 50% chance.

The original communication analyses have two research assistants (RA)—usually PhD or Master students—classify the messages according to a standard procedure of content analysis. From the authors of the study, they are provided written instructions as to which concepts to look for in the text. Initially, they code the messages individually in order not to be influenced by the opinion of the other. Afterwards, they meet or are informed about disagreements and have the chance to reconcile their classification. Finally, only the coding that the two RAs agree upon is entering the messages’ data analysis.

In all analyses of this study, the RAs looked for similar concepts described in the level-k model of strategic reasoning (Nagel Reference Nagel1995; Stahl and Wilson Reference Stahl and Wilson1995). RAs were asked to indicate the lower and upper bound of level of reasoning and in some cases the characteristics of the level-0 belief. Due to a possible ambiguity of messages with respect to the level of reasoning, lower and upper bounds are given that determine the interval within which the level of reasoning is likely to lie.

Here, three datasets will be used to investigate the usefulness of machine learning for the classification. Note that the studies were not chosen based on the particular characteristics of the games, but rather on the kinds of results to be replicated and the content extracted from the text, namely levels of reasoning and level-0 belief characteristics.

First, to see the general features of the computerized level classification, I unite observations from the beauty contest game in Burchardi and Penczynski (Reference Burchardi and Penczynski2014) with observations from the hide and seek game (Penczynski Reference Penczynski2016b). This dataset is referred to as BCHS.

The second, larger dataset is from a study of social learning (SL, Penczynski Reference Penczynski2017) and allows me to investigate whether one of the main results of the paper, namely that the mode behavior is level-2 (or “naïve inference” as in Eyster and Rabin Reference Eyster and Rabin2010), can be found via the computer classification. It features scenarios from the standard social learning framework as introduced by Anderson and Holt (Reference Anderson and Holt1997).

Finally, the third and largest dataset is from a study of asymmetric-payoff coordination games (APC) as investigated in van Elten and Penczynski (Reference van Elten and Penczynski2018) based on games introduced by Crawford, Gneezy, and Rottenstreich (Reference Crawford, Gneezy and Rottenstreich2008, CGR). Beyond the out-of-sample replication of the result that the incidence of level-k reasoning is low in symmetric, pure coordination games and high in asymmetric, “battle of the sexes”-type coordination games, this dataset allows me to go one step further and investigate the classification of level-0 beliefs. Specifically, it can be tested whether the computer classification replicates differences in the relevance of label and payoff salience between symmetric and asymmetric games.

3 Technique

The classification method studied here combines techniques of Natural Language Processing (NLP, Sect. 3.1) and machine learning (ML, Sect. 3.2). Appendix A provides further technical details and annotated example code in the software language R.

3.2 Machine learning

Due to the large number of independent variables T and the possibly nonlinear relationship between word frequencies and level of reasoning, standard linear regression approaches cannot be used. The statistic method of choice should feature a selection of variables and the ability to represent highly nonlinear relationships. The field of machine learning has available a large variety of algorithms for various purposes. Precedent cases of text analysis with random forests (Agrawal et al. Reference Agrawal, Gupta, Prabhu and Varma2013), the ease of their implementation and their general usefulness (Varian Reference Varian2014) let me choose the random forest technique (Breiman Reference Breiman2001; Hastie et al. Reference Hastie, Tibshirani and Friedman2008, henceforth HTF).Footnote ⁸ It does not require prior calibration and has featured good accuracy and little overfitting across applications.

Machine learning is generally used for out-of-sample prediction, in our case for the prediction of reasoning characteristics based on word counts in messages. The out-of-sample performance can be easily and precisely measured and is therefore the deciding measure of the usefulness of a model and guides many if not all of the choices of algorithms and parameters. It is thus indispensable to split the data into two separate sets for training and testing of the model.

For initial analyses and for a very simple linear model that relates the count of a particular token $x_t^m$ to the level of reasoning $y^m$ in message m, $f\left(x_t^m\right)=\beta ·x_t^m$ , I chose to have 70% of the observations to formulate the model in-sample (“train”) and the remaining 30% of observations to test the model out-of-sample.

The in-depth evaluation of the random forest results will make use of cross-validation. For 10 consecutive times, a specific 10% subset of the dataset is taken out for testing and the remaining 90% are used for training. The advantage of this more involved process is that eventually all observations will have been predicted based on a model that was trained exclusively on other observations. In all analyses, the in-sample vs. out-of-sample split is balanced across treatments/games to avoid that results vary due to differences in the number of training observations from particular treatments/games.

As in nature, the concept of a forest is conceptually based on the idea of “trees”. Trees partition the space spanned by the independent variables into subspaces. The splits are performed sequentially, dividing a dimension t along a split point $s_t$ into two subspaces, as shown in the illustrative tree and variable space in Fig. 1. For example, one could divide messages into those with less than one token “team”, $x_{\mathrm{team}}<1$ , and messages with more instances of “team”, $x_{\mathrm{team}}\ge 1$ . The first subspace, $x_{\mathrm{team}}<1$ , could be split again by $x_{\mathrm{urn}}<1$ and $x_{\mathrm{urn}}\ge 1$ , the second by $x_{\mathrm{saw}}<1$ and $x_{\mathrm{saw}}\ge 1$ . The online appendix A.4 gives details on how the trees are grown in random forests. To each subsample, one can now associate a level of reasoning ${\hat{y}}^{R_n}$ , as is done illustratively in Fig. 1a.

Fig. 1 Exemplary decision tree

Models in machine learning are fundamentally different depending on the nature of the dependent variable. With numerical dependent variables for which differences and means are defined like levels of reasoning, one speaks of a “regression model”. When the dependent variable takes a limited number of non-ordered values—“discrete variables” in economics—one speaks of a “classification model”.

A simple regression model reflects the response as a constant $c_n$ in each of the subspaces $R_n$ . The dependent variable y is predicted by

(1) $\begin{array}{c}f\left(x^m\right)=\sum _n{c}_{n}1(x^m\in R_n),\end{array}$

and the model error criterion is the mean squared error $Q_{\mathrm{mse}}=\frac{1}{M}{\sum }_m{(y^m-f\left(x^m\right))}^2$ . In the case at hand, the random forest algorithm grows 500 trees. In regression, the prediction for a message m of the collection of 500 trees is the average over all trees’ predictions, $f\left(x^m\right)=\frac{1}{500}{\sum }_{b=1}^{500}{f}^b\left(x^m\right)$ .

In classification models, the mean cannot be used for aggregation of outcomes in the subspaces. The mode outcome can and therefore the aggregation works like a ballot, each of the randomly generated trees casts one vote for its predicted category. The winner of the ballot turns into the model prediction for the message. In each subspace $R_n$ , the proportion of class d messages is ${\hat{p}}_{\mathrm{nd}}=\frac{1}{N_n}{\sum }_{m:x^m\in R_n}1(y^m=d)$ . The majority class d(n) in $R_n$ determines the response that the tree model attributes to a message, that is,

(2) $\begin{array}{c}f\left(x^m\right)=d(n:x^m\in R_n)=arg\underset{d}{max}({\hat{p}}_{\mathrm{nd}}:x^m\in R_n).\end{array}$

With 500 trees, the majority class d over all 500 trees is the prediction for $x^m$ .

In classification, various error criteria can be conceived. The misclassification error counts the number of misclassified messages and is thus intuitive but not differentiable. I will report the Gini impurity, which gives the error rate not for majority classification, but for a mixture model of classifying a randomly chosen observation in $R_n$ of category d into category $d^{\prime }$ with a probability that corresponds to the proportion ${\hat{p}}_{n{d}^{\prime }}$ : $Q_{\mathrm{Gini}}={\sum }_{d\ne d^{\prime }}{\hat{p}}_{\mathrm{nd}}{\hat{p}}_{n{d}^{\prime }}$ . This criterion measures dispersion in the categorization and is 0 if all messages in $R_n$ fall into one category.

In random forests, many uncorrelated trees are grown and then aggregated. “They can capture complex interactions structures in the data, and if grown sufficiently deep, have relatively low bias. Since trees are notoriously noisy, they benefit greatly from the averaging.” (HTF, p. 587f.).

While a single tree as in Fig. 1a is quite transparent about the modelled relationships, a forest clearly is not. Still, the structure of the model is representable by the so-called variable importance, which tracks over all trees the improvement in the model error thanks to each variable. The higher the reduction in the model error, the more important is the variable for the prediction of the model.

While the level-0 characteristics are discrete variables and hence treated in classification models, the level of reasoning can be treated in either regression or classification models. Given my understanding of levels of reasoning, I would probably see them as categories rather than typical numerical variables. However, in order to also treat and show regression models and results in this paper, I will report both regression and classification results for the levels of reasoning.

5 Economic viability

An important aspect of the presented coding exercise is its economic viability for a research project. What would be the costs and benefits of implementing machine learning?

Regarding the costs, the requirement of a training dataset implies that the manual coding effort cannot be fully substituted. For small projects of the size of the ones treated here the cost of human coding is moderate. Ultimately, the necessary size of the training set relates to the complexity and the quality of the machine coding. For the largest dataset APC, Table 11 shows how the quality of machine coding achieved with a training set of 762 observations (90% of the full dataset) can be achieved with smaller training sets.

Unfortunately, this result does not readily generalize since the determinants of the quality of machine coding can at this point not be identified. The statistical theory points to the number of independent variables, which would relate to the number of tokens and thus the variety of words used in the corpus. One can speculate further that the concepts to be looked for, the level of perceptibility in a given context and the language have a bearing on the model’s performance as well. One possible predictor of performance might be the agreement rate between human coders. Across the three datasets, APC featured the lowest pre-reconciliation agreement rate of 60%. The machine coding might therefore perform less well than in other datasets, holding constant the number of training observations.Footnote ¹⁷

Table 11 Coding performance of regression and classification models in APC ( $N=851$ ) depending on the size of the training set

Training fraction		0.9	0.8	0.7	0.6	0.5	0.4	0.3	0.2	0.1
Regression	$\rho$	0.68	0.67	0.68	0.66	0.66	0.65	0.65	0.63	0.57
	$R^2$	0.69	0.69	0.69	0.68	0.67	0.67	0.66	0.65	0.61
	Correct (%)	0.65	0.66	0.65	0.64	0.66	0.64	0.63	0.64	0.63
Classification	$\rho$	0.58	0.60	0.59	0.57	0.60	0.58	0.55	0.53	0.43
	$R^2$	0.59	0.60	0.59	0.58	0.59	0.58	0.55	0.52	0.40
	Correct (%)	0.66	0.67	0.67	0.65	0.67	0.66	0.65	0.64	0.61
Training set N		762	678	592	509	424	337	252	168	82
Testing set N		89	173	259	342	427	514	599	683	769

Results shown are averages of 10 independent runs of the same regression or classification

For the sake of a conservative cost calculation, let us assume that, for larger projects, the time and money spent on manual coding will not exceed the effort of coding 1000 messages. The extrapolated cost of coding 1000 messages are at the time of writing about 180 Euros and 12 RA student hours. With experimental datasets becoming larger as scientific standards improve and costs of experiments decrease— due to platforms such as Amazon MTurk—the mentioned cap can be valuable. Coding 10000 messages would have resulted in a cost of 1800 Euros and 120 RA student hours, a significant dent in the project’s money and time budget.

Beyond the availability of a training dataset, the costs of implementing machine learning as I present it here are relatively low. The software environment R as well as the required packages are freely available. Machine learning methods are quickly absorbed by quantitatively trained economists. Based on the exposition and references here as well as the example in Appendix A, I estimate that 3-5 researcher hours are enough to generate a first computer coding output. The statistical training of the model implies that the expertise of a linguist or NLP-trained analyst is not needed (Crowston et al. Reference Crowston, Allen and Heckman2012).

For large projects and for researchers that work frequently with text, these numbers suggest that the investment in machine learning expertise is highly economical. Other than reduced labor costs and reduced time of analysis, the computer approach has the additional potential to improve consistency where extended coding or the use of multiple coders jeopardize consistency. Some future developments might shift these numbers further in favor of the investment.

6 Outlook

Economists work with a finite set of concepts to be looked for in text. Linguists have developed off-the-shelf tools like sentiment analysis which do not need further training data and thus work without human coding. It is thus conceivable to eventually have enough training data and validated models for off-the-shelf tools that code strategic sophistication, lying aversion, social preferences, etc. Already now, the body of coded text and messages is considerable and could be used as manually-coded training data.Footnote ¹⁸

Certainly, more research is required to understand the scope of applications and research questions that can be investigated in this way. Since the present study investigates a rather complex phenomenon of strategic sophistication and aspires to code the degree of this sophistication, I view it as a relatively strong test of the feasibility of machine coding. The estimates given in the context of economic viability should be applicable in other coding tasks and possibly understate the benefits. Other concepts that have been studied with communication such as strategicness in Cooper and Kagel (Reference Cooper and Kagel2005) or the extent of social conversation in Abatayo et al. (Reference Abatayo, Lynham and Sherstyuk2017) are probably more easily coded in general, both manually and by machine coding.

An important facilitator in the current study is the researcher’s knowledge of the topic of discussion. In studies with field data, the topic of a text needs to be found out first (Hansen et al. Reference Hansen, McMahon and Prat2017), which imposes further costs of analysis. The control in laboratory studies makes the topic of conversations in a given text to be generally set by the game and thus known by the experimenter. This control makes it particularly simple for experimentalists to use the method shown here for coding.

More work is also required to understand possible differences between human and machine classification. Certainly, the conversion of text data into, here, a document-feature matrix risks losing information that is relevant for the theory at hand. Further, since the machine learning coding cannot easily be reconstructed and intuitively understood, it will require more studies and the input of linguists to clearly see the possibilities and limitations of machine coding.

Further, machine coding might not substitute but rather complement human classification. Existing or to-be-established off-the-shelf models for the coding of specific economic concepts can add evidence or a new perspective beyond the manual classification, as is done, for example in Moellers et al. (Reference Moellers, Normann and Snyder2017) and Abatayo et al. (Reference Abatayo, Lynham and Sherstyuk2017). Finally, the establishment of standard methods has the potential to improve the acceptance of rigorous qualitative analyses by sceptical quantitatively-minded economists.