1 Why care about hyperparameters?

When political scientists work with machine learning models, they want to find a model that generalizes well from training data to new, unseen data.Footnote ¹ Hyperparameters play a key role in this endeavor because they determine the models’ capacity to generalize. Finding a good set of hyperparameters critically affects conclusions about a model's performance. The failure to correctly tune and report hyperparameters has recently been identified as a key impediment to the accumulation of knowledge in computer science (e.g. Henderson et al., Reference Henderson, Islam, Bachman, Pineau, Precup and Meger2018; Melis et al., Reference Melis, Dyer and Blunsom2018; Bouthillier et al., Reference Bouthillier, Laurent and Vincent2019, Reference Bouthillier, Delaunay, Bronzi, Trofimov, Nichyporuk, Szeto, Sepahvand, Raff, Madan, Voleti, Kahou, Michalski, Arbel, Pal, Varoquaux and Vincent2021; Cooper et al., Reference Cooper, Lu, Forde and De Sa2021; Gundersen et al., Reference Gundersen, Coakley, Kirkpatrick and Gil2023). Is political science making the same mistake?

We examined 64 machine learning-related papers published between 1 January 2016 and 20 October 2021 in some of the top journals of our discipline—the American Political Science Review (APSR), Political Analysis (PA), and Political Science Research and Methods (PSRM). Of the 64 publications we analyzed, 36 (56.25 percent) do not report the values of their hyperparameters, neither in the paper nor the appendix. Forty-nine publications (76.56 percent) do not share information about how they used tuning to find the values of their hyperparameters. Only 13 publications (20.31 percent) offer a complete account of the hyperparameters and their tuning. Not being transparent is a dangerous habit because readers and reviewers cannot assess the quality of a manuscript without access to the replication code.

With this paper, therefore, we raise the awareness that hyperparameters and their tuning matter. In statistical inference, the goal is to estimate the value of an unknowable population parameter. Including robustness checks in a paper and its appendix is good practice, allowing others to understand critical choices in research design and statistical modeling. The actual out-of-sample performance of a machine learning model is such an unknown quantity, too. We suggest handling estimates of population parameters and hyperparameters in machine learning models with the same loving care.

First, we explain what hyperparameters are and why they are essential. Second, we show why it is dangerous not to be transparent about hyperparameters. Third, we offer best practice advice about properly selecting hyperparameters. Finally, we illustrate our points by comparing the performance of several machine learning models to predict electoral violence from tweets (Muchlinski et al., Reference Muchlinski, Yang, Birch, Macdonald and Ounis2021).

2 What are hyperparameters and why do they need to be tuned?

Many machine learning models have parameters and also hyperparameters. Model parameters are learned during training, and hyperparameters are typically set before training. Hyperparameters determine how and what a model can learn and how well the model will perform on out-of-sample data. Hyperparameters are thus situated at a meta-level above the models themselves.

Consider the following stylized example displayed in Figure 1.Footnote ² A linear regression approach could model the relationship between X and Y as $\hat Y = \beta _0 + \beta _1 X$. A more flexible model would include additional polynomials in X. For example, choosing λ = 2 encodes the theoretical belief that Y is best predicted by a quadratic function of X, i.e., $\hat Y = \beta _0 + \beta _1 X + \beta _2 X^2$. But it is also possible to rely on data only to find the optimal value of λ. Measuring the generalization error with a metric like the mean squared error helps empirically select the most promising value of λ.

Figure 1. Example with polynomial regression. Data X ~ N(0, 1). Data generating process: $Y = 1 + X + 0.8X^2 + 0.3X^3 + \epsilon$, with $\epsilon \sim N( 0,\; 2)$. Regression Line for Bivariate OLS Model in Blue. Regression Curve for Polynomial Regression with λ = 3 in Teal.

This polynomial regression comes with both parameters and hyperparameters. Parameters are variables that belong to the model itself, in our example, the regression equation coefficients. Hyperparameters are those variables that help specify the exact model. In the context of the polynomial regression, λ is the hyperparameter that determines how many parameters will be learned (Goodfellow et al., Reference Goodfellow, Bengio and Courville2016). Machine learning models can, of course, come with many more hyperparameters that relate not only to the exact parameterization of the machine learning model. Anything part of the function that maps the data to a performance measure and that can be set to different values can be considered a hyperparameter, e.g., the choice and settings of a kernel in a support vector machine (SVM), the number of trees in a random forest (RF), or the choice of a particular optimization algorithm.

3 Misselecting hyperparameters

Research on machine learning has recently identified several problems that may arise from handling hyperparameters without care. The failure to report the chosen hyperparameters impedes scientific progress (Henderson et al., Reference Henderson, Islam, Bachman, Pineau, Precup and Meger2018; Bouthillier et al., Reference Bouthillier, Laurent and Vincent2019, Reference Bouthillier, Delaunay, Bronzi, Trofimov, Nichyporuk, Szeto, Sepahvand, Raff, Madan, Voleti, Kahou, Michalski, Arbel, Pal, Varoquaux and Vincent2021; Gundersen et al., Reference Gundersen, Coakley, Kirkpatrick and Gil2023). In the face of a hyperparameter space marked by the curse of dimensionality, other researchers can only replicate published work if they know the hyperparameters used in the original study (Sculley et al., Reference Sculley, Snoek, Wiltschko and Rahimi2018). In addition, it is essential to tune the hyperparameters of all models, including baseline models. Without such tuning, it is impossible to compare the performance of two different models M _a and M _b: While some may find that the performance of M _a is better than M _b, others replicating the study with different hyperparameter settings could conclude the opposite: that indeed M _a is not better than that of M _b. Such “hyperparameter deception” (Cooper et al., Reference Cooper, Lu, Forde and De Sa2021) has confused scientific progress in various subfields in computer science where machine learning plays a key role, including natural language processing (Melis et al., Reference Melis, Dyer and Blunsom2018), computer vision (Musgrave et al., Reference Musgrave, Belongie and Lim2020), and generative models (Lucic et al., Reference Lucic, Kurach, Michalski, Gelly and Bousquet2018). Reviewers and readers need to comprehend the hyperparameter tuning to assess whether a new model reliably performs better or whether a study tests new hyperparameters (Cooper et al., Reference Cooper, Lu, Forde and De Sa2021).

It is good to see political scientists also discuss and stress the relevance of hyperparameter tuning in their work (e.g., Cranmer and Desmarais, Reference Cranmer and Desmarais2017; Fariss and Jones, Reference Fariss and Jones2018; Chang and Masterson, Reference Chang and Masterson2020; Miller et al., Reference Miller, Linder and Mebane2020; Rheault and Cochrane, Reference Rheault and Cochrane2020; Torres and Francisco, Reference Torres and Francisco2021). But does the broader political science community fulfill the requirements suggested in the computer science literature? To understand how hyperparameters are used in the discipline, we searched for the term “machine learning” in all papers published in APSR, PA, and PSRM after 1 January 2016 and before 20 October 2021. Suppose a paper applies a machine learning model with tunable hyperparameters. In that case, we first annotate whether the authors report the final values of hyperparameters for all models in their paper or its appendix.Footnote ³ We also record whether authors transparently describe how they tuned hyperparameters.Footnote ⁴ Table 1 summarizes the findings from our annotations. We find that 34 (53.12 percent) publications neither report the values of the final hyperparameters nor the tuning regime in the publication or its appendix. Another 15 publications (23.44 percent) offer information about the final hyperparameter values but not how they tuned the machine learning models. In two cases (3.12 percent), we find no information about the final values of the hyperparameters but about the tuning regime. Finally, only 13 publications (20.31 percent) offer a full account of both the final choice of the hyperparameters and the way the tuning occurred in either the paper itself or its appendix.

Table 1. Can readers of a publication learn how hyperparameters were tuned and what hyperparameters were ultimately chosen? Hyperparameter explanations in papers published in APSR, PA, and PSRM between 1 January 2016 and 20 October 2021

Note that we annotated the literature in a way that helps understand whether reviewers and readers can assess the robustness of the analyses based on the manuscript and its appendix. Our analysis does not consider the replication code since it typically does not find consideration in the review process. In addition, we do not make any judgments about correctness. A paper without information about hyperparameter values or their tuning can still be correct. Similarly, a paper that reports hyperparameter values and a complete account of the tuning can still be wrong. It is the realm of reviewers to evaluate the quality of a manuscript. But without a complete account of hyperparameter values and tuning, readers and, in particular, reviewers cannot judge whether hyperparameter tuning is technically sound.

4 Best practice

Hyperparameters are a fundamental element of machine learning models. Documenting their careful selection helps build trust in the insights gained from machine learning models.

4.2 Illustration: Comparing machine learning models to predict electoral violence from tweets

To illustrate our point, we compare machine learning models trained to predict electoral violence from tweets. Muchlinski et al. (Reference Muchlinski, Yang, Birch, Macdonald and Ounis2021) collected Tweets around elections in three countries (Ghana, the Philippines, and Venezuela) and annotated whether these messages described occurrences of electoral violence. We re-scraped the data based on the shared Tweet IDs. To predict these occurrences from the content of these Tweets, we use four different machine learning models—a naive Bayes classifier (NB), random forest (RF), a support vector machine (SVM), and a convolutional neural network (CNN).

Table 2 summarizes our results. In the left column of each country, we report the results from training the models with default hyperparameters. On the right, we show the results after hyperparameter tuning.Footnote ⁹ Hyperparameter tuning improves the out-of-sample performance for most machine learning models in our experiment.Footnote ¹⁰ Table 2 also shows how easy it is to be deceived about the relative performance of different models—if hyperparameters are not properly tuned. The performance gains from tuning are so substantial that most tuned models outperform any other model with default hyperparameters. In the case of Venezuela, for example, comparing a tuned model with all other baseline models at their default hyperparameter settings could lead to different conclusions. Researchers could mistakenly conclude that (a tuned) NB classifier (F1 = 0.308) is at eye-level with a CNN model (F1 = 0.319) and better than any other method; or also that the RF is the better model (F1 = 0.479), or the SVM (F1 = 0.465), or the CNN (F1 = 0.304). In short, model comparisons and model choices are only meaningful if all hyperparameters of all models are systematically tuned and if this tuning is transparently documented.

Table 2. Performance benchmarking of Muchlinski et al. (Reference Muchlinski, Yang, Birch, Macdonald and Ounis2021) on different classifiers using our scraped data.

On the left: results with default values for the hyperparameters. On the right: results from tuned hyperparameters

5 Tuning hyperparameters matters

Hyperparameters critically influence how well machine learning models perform on unseen, out-of-sample data. Despite the relevance of tuned hyperparameters, we found that only 20.31 percent of the papers using machine learning models published in APSR, PA, and PSRM between 2016 and 2021 include information about the ultimate hyperparameter choice and how they were found in the manuscript or the appendix. Furthermore, 34 papers (53.12 percent) neither report the hyperparameters nor their tuning. This is a dangerous habit since handling hyperparameters without care can lead to wrong conclusions about model performance and model choice.

The search for an optimal set of hyperparameters is a vibrant research area in computer science and statistics. For most of the applications in our discipline, acknowledging and discussing how the choice of hyperparameters could influence results in combination with a proper and systematic search for appropriate hyperparameters would go a long way. It would allow others to understand original work, assess its validity, and thus ultimately help build trust in political science that uses machine learning.