2.1 Notation

Consider two sets of textual documents—L, which includes $N^{\text {L}}$ documents labeled with a category number, and U, which includes $N^{\text {U}}$ unlabeled documents—where $N=N^{\text {L}}+N^{\text {U}}$ . When there is no ambiguity, we use i as a generic index for a document in either set and N as a generic description of either set size. Each document falls into category c in a set of mutually exclusive and exhaustive categories ( $c\in \{1,\dots ,C\}$ ), but the category label is only observed in the labeled set. We write $D_i = c$ to denote that document i falls into category c. Denote $N_c^{\text {L}} = \sum _{i=1}^{N^{\text {L}}} \mathbf {1}(D_i = c)$ as the number of documents in category c in the labeled set, $N^{\text {U}}_c$ as the (unobserved) number in c in the unlabeled set, and $N_c$ generically for either set in category c.

The proportion of unlabeled documents in category c is $\pi ^{\text {U}}_c=\operatorname {mean}_{i\in U}[\mathbf {1}(D_i = c)]$ (where for set A with cardinality $\#A$ , the mean over i of function $g(i)$ is $\operatorname {mean}_{i\in A}[g(i)]=\frac {1}{\#A}\sum _{i=1}^{\#A}g(i)$ ). The vector of proportions $\pi ^{\text {U}}\equiv \{\pi _1^{\text {U}},\dots ,\pi _C^{\text {U}}\}$ , which represents our quantity of interest, forms a simplex, that is, $\pi ^{\text {U}}_c \in [0,1]$ for each c and $\sum _{c=1}^C \pi _c^{\text {U}} = 1$ . We also define the analogous (but observed) category proportions for the labeled set $\pi ^{\text {L}}$ .

2.2 Text to Numbers

In this first step, we map the entire labeled and unlabeled corpora, with the document as the unit of analysis, into a constructed space of textual features with the definition of the rows (taking the place of the unit of analysis) a choice to be optimized. Many ways of performing this mapping can be created; we propose one optimized for quantification. For readme, Hopkins and King (Reference Hopkins and King2010) begin with a set of k unigrams, each a binary indicator for the presence (coded 1) or absence (0) of a chosen word or word stem in a document. The number of possible strings of these zeros and ones, called a word stem profile, is $W=2^k$ .

The readme approach then computes a $W\times 1$ feature vector, denoted $S^{\text {L}}$ , by sorting the labeled documents into the W mutually exclusive and exhaustive word stem profiles, and computing the proportion of documents that fall in each. To make the definition of $S^{\text {L}}=\{S_w^{\text {L}}\}$ more precise and easier to generalize later, begin with the $N^{\text {L}}\times W$ document-feature matrix $F=\{F_{iw}\}$ with rows for documents, and columns for features which in this case are unique word stem profiles. Each element of this matrix, $F_{iw}$ , is a binary indicator for whether document i is characterized by word stem profile w. Then, elements of $S^{\text {L}}$ are column means of F: $S_w^{\text {L}}=\operatorname {mean}_{i\in L}(F_{iw})$ . Then, the same procedure is applied, with the same word stem profiles, to the unlabeled set, which we denote $S^{\text {U}}$ .

We also define a $W\times 1$ conditional feature vector as $X_c^{\text {L}}=\{X_{wc}^{\text {L}}\}$ , which results from the application of the same procedure within category c in the labeled set, and $X_c^{\text {U}}=\{X_{wc}^{\text {U}}\}$ within category c in the unlabeled set ( $X_c^{\text {U}}$ is unobserved, because c is unknown in the unlabeled set). These can be computed from $F^c$ , a document-feature matrix representing only documents in category c. We then collect these vectors for all categories into two $W\times C$ matrices, $X^{\text {L}}=\{X_1^{\text {L}},\dots ,X_C^{\text {L}}\}$ and $X^{\text {U}}=\{X_1^{\text {U}},\dots ,X_C^{\text {U}}\}$ , respectively.

2.3 Assumptions

We require two assumptions. First, because the unlabeled conditional feature matrix $X^{\text {U}}$ is unobserved, Hopkins and King (Reference Hopkins and King2010) assume

(1) $$ \begin{align} X^{\text{L}}=X^{\text{U}}. \end{align} $$

Although Assumption 1 is quite restrictive, we can relax it. Begin by supposing that the documents in L are drawn from the same distribution giving rise to U—which is equivalent to the assumption necessary for individual classification.Footnote ³ It turns out we can relax this further: in readme, we could assume that the conditional distribution of features given categories is the same, which is much less restrictive. But we go even further and avoid specifying the full probabilistic model entirely by assuming only that the conditional expectation is the same, or in other words that the labeled conditional feature matrix is an unbiased estimator of the unlabeled conditional feature matrix. That is, we replace Assumption 1 with:

(2) $$ \begin{align} E(X^{\text{L}}) = X^{\text{U}}. \end{align} $$

Second, we must assume that matrix $X^{\text {L}}$ is of full rank, which translates substantively into (a) feature choices with $W>C$ and (b) the lack of perfect collinearity among the columns of $X^{\text {L}}$ . Assumption (a) is easy to control by generating a sufficient number of features from the text. Assumption (b) is only violated if the feature distributions in documents across different categories are identical, which is unlikely with a sufficient number of coded documents. (We prove below that high but not perfect collinearity, which can result if categories are weakly connected to the features or documents are labeled with error, can exacerbate the bias of the readme estimator.)

2.4 Estimation

Our goal is to estimate the vector of unlabeled set category proportions $\pi ^{\text {U}}=\{\pi _1^{\text {U}}, \dotsc , \pi _C^{\text {U}}\}$ given $S^{\text {L}}$ , $S^{\text {U}}$ , and $X^{\text {L}}$ . Begin with an accounting identity (i.e., true by definition), $ S_w^{\text {U}} = \sum _{c=1}^C X_{wc}^{\text {U}}\pi _c^{\text {U}},\; \forall \; w$ , or equivalently in matrix form:

(3) $$ \begin{align} S^{\text{U}}=X^{\text{U}}\pi^{\text{U}}, \end{align} $$

which is a linear regression, but where the linearity is a property rather than assumption. Then, readme estimates $\pi ^{\text {U}}$ by the least-squares estimator

(4) $$ \begin{align} \widehat{\pi^{\text{U}}} = (X^{\text{L}^{\prime}}X^{\text{L}})^{-1}X^{\text{L}^{\prime}}S^{\text{U}} \end{align} $$

(or a modified version that explicitly preserves the simplex constraint).Footnote ⁴

2.5 Properties

In the classic errors-in-variables linear regression model, with random measurement error only in the explanatory variables, least squares is biased and inconsistent. However, in readme, the dimensions of $X^{\text {L}}$ remain fixed as $N^{\text {L}}$ grows, and so Assumption 2 also implies measurement consistency: $\lim _{N^{\text {L}}\to \infty } X^{\text {L}} = X^{\text {U}}$ . This means that the readme estimator in Equation (4), which is also a linear regression with random measurement error in the explanatory variables, is statistically consistent: as we gather and code more documents for the labeled set (and keep W fixed, or at least growing slower than n), its estimator converges to the truth:

$$ \begin{align*}\lim_{N^{\text{L}}\to\infty}\widehat{\pi^{\text{U}}} = \lim_{N^{\text{L}}\to\infty}(X^{\text{L}^{\prime}}X^{\text{L}})^{-1}X^{\text{L}^{\prime}}S^{\text{U}}=(X^{\text{U}^{\prime}}X^{\text{U}})^{-1}X^{\text{U}^{\prime}}S^{\text{U}}=\pi^{\text{U}}.\end{align*} $$

This is a useful result suggesting that, unlike classic errors-in-variables, labeling more observations can reduce bias and variance. It also suggests that, to improve readme, we should focus on reducing finite sample bias rather than consistency, which is already guaranteed.

3.1 Issues to Address

Readme is affected by three issues that arise in practice. First is a specific type of “concept drift” (Gama et al. Reference Gama2014) that we refer to as semantic change—the difference in the meaning of language between the labeled and unlabeled sets. Authors and speakers frequently morph the semantic content of their prose to be clever, get attention, be expressive, curry political favor, evade detection, persuade, or rally the masses. For these or other purposes, the content, form, style, and meaning of every symbol, object, or action in human language can always evolve.

We address two types of semantic change that impact readme: emergent discourse, where new words and phrases, or the meanings of existing words and phrases, appear in the unlabeled set but not the labeled set, and vanishing discourse, where the words, phrases, and their meanings exist in the labeled set but not the unlabeled set. “Russian election hacking” following the 2016 U.S. presidential election is an example of emergent discourse, language which did not exist a few years before, whereas “Russian Communism” is an example of vanishing discourse, a usage that has been disappearing in recent decades. However, emergent and vanishing discourse can reverse their meanings if the researcher swaps which set is labeled. For example, in analyzing a large historical dataset, a researcher may find it more convenient to read and label a contemporary dataset and infer to the historical datasets (e.g., as they are recovered from an archive); to label documents at the start of the period and infer to subsequent periods; or to code a sample spread throughout the period and to infer to the full dataset. Either vanishing or emergent discourse can bias readme, but only if such discourse is present in a specific way (we describe below) across the categories.

Second is the lack of textual discrimination, where the language used in documents falling in different categories or across features is not clearly distinguishable. For clarity, we divide textual discrimination into two separate concepts that, in the readme regression, are related to the minimal requirement in least squares that, to reduce variance and model dependence, X must be of full rank and, ideally, far from degeneracy. Since X represents categories as variables and features of the text as rows, we refer to these two variables as category distinctiveness (CD) and feature distinctiveness (FD), respectively.

The lack of textual discrimination may arise, because the conceptual ideas underlying the chosen categories or features are not distinct. Hand coding errors can also lead to this problem, which is commonly revealed by low levels of intercoder reliability. The problem can also occur because of heterogeneity in how authors express information or a divergence between how authors of the documents express this information and how the analyst conceptualizes the categories or codes the features. Also common is where the analyst begins with distinct and well-defined conceptual definitions for the set of C categories, with examples of documents that fall unambiguously into each one, but where it turns out upon large-scale coding that large numbers of documents can only be described as falling into multiple categories. Adding categories to represent these more complicated expressions (so that the resulting set is still mutually exclusive and exhaustive) is a logical solution, but this step often leads to a more cognitively demanding hand-coding problem that reduces intercoder reliability.

A final problematic situation for readme occurs due to interactions with the other two problems. This issue is proportion divergence, when the category proportions in the labeled set ( $\pi ^{\text {L}}$ ) diverge from those in the unlabeled set ( $\pi ^{\text {U}}$ ). To understand this issue, consider a dataset with massive semantic change and no textual discrimination—so the document texts are largely uninformative—but where $\pi ^{\text {L}}\approx \pi ^{\text {U}}$ , such as when the labeled set is a random sample from the unlabeled set. In this situation, readme will return the observed proportion vector in the labeled set, $\pi ^{\text {L}}$ , which is a good estimate of $\pi ^{\text {U}}$ . This means that we can sometimes protect ourselves from semantic change and the lack of textual discrimination by selecting a labeled set with a similar set of category proportions as the unlabeled set. This protective measure is impossible to put into practice in general, as it requires a priori knowledge of category membership, but it can be useful in some cases when designing training sets, and we show below that it can be corrected for.

3.2 Analytical Solutions

We now propose improvements to readme, including (1) a dimension reduction technique for direct estimation of category proportions (analogous to techniques that have been used for improving classifier performance; e.g., Brunzell and Eriksson Reference Brunzell and Eriksson2000; Vincent et al. Reference Vincent, Larochelle, Lajoie, Bengio, Manzagol and Bottou2010) and (2) an adaptation of matching methods for causal inference. (We also show in Appendix D in the Supplementary Material how the methods developed here may have uses in the causal inference literature.) Figure 2 gives a schematic summary of step (1), and Figure 3 summarizes the whole algorithm including step (2).

Figure 2 Summary schematic of the quantification-targeted text processing in readme2.

Figure 3 Summary schematic of readme2. See Figure 3 for details of step (2). The matching step is done prior to the formation of $\underline {X}^L$ .

3.2.1 Dimension Reduction

Dimension reduction is especially important here, because numerical representation of text documents can have an outsized impact on the results (see Denny and Spirling Reference Denny and Spirling2018; Levy, Goldberg, and Dagan Reference Levy, Goldberg and Dagan2015). The idea behind our dimension reduction approach is that even if large numbers of individual features perform poorly, lower-dimensional linear combinations of features may do considerably better. We begin with the $N\times W$ document-feature matrix F defined in Section 2. Our goal then is to project this matrix to a lower-dimensional $N\times W^{\prime }$ document-feature matrix , where $\Gamma $ is a $W \times W^{\prime }$ matrix of transformation weights defined below and $W'\ll W$ . Once we obtain , we can take conditional expectations of the features given the category labels to generate the readme regression matrix as before. We denote conditional regression matrix obtained from as , in parallel to notation F and X.Footnote ⁵

The key question, then, is how to define $\Gamma $ to choose among the many possible lower-dimensional matrices and thereby reduce readme’s bias? To answer this question, we develop intuition by studying readme’s bias in simplified data with only two categories. Because of the simplex constraint, the unlabeled set category proportions can then be characterized by a single parameter, $\pi ^{\text {U}}_1$ . The accounting identity for each feature mean w, $S_w^{\text {U}}$ , can be written as $S_w^{\text {U}} = X_{w2}^{\text {U}} + B_w^{\text {U}}\pi ^{\text {U}}_1$ , where

(5) $$ \begin{align} B_w^{\text{U}}=X_{w1}^{\text{U}} - X_{w2}^{\text{U}} \end{align} $$

quantifies the systematic component of textual discrimination between the two categories. The readme estimator is then the least-squares estimator of $\pi ^{\text {U}}_1$ (see Appendix A in the Supplementary Material for proofs of all propositions in this section).

Proposition 1. The two-category readme estimator is

$$ \begin{align*} \widehat{\pi^{\text{U}}_1} = \frac{\sum_{w=1}^W B_w^{\text{L}}\left(S_w^{\text{U}} - X_{w2}^{\text{L}}\right)} {\sum_{w=1}^W (B_w^{\text{L}})^2}. \end{align*} $$

If $X^{\text {L}} = X^{\text {U}}$ , the above expression equals $\pi ^{\text {U}}_1$ and readme is unbiased. We can relax this assumption by making Assumption 2, which we express as $X_{wc}^{\text {L}} = X_{wc}^{\text {U}} + \epsilon _{wc}$ , where $\epsilon _{wc}$ is a random variable with mean zero and variance inversely proportional to $N_c$ . We then write the readme estimator in terms of $X^{\text {U}}$ , the true unlabeled set category proportion $\pi _1^{\text {U}}$ , and the sample category size $N_c^{\text {L}}$ , which we can use to obtain the approximate bias.

Proposition 2. The approximate bias of the readme estimator is

$$ \begin{align*} \text{Bias}\left(\widehat{\pi^{\text{U}}_1}\right) \approx \frac{\sum_{w=1}^W \left[\text{Var}(\epsilon_{w2}) - \text{Cov}(\epsilon_{w1}, \epsilon_{w2})\right] (1 - \pi_1^{\text{U}}) - \left[\text{Var}(\epsilon_{w1}) - \text{Cov}(\epsilon_{w1}, \epsilon_{w2})\right]\pi_1^{\text{U}}} {\sum_{w=1}^W \left[(B_w^{\text{U}})^2 + \text{Var}(\nu_w)\right]}, \end{align*} $$

with combined error variance $\nu _w=\epsilon _{w1} - \epsilon _{w2}$ .

Proposition 2 suggests several factors to consider when defining $\Gamma $ . As the systematic component of textual discrimination, $B_w^{\text {U}}$ , increases relative to the variance of the error terms, $\epsilon _{w1}$ and $\epsilon _{w2}$ , the bias approaches $0$ . In other words, readme works better with distinct language across categories. We should therefore define $\Gamma $ to maximize CD, the difference in conditional means across categories. We do this by directly generalizing to C categories Equation (5):

where the inequalities in the summations prevent double-counting. This expression is simply the sum of all the absolute values of all the $B_w^{\text {U}}$ terms.

Proposition 2 also suggests defining $\Gamma $ to increase FD, which involves making the rows of our regression matrix

closer to independent. To see why, consider how the numerator of the approximate bias contains the sum across all the features of the difference in the differences between the error variance of the categories and the covariance between the error variances (we can call this latter difference “excess variance”). If the features all were to capture the same latent quality, we would encounter a situation where the differences in excess variance would tend to go in the same direction (either all positive or all negative) across the features. Dependencies between one feature and another would increase the bias. However, when the features capture distinctive latent qualities of the text, the differences in excess variance between the categories will sometimes be positive and sometimes negative—so their sum will be closer to 0. We define this criterion, in parallel to CD, as follows:

where again the inequalities in the summations prevent double-counting.

We then choose $\Gamma $ by optimizing both CD and FD:

$$ \begin{align*} \Gamma^* = \underset{\Gamma \in \mathbb{R}^{W \times W'}}{\text{arg max}} \;\; \text{CD}(\Gamma) + \text{FD}(\Gamma). \end{align*} $$

As Appendix B in the Supplementary Material demonstrates, using only one of these two criteria alone does not optimize as desired. Thus, although we can apply readme in any feature space summary of the documents, using the lower-dimensional feature space defined by $\Gamma $ should reduce bias in readme.

3.2.2 Adapting Matching Methods for Text Analysis

Finally, we add one last insight by considering the special case of independence of measurement errors across categories (i.e., $\text {Cov}(\epsilon _{w1}, \epsilon _{w2}) = 0$ ). In this situation, readme bias is minimized when the following relationship holds between the labeled and unlabeled set category proportions.

Proposition 3. When measurement errors are independent across categories, the bias of readme is minimized at

$$ \begin{align*} \pi_1^{\text{L}} = \frac{\pi_1^{\text{U}} \sum_{w=1}^W \sigma^2_{w1}}{\pi_1^{\text{U}}\sum_{w=1}^W \sigma^2_{w1} + (1 - \pi_1^{\text{U}})\sum_{w=1}^W \sigma^2_{w2}}. \end{align*} $$

Thus, when the measurement error variances are roughly equivalent across categories, the bias of readme is minimized when proportion divergence is smallest.

We use this result to simultaneously reduce the biasing effect of vanishing discourse and to reduce proportion divergence. We do this borrowing the idea of matching from the causal inference literature (Ho et al. Reference Ho, Imai, King and Stuart2007; Iacus, King, and Porro Reference Iacus, King and Porro2012). In causal inference, matching “prunes” control observations from the data that have covariate values far from treated observations. In our case, we fix the unlabeled set and selectively prune the labeled set to remove documents with covariate profiles far from those in the unlabeled set.

To implement this idea, we search the labeled set L for a matched subset, $\mathcal {M}$ , that more closely resembles the unlabeled set. We do this without any information from the category labels (since they are unobserved in the unlabeled set). The goal is to remove observations from the labeled set that are so far in their content from the observations in the unlabeled set that they likely come from an entirely different data-generating process. If the texts of the documents are meaningful, exact one-to-one matching of documents in this way will eliminate all error, since it will mean that we have a hand code for each unlabeled document. In practice, approximate matching is usually needed, and any labeled unmatched documents are pruned and not used further. Our matching is in a space (of the transformed word vector summaries) designed for our specific problem; aligning the space of matching and the goals of the analysis are, of course, also key in matching for causal inference (King and Nielsen Reference King and Nielsen2017). Finally, we recompute and the matched , which we denote , and apply the readme regression.

As a result, the assumption in Equation (2) now needs only to hold in the matched subset of the labeled set rather than for the entire labeled set:

(6) $$ \begin{align} E[X^{\text{L}_{\mathcal{M}}}] = X^{\text{U}}. \end{align} $$

Matching thus considerably weakens the assumptions necessary for readme estimation and predictably reduces its bias.

4.1 Design

We performed 18 large-scale evaluations of our methodology, each following a different design protocol for allocating documents to membership in the labeled and unlabeled sets. Prior approaches in the literature have almost always used a single evaluation design, as compared to our 18, and only a few datasets, compared to our 73. The resulting 19,710 empirical evaluations in our replication data and code thus increase the rigor which future scholars can bring to bear on new methods developed to improve on those proposed here.

For each design protocol, we estimate readme2 and 32 alternative statistical methods that can be used to estimate category proportions (including readme). Each method is analyzed on 19,710 ( $=73 \times 15 \times 18$ ) datasets, because we have $73 $ corpora, $15 $ iterations per corpora per design, and 18 designs. The total number of method-design observations is therefore $630,720 $ ( $=19,710 \times 32$ ).

The 32 alternative methods of estimating category proportions are of five types. The first four types comprise six classifiers each run within each of the four possible combinations of (a) a discrete or continuous feature space and (b) a classification of whole documents and counting or averaging continuous probability estimates to yield estimates of the category proportions. The six classifiers include SVMs, random forests, Naive Bayes, and L1- and L2-regularized multinomial regression (James et al. Reference James, Witten, Hastie and Tibshirani2013), and an ensemble of these classifiers based on an average of classifiers within each of the two cells of (b). The fifth type of alternative method includes eight methods tuned for quantification. Among these, only readme is designed for more than two categories. We adapt the remaining seven—Friedman, Adjusted Counts, HDX, Median Sweep, Mixture HPMF, Mixture L1, and Mixture L2 (each detailed in Firat Reference Firat2016)—to multiple categories via estimation of repeated dichotomizations of the set of categories. When a method has adjustable parameters, we either optimized over them or used the software default.

Each of the 19,710 datasets we analyze, constructed as a subset of 1 of our 73 corpora, has a labeled out-of-sample test set that plays the role of the unlabeled set, except that we are able to use its labels after estimation to evaluate performance. The 73 corpora include three used in Hopkins and King (Reference Hopkins and King2010): (1) 1,426 emails drawn from the broader Enron Corporation corpus made public during the Federal Energy Regulatory Commission’s investigation of the firm’s bankruptcy and hand-coded by researchers into five categories; (2) a set of 462 newspaper editorials about immigration (with 3,618 word stems and 5 categories); and (3) a set with 1,938 blog posts about candidate Hillary Clinton from the 2008 presidential election (with 3,623 word stems and 7 categories). We also include 11,855 sentences (with 5 categories and 3,618 word stems) from the Stanford Sentiment Treebank (Socher et al. Reference Socher2013). Finally, we include 69 separate social media datasets (most from Twitter and a few from diverse blogs and Facebook posts), each created by a different political candidate, private company, nonprofit, or government agency for their own purposes, covering different time frames and categorization schemes (see Firat Reference Firat2016); these data cover 150–4,200 word stems, 3–12 categories, and 700–4,000 documents. (All data are in our replication dataset, except that for privacy reasons the raw text of the last set has been coded as numbers.)

Nearly all documents in the 73 corpora are labeled with a time stamp. For the empirical design, we randomly select a time point and pick the previous 300 documents as the labeled set and the next 300 documents as the out-of-sample evaluation set (wrapping in time if necessary). For each corpus, we repeat this process 50 times. This procedure keeps the evaluation highly realistic while also ensuring that we have many types of datasets with variation in proportion divergence, textual discrimination, and semantic change. The joint distribution of these quantities is crucial in determining the overall error dynamics, so accurately simulating this distribution is of the utmost importance in this exercise.

Although the empirical design emphasizes realism about what we see in practice, we replicate this analysis across the 18 designs described in Table 1 which make different assumptions about the joint distribution. Each of the 18 evaluation designs offers a different way of generating 19,710 datasets as subsets of the 73 corpora described in Section 4. Each dataset is divided into a labeled set as well as a test set that serves the purpose of the unlabeled set during estimation, but can also be used for evaluation, since all its document labels are observed.

Table 1 Alternative evaluation designs.

4.2 Implementation Details

For word embeddings, we use the 200-dimensional global word vector (“GloVe”) model, estimated from about 2 billion Twitter posts with a vocabulary of about 1.2 million terms. We follow standard approaches by summarizing the vectors, in our case with the $10{\text {th}}$ , $50{\text {th}}$ , and $90{\text {th}}$ quantiles of each dimension (Templeton and Kalita Reference Templeton and Kalita2018), resulting, for any given corpus, in F with $W = 200 \times 3 = 600$ unique features observed for each document. Because we are optimizing over these factors, we do not require the preprocessing steps for textual data (removing stop words, punctuation, etc.); instead, we are only constrained by the choice of the preprocessing for the original word embedding training, which in the case of GloVe means making the text all lower case. For optimization, we use stochastic gradient descent with momentum, minimizing overfitting with dropout, gradient clipping, normalizing to mean zero and variance $1$ , and using a standard confidence penalty (King, Gebbie, and Melosh Reference King, Gebbie and Melosh2019; Kingma and Ba Reference Kingma and Ba2017; Pereyra et al. Reference Pereyra2017; Srivastava et al. Reference Srivastava2014). We set the number of final features, $W^{\prime }$ , to $20$ , although performance does not greatly depend on this parameter as long as it is much smaller than the number of input features W.