Main concepts

A meta-evaluation is an evaluation of evaluations – impact evaluations in this case. Impact evaluations compare the situation of individuals exposed to a policy intervention with the situation of those not exposed to it, ceteris paribus.

In impact evaluations, different techniques are used to measure the net effect of active policies on a treatment group relative to a control group with statistically similar characteristics. Such techniques include randomised controlled trials, matching, difference in differences, discontinuous regression and instrumental variables. Under certain assumptions, these techniques ensure that control and treatment groups (non-beneficiaries and beneficiaries, respectively) have similar characteristics, making it possible to measure the net effect or average treatment effect of a policy.

Our meta-evaluation allows us to aggregate and compare the findings of a series of impact evaluations on unemployed people over 50. To be included in the meta-evaluation, studies must meet two fundamental conditions. First, they must be conceptually comparable; that is, they must study the same policy for the same population – in this case, active policies for unemployed people over 50. Second, they must use similar statistical methods and present their results in a similar way (Lipsey and Wilson, Reference Lipsey and Wilson2000). In particular, they must be impact evaluations in the sense that active policy beneficiaries are compared with a group of non-beneficiaries with the same average characteristics.

In our meta-evaluation, the measure used to analyse the average effect of active policies is the probability of finding a job. The average effect size allows us to measure the aggregate effect of different active policies aimed at unemployed people over 50 according to different variables: type of policy, methodology, time elapsed since implementation and gender. The average effect size is calculated by computing the weight of each effect size by the inverse of the variance for each of the studies. Therefore, the average size of the weighted effect can be calculated using the following formula:

$$\overline {ES} = \displaystyle{{\mathop \sum \nolimits ( {\omega_iES_i} ) } \over {\mathop \sum \nolimits^ \omega _i}}, \, \rm where \,\omega = \displaystyle{1 \over {SE^2}}.$$

ES_i represents the average effect of each evaluation included in the analysis, and ω_i represents the inverse of the variance of each evaluation. Thus, the sample size is present for each of the studies, with SE being the standard error for each evaluation. The value of i ranges from 1 to k, where k represents the total number of impacts or effects obtained from all the evaluations included in the analysis. For interpretation purposes, we present our results using so-called ‘forest plots’, which are simply error-bar charts with effect sizes and 95 per cent confidence intervals from the different studies (their name comes from their vague resemblance to trees).

We also run a meta-regression to analyse the relationship between study characteristics and the effect sizes of active policies. This is done to check whether and how heterogeneous study characteristics affect the size of the effects estimated in the evaluations (Thompson and Higgins, Reference Thompson and Higgins2002).

Systematic selection of studies

We conducted our selection of studies in accordance with widely adopted research guidelines, especially in the analysis of active labour market policies (Card et al., Reference Card, Kluve and Weber2010, Reference Card, Kluve and Weber2017; Kluve, Reference Kluve2010). We started by selecting all impact evaluations aimed at the older unemployed and then reviewed them to obtain only comparable evaluations for analysis.

For the selection of impact evaluations, we consulted up to six data sources. The first source was the Centre for Research on Impact Evaluation (CRIE), a repository created by the European Commission that seeks to improve the evaluation of policies by promoting the use of counterfactuals in analysis. This repository stores all impact evaluations that meet a set of standards and requirements to ensure the quality of the studies. A search was carried out in this database using certain key words (e.g. labour market and older unemployed) to narrow the focus to impact evaluations aimed at unemployed people over 50. We also reviewed impact evaluations targeting the entire unemployed population to check for any subgroup-level analyses focusing on the older unemployed. This first review yielded four studies on unemployed people over 50 years of age.

Our second source was the work published by Boockmann and Brandle (Reference Boockmann and Brandle2015). This article reviews a series of evaluations focused exclusively on the older unemployed. Our selection criterion was the same: studies using impact evaluation techniques to examine unemployment for people over 50. Our review of the bibliography of Boockmann and Brandle (Reference Boockmann and Brandle2015) yielded a total of ten possible impact evaluations to include in our analysis.

Our third source was Google Scholar. The search criteria remained the same: the keywords were older unemployed, active labour market policies and impact evaluations. This search yielded two additional impact evaluations for potential inclusion in our analysis.

Our fourth and fifth sources were the Dialnet and Scopus platforms. However, our search of these two sources yielded no additional impact evaluations concerning active policies for the older unemployed beyond those already found.

Finally, as our sixth source of impact evaluations, we exhaustively checked the bibliographies of the studies already selected in the five previous sources. The search criteria remained the same in that we focused only on impact evaluations of active policies aimed at unemployed people over 50.

After completing our search of the above sources, we had a total of 20 studies available to include in our analysis. However, after a detailed examination, we determined that some of the studies did not meet certain requirements for inclusion in the meta-evaluation (see Table A1 in the Appendix).

First, we excluded Baumgartner and Caliendo (Reference Baumgartner and Caliendo2008), Caliendo and Kunn (Reference Caliendo and Kunn2011) and Cahuc et al. (Reference Cahuc, Carcillo and Le Barbanchon2019). Despite being impact evaluations with rigorous methodologies, these evaluations do not study the net effect of policies on unemployed people over 50 but rather focus on the entire unemployed population.

Second, we excluded Heyma et al. (Reference Heyma, Van der Werff, Nauta and Van Sloten2014), Koning and Raterink (Reference Koning and Raterink2013) and Hullegie and Van Ours (Reference Hullegie and Van Ours2014), which had been extracted from the bibliography of Boockmann and Brandle (Reference Boockmann and Brandle2015). These three studies were excluded for methodological reasons: although they use rigorous analysis techniques, they are not impact evaluations.

Third, Bernhard et al. (Reference Bernhard, Gartner and Stephan2008) was ruled out because this study examines the effect of a subsidy programme on unemployed individuals between the ages of 35 and 49, which is not our group of interest. Boockmann et al. (Reference Boockmann, Zwick, Ammermuller and Maier2012), Centeno et al. (Reference Centeno, Centeno and Novo2009) and Heinrich et al. (Reference Heinrich, Mueser, Troske, Jeon and Kahvecioglu2013) were also excluded from the analysis because they estimate the effect of active policies in terms of the level of income or time spent in unemployment and not in terms of the transition into employment.

Fourth, we discarded two studies due to a lack of information necessary for our meta-evaluation. Boockmann and Brandle (Reference Boockmann and Brandle2018), despite fulfilling all desired characteristics, does not provide information on standard errors. Romeu-Gordo and Wolff (Reference Romeu-Gordo and Wolff2011) was excluded for the same reason.

Furthermore, Huttunen et al. (Reference Huttunen, Pirttilä and Uusitalo2013) was excluded because the individuals in the control group are of a different age than the individuals in the treatment group. Lastly, Lammers et al. (Reference Lammers, Bloemen and Hochguertel2013) was discarded because the results are only available disaggregated by gender.

Following the review, the database for this investigation consisted of 82 evaluated impacts extracted from six studies that fulfilled the requirements for inclusion. The review was minutely carried out, extracting information on the average estimated effects of active policies, the standard errors, the sample sizes, the methodologies used, the country and the time elapsed since policy implementation. In short, we included impact evaluations that have control groups, focus only on unemployed people over 50, provide all information detailed above and measure impacts in terms of the probability of accessing a job. There is not a standard minimum of studies that should be considered in meta-analysis, but the small number of studies available about older workers leads to not much heterogeneity in the methodology within each country considered. The number of evaluated impacts is large enough to carry out a meta-evaluation but results on country and methodology should be taken carefully and as provisional.

As shown in Table 1, our database covers a wide variety of methodological techniques. The studies we included use difference in differences, matching, randomised controlled trial and instrumental variable approaches. In addition, they provide data on the effects of active policies until 36 months after implementation. Regarding countries, bear in mind that there are different estimations by region in the case of Germany (East versus West), following the pattern of some previous studies (Caliendo et al., Reference Caliendo, Hujer and Thomsen2008). Although in almost all the studies included in the meta-analysis both institutional context (country) and methodology differ, most countries analysed (Austria, Belgium, Germany, Switzerland) share many important institutional features (Esping-Andersen Reference Esping-Andersen1990; Sapir, Reference Sapir2006; Card et al., Reference Card, Kluve and Weber2010). As Card et al. (Reference Card, Kluve and Weber2010) point out, there might be differences in the design of active labour market policies between countries. These differences cannot be analysed herein because of the small sample of studies considered.

Table 1. Characteristics of the evaluations included in the database

Source: Authors’ calculations.

Descriptive analysis

Table 2 presents the raw information on the average treatment effects collected from the selected studies. There are some striking differences when analysing the obtained effects according to policy type. Training and training jointly provided with counselling and job search assistance show a positive average effect of 2.6 and 2 percentage points on the probability of finding a job for unemployed people over 50, respectively. However, for direct job creation policies, the average effect obtained is negative, reaching −3.2 percentage points, and for training jointly with only counselling the average effect is extremely negative, −5.9 percentage points, although this average effect is the result of only four impacts.

Table 2. Average effects on the probability of finding a job for the older unemployed according to impact measurement variables

Notes: Total number of observations = 82. ATE: average treatment effect.

Source: Authors’ calculations.

Table 2 also reveals interesting patterns when analysing the effects of active policies according to the time horizon. The effect is practically null up to the first 12 months and then turns negative as the time horizon increases to 24 months. However, between 24 and 36 months after policy implementation, we observe a positive impact of 2.3 percentage points on the probability of finding a job for unemployed people over 50. This evidence seems to be consistent with the results of other studies, which find a greater effect of active policies in the long term (Card et al., Reference Card, Kluve and Weber2017).

In terms of the methodological approach, Table 2 shows important differences between matching and difference in differences estimations. The evaluations using matching to calculate the impact of active policies show a negative average effect of 1.5 percentage points on the probability of finding a job. In contrast, the difference in differences and instrumental variable methods show positive average effects of 1.9 and 2.7 percentage points, respectively.

Table 2 also reveals slight differences in the estimated impact of active policies for unemployed people over 50 according to gender. Specifically, the estimated average effect is lower for men than for women. This result is consistent with the literature on the entire unemployed population, regardless of age (Card et al., Reference Card, Kluve and Weber2017).

Meta-evaluation

Now, we present the weighted average effect obtained for the whole set of active policies, which is −0.8 percentage points (see Table 3, last row). This effect is between −1 and −0.5 percentage points with a 95 per cent confidence interval. Therefore, we find a negative average effect on the probability of finding a job for unemployed people over 50. A hasty interpretation that may be drawn from this result is that active policies for the older unemployed are ineffective and even detrimental. However, we observe an important heterogeneity across different active policies. Whereas two versions of training policies show a positive impact on the probability of finding a job (2.4 and 1.7 percentage points), direct job creation policies show a negative impact (−3.9 percentage points). For training policies combined exclusively with counselling, we found an effect of 0.1 percentage points but this effect is not statistically significant different from zero.

Table 3. Weighted average effect (WAE) of active policies on the probability of finding a job for the older unemployed according to methodology, type of policy and time

Note: Calculations for the lower and upper limits correspond to the 95 per cent confidence interval.

Source: Authors’ calculations.

Table 3 also presents data obtained for the set of active policies according to the methodology used. Impact evaluations using the difference in differences method show a positive effect of 1.6 percentage points on the probability of finding a job for the older unemployed. In contrast, evaluations using matching show a negative effect of 3.6 percentage points on this probability.

As can be seen in Figure 1, for impacts obtained using the difference in differences method, the deviation is very small. Therefore, we can affirm that impact evaluations using this methodology show a positive impact of active policies.

Figure 1. Forest plot of the estimated impacts and average treatment effect of the difference in differences method.

Notes: ES: effect size. CI: confidence interval.

Source: Authors’ calculations.

The weighted average effect obtained for the matching method is −3.6 percentage points, as can be observed in Figure 2. Again, the effect deviation is very small. Thus, we can affirm that impact evaluations using this method show a negative estimated effect of active policies. Therefore, we find important differences between the difference in differences and matching methods.

Figure 2. Forest plot of the estimated impacts and average treatment effect of the matching method.

Notes: ES: effect size. CI: confidence interval.

Source: Authors’ calculations.

The results for training policies are very encouraging (Figure 3), but those for direct job creation policies are in the opposite direction. As shown in Figure 4, the weighted average effect of direct job creation policies is −3.9 percentage points. Taking into account the fact that this type of policy is mainly aimed at unemployed people with unique problems re-entering the labour market, the results obtained regarding the effectiveness of these policies for people over 50 are not very encouraging. However, some studies show large differences in the effects of these policies according to the characteristics of unemployed individuals (Caliendo et al., Reference Caliendo, Hujer and Thomsen2008).

Figure 3. Forest plot of the estimated impacts and average treatment effect of training, counselling and search assistance policies.

Notes: ES: effect size. CI: confidence interval.

Source: Authors’ calculations.

Figure 4. Forest plot of the estimated impacts and average treatment effect for direct job creation policies.

Notes: ES: effect size. CI: confidence interval.

Source: Authors’ calculations.

Finally, there are some differences when analysing the effect of active policies according to the time horizon. As shown in Table 3, the results are slightly different from those obtained in the descriptive analysis. Although a negative effect is still observed in the ranges of 7–12 months (−2.3 percentage points) and 13–24 months (−1.4 percentage points), there is no longer a positive impact in the ranges of 0–6 and 25–36 months. For these two ranges, we cannot distinguish the average effect from zero; although the value of the weighted average effect is negative, the range obtained by the effect deviation exceeds zero.

The above evidence indicates that the effect of active policies on unemployed people over 50 displays a changing trend as the time horizon lengthens. The effect first decreases from null to negative until the lowest estimation for 7–12 months and then increases becoming null for the range of 25–36 months. Despite this, several positive effects are observed, which suggests that effects become greater as time increases. This is particularly true four or five years after policy implementation.

Next, we turn our attention to differences by gender. The results of the meta-evaluation show that active policies have a negative impact on women's probability of finding a job, with an effect size of −1.8 percentage points. For men, the negative impact is greater, at −3.2 percentage points.

Apart from the evaluations included in the database, we also consider other evaluations that, despite being excluded from the main analysis due to a lack of data, provide interesting information to interpret the above results. For training policies, impact evaluations that use both the difference in differences and matching methods (Romeu-Gordo and Wolff, Reference Romeu-Gordo and Wolff2011; Boockmann and Brandle, Reference Boockmann and Brandle2018) show similar effects to those obtained in our analysis, i.e. positive effects. Regarding wage subsidy policies, some impact evaluations show evidence of positive impacts (Huttunen et al., Reference Huttunen, Pirttilä and Uusitalo2013), but there is also conflicting evidence showing a negative effect (Boockmann et al., Reference Boockmann, Zwick, Ammermuller and Maier2012).

It is also interesting to compare our meta-evaluation of studies on the older unemployed with other meta-evaluations that consider the entire unemployed population. Among these meta-evaluations, the one conducted by Card et al. (Reference Card, Kluve and Weber2017) stands out as a benchmark work in the field of active labour market policies. Furthermore, given that we followed the same analysis guidelines for our meta-evaluation, we can compare results between our study (unemployed people over 50) and theirs (all unemployed).

First, we focus on differences in the effects of active policies according to the methodology used in impact evaluations. For two of the most commonly used methods, i.e. matching and difference in differences, we find a difference of 5 percentage points. A potential problem when using the matching method is that non-observable variables that can have a considerable influence on the population under study are not considered. In some areas of knowledge, such as economics, matching is done in a rigorous step-by-step fashion, especially propensity score matching, due to the importance of non-observable variables when making inferences (Cunningham, Reference Cunningham2021: 561–572).

Although any technique that falls under the matching method faces non-observable variable issues, we must remember that this method allows evaluation in cases where other methods are not possible. In addition, certain techniques, such as exact matching, reduce the dependency and bias of the model while providing for simpler analysis (Iacus et al., Reference Iacus, King and Porro2012).

Given the potential impact of non-observable variables, the difference in differences method is often chosen because it reduces selection bias. However, this method also has certain limitations. For example, one of the assumptions on which it is based, namely the parallel trends assumption, requires that policy beneficiaries and non-beneficiaries display similar trends in the period before the policy intervention; this assumption has been criticised because it is not always true (Kahn Lang et al., Reference Kahn Lang and Lang2018). However, in most cases, problems arise not from the technique chosen but from data quality or research design issues. These differences by methodology might also be affected by differences in the policy design of each country (Card et al., Reference Card, Kluve and Weber2010). Although policy design may differ leading to differences in the results of the evaluation, in our case we do not have enough heterogeneity to obtain conclusions about the effect of policy design.

Second, the results obtained according to policy type show that direct job creation policies are less effective than training policies in terms of increasing the probability of finding a job for unemployed people over 50. Our results are in line with those of other meta-evaluations considering the entire unemployed population, where policies emphasising the accumulation of human capital are found to have a greater effect than direct job creation policies (Heckman et al., Reference Heckman, Lalonde and Smith1999; Card et al., Reference Card, Kluve and Weber2010, Reference Card, Kluve and Weber2017).

However, although we observe a smaller or negative effect of direct job creation policies, the positive aspects of these policies should be kept in mind. They are designed to help individuals who have been unable to enter the labour market to obtain a job, and in some cases they spring from a social equity motivation to help the disadvantaged (Auer et al., Reference Auer, Efendioglu and Leschke2008). In addition, some studies show evidence of a positive impact of direct job creation policies on other labour market outcomes, such as earnings (Jespersen et al., Reference Jespersen, Munch and Skipper2008).

For unemployed people over 50 exposed to training policies, our meta-evaluation results show a positive impact of 1.7 percentage points on the probability of finding a job. Although this value may seem small, it actually represents an important effect for this subset of the unemployed, who face great disadvantages in the labour market. Furthermore, the analysis shows that the training policies with the greatest effect are comprehensive ones that also incorporate search assistance and counselling.

Third, regarding the time horizon, the effects of active policies show an interesting pattern: the effects are negative in the short term but then increase 25–36 months after policy implementation. The literature reveals a similar trend for the entire unemployed population (Card et al., Reference Card, Kluve and Weber2010, Reference Card, Kluve and Weber2017; Vooren et al., Reference Vooren, Haelermans, Groot and Van den Brink2018). As discussed earlier, the timing of effects is crucial for the older unemployed, who are near the end of their working lives. Therefore, the fact that positive impacts are observed only after 25–36 months raises the question of how profitable these policies are for them. However, considering that, behind the non-significant effects we have not null impacts but a mix of negative and positive impacts. This lack of precision may be related to the crucial role of policy implementation to obtain a positive impact (Malo and Cueto, Reference Malo and Cueto2017; Malo, Reference Malo2018).

Finally, with regard to gender, results obtained for unemployed people over 50 show a greater negative effect of active policies for men than for women. Card et al. (Reference Card, Kluve and Weber2017) similarly observe a greater impact of active policies on women. Furthermore, in some cases, we find a strong dispersion of effects. Together, these results suggest that it is essential to maximise the effects of active policies by focusing adequately on a target population, as recommended by Malo and Cueto (Reference Malo and Cueto2017).

Meta-regression

The meta-regression is an ordinary least squares regression that sheds light on the differences and relationships among the variables included in the analysis and the estimated effects of the active policies.

The dependent variable is the estimated effect of the active policies, and the independent variables (all categorical) are policy type, methodology, time horizon, gender and country. However, the country variable is not included in the model due to collinearity problems. These problems may arise due to the small sample of studies and because there is not much heterogeneity in terms of country and methodology. The following categories are taken as a reference for each variable: direct job creation policies for policy type, difference in differences for methodology, men for gender and 0–6 months for time horizon. The methodology was defined as a dummy variable that takes a value of 1 for the instrumental variable method and 0 for the rest. Because of collinearity problems in estimations, we cannot introduce a variable on methodology with more categories. Again, these collinearity problems may arise due to the small number of studies that could be considered in the analysis.Footnote ¹

As shown in Table 4, we obtain significant differences for policy type, gender and time horizon. The estimated effects are greater for training policies, with particularly large effects for more comprehensive training policies that incorporate search assistance and counselling. This is in line with Malo (Reference Malo2018), who states that international evidence shows that active policy packages are more effective than isolated policies.

Table 4. Meta-regression on the size of the policy impacts

Notes: Total number of observations = 82. Adjusted R ² = 87.54 per cent. Ref.: reference category.

Source: Authors’ calculations.

The estimated effect becomes greater as the time horizon lengthens to the range of 25–36 months after policy implementation, and the estimated effect is greater for women than for men. The first result is in line with evidence shown in the meta-evaluation of Vooren et al. (Reference Vooren, Haelermans, Groot and Van den Brink2018), and the worse result for men is also in line with other meta-evaluations that consider all unemployed people (Card et al., Reference Card, Kluve and Weber2017). All of this evidence is in line with the data obtained in the meta-evaluation, although differences observed according to the methodology used were not significant. This is important, as we are unable to affirm that the method chosen has any influence on the estimated effect.

As a robustness check, we also run meta-regressions by the type of active policy. The results obtained are consistent with those in the previous model and are available upon request.