Systematic reviews play important roles but manual efforts can be time-consuming given a growing literature. There is a need to use and evaluate automated strategies to accelerate systematic reviews. Here, we comprehensively tested machine learning (ML) models from classical and deep learning model families. We also assessed the performance of prompt engineering via few-shot learning of GPT-3.5 and GPT-4 large language models (LLMs). We further attempted to understand when ML models can help automate screening. These ML models were applied to actual datasets of systematic reviews in education. Results showed that the performance of classical and deep ML models varied widely across datasets, ranging from 1.2 to 75.6% of work saved at 95% recall. LLM prompt engineering produced similarly wide performance variation. We searched for various indicators of whether and how ML screening can help. We discovered that the separability of clusters of relevant versus irrelevant articles in high-dimensional embedding space can strongly predict whether ML screening can help (overall R = 0.81). This simple and generalizable heuristic applied well across datasets and different ML model families. In conclusion, ML screening performance varies tremendously, but researchers and software developers can consider using our cluster separability heuristic in various ways in an ML-assisted screening pipeline.

The therapeutic effects of soy consumption on adipokine concentrations have yielded inconsistent results in previous meta-analyses. This umbrella meta-analysis aims to investigate the impact of soy and its isoflavones on serum adiponectin and leptin levels in adults. We searched the Cochrane Central, Web of Science, PubMed, and Scopus databases until October 10, 2024. The articles were restricted to those written in English. We included meta-analysis studies that evaluated the effects of soy and its isoflavones on levels of adiponectin and leptin and reported effect sizes (ES) and corresponding confidence intervals (CI). Two independent reviewers screened all articles based on eligibility criteria and extracted the required data from the included meta-analyses. The meta-analysis was performed using a random-effects model in STATA software. Six meta-analyses of randomized controlled trials meeting the inclusion criteria were included in the current umbrella meta-analysis. The findings indicated that soy and its isoflavones did not have a significant effect on adiponectin (ES= 0.10; 95% CI: -0.22, 0.41; P=0.55; I2= 51.8%) and leptin (ES= -0.37; 95% CI: -1.35, 0.61; P=0.46; I2= 71.2%) concentrations. Subgroup analysis based on participants’ mean age, total sample size, and duration was conducted. Results showed that the effect is not statistically significant in any of the subgroups. In conclusion, soy and its isoflavones could not improve the adipokines mentioned above. However, further high-quality research in different countries is required to substantiate these findings.

Bibliographic aggregators like OpenAlex and Semantic Scholar offer scope for automated citation searching within systematic review production, promising increased efficiency. This study aimed to evaluate the performance of automated citation searching compared to standard search strategies and examine factors that influence performance. Automated citation searching was simulated on 27 systematic reviews across the OpenAlex and Semantic Scholar databases, across three study areas (health, environmental management and social policy). Performance, measured by recall (proportion of relevant articles identified), precision (proportion of relevant articles identified from all articles identified), and F1–F3 scores (weighted average of recall and precision), was compared to the performance of search strategies originally employed by each systematic review. The associations between systematic review study area, number of included articles, number of seed articles, seed article type, study type inclusion criteria, API choice, and performance was analyzed. Automated citation searching outperformed the reference standard in terms of precision (p < 0.05) and F1 score (p < 0.05) but failed to outperform in terms of recall (p < 0.05) and F3 score (p < 0.05). Study area influenced the performance of automated citation searching, with performance being higher within the field of environmental management compared to social policy. Automated citation searching is best used as a supplementary search strategy in systematic review production where recall is more important that precision, due to inferior recall and F3 score. However, observed outperformance in terms of F1 score and precision suggests that automated citation searching could be helpful in contexts where precision is as important as recall.

The number of published systematic reviews has increased over the last years, with a non-negligible proportion displaying methodological concerns. We aimed to develop and evaluate a tool to assess the reported methodological quality of medical systematic reviews. The developed tool (ReMarQ) consists of 26 dichotomous items. We applied an item response theory model to assess the difficulty and discrimination of the items and decision tree models to identify those items more capable of identifying systematic reviews with higher reported methodological quality. ReMarQ was applied to a representative sample of medical systematic reviews (excluding those published in the Cochrane Database of Systematic Reviews) to describe their methodological quality and identify associated factors. We assessed 400 systematic reviews published between 2010 and 2020, of which 196 (49.0%) included meta-analysis. The most discriminative items were (i) conducting a risk of bias assessment, (ii) having a published protocol and (iii) reporting methods for solving disagreements. More recent systematic reviews (adjusted yearly RR=1.03; 95%CI=1.02 −1.04, p<0.001) and those with meta-analysis (adjusted RR=1.34; 95%CI=1.25 −1.43, p<0.001) were associated with higher reported methodological quality. Such an association was not observed with the journal impact factor. The items most frequently fulfilled were (i) reporting search dates, (ii) reporting bibliographic sources and (iii) searching multiple electronic bibliographic databases. ReMarQ, consisting of dichotomous items and whose application does not require subject content expertise, may be important (i) in supporting an efficient quality assessment of systematic reviews and (ii) as the basis of automated processes to support that assessment.

The so-called credibility revolution dominates empirical economics, with its promise of causal identification to improve scientific knowledge and ultimately policy. By examining the case of rural electrification in the Global South, this opinion paper exposes the limits of this evidence-based policy paradigm. The electrification literature boasts many studies using the credibility revolution toolkit, but at the same time, several systematic reviews demonstrate that the evidence is divided between very positive and muted effects. This bifurcation presents a challenge to the science-policy interface, where policymakers, lacking the resources to sift through the evidence, may be drawn to the results that serve their (agency's) interests. The interpretation is furthermore complicated by unresolved methodological debates circling around external validity as well as selective reporting and publication decisions. These features, we argue, are not particular to the electrification literature but inherent to the credibility revolution toolkit.

It is estimated that electroconvulsive therapy is still administered to approximately a million people a year. It involves passing enough electric current through the human brain, eight to twelve times, to cause convulsions, in the hope of somehow alleviating emotional suffering, primarily depression. There have only ever been 11 placebo-controlled studies (where general anaesthesia is administered but the electric shock is withheld), all of which were pre-1986, had very small sample sizes and were seriously methodologically flawed. Five of these studies found no difference between the two groups at the end of treatment, four found ECT produced better outcomes for some patients, and two produced mixed results, including one where psychiatrists' ratings produced a difference, but the ratings of nurses and patients did not. In the 80 years since the first ECT no studies have found any evidence that ECT is better than placebo beyond the end of treatment. Nevertheless, all five meta-analyses relying on these studies have somehow concluded that ECT is more effective than placebo despite the studies' multiple failings. Meanwhile, evidence of persistent or permanent memory loss in 12% to 55% of patients has accumulated. Attempts to highlight this failure of ECT proponents to provide robust evidence that their treatment is effective and safe are routinely dismissed, diminished, denied and denounced. This paper responds to one such attempt, by Drs Meechan, Laws, Young, McLoughlin and Jauhar, to discredit two systematic reviews of the eleven pre-1986 studies, in 2010 and 2019, the latter of which also reviewed five meta-analyses that had ignored the studies' failings. The criticisms and claims of the recent crtiique of the two systematic reviews are examined in detail, by the first author of both reviews, for accuracy, relevance and logic. The critique is found to include multiple errors, misrepresentations, omissions, inconsistencies and logical flaws. It is concluded that Meechan et al. fail to make a fact-based, coherent argument against suspending ECT pending a series of large, carefully designed placebo-controlled studies to establish whether ECT does have any beneficial effects against which to weigh the significant established adverse effects.

This is a systematic review of systematic reviews of secondary health conditions, health promotion interventions, and employment in people with intellectual disabilities. Articles were included if they reported a systematic review of health and employment, secondary health conditions, and health promotion interventions for people with intellectual disabilities. The methodological quality of the included reviews was reviewed using the A MeaSurement Tool to Assess systematic Reviews quality rating system, a measurement tool to assess systematic reviews. Twenty-five systematic reviews were included. There was evidence that people with intellectual disabilities (ID) were at elevated risk for secondary health conditions, health promotion interventions can improve physical and mental health conditions, and employment is associated with better health-related quality of life. Health promotion intervention to help people with ID engage in health promoting behaviors can improve health and their ability to find and maintain employment.

Chronic inflammation has been considered as the main cause of chronic diseases. Zn has anti-inflammatory effects by decreasing the expression of inflammatory markers. The present systematic review and meta-analysis study aims to evaluate the impact of Zn supplementation on inflammation. PubMed (Medline), Scopus, Web of Science, and Embase databases were searched up to 10 December 2020. Controlled trials which have investigated the effects of Zn supplementation on serum/plasma levels of inflammatory cytokines in subjects aged >15 years were included. A pooled meta-analysis was performed using a random effect model. Sensitivity analysis was performed to determine the robustness of the observed effect sizes. A total of twelve studies was included in meta-analysis. Zn could decrease IL-6 levels (standardised mean difference (SMD) = −0·76 pg/ml; 95 % CI −1·28, −0·24; P = 0·004). There was no significant change in TNF-α (SMD = 0·42 pg/ml; 95 % CI −0·31, 1·16; P = 0·257) and IL-2 levels (SMD = 1·64 pg/ml; 95 % CI −1·31, 4·59; P = 0·277) following Zn supplementation. However, Zn could increase IL-2 significantly after the deletion of one arm in sensitivity analysis (SMD = 2·96 pg/ml; 95 % CI 2·03, 3·88; P < 0·05). Conclusively, Zn supplementation can decrease the IL-6 level. Zn increased IL-2 level after the sensitivity analysis. Zn supplementation has not ameliorative effects on TNF-α.

The majority of psychological treatment research is dedicated to investigating the effectiveness of cognitive behavioural therapy (CBT) across different conditions, population and contexts. We aimed to summarise the current systematic review evidence and evaluate the consistency of CBT's effect across different conditions. We included reviews of CBT randomised controlled trials in any: population, condition, format, context, with any type of comparator and published in English. We searched DARE, Cochrane, MEDLINE, EMBASE, PsycINFO, CINAHL, CDAS, and OpenGrey between 1992 and January 2019. Reviews were quality assessed, their data extracted and summarised. The effects upon health-related quality of life (HRQoL) were pooled, within-condition groups. If the across-condition heterogeneity was I2 < 75%, we pooled effects using a random-effect panoramic meta-analysis. We summarised 494 reviews (221 128 participants), representing 14/20 physical and 13/20 mental conditions (World Health Organisation's International Classification of Diseases). Most reviews were lower-quality (351/494), investigated face-to-face CBT (397/494), and in adults (378/494). Few reviews included trials conducted in Asia, South America or Africa (45/494). CBT produced a modest benefit across-conditions on HRQoL (standardised mean difference 0.23; 95% confidence intervals 0.14–0.33, I2 = 32%). The effect's associated prediction interval −0.05 to 0.50 suggested CBT will remain effective in conditions for which we do not currently have available evidence. While there remain some gaps in the completeness of the evidence base, we need to recognise the consistent evidence for the general benefit which CBT offers.

This meta-analysis aimed to study the relationship between abdominal obesity and the risk of CVD by waist circumference (WC), waist:hip ratio (WHR) and waist:height ratio (WHtR). We systematically searched PubMed, Embase and Web of Science. Prospective studies that estimated cardiovascular events by WC, WHR and WHtR were included in this study. Pooled relative risks with 95 % CI were calculated using random effects models. A total of thirty-one studies were included in the meta-analysis, including 669 560 participants and 25 214 cases. Compared the highest with the lowest category of WC, WHR and WHtR, the summary risk ratios were 1·43 (95 % CI, 1·30, 1·56, P < 0·001), 1·43 (95 % CI, 1·33, 1·54, P < 0·001) and 1·57 (95 % CI, 1·37, 1·79, P < 0·001), respectively. The linear dose–response analysis revealed that the risk of CVD increased by 3·4 % for each 10 cm increase of WC, and by 3·5 and 6·0 % for each 0·1 unit increase of WHR and WHtR in women, respectively. In men, the risk of CVD increased by 4·0 % for each 10 cm increase of WC, and by 4·0 and 8·6 % for each 0·1 unit increase of WHR and WHtR, respectively. Collectively, abdominal obesity is associated with an increased risk of CVD. WC, WHR and WHtR are good indicators for the prediction of CVD.