Introduction
Rationale
In the dairy industry, a large proportion of total antimicrobial use is for the prevention and treatment of intramammary infections (IMI), with a large portion of the total mass used aimed at controlling IMI during the dry period (Lam et al., Reference Lam, van Engelen, Scherpenzeel and Hage2012). At the end of lactation, colloquially known as dry-off, formation of the teat-canal keratin plug plays an important role in susceptibility to IMI (Huxley et al., Reference Huxley, Greent, Green and Bradley2002), but there is wide variation among cows in time taken to complete closure of the teat-canal, or whether closure occurs at all (Dingwell et al., Reference Dingwell, Kelton and Leslie2003). Prepartum IMI is an important risk factor for the development of clinical mastitis in early lactation (Piepers et al., Reference Piepers, De Vliegher, de Kruif, Opsomer and Barkema2009). In the United States, clinical mastitis represents the most common disease treated with antimicrobials in adult dairy cattle, with 16.4% of cows reported as treated for this disease with antimicrobials in 2007, and cephalosporins the most commonly selected drug class (United States Department of Agriculture, 2008). As a consequence of this mastitis risk, teat sealants can be employed to close the teat canal in a more consistent and timely manner.
Teat sealants applied internally or externally to close the teat canal provide a non-antimicrobial means to prevent new IMI in the pre-calving period, which is of increasing importance due to concern over antimicrobial use and its relationship with the development of antimicrobial resistance (World Health Organisation, 2015). Understanding the efficacy of teat sealants is essential for optimizing their use in order to decrease reliance on antimicrobials for both treatment and prevention of disease.
Systematic reviews and meta-analyses of well-executed and well-reported randomized controlled trials yield the highest level of evidence for the efficacy of interventions under field conditions (Sargeant et al., Reference Sargeant, Kelton and O'Connor2014). If sufficient primary studies on a given comparison are available, a pairwise meta-analysis provides the relative efficacy of the two treatments. Previous work has typically involved this method of meta-analysis to evaluate the efficacy of antimicrobial and non-antimicrobial interventions for dairy cattle at dry-off, including teat sealants (Halasa et al., Reference Halasa, Osteras, Hogeveen, van Werven and Nielen2009; Rabiee and Lean, Reference Rabiee and Lean2013; Naqvi et al., Reference Naqvi, Nobrega, Ronksley and Barkema2018), antimicrobials (Robert et al., Reference Robert, Seegers and Bareille2006; Halasa et al., Reference Halasa, Osteras, Hogeveen, van Werven and Nielen2009), and dry-period length (van Knegsel et al., Reference van Knegsel, van der Drift, Cermakova and Kemp2013). However, pairwise comparisons are often between treated animals and non-treated controls (NTCs), and direct comparisons of potentially comparable interventions may be limited (Roy and Keefe, Reference Roy and Keefe2012). In the case of intramammary treatments of cattle at dry-off, numerous interventions are available, including teat sealants used with or without intramammary antimicrobials. Pairwise meta-analyses can only provide information about a single comparison, and do not provide a summary of evidence across multiple interventions (Cipriani et al., Reference Cipriani, Higgins, Geddes and Salanti2013).
Network meta-analysis provides a method of assessing relative efficacy across many treatments by using both direct evidence (from studies that compare given treatments) and indirect evidence (from studies that share common comparators), and is a commonly used approach in the human medicine literature (Caldwell et al., Reference Caldwell, Ades and Higgins2005; Cipriani et al., Reference Cipriani, Higgins, Geddes and Salanti2013). Establishing the relative efficacy of teat sealants administered at dry-off in cows, or prepartum in heifers, to reduce the incidence of clinical mastitis or IMI, will improve decision makers’ ability to engage in effective stewardship of antimicrobials through the strategic use of non-antimicrobial alternatives with knowledge of implications for animal health and welfare.
This systematic review was conducted based on guidelines from the Cochrane Collaboration (Higgins and Green, Reference Higgins, Green, Chandler, Cumpston, Li, Page and Welch2011) and recommendations for conducting systematic reviews in animal agriculture and veterinary medicine (O'Connor et al., Reference O'Connor, Anderson, Goodell and Sargeant2014a, Reference O'Connor, Sargeant and Wang2014b; Sargeant and O'Connor, Reference Sargeant and O'Connor2014a, Reference Sargeant and O'Connor2014b). This review is reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Extension Statement for Reporting of Systematic Reviews Incorporating Network Meta-analyses of Health Care Interventions (PRISMA-NMA) (Hutton et al., Reference Hutton, Salanti, Caldwell, Chaimani, Schmid, Cameron, Ioannidis, Straus, Thorlund, Jansen, Mulrow, Catala-Lopez, Gotzsche, Dickersin, Boutron, Altman and Moher2015).
Objectives
The objective of this review was to assess the efficacy of internal or external teat sealants, administered with or without antimicrobial therapy, given at dry-off in cows or prepartum in heifers to prevent new IMI and clinical mastitis early in the subsequent lactation.
Methods
Protocol
A review protocol, established in advance and reported in accordance with the PRISMA guidelines for review protocols (PRISMA-P) (Moher et al., Reference Moher, Shamseer, Clarke, Ghersi, Liberati, Petticrew, Shekelle and Stewart2015), was published in the University of Guelph's institutional repository (https://atrium.lib.uoguelph.ca/xmlui/handle/10214/10046) on 25 June 2018. The protocol is also available through Systematic Reviews for Animals and Food (SYREAF) (http://www.syreaf.org/contact/).
Eligibility criteria
Primary research studies available in English were eligible for inclusion. Studies must have been conducted in prepartum dairy heifers or cows after their first (or greater) lactation, without existing IMI (for IMI outcomes, and based on the trial authors’ definition of IMI at dry-off) or clinical mastitis (for the clinical mastitis outcome). Studies must have included at least one treatment arm with an internal or external teat sealant given at the time of dry-off, or prepartum in heifers, and may include combination treatment with teat sealant and an antimicrobial preparation, compared to no treatment, placebo, or another treatment (such as an antimicrobial dry-cow preparation). To be eligible, studies must have included at least one of the following outcomes: (i) incidence of IMI (using the authors’ definition of incident cases) during the pre-calving period following the intervention, (ii) incidence of IMI (using the authors’ definition of incident cases) during the first 30 days of the subsequent lactation, and (iii) incidence of clinical mastitis during the first 30 days of the subsequent lactation. For the clinical mastitis outcome, cows were assumed to be free of clinical mastitis at dry-off if this was not explicitly stated (i.e. all cases were considered incident). Controlled trials with natural disease exposure were the only eligible study design, although challenge trials and analytical observational studies were documented during the full-text screening stage.
Information sources
The following databases were searched: Agricola (via ProQuest, 1970 to current), CAB Abstracts and Global Health (via Web of Science, 1910 to current), Ovid MEDLINE(R) Daily, and Ovid MEDLINE(R) (via Ovid, 1946 to current), Conference Proceedings Citation Index – Science (via Web of Science, 1990 to current), and Science Citation Index (via Web of Science, 1900 to current). A reviewer hand-searched the table of contents of the following conferences from 1997 to 2018: Proceedings of the American Association of Bovine Practitioners, World Association for Buiatrics, and the National Mastitis Council Proceedings. The Food and Drug Administration (FDA) website containing the Freedom of Information New Animal Drug Approvals (NADA) summaries was also searched, and all available summaries were examined.
Search
The search strategy was initially developed for the Science Citation Index (Web of Science). The conceptual structure was as follows: (dairy cattle OR mastitis) AND teat sealants (Table 1). To maximize sensitivity, the dry-off period was not included as a search concept. The Science Citation Index strategy was translated appropriately for the other databases searched. Database searches were conducted on 26 June 2018 and accessed through the University of York in the United Kingdom. Search results were uploaded to EndNoteX7 (Clarivate Analytics, Philadelphia, PA, USA) and duplicate results were documented and removed. Records were then uploaded to DistillerSR (Evidence Partners Inc., Ottawa, ON, USA) and additionally de-duplicated. If the same study and data were available as a conference abstract and as a full publication, the conference abstract was removed. Data only available as a conference abstract were eligible if the full text was >500 words, to allow sufficient detail for data extraction and risk-of-bias assessment. Validation of the search was done by identifying all articles included in the qualitative syntheses of reviews in the area of dry-cow management as identified from the following papers, selected by the review content experts: Robert et al. (Reference Robert, Seegers and Bareille2006), Halasa et al. (Reference Halasa, Osteras, Hogeveen, van Werven and Nielen2009), Pereira et al. (Reference Pereira, Oliveira, Mesquita, Costa and Pereira2011), Rabiee and Lean (Reference Rabiee and Lean2013), van Knegsel et al. (Reference van Knegsel, van der Drift, Cermakova and Kemp2013), Enger et al. (Reference Enger, White, Nickerson and Fox2016), Naqvi et al. (Reference Naqvi, Nobrega, Ronksley and Barkema2018). All relevant articles identified in these reviews were captured in the search.
Table 1. Full electronic search strategy used to identify studies of the effectiveness of teat sealants during the dry-off period in dairy cattle in Science Citation Index (Web of Science) conducted on 18 June 2018
Study selection
The online systematic review management program DistillerSR was used for relevance screening and data extraction. Title and abstracts were initially screened for eligibility. Two reviewers independently evaluated each citation, and all reviewers were trained by CBW and JMS on a pre-test of the title and abstracts of the first 250 citations to ensure clarity of understanding and consistency of question application. The following questions were used to assess relevance:
(1) Is this a primary study which evaluates the use of an internal or external teat sealant in prepartum heifers or at dry-off in dairy cows following the first or greater lactation? YES (neutral), NO (exclude), UNCLEAR (neutral)
(2) Is there a concurrent comparison group (i.e. controlled trial with natural or deliberate disease exposure, or analytical observational study)? YES (neutral), NO (exclude), UNCLEAR (neutral)
(3) Is the text available in English? YES (include for full-text screening), NO (exclude), UNCLEAR (include for full-text screening)
Agreement was at the level of the form, and therefore citations were excluded if both reviewers responded ‘NO’ to any of the questions. Disagreements were resolved by consensus with mediation by JMS or CBW if an agreement could not be reached. Secondary screening was conducted on the full text of remaining studies independently by two reviewers, using the first 10 citations as a pre-test by all reviewers. This level of screening included the initial three questions with only YES (neutral) or NO (exclude) options, and additionally:
(1) Does the study evaluate any of the following outcomes: incidence of clinical mastitis at 30 days in milk (DIM), incidence of IMI or subclinical mastitis at calving, or incidence of IMI or subclinical mastitis at 30 DIM? YES (neutral) NO (exclude)
(2) What is the study design? Experimental – natural disease exposure (include), Experimental – deliberate disease exposure (exclude), Analytical observational study (exclude)
The term ‘subclinical mastitis’ was included as authors may have referred to this instead of IMI. Agreement was at the question level, with conflicts resolved by consensus or with mediation by JMS or CBW if an agreement could not be reached.
Data collection
Data from citations meeting the full-text screening inclusion criteria were independently extracted by two reviewers using a standardized form, which was piloted on the first five citations by all reviewers to ensure consistency. Discrepancies in data extraction were resolved by consensus, with mediation by JMS and CBW if an agreement could not be reached. Hierarchical forms were used in DistillerSR for data extraction, with the forms nested as: (Study Characteristics (Outcome (Arm, Contrast, Risk of bias))). A PDF version of the full data extraction tool is available as Supplementary File S1.
Data items
Study characteristics
Study-level data included study design, country of conduct, year and months of study conduct, setting (research or commercial herd), breed of cattle, number of herds enrolled, inclusion criteria at the cow and herd level, and parity of enrolled animals. Study characteristics were extracted for all studies included after full-text screening.
Interventions and comparators
Details on the interventions, including the name of the product(s) administered (both trade and generic, if available), dose, route, duration, concurrent therapy, dry-period length, level of treatment allocation, and level of analysis were recorded. Baseline characteristics and loss to follow-up were captured. Case definitions were recorded, including methods used to identify IMI, as were the times at which the outcomes were measured. Following data extraction, interventions were identified and labeled on a treatment map (Table 2). To provide strength to the network, interventions in the same antimicrobial family (World Organization for Animal Health, 2007) were considered to be the same treatment protocol.
Table 2. Description of treatment groups as labeled in subsequent figures and tables
Although the results of all comparisons in the network were included in the analysis, relative efficacy rankings are presented only for those treatment arms with a teat sealant, or a NTC group (i.e. antimicrobial dry-cow therapies given without teat sealants were not ranked, but information captured on these comparator arms provided evidence to the network).
Eligible outcomes
Outcomes eligible for inclusion in the meta-analysis were:
• Incidence of clinical mastitis in the first 30 days of lactation
• Incidence of IMI between treatment and calving, and
• Incidence of IMI in the first 30 days of lactation
Prioritization of these outcomes for meta-analysis was determined during protocol development in consultation with content experts based on the anticipated frequency of use in the primary literature and as being proxies to reflect the effects of infection during the dry period. Data reported for clinical mastitis were considered as incidence; cows were assumed to be free of clinical mastitis at dry-off unless otherwise reported in the study. For IMI incidence, cows were not assumed to be free of IMI at dry-off, and studies had to report results separately for ‘new’ infections to proceed to data extraction. The trial authors’ definition on what constituted a ‘new’ infection was recorded: no pathogen growth initially followed by any pathogen growth; a new pathogen isolated on the follow-up sample; or not reported.
For included studies, information on other outcomes was extracted to describe their use in the literature, but data were not extracted for synthesis. These secondary outcomes were: total antimicrobial use during the first 30 days of lactation, total milk production over the next lactation, somatic cell count at the first milk recording test of next lactation, average somatic cell count of the first three milk recording tests of the next lactation, and the risk of culling over the next lactation.
For outcomes for which data were extracted, the prioritized outcome measure was an adjusted summary effect (adjusted odds ratio (OR) or relative risk or risk ratio (RR) for dichotomous outcomes, or adjusted least square mean differences for continuous outcome). Variables included in adjustment and the corresponding precision estimate were recorded. If an adjusted measure was not reported, unadjusted summary effect size (second priority) or treatment arm-level (raw) data (third priority) were recorded, with an applicable variance measure. Continuous data presented without variance measures, and for which a measure of variance could not be calculated, were not extracted.
For multi-farm studies where clustering at the farm level was not adjusted for (i.e. those reporting raw data for multiple farms), if raw data were available by the farm, each farm was extracted as a unique study.
Geometry of the network
We visually evaluated the geometry of the network, to determine if some pairwise comparisons dominated and to determine the network structure. We evaluated if there were intervention comparisons that were not linked to the network (i.e. did not have an intervention in common with one or more other published studies).
Risk of bias in individual studies
Risk of bias was assessed by outcome for all three outcomes extracted, using the Cochrane Risk of Bias 2.0 tool (Higgins et al., Reference Higgins, Sterne, Savovic, Page, Hróbjartsson and Boutron2016), with signaling questions modified to be specific to the topic of the review. This tool assesses the potential for bias arising from five areas or domains: bias arising from the randomization process, bias due to deviations from intended interventions, bias due to missing outcome data, bias in the measurement of the outcome, and bias in the selection of the reported results. While for some commodity groups’ individual animal value is likely to be unknown, or equal, at the time of treatment allocation, the Cochrane Risk of Bias 2.0 algorithm has been modified to exclude the question 1.2 on allocation sequence concealment (Moura et al., Reference Moura, Totton, Sargeant, O'Sullivan, Linhares and O'Connor2019). In the case of dairy cattle, a decision was made to include the question on allocation concealment in the risk-of-bias assessment, as individual animal value is likely unequal and known at the time of treatment allocation in most (or all) studies. As well, an additional answer option was provided for the question on random allocation sequence, for studies using the word ‘random’ to describe the allocation sequence but not providing details on the method used to generate the random sequence.
Risk of bias was assessed independently in duplicate, with disagreement resolved by consensus and mediation by JMS or CBW if needed. The risk-of-bias tool is available as Supplementary File S2. Risk of bias is presented separately for each outcome, and then by the domain of bias.
Summary measures
After extracting the outcomes, the analysis was conducted on the log OR for the analysis. For presentation purposes, the log OR was back transformed to the RR using the baseline risk from the model data. The posterior mean and standard deviation of the baseline risk mean were −1.0222 and 2.0967. The posterior mean and standard deviation of the baseline risk standard deviation were 1.6334 and 0.9036. When studies had zero cells for some data points, and the OR could not be calculated, the trial results could not be included in the analyses.
Pairwise meta-analysis
For outcomes where insufficient data were found, network meta-analysis was not conducted, but pairwise meta-analysis was performed when multiple studies evaluated the same comparison. Meta-analysis was conducted in R 3.5.2 (R Foundation for Statistical Computing, Vienna, Austria) using RStudio version 1.0.136 (RStudio Inc., Boston, MA, USA) using the ‘metafor’ package (Viechtbauer, Reference Viechtbauer2010). A random-effect approach was used, with weighting of studies using the inverse variance method. Heterogeneity was assessed by the I 2 statistic (Viechtbauer, Reference Viechtbauer2010).
Network meta-analysis
Planned method of statistical analysis
A network meta-analysis was conducted for the outcome of IMI at calving, using methodology described by Dias et al. (Reference Dias, Welton, Caldwell and Ades2010) and O'Connor et al. (Reference O'Connor, Coetzee, da Silva and Wang2013). Raw data or ORs were converted to a log OR, and RRs were converted to a log OR using the risk of disease in the control group. If probabilities were reported, the values were back converted to a log OR, using a process described by Hu et al. (Reference Hu, Wang and O'Connor2019).
Selection of prior distributions in Bayesian analysis
The prior distributions were originally based on the approach reported previously (Dias et al., Reference Dias, Welton, Sutton and Ades2011). For the model, we assessed σ ~ U (0,2) and σ ~ U (0,5). The analysis suggested σ ~ U (0,5) was preferred, so this prior was retained in the model.
Implementation and output
All posterior samples were generated using Markov Chain Monte Carlo (MCMC) simulation, which was implemented using Just Another Gibbs Sampler (JAGS) software (version 3.4.0) (Plummer, Reference Plummer2015). All statistical analyses were performed using R software (version 3.2.1) (R Core Team, 2018) in a Linux system. The model was fit by calling JAGS from R through the RJAGS package (Plummer, Reference Plummer2015). Three chains were simulated and the convergence was assessed using Gelman–Rubin diagnostics. A total of 5000 ‘burn-in’ iterations were discarded, and the inferences were based on a further 10,000 iterations. The model output included all possible pairwise comparisons using log ORs for the inconsistency assessment, RRs for comparative efficacy reporting, rankings for comparative efficacy, and the probability of being the worst treatment option for comparative efficacy.
Assessment of model fit
The fit of the model was assessed based on the log OR, by examining the residual deviance between the predicted values from the mixed-treatment comparison model and the observed value for each study (Dias et al., Reference Dias, Welton, Caldwell and Ades2010).
Assessment of inconsistency
Inconsistency was assessed by examining the consistency between direct and indirect evidence for all pairwise comparisons, using the method described by Dias et al. (Reference Dias, Welton, Caldwell and Ades2010). Means and standard deviations of log OR of treatment effects were calculated using direct (head-to-head) evidence only, indirect evidence only, and the combined evidence. We compared the estimates from the direct and indirect models and considered the standard deviation of each estimate, rather than relying on the P-values.
Risk of bias in the overall network
Risk of bias in the overall network of evidence was assessed using the Confidence In Network Meta-Analysis (CINeMA) platform (http://cinema.ispm.ch), which uses a frequentist approach through the ‘metafor’ package (Viechtbauer, Reference Viechtbauer2010) to determine the basis for the contribution matrix for the risk of bias. CINeMA evaluates within-study bias, across-study bias, indirectness, imprecision, heterogeneity, and incoherence. As opposed to presenting an overall assessment of bias and of indirectness, we reported the contribution of studies based on the approach to allocation to groups and blinding, as there is evidence in animal health that failure to include these design elements is associated with exaggerated treatment effects (Burns and O'Connor, Reference Burns and O'Connor2008; Sargeant et al., Reference Sargeant, Elgie, Valcour, Saint-Onge, Thompson, Marcynuk and Snedeker2009). Risk of bias due to randomization was assessed as ‘low’ if the authors reported randomization and details of the method used to generate the sequence; ‘some concerns’ if random allocation was reported but no details on how the random sequence was generated were reported; and ‘high’ if no information on allocation was provided or if a non-random method was used. Risk of bias due to blinding was assessed as ‘low’ if both caregivers and outcome assessors were blind to the treatment group, ‘unclear’ if caregivers or outcome assessors were blinded, but not both, and ‘high’ if neither caregivers nor outcome assessors were blinded.
Indirectness (how closely the populations studied resemble the target populations for the intervention) was not considered to be an issue due to the eligibility criteria for the review, and therefore the risk of bias was considered ‘low’ for all studies. Bias due to imprecision was assessed using 0.8 and 1.25 as the clinically important ORs. Similarly, ORs of 0.8 and 1.25 were used to assess heterogeneity. The incoherence (inconsistency) analysis from CINeMA was not reported from as this was conducted based on the Bayesian analysis described elsewhere in this paper.
The process recommended to assess across-study bias in an NMA is not well developed. Further, no pairwise comparisons in this review included more than 10 trials, which is the number typically believed to be necessary for an accurate across-study bias assessment (Sterne et al., Reference Sterne, Gavaghan and Egger2000). Therefore, across-study bias was not evaluated.
Results
Study selection
Results of the search and flow of studies through the screening process are presented in Fig. 1, including reasons for full-text exclusions. Details on all searches are available as Supplementary File S3. From an initial 2280 articles screened by title and abstract, 199 full texts were reviewed, with 152 articles not meeting full-text eligibility criteria, and 47 studies reflecting 50 separate trials included after full-text screening. Of these, 18 trials had data that were not extractable (e.g. complete data were not presented, no variance measure was provided, data were presented in graphs or figures only, etc.). Therefore, data were extracted for one or more outcomes from 32 trials.
Fig. 1. Preferred Reporting Items for Systematic Review and Meta-analysis (PRISMA) study flow diagram (Moher et al., Reference Moher, Shamseer, Clarke, Ghersi, Liberati, Petticrew, Shekelle and Stewart2015) for the systematic review of trials examining the efficacy of teat sealants given prepartum.
Study characteristics
Full details on study characteristics are available as Supplementary File S4. Trials were conducted in eight countries, most frequently in the United Kingdom (7/32), New Zealand (6/32), and the United States (5/32). The country of conduct was not reported in 22% of trials (n = 7). The trial setting was most commonly a commercial dairy (28/32; 88%), with one trial conducted at a research facility. In three trials (9%), the setting was not reported. The majority were conducted in the past two decades, with six (19%) conducted in 2010 or more recently, seven from 2000 to 2010 (22%), two from 1990 to 2000 (6%), and three prior to 1990. A substantial number of trials (14/32; 44%) did not report the year of conduct. Breed was reported in 16 (50%) of trials, with these trials conducted in cross-bred or multiple breeds (n = 9; 28%) and Holstein/Friesian (n = 7; 22%). Six trials were conducted in a single herd (19%), with the number of herds ranging from 1 to 30. The number of herds was not reported in one trial. Six trials were conducted in prepartum heifers only (19%), while 18 trials enrolled cows following their first or greater lactation (56%) and five trials had different parity inclusion criteria. Three trials did not report the parity of animals enrolled in the study.
Outcomes
IMI at calving was the most commonly reported outcome (n = 27), with nine studies reporting the incidence of clinical mastitis in the first 30 DIM, and no studies included that had extractable data for IMI in the first 30 DIM. Of the included trials, four reported LS or SCC at first test after calving, one reported milk production over the subsequent lactation, and none reported LS or SCC average over the first three tests or total antimicrobial use over the first 30 DIM. Definitions of new infections and timing of the follow-up sample used by the authors for trials measuring IMI at calving are presented in Tables 3 and 4.
Table 3. Definition of new intramammary infection (IMI) from 27 studies reporting the efficacy of teat sealant treatments given at dry-off on the incidence of IMI at calving
Table 4. Timing of the follow-up sample from 27 studies reporting the efficacy of teat sealant treatments given at dry-off on the incidence of IMI at calving
Risk of bias within studies by outcome
The results of the risk-of-bias assessment for the 23 trials included in the network meta-analysis for IMI at calving are presented in Fig. 2, showing risk in the five evaluated domains assessed in the network meta-analysis of IMI at calving. Risk of bias for the nine trials presenting outcome data for clinical mastitis in the first 30 DIM is included as Supplementary File S5. All trials for both outcomes were rated overall either as ‘some concerns’ or ‘high’.
Fig. 2. Risk of bias by domain for trials included in the network meta-analysis assessing the efficacy of teat sealants given prepartum to prevent intramammary infections (IMI) at calving (n = 23). Risk of bias was assessed according to the Revised Cochrane risk-of-bias tool for randomized trials (RoB 2) (Higgins et al., Reference Higgins, Sterne, Savovic, Page, Hróbjartsson and Boutron2016).
Risk of bias – IMI at calving
For bias arising from the randomization process, all trials were assessed as ‘some concerns’ (Fig. 2). This was driven by incomplete reporting, as only two trials reported if the allocation sequence was concealed when cows were assigned to intervention groups, and random allocation of treatment was reported in 9/23 trials (39%). An additional six trials reported random assignment of cows or quarter to treatment, but did not provide evidence of randomization, three reported a non-random process (such as even and odd ear tags), and five did not provide sufficient information to assess this area.
Bias due to deviations from intended interventions in many studies was assessed as low (14/23; 61%), as blinding of caregivers and study personnel was commonly used, treatments were commonly co-mingled in an environmental group, where differential care would be unlikely, and interventions were short-term, with deviations from intended groups unlikely. Bias due to missing outcome data was generally assessed as low risk (16/23), with six trials assessed as ‘some concerns’, and one with high risk of bias. ‘Some concerns’ resulted from a lack of reported information on loss to follow-up, and a ‘high’ risk of bias was due to a high level of missing data that was non-random or unequal between groups where results were not robust to the presence of missing data.
Bias due to the measurement of the outcome was considered low in all trials; although blinding of outcome assessors was rarely reported (17/23), laboratory diagnosis was often used. As laboratory methods are relatively objective in their measurement, this resulted in a low overall risk of bias in this domain.
For bias arising from the selection of the reported results, information regarding a priori intentions of outcome measurements and analyses was not available for any studies; this domain generally requires the examination of a trial protocol or statistical analysis plan documented ahead of the trial if there are multiple ways an outcome could be measured or analyzed. As a result, all trials were assessed as ‘some concerns’ in this area.
Results of individual trials
Of trials examining IMI at calving, 12 reported adjusted estimates of the treatment effect and 16 reported unadjusted (crude) estimates of the treatment effect. Only one trial reported results at the cow level and utilized a single herd; all other trials either had multiple herds enrolled and/or quarter-level data. Controlling for clustering for herd and/or cow (when appropriate) was done in 9/28 trials; all others did not adjust for a lack of independence in both factors (if present).
Quantitative summary
A network meta-analysis was conducted for trials examining the incidence of IMI at calving; no trials were identified examining the incidence of IMI in the first 30 DIM, and too few trials examining clinical mastitis in the first 30 DIM were found to inform a treatment network.
Pairwise meta-analysis – incidence of clinical mastitis in the first 30 DIM
Of the nine trials included, four were trials comparing internal teat sealant (bismuth subnitrate) to an NTC. The other five trials described single intervention comparisons which were not replicated. Therefore, pairwise meta-analysis was conducted only for the comparisons between internal teat sealants and an NTC. Of the four included trials, one contained no clinical mastitis events in either treatment group and therefore could not contribute to the overall effect estimate (Fig. 3). Substantial heterogeneity was observed (I 2 = 77%), but further exploration to identify subgroups associated with heterogeneity could not be explored due to the limited number of included trials. The overall effect of teat sealant was protective (RR = 0.43, 95% CI 0.17–1.10).
Fig. 3. Forest plot showing the effect of treatment with an internal teat sealant (bismuth subnitrate) compared to a non-treated control group on the incidence of clinical mastitis over the first 30 DIM. Each study is listed by the first author's last name and year of publication. The squares indicate the individual study's effect size as a risk ratio. The horizontal line shows the corresponding confidence interval. The center of the diamond shows the overall effect estimate, with the width of the diamond showing the confidence interval of this estimate.
Network meta-analysis – incidence of intramammary infection at calving
The full network plot of treatments assessed for IMI at calving is shown in Fig. 4. Two treatments, intramammary penicillin (PEN) and intramammary penicillin with an internal teat sealant (TS_PEN) were not connected to the larger network, and so could not be included in the network meta-analysis. The network of evidence used in the network meta-analysis is shown in Fig. 5, and represents 54 intervention arms from 23 trials, including 18 two-arm trials, two three-arm trials, and three four-arm trials. Trials included in the network meta-analysis are bolded and underlined in Supplementary File S4.
Fig. 5. Treatment arm network for the examination of the relative efficacy of teat sealant treatments at dry-off to prevent intramammary infections (IMI) at calving. The size of the circle indicates the relative number of arms and the width of the lines indicates the relative number of direct comparisons. Full treatment arm descriptions are found in Table 2.
Assessment of consistency
The consistency assessment for all direct and indirect comparisons is shown in Table 5. Means and standard deviations of log OR of treatment effects are shown using direct (head-to-head) evidence only, indirect evidence only, and the combined evidence. The inconsistency estimate and standard deviation are presented; there was no evidence of significant inconsistency between direct and indirect estimates. The contribution of studies to estimates based on the randomization status of the study is presented in Fig. 6, and contribution of studies to estimates based on blinding is presented in Fig. 7. Although most pairwise comparisons included a roughly equal contribution from studies which randomly allocated to treatment and provided evidence of random sequence generation, those which described random allocation with no supporting evidence, and trials where the allocation method was not reported or a non-random method was described (Fig. 6), the majority contribution (largest component) for 32 of 36 comparisons was from those describing random allocation without supporting evidence. For the contribution of trials to estimates based on blinding (Fig. 7), in most pairwise comparisons, there was only a very small (or no) contribution from trials reporting blinding of both caregiver and outcome assessors. Although most pairwise comparisons had some contributions from studies reporting blinding of either caregivers or outcome assessors, the majority contribution in 30/36 pairwise comparisons was from trials not reporting blinding of either caregivers nor outcome assessors. Table 6 further summarizes the risk-of-bias conclusion for each pairwise comparison for randomization and blinding, imprecision, and heterogeneity.
Fig. 6. The contribution of studies to the point estimate based on the description of allocation approach for studies contributing to the network meta-analysis examining the relative efficacy of teat sealant treatments given at dry-off to prevent intramammary infections (IMI) at calving (n = 23). Green indicates studies that randomly allocated to treatment and provided evidence of random sequence generation, yellow indicates studies that reported random allocation but did not provide supporting evidence, and red indicates studies that did not report allocation approach or reported a non-random method. White vertical lines indicate the percentage contribution of separate studies.
Fig. 7. The contribution of studies to the point estimate based on the description of blinding for studies contributing to the network meta-analysis examining the relative efficacy of teat sealant treatments given at dry-off to prevent intramammary infections (IMI) at calving (n = 23). Green indicates studies that reported both caregivers and outcome assessors were blinded to treatments, yellow indicates studies that reported caregivers or outcome assessors were blinded to treatment (but not both), and red indicates studies where blinding was not used, or not reported, for both caregivers and outcome assessors. White vertical lines indicate the percentage contribution of separate studies.
Table 5. Direct (dir) and indirect (rest) comparisons for the consistency assumption of pairwise comparisons within the network of studies examining the efficacy of teat sealant protocols given at dry-off to prevent new intramammary infections (IMI) at calving
The inconsistency estimate (ωXY) and standard deviation (SDωXY) are shown. Posterior means (d) and standard deviation (SD) of the log odds ratio of intervention effects calculated for direct (head-to-head) evidence only (dir), indirect evidence only (rest), and a combination of all evidence (MTC). The first treatment listed is the referent (denominator) and the second listed is the comparator (numerator).
Table 6. Summary of the overall quality of evidence of the network of studies examining the efficacy of teat sealant protocols to prevent new intramammary infections (IMI) at calving, using the Confidence In Network Meta-Analysis (CINeMA) platform (http://cinema.ispm.ch), with a modified approach, to determine the risk of bias due to the approach to randomization, blinding, imprecision, and heterogeneity
Imprecision and heterogeneity were determined using a clinically important odds ratio of 0.8.
Rankings and distribution probability of IMI at calving
RRs from the network meta-analysis comparing all treatments are shown in Table 7. The RR is the risk of the event (treatment failure corresponding to a new IMI at calving) in the column header (numerator), divided by the risk of the event in the row header (denominator). For example, the estimated risk of IMI at calving is three times higher for cows given an NTC compared to an internal teat sealant and intramammary cloxacillin (TS_CLOX). The corresponding confidence interval is located at the lower left-hand section of the table, with rows and column reversed (95% CI 1.42–5.28). Mean rankings and 95% credibility intervals are presented as a forest plot (Fig. 8), and as a table in Supplementary File S6. The distribution of the probability of treatment failure (probability of an IMI event at calving) is presented for each treatment in the network meta-analysis in Supplementary File S7. Although better than NTCs, the RRs of a new IMI occurring at calving were very imprecisely estimated because of the low number of replicated interventions. Therefore, although point estimates do differ, it is difficult to reach a conclusion of different effects between cows given teat sealants alone, or teat sealants combined with intramammary cloxacillin (RR = 1.12, 95% CI 0.40–1.79), cephalosporins (RR = 1.35, 95% CI 0.69–2.51), or fluroquinolones (RR = 2.07, 95% CI 0.85–4.53). There was also no difference in risk reduction among antimicrobial categories.
Fig. 8. Forest plot of mean rank and 95% credibility interval for the network meta-analysis examining the relative efficacy of teat sealant treatments given at dry-off to prevent intramammary infections (IMI) at calving. Full treatment arm descriptions are found in Table 2.
Table 7. Risk ratio comparison of all interventions assessed in the network meta-analysis for the outcome of IMI at calving
The upper right-hand section of the table represents the risk ratio between the numerator (upper left treatment) and denominator (lower right treatment). The lower left section of the table represents the 95% credibility interval for the comparison, with the rows and columns reversed. For example, the risk ratio for IMI at calving for a non-treated control (NTC) compared to an internal teat sealant and intramammary cloxacillin (TS_CLOX) is 3.09 (95% CI 1.42–7.51).
Discussion
As multiple intervention options exist for cows at dry-off to prevent IMI and clinical mastitis and comparative efficacy is an important part of choosing a preventative strategy, network meta-analysis is an appropriate instrument to provide veterinarians and other decision makers with information regarding relative efficacy. Treatment decisions may be driven by multiple additional factors, including availability, cost (e.g. direct costs, discarded milk, residue risk, etc.), importance to human health, and other considerations. With these in mind, relative efficacy can help inform decision making; for example, if two treatments appear to be similar in efficacy, the treatment with a lower cost, or lower importance to human health, can be selected. Similarly, the use of apparently ineffective products can be avoided to decrease unnecessary antimicrobial use.
Summary of evidence
From the network of evidence included in this analysis, it was apparent that internal teat sealants (all made with bismuth subnitrate) provided significant protection against developing new IMI at calving compared to NTCs (RR = 0.36, 95% CI 0.25–0.72), similar to results from previous work (RR = 0.27, 95% CI 0.13–0.55; Rabiee and Lean, Reference Rabiee and Lean2013). While in our analysis there was no significant additional benefit of the provision of any antimicrobial group in addition to the use of an internal teat sealant, a lack of replication of interventions means that we cannot reach a definitive conclusion of the efficacy of additional antimicrobial administration, nor if differences exist between antimicrobial groups.
For the comparison of NTCs to teat sealants, imprecision was assessed as ‘no concerns’, which indicates that the boundaries of the 95% CI around the point estimate did not include values that would be clinically ambiguous (e.g. spanning values representing both clinically beneficial and equivalent, or clinically beneficial and clinically harmful), based on a clinically significant OR of <0.80 representing clinically beneficial and >1.25 representing clinically harmful. However, some concerns were noted due to heterogeneity, as the predictive interval did not agree in relation to clinically important effects. This indicates there are some between-study variations within this comparison, which could be due (in part) to different study populations.
Examining the pairwise comparisons between teat sealant and teat sealants plus antimicrobials, many had ‘some’ or ‘major’ concerns in regards to imprecision, meaning the 95% CI extends into the margin of equivalence (‘some concerns’) or extends into estimated ORs favoring either treatment (‘major concerns’). This means that although the point estimates may be clinically meaningful, the actual effect may lie outside of a clinically meaningful range, which is likely driven by the small number of studies included for each unique treatment (Fig. 5).
Blinding of caregivers and outcome assessors was uncommonly reported for studies evaluating the incidence of IMI at calving (Fig. 7). However, as this outcome was considered relatively objective, this resulted in a low overall risk of bias due to the assessment of the outcome (Fig. 2). Bias arising from missing outcome data was observed in some trials, which in some cases was due to a lack of reporting of the number of study units analyzed. The Reporting guidElines For randomized controL trials in livEstoCk and food safTey (REFLECT) statement recommends that the authors report the flow of study units through each stage of the study, including the number allocated, receiving the intervention, completing the protocol, and analyzed for each outcome, with the use of a diagram recommended (O'Connor et al., Reference O'Connor, Sargeant, Gardner, Dickson, Torrence, Dewey, Dohoo, Evans, Gray, Greiner, Keefe, Lefebvre, Morley, Ramirez, Sischo, Smith, Snedeker, Sofos, Ward and Wills2010; Sargeant et al., Reference Sargeant, O'Connor, Gardner, Dickson and Torrence2010).
Randomization was done in some (5/23) trials, but non-random allocation, such as assignment by even or odd ear tag number, was conducted in several, and many did not report the method of allocation. While there is evidence that since the publication of the REFLECT statement reporting guidelines, reporting of randomization is improving (Totton et al., Reference Totton, Cullen, Sargeant and O'Connor2018), reporting specific to dairy science revealed that while 104 of a sample of 137 trials published in 2017 reported random allocation to study group, only seven reported the method of randomization (Winder et al., Reference Winder, Churchill, Sargeant, LeBlanc, O'Connor and Renaud2019). Assumptions for many statistical methods rely on the interchangeable group, and failure to randomize has been associated with exaggerated treatment effects (Burns and O'Connor, Reference Burns and O'Connor2008; Sargeant et al., Reference Sargeant, Elgie, Valcour, Saint-Onge, Thompson, Marcynuk and Snedeker2009; Brace et al., Reference Brace, Taylor and O'Connor2010). Even in trials of genetically identical mice, failure to randomize has shown similar exaggerated associations (Egan et al., Reference Egan, Vesterinen, Beglopoulos, Sena and Macleod2016).
Limitations of the body of literature
Although udder health is arguably one of the most important topics in the realm of dairy cattle health and production, and despite a large number of trials in this area, there was a limited number of trials eligible to be combined in the meta-analysis (Fig. 1). Lack of comparable outcomes and inadequate presentation of required data were the most common reasons that trials could not be included in the network, as well as trials without treatment arms linking them to the larger network. However, limitations of a sparse body of comparable work pertain to any research synthesis approach.
Both case definition (Table 3) and risk period (Table 4) varied within the single outcome of IMI at calving. The exact role of existing minor pathogen IMI on the risk of new major pathogen IMI is unclear; based on a systematic review and meta-analysis, a protective effect has been reported in challenge trials, but not observational studies, and there is a large amount of heterogeneity in these meta-analyses (Reyher et al., Reference Reyher, Haine, Dohoo and Revie2012). If the existing infection does influence the risk of a new infection, then it is important that primary research consider this and ensure adequate reporting of the case definition. Risk period was variable among studies, which, assuming this has an influence on outcomes, limits the ability to further utilize this body of research. Standardized outcomes with biological meaning for a given intervention would strengthen the value of primary research. In human health, efforts to standardize outcome measures exist in multiple research areas (Williamson et al., Reference Williamson, Altman, Blazeby, Clarke, Devane and Gargon2012; Macefield et al., Reference Macefield, Jacobs, Korfage, Nicklin, Whistance, Brookes, Sprangers and Blazeby2014). Our network included trials in prepartum heifers as well as those restricted to multiparous animals. If the relative effect of these interventions is different in these populations, this may be a source of heterogeneity.
The use of an internal herbal teat sealant, or internal teat sealants (bismuth subnitrate) given in combination with tylosin, or cephalosporin and tylosin, came from single arms and therefore their relative rankings have wide confidence intervals, overlapping both the best and worst treatments. This does not provide useful evidence for relative efficacy, and highlights the need for replication, if these interventions are of interest to end users. As well, the efficacy of teat sealants given with intramammary penicillin was unable to be determined as there were no common treatment arms which connected them to the larger network. Without intervention arms common to multiple trials, it is not possible to provide estimates for relative efficacy using the network meta-analysis approach, and this in turn impairs the utility of this body of primary research.
Limitations of the review
This review only included studies in English, and as a result, our conclusions may not represent the entire body of literature assessing the efficacy of teat sealants on the prevention of IMI and clinical mastitis. Additionally, our intervention arms were collapsed based on OIE antimicrobial categories, and some arms contained differing dosages. Therefore, it is possible there may be differential effects of specific treatment protocols (e.g. product, dose) within the collapsed arms. However, assigning each product formulation and dose would have resulted in an increasingly disparate network, and we attempted to be transparent with how these data were grouped for analysis.
Conclusions
From the network of evidence produced by this analysis, it was apparent that the use of an internal teat sealant (bismuth subnitrate) was significantly protective for the development of new IMI at calving, compared to NTCs. There was no additional effect shown of adding any category of intramammary antimicrobial to the teat sealant, and so for cows without existing IMI, there did not appear to be an additional benefit of these added strategies to prevent new IMIs at calving. However, a lack of precision of the estimates of the comparisons between teat sealants and teat sealants plus antimicrobials meant that it is possible the true effects of some of these treatments are not equivalent. Synthesis of the primary research revealed challenges with comparable outcomes, replication and connection of interventions, and quality of reporting of study conduct sufficient to assess the potential risk of bias in the reported results. Consideration of the use of reporting guidelines, standardization of outcomes, and inclusion of at least one intervention arm used in other research would increase the value of primary research in this area.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S1466252319000276
Author contributions
CBW assisted with the development of the review protocol, co-coordinated the research team, assisted with data screening, extraction, and risk-of-bias assessment, interpreted results, and wrote the manuscript drafts. JMS developed the review protocol, co-coordinated the research team, interpreted the results, commented on the manuscript drafts, and approved the final manuscript. DH conducted the data analysis, provided guidance for the interpretation of the results, commented on the manuscript drafts, and approved the final manuscript. CW assisted with the development of the review protocol, provided guidance on the conduct of the analysis and interpretation of the results, and approved the final manuscript. JG and HW developed the search strings, conducted all searches, commented on the manuscript drafts, and approved the final manuscript. KJC, MdB, JD, KD, SM, BD, MR, and CM conducted relevance screening, extracted data, conducted risk-of-bias assessments, commented on the manuscript drafts, and approved the final manuscript version. AMOC, DFK, SJL, and TFD co-developed the review protocol, provided guidance on the interpretation of the results, commented on the manuscript drafts, and approved the final manuscript.
Financial support
Support for this project was provided by The Pew Charitable Trusts.
Conflict of interest
None of the authors have conflicts to declare.
Open access
