Study Design: Overview

It was determined that the annual LEADS survey would consist of a set of core items, to permit the assessment of changes over time, along with a set of “snapshot” items, to deal with emerging topics of interest. In addition to the annual LEADS survey, the LEADS project developed two other types of data collection instruments: the LEADS exit survey, an instrument administered to LEADS respondents who indicated that they had left the EMS profession on both a temporary or permanent basis, and an annual follow-up survey of NRs (those who did not respond to the annual LEADS survey). This follow-up survey instrument consisted of an abbreviated version of the annual LEADS survey so that a determination about representativeness of the LEADS survey respondents could be rendered.

The following considerations and procedures were undertaken to counter potential threats to data validity. First, the sample had to be representative of the EMS workforce to whom findings would be generalized. Second, survey items had to be comprehensible to potential respondents who would also have to be knowledgeable informants about the topic. Third, respondents had to be representative of the sampling strata from which they were selected. Finally, analytic rigor was to be ensured so that data were not incorrectly analyzed.

With these concerns in mind, a special study of EMS professionals who were not nationally certified was undertaken. In 2000, a random sample of state-licensed EMS professionals who were not nationally certified were administered the annual LEADS survey. Their responses to the LEADS survey were compared with the survey responses of nationally certified respondents. Methods and results of this special study are discussed subsequently. To address the second concern, all items on the survey were cognitively tested, using customized protocols with EMT-Basics and EMT-Paramedics. To address the third concern, NRs were given an abbreviated version of each year’s LEADS survey. This was sent as a follow-up to NRs in an effort to estimate the magnitude and direction of NRs bias. To address the final concern, use of LEADS data were restricted. Researchers who wanted to use the LEADS data were required to prepare data use requests, specifically outlining their research questions. All submitted requests were reviewed by LEADS Committee members according to criteria identified by the LEADS Committee. Technical assistance was provided to facilitate analyses, ensuring that survey weights were properly employed, and that statistics were properly interpreted.

Sampling

Annual LEADS Survey

The initial sampling frame for the LEADS study was comprised of all EMT-Basics and EMT-Paramedics who were certified by the National Registry of Emergency Medical Technicians (NREMT; Columbus, Ohio USA) on September 15, 1999. Separate sampling frames were established for EMT-Basics and EMT-Paramedics. These frames were stratified by duration of continuous certification at each level, with “new” referring to those certified at the given level for less than one year, and “experienced” referring to those certified for one year or longer at the given level. Additionally, the sample was stratified by race/ethnicity with “white” referring to those who self-identified as white, other, or did not self-identify. “Minority” was reserved for those who self-identified as Asian, Black, Hispanic, or Native American.

Disproportionate stratified sampling was employed to enhance the power of the sample for making comparisons between EMT-Basics and EMT-Paramedics, between white and minority (non-white) respondents, and between “new” and “experienced” EMT-Basics and EMT-Paramedics. In order to generalize results to all EMS professionals, weights were calculated to reflect a stratum member’s probability of selection. These weights were adjusted for differential non-response rates within strata. The sizes of the strata, sampling fractions, sample, and return rates for all years of the LEADS survey are presented in Tables 1a-2b. Response rates were calculated using the American Association of Public Opinion Researchers’ (Oakbrook Terrace, Illinois USA) Response Rate 2 formula.Reference Levine ¹

Table 1a LEADS EMT-Basic Sample and Response Rates, 1999-2003

Abbreviations: EMT, Emergency Medical Technician; LEADS, Longitudinal EMT Attributes and Demographic Study.

Table 1b LEADS EMT-Basic Sample and Response Rates, 2004-2008

Abbreviations: EMT, Emergency Medical Technician; LEADS, Longitudinal EMT Attributes and Demographic Study.

Table 2a LEADS Paramedic Sample and Response Rates, 1999-2003

Abbreviation: LEADS, Longitudinal Emergency Medical Technician Attributes and Demographic Study.

Table 2b LEADS Paramedic Sample and Response Rates, 2004-2008

Abbreviation: LEADS, Longitudinal Emergency Medical Technician Attributes and Demographic Study.

To enable generalizability of findings from LEADS to the current population of EMS professionals for each year that the LEADS survey was administered, the sample of EMS professionals was freshened on an annual basis with individuals who had entered the profession (ie, became nationally certified) in the preceding 12 months. For example, in 2000, the second year of LEADS project, a sampling frame of new EMS professionals was used to freshen the sample, since the “new” 1999 survey respondents were no longer new. By 2000, these individuals had been nationally certified for at least one year, while many other EMT-Basics and EMT-Paramedics had entered the profession. So, the year 2000 frame of “new” EMS professionals was comprised of all EMS professionals who became nationally certified between September 16, 1999 and September 15, 2000. The “white, experienced” 2000 sample was comprised of the white, experienced EMS providers and the white, new EMS providers who responded to the 1999 survey. Analogous freshening procedures were employed in all subsequent years.

Sample attrition within certain “experienced” strata became a concern. Accordingly, starting in 2002, the samples for the experienced strata were augmented through a random selection of experienced EMS professionals within each stratum who were not respondents to the preceding year’s study.

Exit Survey

In each year, except for 2002, an exit survey of 34 closed-ended items was prepared and sent to EMS professionals who indicated on the year’s core survey that they were either temporarily or permanently not practicing as an EMS professional.

Non-responder Survey

In each year, except for 2000, an abbreviated version of the LEADS survey, containing selected demographic items from the core survey and selected items from the year’s snapshot survey, was mailed out to a sample of respondents who did not respond within six months after the original LEADS survey was mailed out. In 2000, LEADS conducted a special study of state-registered EMS providers rather than conducting a non-response follow-up survey.

Before 2003, NRs from all of the sampling strata were sent follow-up surveys. For example, for the 1999 survey, 500 EMT-Basics and 500 EMT-Paramedics NRs were sent follow-up surveys. For this survey administration, follow-up surveys were received from 154 EMT-Basics (30.8%) and 207 EMT-Paramedics (41.4 %), optically scanned, and used to create analytical data files. This enabled comparisons of LEADS respondents and NRs to the annual survey. In subsequent years, surveys were mailed to non-response respondents from two selected strata. This procedural change was implemented to reduce burden and costs. Samples of about 200 NRs per stratum were selected for follow-up surveys.

Survey Instrument Development

Annual Survey

The annual LEADS survey consisted of two sections. The first section was the “core” survey and consisted of 79 items that were intended to be readministered annually. The LEADS Committee members selected the following areas for the development of core items: (1) employment characteristics (location, community size, service mix); (2) workforce issues (level of employment, hours worked, job satisfaction, likelihood of leaving profession, salary, experience); (3) personal characteristics (basic demographics, self-assessed health, self-assessed fitness, health issues, reasons for entering profession); and (4) continuing education (amount, delivery modes, instructor characteristics).

Draft items were prepared, circulated among LEADS Committee members for review and comment, and a version for pilot testing was prepared. The initial LEADS survey (1999) was pilot tested on 42 EMT-Basics and EMT-Paramedics at eight different locations throughout the United States, using a cognitive interviewing protocol. ^{2 ,} Reference Willis ³ This protocol employed a variety of cognitive interviewing probes, which were intended to identify issues associated with respondents’ comprehension of the items, their ability to provide the requested information, difficulties associated with response formation, and issues associated with response production (including scaling issues). This protocol elicited extensive feedback about specific survey items as well as general feedback about the instrument and the proposed co-operation elicitation procedures. As a result of the pilot testing, additional response categories were developed for several items, several items underwent minor revisions, and several items were deleted to reduce respondent burden.

In 2004, the LEADS core survey underwent minor modifications including the expansion of response categories for annual income to compensate for inflation; the elimination of certain items that showed little or no change over time; and the addition of items related to volunteerism and job tasks. The new core survey contained 77 items. Copies of the annual surveys are posted on the NREMT website (https://www.nremt.org/nremt/about/lead_survey.asp).

The second component of the main LEADS survey, the “snapshot,” had a different topic (or focus) in each administration. As with the core survey, draft items were prepared, circulated among LEADS Committee members for review and comment, and a version for pilot testing was prepared. Table 3 presents the snapshot topics by year, with the number of items in each snapshot.

Table 3 LEADS Snapshot Surveys by Year

Abbreviations: EMS, Emergency Medical Services; EMT, Emergency Medical Technician; LEADS, Longitudinal EMT Attributes and Demographic Study.

In each year, the new snapshot items were pilot tested, using the procedures employed for developing and testing core survey items, with a protocol that included cognitive interviewing probes developed specifically for the new items. The 2008 snapshot did not undergo cognitive testing, as it was made up entirely of items that had been previously pilot tested. Proposed revisions to core items also underwent similar cognitive testing. The number of EMS professionals participating in cognitive testing in subsequent years varied from seven to nine. All of these pilot tests were successful in identifying problems and informing item revisions.

Exit Survey

Items on the exit survey were concerned with the importance of different factors in the individual’s decision to leave the EMS profession. The exit survey also included an item about the likelihood of returning to the profession. The exit surveys were developed by the LEADS Committee, using procedures analogous to the core and snapshot survey development process. Response rates to the exit surveys are presented in Table 4.

Table 4 LEADS Exit Survey Sample and Response Rates by Year

Abbreviation: LEADS, Longitudinal Emergency Medical Technician Attributes and Demographic Study.

^aExit survey was not administered in 2002 due to administrative error.

Non-responder Survey

The NR survey was an abbreviated version of the annual LEADS survey, containing selected demographic items from the core survey and selected items from the year’s snapshot survey. Shortened NR surveys (a follow-up survey) are an approach recommended by the US Department of Education’s National Center for Education Statistics (Washington, DC USA) to deal with high unit non-response.Reference Willis ⁴

Certain core items—type of employer, satisfaction with one’s assignment, satisfaction with the profession, and annual EMS earnings—were included in all follow-up surveys. The length of the follow-up survey was restricted to a single sheet of paper, and varied from 15 to 28 items, year to year.

Analysis and Results

Non-responder Surveys

The responses of EMT-Basics and EMT-Paramedics who completed and returned the NR follow-up survey were compared with the responses of demographically similar (ie, EMS professional level, experience, and race/ethnicity) annual LEADS survey respondents. Discriminant analyses were used to compare the overall response patterns of non-response survey respondents with comparable annual LEADS survey respondents. Discriminant analyses attempt to classify individuals as being either a “non-response survey respondent” or a “regular survey respondent” based on response patterns characteristic of each type of respondent. The accuracy of the classification criterion can be tested statistically through calculation of multivariate statistics such as Wilk’s λ or Pillai’s trace. ⁵ PROC DISCRIM (SAS Version 9.1) was used to conduct these analyses. Additional analyses, comparing mean responses for NR and regular survey respondents, within each stratum, were conducted using PROC TTEST (SAS Version 9.1). Discriminant analyses are thought to be a better way to test for differences between types of survey respondents. Given the large numbers of comparisons made, about 5% of the comparisons, by chance alone, would appear to be statistically significant. The results of the t test comparisons are presented for their potential value in identifying specific attributes that appear to reliably distinguish regular annual survey and NR survey respondents.

Analysis of the 1999 survey indicated differences in the response patterns of three types of EMT-Paramedic survey respondents: (1) new, minority; (2) experienced, minority; and (3) new, white. These differences were associated with responses to an item about EMS income in the past 12 months, for which NRs reported significantly higher incomes than regular survey respondents. Accordingly, estimated EMS earnings for regular survey EMT-Paramedic respondents may underestimate income. When this income item was excluded, there was little evidence that NRs differed from respondents on demographic, attitudinal, or education items; that is, Wilk’s λ was no longer statistically significant, indicating the absence of a criterion for distinguishing between regular and NR survey respondents.

Similar findings were noted for most years’ survey in that there were certain variables which distinguished regular respondents and respondents to the non-response survey (within certain strata). However, for most variables no differences were detected (Table 5).

Table 5 Non-Respondent (NR) Survey Findings by Year

Abbreviations: EMS, Emergency Medical Services; EMT, Emergency Medical Technician.

Given the large number of comparisons between NR and regular (annual) survey respondents, it is not surprising that some statistically significant differences were found. However, there were several relationships which occurred in more than one group and in more than one year: NRs tended to report higher annual EMS incomes, were younger, healthier, more physically fit, and were more likely to report that they were not practicing EMS.

Special Study: Non-nationally Certified EMS Professionals

In September 2000, a special study was conducted to compare the responses of regular, annual LEADS survey respondents (nationally certified) with those of EMT-Basics and EMT-Paramedics who were not nationally certified. Samples of EMT-Basics and EMT-Paramedics not nationally certified, of predetermined sizes, were provided by 19 states. A total of 620 randomly selected EMT-Basics and 598 EMT-Paramedics comprised these samples.

Response rates for non-nationally certified EMT-Basics (22.1%) and EMT-Paramedics (27.0%) were substantially lower than comparable response rates for nationally certified LEADS respondents (35.6% for EMT-Basics and 46.4% for EMT-Paramedics). Weights were calculated for respondents to these surveys, reflecting their probability of selection and were adjusted for non-response.

Respondents were compared on 65 core survey variables. Non-nationally certified EMT-Basics differed from nationally certified EMT-Basics on 11 items (16.9%), while non-nationally certified EMT-Paramedics differed on 17 items (26.2%). Non-nationally certified EMS providers were significantly older, significantly more experienced, and significantly more likely to be employed by a fire service (Table 6).

Table 6 Comparisons of Non-nationally Certified and Nationally Certified EMS Professionals

Abbreviations: EMS, Emergency Medical Services; EMT, Emergency Medical Technician; NREMT, National Registry of Emergency Medical Technicians.

^a P<.05.

^b P<.01.

^c P<.001.

Since many of the variables on the survey were associated with these characteristics (age, experience, hours worked, and type of employer) multivariate analyses were conducted to control for these factors. After controlling for these variables, non-nationally certified EMT-Basics differed from nationally certified EMT-Basics on two items (3.3%), and non-nationally certified EMT-Paramedics differed on five items (8.3%). By chance, one would expect to find statistically significant differences between non-nationally certified and nationally certified EMS providers 5% of the time.

Discussion: Limitations

The LEADS program has resulted in the creation of a large, longitudinal data file that can be analyzed in regard to issues of concern for EMS. As with all self-reported survey data, questions about respondents’ ability to understand and accurately respond to the items may arise. To deal with these concerns, survey items used in LEADS were cognitively tested on EMT-Basics and EMT-Paramedics, and modified to incorporate results of this testing, bolstering one’s confidence that the LEADS survey items provide reliable and valid measures of the constructs they were intended to measure. In addition, selected snapshot survey items were taken from validated scales and instruments. These items were also cognitively tested to confirm their relevance for the population being studied.

Over the 10 years of data collection, annual LEADS survey response rates ranged from 40.7% to 23.0% and have trended downwards in recent years. These rates are consistent with general trends noted by survey researchers; in the past decade, response rates have declined and are continuing to decline. ^{6 ,} Reference de Leeuw and de Heer ⁷ Declining response rates have caused increasing concern about data quality and the generalizability of survey research findings. These concerns, based on the decline of a simplistic indicator (response rate) may not be justified. Recent empirical research has shown that non-response bias is not necessarily a function of non-response rates.Reference Curtin, Presser and Singer ^{8 -} Reference Merkle and Edelman ¹¹ A number of researchers feel that sample representativeness is a more important consideration.Reference Groves ¹² Under certain conditions, the use of incentives or intensive follow-up procedures, through a disproportionate impact on the response propensity of members of certain groups, may result in a higher response rate but a less representative sample.Reference Keeter, Kennedy, Dimock, Best and Craighill ^{13 ,} Reference Levine, Garfinkel and Evensen ¹⁴

In order to address concerns of non-response bias, the LEADS project conducted a series of annual follow-up surveys to LEADS NRs, using an abbreviated version of the annual LEADS questionnaire. Although such studies cannot prove the absence of bias, they can help to increase confidence in the representativeness of the sample and the generalizability of findings. The LEADS follow-up NR surveys suggested that NRs were, in general, younger, in better health, more physically fit, and earned more money in their EMS jobs than regular LEADS survey respondents. These tendencies seemed to be stable over time. There was also some evidence that recent NRs were more satisfied with their EMS position and showed greater occupational commitment.

As with all probability surveys, some caution in interpreting LEADS findings merits discussion. For most years, it was not possible to discriminate respondents from NRs, and when such discriminations were possible, they were generally associated with a specific respondent characteristic. In other words, while the use of LEADS data for estimating EMS earnings, overall health, age, or physical fitness of the population will probably produce underestimates, the LEADS data can be more safely used to compare different types of EMS professionals with respect to these characteristics and to assess how these characteristics have changed over time.

In 2008, there were over one-quarter of a million people who were nationally certified as EMT-Basics and Paramedics. The National Association of State Emergency Medical Services Officials (Falls Church, Virginia USA) reported that there were nearly 800,000 EMT-Basics and Paramedics holding valid licenses to practice in 2007.Reference Pearson, Levine and Krosnick ¹⁵ It is, however, very unlikely that all of these licensed EMT-Basics and Paramedics were practicing EMS. Nonetheless, there are clearly more individuals practicing as EMS professionals than individuals with current National EMS Certification.

In order to determine how generalizable LEADS results are to all EMS professionals in the United States, rather than only to the universe of nationally certified EMS professionals, the NREMT conducted a special study. This study compared nationally certified EMS professionals with similar EMS professionals who were not nationally registered. There were several significant demographic differences, including age, experience, and type of employer, indicating that generalizing from LEADS data to the universe of practicing EMS professionals should be done cautiously. These differences may reflect state and local policy and practice. Many states and local agencies do not require maintenance of National EMS Certification as a requirement to practice. Since maintenance of National EMS Certification requires continuing education, and extra time, it is likely that some EMS professionals will choose to avoid these additional time and financial burdens (non-nationally certified EMS professionals were significantly older, more experienced, and more likely to have a fire-based employer). When these differences were controlled for statistically, there were few systematic differences between non-nationally certified and nationally certified EMS professionals. In other words, older, more experienced, and fire-based non-nationally certified EMS professionals do not appear to be different from their nationally certified peers. However, the LEADS data should not be used to produce estimates of the average age or the average amount of experience possessed by the typical American EMS professional. With respect to the other workforce issues assessed in the LEADS survey, such as EMS position satisfaction, reasons for joining the profession, and most of the other core survey items, LEADS respondents appear to be similar to non-nationally certified EMS professionals.

In spite of some weaknesses, the LEADS Committee and research team urges the reader and interested researchers to recognize the strengths and value of LEADS. The EMS Workforce for the 21 ^st Century: A National Assessment noted that the LEADS study was the only national, randomized, and longitudinal data source for studying EMS professionals in the United States. ¹⁶ This document also noted that LEADS was one of the few data sources that distinguished EMT-Basics from Paramedics, and that this project was the only EMS professional data source useful for comparative analyses of minority and non-minority EMS professionals. ¹⁶ Given the diversity of topics covered, the rigor employed in developing and testing survey items, the representativeness of the sample, the size of the sample (and its associated power for detecting differences), its longitudinal nature, and the generalizability of its findings, others are encouraged to consider using LEADS data to investigate areas of interest and to use LEADS data to help inform their decision making.