Introduction: Despite their widespread use, measures of classification accuracy (i.e. sensitivity and specificity) have several limitations that conceals relevant information and may bias decision-making. Assessing the predictive ability of clinical tools instead may provide more useful prognostic information to support decision-making, particularly in an Emergency setting. We sought to contrast classification accuracy versus predictive ability of the Systemic Inflammatory Response Syndrome (SIRS) and quick Sepsis-related Organ Failure Assessment (qSOFA) Sepsis scores for determining mortality risk among patients with infection transported by paramedics. Methods: A one-year cohort of patients with infections transported to the Emergency Department by paramedics was linked to in-hospital administrative databases. Hospital mortality was determined for each patient at the time of discharge. We calculated sensitivity and specificity of SIRS and qSOFA for classifying hospital mortality across different score thresholds, and estimated discrimination (assessed using the C statistic) and calibration (assessed visually) of prediction. Prediction models for hospital mortality were constructed using the aggregated SIRS or qSOFA scores for each patient as a predictor, while accounting for clustering by institution and adjusting for differences in patient age and sex. Predicted and observed risk were plotted to assess calibration and change in risk across levels of each score. Results: A total of 10,409 patients with infection who were transported by paramedics were successfully linked, with an overall mortality rate of 9.2%. The median SIRS score among non-survivors was 2, while the median qSOFA score was 1. SIRS score had higher sensitivity estimates than qSOFA for classifying hospital mortality at all thresholds (0.11 – 0.83 vs. 0.08 – 0.80), but the qSOFA score had better discrimination (C statistic 0.76 vs. 0.71) and calibration. The risk of hospital mortality predicted by the SIRS score ranged from 6.6-24% across score values, whereas the risk predicted by the qSOFA score ranged from 8.6-53%. Conclusion: Assessing the SIRS and qSOFA scores predictive ability reveals that the qSOFA score provides more information to clinicians about a patient's mortality risk despite having worse sensitivity. This study highlights important limitations of classification accuracy for diagnostic test studies and supports a shift toward assessing predictive ability instead. Character count 2490