The performance of diagnostic health technologies is usually assessed by comparing them with standard care using the kappa statistic. These comparisons are made based on comprehensive clinical information (e.g., anamnesis and complementary tests). However, not all digital applications (DAs) execute over complete information, which leads to work under non-uniform distribution of values. Using kappa statistic in this situation has serious methodological limitations. Kappa assumes that the marginal values are uniformly distributed and highly weights the discordant values when calculating concordance, which underestimates the real effectiveness of DAs (i.e., observed concordance). We aimed to present the application of the B statistic to WtsWrng, a symptom triage DA for individuals.

WtsWrng was used by 382 patients at the emergency department of a hospital. Diagnoses provided by WtsWrng, given 19 symptoms, were compared with those logged in the hospital’s electronic clinical records at discharge. Observed concordance was calculated using contingency tables. The concordance using the kappa and B statistics were compared for the 12 most frequent diagnoses at hospital discharge. Sensitivity and specificity were also calculated.

Real observed concordance fluctuated from 0.4 to 0.98 for the 12 most frequent diagnoses, eight of which had a concordance greater than 0.8. The results ranged from -0.005 to 0.37 when using the kappa statistic and from 0.36 to 0.99 when using the B statistic. The sensitivity and specificity of WtsWrng were greater than 0.8 for three and eight of the 12 diagnoses, respectively.

The results show that the B statistic is closer to the real observed concordance when kappa statistic assumptions are not fulfilled by a DA. Therefore, the B statistic is better suited for assessing the effectiveness of this type of technology. Analysis of WtsWrng using the B statistic showed that its diagnoses were close to those provided by clinicians, which were arrived at using complete clinical information. Moreover, the high specificity of the WtsWrng DA suggests that it is a good tool for determining the appropriate use of healthcare resources.