Artificial intelligence (AI)-supported technologies are rapidly developing and have the potential to improve healthcare quality at reduced cost. However, few examples exist of successfully deployed AI-technologies in a real-world context that have been adequately assessed. Therefore, the objective of this research is to: (i) identify existing health technology assessment (HTA) methods developed or adapted to assess AI-supported health technologies, (ii) identify new assessment topics or domains relevant for AI-technology uptake, and (iii) take the first step in developing a framework applicable for new challenges that emerge with the introduction of AI.

A systematic literature review of studies describing methods or frameworks to assess AI-supported health technologies was performed on PubMed from January 2010 until February 2021. Furthermore, a web page search of international HTA agencies and international organizations such as the World Health Organization, Organziation for Economic and Co-ordination and Development, and the European Commission was performed to identify important aspects to consider when implementing and assessing AI technologies.

No assessment frameworks for AI technologies were identified from the systematic literature review or web page searches of international HTA agencies. Reports from international organizations highlight limitations or inability of most AI technologies to ‘explain’ their decision-making process (black box issue), leading to lack of trust in the technology that affects its adoption. It is recommended to put more emphasis on assessing transparency and ‘explainability’ of the AI solution as well as aspects of safety, ethical, legal, and social issues related to implementation and the development/training phase of the AI technology.

The results from this study uncover key gaps in frameworks posed for performing a systematic and holistic assessment of AI in a real-world context of health care. However, valuable information on relevant assessment aspects for AI-supported technologies have been identified.

The results will form the basis for the development of a framework to assist decision-makers in assessing AI-supported technologies in a holistic manner for a responsible deployment – the HTA AI Framework.