Hostname: page-component-586b7cd67f-2plfb Total loading time: 0 Render date: 2024-11-26T12:24:33.583Z Has data issue: false hasContentIssue false
  • English
  • Français

When Medical Devices Have a Mind of Their Own: The Challenges of Regulating Artificial Intelligence

Published online by Cambridge University Press:  17 March 2022

Jessa Boubker
Jessa Boubker*
Affiliation:
Foley & Lardner LLP, Boston, MA, USA
Get access Rights & Permissions [Opens in a new window]

Abstract

How can an agency like the U.S. Food & Drug Administration (“FDA”) effectively regulate software that is constantly learning and adapting to real-world data? Continuously learning algorithms pose significant public health risks if a medical device can change overtime to fundamentally alter the nature of a device post-market. This Article evaluates the FDA’s proposed regulatory framework for artificially intelligent medical devices against the backdrop of the current technology, as well as industry professionals’ desired trajectory, to determine whether the proposed regulatory framework can ensure safe and reliable medical devices without stifling innovation. Ultimately, the FDA succeeds in placing effective limits on continuously learning algorithms while giving manufacturers freedom to allow their devices to adapt to real-world data. The framework, however, does not give adequate attention to protecting patient data, monitoring cybersecurity, and ensuring safety and efficacy. The FDA, medical device industry, and relevant policymakers should increase oversight of these areas to protect patients and providers relying on this new technology.

Type
Articles
Information
American Journal of Law & Medicine , Volume 47 , Issue 4 , December 2021 , pp. 427 - 454
Copyright
© 2022 The Author(s)

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

†

Jessa graduated with a JD/MPH in May 2021 from Boston University School of Law and Boston University School of Public Health. Jessa would like to thank Professor Frances Miller both for her help advising the writing of this Article and for her boundless encouragement and mentorship throughout Jessa’s law school career. The author can be contacted at [email protected]

References

1 U.S. Food & Drug Admin., Statement from FDA Commissioner Scott Gottlieb, M.D. on Steps Toward a New, Tailored Review Framework for Artificial Intelligence-based Medical Devices (Apr. 2, 2019) [hereinafter Commissioner Statement], https://www.fda.gov/news-events/press-announcements/statement-fda-commissioner-scott-gottlieb-md-steps-toward-new-tailored-review-framework-artificial [https://perma.cc/N6D4-FG3H].

2 U.S. Food & Drug Admin., FDA Permits Marketing of Artificial Intelligence-based Device to Detect Certain Diabetes-Related Eye Problems (Apr. 11, 2018), https://www.fda.gov/news-events/press-announcements/fda-permits-marketing-artificial-intelligence-based-device-detect-certain-diabetes-related-eye [https://perma.cc/26VQ-TQM7] (authorizing a device to detect diabetic retinopathy, an eye complication caused by high levels of blood sugar resulting in retinal damage).

3 Id.

4 Id.

5 U.S. Food & Drug Admin., FDA Permits Marketing of Clinical Decision Support Software for Alerting Providers of a Potential Stroke in patients (Feb. 13, 2018), https://www.fda.gov/news-events/press-announcements/fda-permits-marketing-clinical-decision-support-software-alerting-providers-potential-stroke [https://perma.cc/95MK-Z83K].

6 Id.

7 Id.

8 U.S. Food & Drug Admin., supra note 5; U.S. Food & Drug Admin., supra note 2.

9 U.S. Food & Drug Admin., supra note 5.

10 U.S. Food & Drug Admin., Guidance Document: Deciding When to Submit a 510(k) for a Software Change to an Existing Device (Oct. 2017) [hereinafter 510(k) Guidance], https://www.fda.gov/regulatory-information/search-fda-guidance-documents/deciding-when-submit-510k-software-change-existing-device [https://perma.cc/9N6N-KBVF] (providing guidance as to when a manufacturer needs to submit a new 510(k)).

11 Commissioner Statement, supra note 1. As of January 2021, the FDA is still responding to feedback from the original proposed regulatory framework. The FDA has issued an action plan committing to updating its proposed regulatory framework and will issue Draft Guidance on the Predetermined Change Control Plan. See U.S. Food & Drug Admin., Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) Action Plan (Jan. 2021) [hereinafter Action Plan], https://www.fda.gov/media/145022/download [https://perma.cc/8FK6-E4YE].

12 Commissioner Statement, supra note 1.

13 See U.S. Food & Drug Admin., supra note 11.

14 U.S. Food & Drug Admin., Proposed Regulatory Framework for Modification to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) 2 (2019) [hereinafter Proposed Framework], https://www.fda.gov/media/122535/download [https://perma.cc/8CQV-M6PJ].

15 Id. at 4 (noting that AI learns by tracking its execution on a specific task to improve over time).

16 Id.; Food, Drug & Cosmetic Act, 21 U.S.C. §321(h).

17 Proposed Framework, supra note 14, at 3.

18 See Oxford Dictionary, Algorithm Definition, Lexico, https://www.lexico.com/en/definition/algorithm [https://perma.cc/FRL5-CU6S] (last visited Nov. 23, 2019).

19 Proposed Framework, supra note 14, at 3. Locked algorithms do not change with use and manufacturers must approve any modifications before it’s used. Id.

20 Id.

21 Id. at 5.

22 Continuously learning devices improve as they gather more data. These improvements could possibly modify the original intended use. See Id.

23 IMDRF SaMD Working Group, Software as a Medical Device (SaMD): Key Definitions 6 (Dec. 4, 2013), http://www.imdrf.org/docs/imdrf/final/technical/imdrf-tech-131209-samd-key-definitions-140901.pdf [https://perma.cc/Q32J-BPN7].

24 U.S. Food & Drug Admin., Software as a Medical Device (SaMD) (Dec. 4, 2018), https://www.fda.gov/medical-devices/digital-health-center-excellence/software-medical-device-samd [https://perma.cc/RFT5-ARMA].

25 IMDRF SaMD Working Group, supra note 23.

26 Proposed Framework, supra note 9, at 17-18; U.S. Food & Drug Admin., Policy for Device Software Functions and Mobile Medical Applications 3-5 (Sept. 27, 2019), https://www.fda.gov/media/80958/download [https://perma.cc/9L5U-5B9V] [hereinafter Policy for Device Software].

27 Policy for Device Software, supra note 26, at 26.

28 Proposed Framework, supra note 9.

29 Kumba Sennaar, AI in Medical Devices— Three Emerging Industry Applications, EMERJ (Feb. 10, 2019), https://emerj.com/ai-sector-overviews/ai-medical-devices-three-emerging-industry-applications/ [https://perma.cc/TRR3-NXSR].

30 Irene Dankwa-Mullan et al., Transforming Diabetes Care Through Artificial Intelligence: The Future is Here, 22 Population Health Mgmt. 229, 229 (2019).

31 Id.

32 Id.

33 Arundhati Parmar, Powered by AI, Medtronic’s Sugar.IQ diabetes assistant shows better outcomes, MedCity News (June 10, 2019), https://medcitynews.com/2019/06/powered-by-ai-medtronics-sugar-iq-diabetes-assistant-shows-better-outcomes/?rf=1 [https://perma.cc/EEH5-2MYF]. The Sugar.IQ diabetes assistant uses AI to process the Sugar.IQ data to find patterns, predict highs and lows, and alert at-risk patients. Patients receive personalized feedback and statistics based on their own data and thus can better manage their own care. Id.

34 Laura Lovett, Medtronic, IBM Watson launch Sugar.IQ diabetes assistant, Mobi Health News (June 25, 2018), https://www.mobihealthnews.com/content/medtronic-ibm-watson-launch-sugariq-diabetes-assistant [https://perma.cc/HRL5-TNLR]. Sugar.IQ is available as a mobile medical application to work alongside Guardian Connect. Id.

35 Parmar, supra note 33.

36 U.S. Food & Drug Admin., Certain Medtronic MiniMed Insulin Pumps Have Potential Cybersecurity Risks: FDA Safety Communication (June 27, 2019), https://www.fda.gov/medical-devices/safety-communications/certain-medtronic-minimed-insulin-pumps-have-potential-cybersecurity-risks-fda-safety-communication [https://perma.cc/V9XC-72YP] (noting that cybersecurity risks can potentially allowing hackers to change the pump’s settings and either over-deliver insulin, leading to low blood sugar, or stop insulin, leading to high blood sugar and diabetic ketoacidosis).

37 See id.

38 Id.

39 U.S. Food & Drug Admin., FDA approves first automated insulin delivery device for type 1 diabetes (Sept. 28, 2016), https://www.fda.gov/news-events/press-announcements/fda-approves-first-automated-insulin-delivery-device-type-1-diabetes?source=govdelivery&utm_medium=email&utm_source =govdelivery [https://perma.cc/85XR-X5GR].

40 Filippo Pesapane, Marina Codari & Francesco Sardanelli, Artificial Intelligence in Medical Imaging: Threat or Opportunity?, 2 Eur. Radiology Experimental 1 (2018), https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6199205/pdf/41747_2018_Article_61.pdf [https://perma.cc/V6VH-NMBR].

41 Id. at 4.

42 Id. at 5 (citing Vladimir Golkov et al., Q-Space Deep Learning: Twelvefold Shorter and Model-Free Diffusion MRI Scans, 35 IEEE Trans Med. Imaging 1344-51 (2016); Paras Lakhani et al., Machine learning in radiology: applications beyond image interpretation, 15 J. Am. Coll Radiology 350–59 (2018)).

43 U.S. Food & Drug Admin., Critical Care Suite 510(k) Approval Letter (July 12, 2019), https://www.accessdata.fda.gov/cdrh_docs/pdf18/K183182.pdf [https://perma.cc/8S6T-35QN]; GE Healthcare Receives FDA Clearance of First Artificial Intelligence Algorithms Embedded On-Device to Prioritize Critical Chest X-ray Review, GE Reps., Sept. 12, 2019, https://www.genewsroom.com/press-releases/ge-healthcare-receives-fda-clearance-first-artificial-intelligence-algorithms [https://perma.cc/MP29-V5U6].

44 See sources cited supra note 43.

45 See sources cited supra note 43.

46 Nina Bai, Artificial Intelligence That Reads Chest X-rays is Approved by FDA, UCSF (Sept. 12, 2019), https://www.ucsf.edu/news/2019/09/415406/artificial-intelligence-reads-chest-x-rays-approved-fda [https://perma.cc/JFP6-CF6E].

47 Id.

48 Nicolas P. Terry, Will the Internet of Things Transform Healthcare?, 19 Vand. J. Ent. & Tech. L. 327, 327 (2016) [hereinafter Terry, IoT] (noting that IoT products generate a great deal of data by monitoring patients from multiple, nonstop sensors and learning through analytics).

49 David W. Opderbeck, Artificial Intelligence in Pharmaceuticals, Biologics, and Medical Devices: Present and Future Regulatory Models, 88 Fordham L. Rev. 553, 567 (2019).

50 Terry, IoT, supra note 48, at 330.

51 U.S. Food & Drug Admin., De Novo Summary (DEN18004) (Sept. 11, 2018), https://www.accessdata.fda.gov/cdrh_docs/pdf18/DEN180044.pdf [https://perma.cc/USB7-YA9R] [hereinafter De Novo Summary]; Deloitte, 2019 Global Life Sciences Outlook 23 (2019), https://www2.deloitte.com/content/dam/Deloitte/global/Documents/Life-Sciences-Health-Care/gx-lshc-ls-outlook-2019.pdf [https://perma.cc/K9R3-8EDC] [hereinafter Deloitte, Life Sciences].

52 De Novo Summary, supra note 51 (identifying risks such as a poor-quality ECG signal, misinterpretation or over-reliance on device output, false negatives, and false positives). See also U.S. Food & Drug Admin., Evaluation of Automatic Class III Designation (De Novo) (Dec. 5, 2018), https://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/HowtoMarketYourDevice/PremarketSubmissions/ucm462775.htm [https://perma.cc/UJ68-SCDB].

53 See Terry, IoT, supra note 48, at 330.

54 Terry, IoT, supra note 48, at 328.

55 About da Vinci Systems, Intuitive (Mar. 2019), https://www.davincisurgery.com/da-vinci-systems/about-da-vinci-systems [https://perma.cc/A7U6-AYYV].

56 Anna Sayburn, Will the Machines Take Over Surgery?, 99 The Bulletin 88-90 (2017), https://publishing.rcseng.ac.uk/doi/full/10.1308/rcsbull.2017.87 [https://perma.cc/KV6G-76JC] (highlighting the development of suturing robots called the Raven Robot, PR2 Robot, and Smart Tissue Autonomous Robot).

57 Shane O’Sullivan, et al., Legal, Regulatory, and Ethical Frameworks for Development of Standards in Artificial Intelligence and Autonomous Robotic Surgery, 15 Int’l J. Med. Robotics & Comput. Assisted Surgery (2018).

58 Unlike the da Vinci robots, AI-assisted robots could make strategic, autonomous decisions for surgeons instead of just offering “rudimentary guidance.” See D. T. Max, Paging Dr. Robot, The New Yorker (Sept. 30, 2019), https://www.newyorker.com/magazine/2019/09/30/paging-dr-robot [https://perma.cc/Q8QJ-R896].

59 Id.

60 Proposed Framework, supra note 14.

61 Id. at 5.

62 Action Plan, supra note 11.

63 IMDRF SaMD Working Group, supra note 23, at 4. Manufacturers tend to develop software faster than other products, like pharmaceuticals, and introduce frequent changes through mass updates. Id. The FDA discussion paper bases modifications to AI/ML-based SaMD on the IMDRF risk categorization principles. See Proposed Framework, supra note 14, at 5.

64 Proposed Framework, supra note 14.

65 U.S. Food & Drug Admin., Artificial Intelligence and Machine Learning in Software as a Medical Device (Sept. 22, 2021), https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-software-medical-device [https://perma.cc/EZ6X-QJDQ].

66 IMDRF SaMD Working Group, supra note 23, at 22 (including adaptive, perfective, corrective, or preventive changes that must be clearly identified and traced to a specific aspect of the software). As a member of IMDRF, the FDA has adopted much of the IMDRF Working Group’s guidance suggestions in its own SaMD guidance. See U.S. Food & Drug Admin., Software as a Medical Device (SaMD): Clinical Evaluation (June 22, 2017), https://www.fda.gov/media/100714/download [https://perma.cc/N3MY-UKND]. See generally U.S. Food & Drug Admin., Global Approach to Software as a Medical Device, https://www.fda.gov/medical-devices/software-medical-device-samd/global-approach-software-medical-device [https://perma.cc/58U9-Q725].

67 IMDRF SaMD Working Group, supra note 23, at 25-26.

68 Id. at 14.

69 IMDRF SaMD Working Group, supra note 23, at 14.

70 Proposed Framework, supra note 14, at 3.

71 Id.

72 Id. at 2.

73 Sayburn, supra note 56, at 88-90.

74 Proposed Framework, supra note 14, at 3.

75 Id. at 2.

76 U.S. Food & Drug Admin., De Novo Classification Process (Evaluation of Automatic Class III Designation): Guidance for Industry and Food and Drug Administration Staff 5 (2017), https://www.fda.gov/media/72674/download [https://perma.cc/XB7F-C83T].

77 Proposed Framework, supra note 14, at 2.

78 Id. at 3.

79 Id. at 6.

80 IMDRF SaMD Working Group, supra note 23, at 14; Id. at 5.

81 Proposed Framework, supra note 14, at 3. See also 510(k) Guidance, supra note 10, at 12.

82 Proposed Framework, supra note 14, at 5.

83 U.S. Food & Drug Admin., Developing a Software Precertification Program: A Working Model 7, 12 (Jan. 2019) [hereinafter Precertification Program], https://www.fda.gov/media /119722/download [https://perma.cc/3AS4-PTPN] (proposing a streamlined premarket review that is absent in FDA’s AI/ML proposed regulations). The FDA intended the Pre-Certification Program as a flexible way to efficiently regulate software to streamline patient access. Id. at 6. The Pre-Certification Program acknowledges that traditional medical device regulations are “not well-suited for the faster, iterative design and development” unique to SaMD. Id. Instead of focusing on regulating specific software or devices, the program evaluates the organization creating the software and its “robust culture of quality and organizational excellence” and commitment “to monitoring real-world performance.” Id.

84 Commissioner Statement, supra note 1.

85 Proposed Framework, supra note 14, at 7.

86 Id. at 7.

87 Id. at 7-8. See also Precertification Program, supra note 83, at 7.

88 Precertification Program, supra note 83, at 31.

89 It remains unclear as to whether the FDA has the statutory authority to implement a streamlined review. See generally U.S. Food & Drug Admin., Precertification (Pre-Cert) Pilot Program: Frequently Asked Questions (Sept. 14, 2020), https://www.fda.gov/medical-devices/digital-health-software-precertification-pre-cert-program/precertification-pre-cert-pilot-program-frequently-asked-questions [https://perma.cc/48YA-BHDP] (indicating that the pilot program will inform the FDA as to whether new regulations or legislation will be needed to make the program permanent, as well as the FDA’s desire to test different approaches to regulating software).

90 Proposed Framework, supra note 14, at 9-10 (offering examples of SaMD GMLPs applicable to most manufacturers: clinically relevant data; consistent data with intended use and modification plans; training, tuning, and testing datasets remains separate; and output and algorithm transparency).

91 Id. at 10.

92 Id.

93 Id. (calling these predicted modifications “a region of potential changes,” all surrounding the “initial specifications and labeling of the original device”).

94 Id. at 11.

95 Id. at 10-11.

96 Id. at 12 (providing that changes from low risk to high risk, like from managing scars to diagnosing melanoma, would not be appropriate. Changes expanding use to a new patient population “for which there had been insufficient evidence available to initially support that indication for use” may be documented if the manufacturer can demonstrate a clinical association and plan, for new data collection and testing for that expanded patient population).

97 See 510(k) Guidance, supra note 7.

98 Proposed Framework, supra note 14, at 13 (requiring that changes not specified in the plan need a new 510(k), but manufacturers can refine their plans over time and ask for a “focused review”).

99 Id. at 14 (requiring manufacturers to further commit to “transparency” and include SaMD updates and label changes for modifications and changes to inputs or performance. Transparency also includes updating “supporting devices, accessories, or non-device components” and establishing communication procedures to notify users of modifications).

100 Id. at 15.

101 Id. at 7.

102 The AI/ML discussion paper suggests that reporting type and frequency would vary depending on the device’s risks, modifications, and maturity, but could include a number of mechanisms, such as real-world performance analytics. Id. at 4.

103 Sayburn, supra note 56, at 88-90.

104 Id.

105 Id.

106 Christopher James Vincent et al., Can Standards and Regulations Keep Up with Health Technology?, 64 JMIR Mhealth & Uhealth 1 (2015).

107 FDA Approvals for Smart Algorithms in Medicine in One Giant Infographic, Med. Futurist https://medicalfuturist.com/fda-approvals-for-algorithms-in-medicine/ [https://perma.cc/S9FT-CCQE] (last visited Oct. 21, 2021).

108 Id. See also MedGadget Editors, Arterys FDA Clearance for Liver AI and Lung AI Lesion Spotting Software, MedGadget (Feb. 19, 2018), https://www.medgadget.com/2018/02/arterys-fda-clearance-liver-ai-lung-ai-lesion-spotting-software.html [https://perma.cc/P4VN-YNY6] (clearing a device that can diagnose liver and lung cancer through AI software); AliveCor Named No.1 Artificial Intelligence Company in Fast Company’s 2018 Most Innovative Companies Ranking, AliveCor (Feb. 20, 2018), https://www.alivecor.com/press/press_release/alivecor-named-no-1-artificial-intelligence-company-in-fast-companys-2018-most-innovative-companies/ [https://perma.cc/4GH8-5ZG9] (discussing a device that can detect atrial fibrillation with KardiaMobile app).

109 See DreaMed Diabetes (Israel) Receives CE Mark for Platform for the management of Type 1 Diabetes, Israel Science Info (Feb. 15, 2018), http://www.israelscienceinfo.com/en/medecine/dreamed-diabetes-israel-recoit-le-marquage-ce-pour-sa-plateforme-de-gestion-du-diabete-de-type-1/ [https://perma.cc/23AV-97UX].

110 Kerstin N. Vokinger et al., Continual Learning in Medical Devices: FDA’s Action Plan and Beyond, 3 Lancet E337 n. 2-3 (June 1, 2021), https://www.thelancet.com/journals/landig/article/PIIS2589-7500(21)00076-5/fulltext [https://perma.cc/5474-SP6B] (citing Cecilia S. Lee & Aaron Y Lee, Clinical Application of Continual Learning Machine Learning, 2 Lancet Digital Health e279-e281 (June 2020); Samantha Cruz Rivera et al., Guidelines for Clinical Trial Protocols for Interventions Involving Artificial Intelligence: the SPIRIT-AI Extension, 370 BMJ m3210 (Sept. 2020)). See also PEW, How FDA Regulates Artificial Intelligence in Medical Products (Aug. 5, 2021), https://www.pewtrusts.org/en/research-and-analysis/issue-briefs/2021/08/how-fda-regulates-artificial-intelligence-in-medical-products [https://perma.cc/E8JD-CG4J].

111 Geralyn Miller, AI and Health Care are Made for Each Other, Time (Oct. 24, 2019), https://time.com/5709346/artificial-intelligence-health/ [https://perma.cc/9C5D-VPGU].

112 Id.

113 Id.

114 World Health Org., Fact Sheet: Breast Cancer (March 26, 2021), https://www.who.int/news-room/fact-sheets/detail/breast-cancer [https://perma.cc/NGJ7-NU7T].

115 Joshua J. Fenton, Is It Time to Stop Paying for Computer-Aided Mammography?, 175 JAMA Intern Med. 1837-38 (2015), https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2443366 [https://perma.cc/66NC-EQF3].

116 Ajay Kohli & Saurabh Jha, Why CAD Failed in Mammography, 15 J. Am. C. Radiology (JACR) 535, 535-37 (2017). See also iCAD, What is CAD? iCAD, inc., www.icadmed.com/what-is-cad.html [https://perma.cc/5GHW-TJKJ] (last visited Dec. 30, 2021) (citing Matthew Gromet, Comparison of Computer-Aided Detection to Double Reading of Screening Mammograms: Review of 231,221 Mammograms, 190 AJR 854-59 (2008)).

117 Fenton, supra note 115, at 1837-38.

118 Id.; Hiroshi Fujita et al., Computer-aided diagnosis: The emerging of three CAD systems induced by Japanese health care needs, 92 Comput. Methods & Programs Biomedicine 238, 238(2008) https://www.sciencedirect.com/science/article/abs/pii/S0169260708000977?via%3Dihub [https://perma.cc/7RUS-CMUM].

119 Kohli & Jha, supra note 116, at 535-37.

120 Fenton, supra note 115, at 1837-38.

121 Constance D. Lehman et al., Diagnostic Accuracy of Digital Screening Mammography With and Without Computer-aided Detection, 175 JAMA Intern Med. 1828-37 (2015).

122 Id.

123 Id. at 1837.

124 Fenton, supra note 115, at 1837-38.

125 Id.

126 Id.

127 Id.

128 See iCAD, supra note 116 (citing Gromet, supra note 116, at 854-59 (finding that a single CAD reading, compared with a double reading without CAD, resulted in a small, but not statistically significant, increase in sensitivity. CAD improves performance of a single reader and yields statistically significant increased sensitivity)).

129 Jeremy Hsu, Computers Match Accuracy of Radiologists in Screening for Breast Cancer Risk, IEEE Spectrum (Apr. 30, 2018), https://spectrum.ieee.org/computers-match-human-accuracy-in-screening-for-breast-cancer-risk [https://perma.cc/GB3E-PVT2].

130 See Social Security Act § 1862(l), 42 U.S.C. 1395y(l) (2021) (outlining CMS national and local coverage determination process to determine whether or not CMS will cover a particular item or service); 78 Fed. Reg. 48165 (Aug. 7, 2013) (describing differences between FDA and CMS review).

131 Rachel Sachs, Your Weekly Reminder that FDA Approval and Insurance Coverage are Often Linked, Bill of Health (Nov. 30, 2016), https://blog.petrieflom.law.harvard.edu/2016/11/30/your-weekly-reminder-that-fda-approval-and-insurance-coverage-are-often-linked/ [https://perma.cc/KH73-AYRU].

132 Fenton, supra note 115, at 1837-1838.

133 Medicare Coverage of Innovative Technology (CMS-3372-F), CMS.gov (Jan. 12, 2021), https://www.cms.gov/newsroom/fact-sheets/medicare-coverage-innovative-technology-cms-3372-f https://perma.cc/5HCS-MACC; see also Final Rule, Medicare Program; Medicare Coverage of Innovative Technology (MCIT) and Definition of “Reasonable and Necessary”, 86 Fed. Reg. 405 (Jan. 14, 2021) (codified in 42 CFR 405).

134 Glenn G. Lammi, CMS Should Offer Immediate Reimbursement Coverage to FDA-Approved Breakthrough Devices, Forbes (April 29, 2021), https://www.forbes.com/sites/wlf/2021/04/29/cms-should-offer-immediate-reimbursement-coverage-to-fda-approved-breakthrough-devices/ [https://perma.cc/PW2U-PE5U].

135 Radiology Devices; Reclassification of Medical Image Analyzers, 83 Fed. Reg. 25598-25604 (proposed June 4, 2018) (to be codified at 21 C.F.R. 892) (seeking to reclassify medical image analyzers applied to mammography breast cancer, ultrasound breast lesions, radiograph lung nodules, and radiograph dental caries detection).

136 See id.

137 Id.

138 Id.

139 See U.S. Food & Drug Admin., DEN170022, Decision Summary: Evaluation of Automatic Class III Designation for QuantX, 1 (decided July 19, 2017) [hereinafter Quant X Decision], https://www.accessdata.fda.gov/cdrh_docs/reviews/DEN170022.pdf [https://perma.cc/PL9N-RWBA].

140 Qlarity Imaging (2019), www.qlarityimaging.com [https://perma.cc/JU4A-5DC7]. See also Daneet Steffens, From Research to Commercialization: AI Diagnostic Tool Aims to Improve Breast Cancer Diagnosis, SPIE (Sept. 24, 2019), https://spie.org/news/from-research-to-commercialization-ai-diagnostic-tool-aims-to-improve-breast-cancer-diagnosis?SSO=1 [https://perma.cc/5YPH-VR8X] (explaining that QuantX was acquired by Qlarity Imaging, a subsection of Paragon Biosciences, post-FDA clearance).

141 See Quant X Decision, supra note 139, at 2.

142 See id.

143 See id. at 21-22.

144 Id. at 9; See U.S. Food & Drug Admin., Content of Premarket Submissions for Management of Cybersecurity in Medical Devices, Guidance for Industry and Food and Drug Administration Staff (2014), https://www.fda.gov/media/86174/download [https://perma.cc/PBJ2-QUVR] [hereinafter Cybersecurity Guidance].

145 Qlarity Imaging, supra note 140 (citing an unnamed clinical study from the De Novo submission).

146 Quant X Decision, supra note 139, at 3.

147 Id. at 7-24.

148 Id. at 23.

149 Melissa Locker, This AI breast cancer diagnostic tool is the first to get FDA clearance, Fast Co. (July 17, 2019), https://www.fastcompany.com/90377791/quantx-is-first-ai-breast-cancer-diagnostic-tool-cleared-by-fda [https://perma.cc/7NKT-PNWY].

150 Adam Conner-Simons & Rachel Gordon, Using AI to Predict Breast Cancer and Personalize Care, MIT News (May 7, 2019), http://news.mit.edu/2019/using-ai-predict-breast-cancer-and-personalize-care-0507 [https://perma.cc/N6SJ-WS5Z].

151 Id.

152 Richard Harris, Training a Computer to Read Mammograms as Well as a Doctor, NPR (Apr. 1, 2019), https://www.npr.org/sections/health-shots/2019/04/01/707675965/training-a-computer-to-read-mammograms-as-well-as-a-doctor [https://perma.cc/5YWR-PK6L].

153 Susan Gubar, Using AI to Transform Breast Cancer Care, N.Y. Times (Oct. 24, 2019), https://www.nytimes.com/2019/10/24/well/live/machine-intelligence-AI-breast-cancer-mammogram.html?auth=login-email&login=email [https://perma.cc/K2TZ-Y2UE].

154 Id.

155 Id.

156 Id.

157 Adam Yala, et al., A Deep Learning Mammography-based Model for Improved Breast Cancer Risk Prediction, 292 RSNA Radiology 60, 62 (July 2019), https://pubs.rsna.org/doi/pdf/10.1148/radiol.2019182716 [https://perma.cc/6ELB-EUEY].

158 Gubar, supra note 153.

159 Proposed Framework, supra note 14, at 10.

160 Id.

161 Id. at 13 (citing 21 C.F.R. 807.81(a)(3).

162 Id. at 14.

163 Harris, supra note 152.

164 Algorithm bias refers to the issue that current biases already present in the U.S. health care system, “such as race, ethnicity, and socio-economic status,” can be “inadvertently introduced into the algorithms,” furthering systemic harm to these patients. See Action Plan, supra note 62, at 6. Algorithms are “vulnerable to bias” because algorithms mirror biases already present in the data. Id. After receiving numerous comments on this issue, the FDA has added this issue to their Action Plan to address in the final AI/ML SaMD guidance. Id.

165 See Cybersecurity Guidance, supra note 144.

166 See Opderbeck, supra note 49, at 576. See also Nicolas P. Terry, Regulatory Disruption and Arbitrage in Health-Care Data Protection, YALE J. HEALTH POL’Y L. & ETHICS 143, 180-82 (2017) [hereinafter Terry, Regulatory Disruption] (explaining that many mobile health apps are not even subject to HIPAA regulations).

167 See Fenton, supra note 115, at 1838.

168 See generally Quant X Decision, supra note 139.

169 Proposed Framework, supra note 14, at 6.

170 Id. at 11-12.

171 Id. at 14.

172 FDA only requires clinical data for 10-15% of 510(k) premarket notifications. See U.S. Food & Drug Admin., Clinical trials for medical devices: FDA and the IDE process, Clinical Investigator Training Course, https://www.fda.gov/media/87603/download [https://perma.cc/EZE8-JM5V] (last visited Nov. 24, 2019).

173 See U.S. Food & Drug Admin., The Least Burdensome Provisions: Concept and Principles Guidance for Industry and Food and Drug Administration Staff (2019).

174 Proposed Framework, supra note 14, at 10.

175 U.S. Food & Drug Admin., FDA Informs Patients, Providers and Manufacturers About Potential Cybersecurity Vulnerabilities in Certain Medical Devices with Bluetooth Low Energy (Mar. 3, 2020), https://www.fda.gov/news-events/press-announcements/fda-informs-patients-providers-and-manufacturers-about-potential-cybersecurity-vulnerabilities-0 [https://perma.cc/8ZXU-NV8L].

176 Christina Farr, The Apple Watch is giving patients control over their health, but some doctors say consumers are taking it too far, CNBC (Dec. 20, 2018), https://www.cnbccom/2018/12/19/apple-watch-ecg-is-putting-a-lot-of-health-controlin-consumers-hands.html. [https://perma.cc/9PGZ-R7VG] (arguing that excess health data can result in both overuse of physician time if a patient goes to the doctor every time the app output is abnormal or underuse if a patient fails to go to the doctor despite other symptoms because the app output is normal).

177 Id.

178 Terry, Regulatory Disruption, supra note 166, at 199 (2017) (noting that the harm of unregulated data goes beyond just targeted advertising and that unregulated data paves the way for health scoring and discrimination).

179 See 21st Century Cures Act, Pub. L. No. 114-255, 130 Stat. 1033 (removing wellness apps and other low-risk software from the definition of medical device).

180 U.S. Food & Drug Admin., General Wellness: Policy for Low-Risk Devices (Sept. 2019), https://www.fda.gov/media/90652/download [https://perma.cc/QA5H-N9PH].

181 Action Plan supra note 62.

182 Benjamin Harris, FDA issues new alert on Medtronic insulin pump security, Healthcare IT News (July 1, 2019), https://www.healthcareitnews.com/news/fda-issues-new-alert-medtronic-insulin-pump-security [https://perma.cc/EA8U-LB5E].

183 Id.

184 Id.

185 U.S. Food & Drug Admin., Cybersecurity, https://www.fda.gov/medical-devices/digital-health/cybersecurity [https://perma.cc/9HUK-L392] (last visited Dec. 17, 2021).

186 Id.

187 Combination Products Coalition, Comment Letter of Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) 5 (May 31, 2019), https://www.regulations.gov/document?D=FDA-2019-N-1185-0048 [https://perma.cc/6LGV-X6A7]. The Combination Products Coalition represents a group of drug, device, and biologics industries that advocate for policy and regulatory issues affecting combination products. See About CPC, Combination Products Coalition, http://combinationproducts.com/about/ [https://perma.cc/M89A-GHVC] (last visited Nov. 24, 2019).

188 U.S. Food & Drug Admin., Computer-Assisted Surgical Systems, Mar. 13, 2019, https://www.fda.gov/medical-devices/surgery-devices/computer-assisted-surgical-systems#3 [https://perma.cc/W4WF-68JG] (last visited Nov. 24, 2019).

189 See Quality Management and Corresponding General Aspects for Medical Devices Technical Committee: ISO/TC 210, Int’l Org. for Standardization, https://www.iso.org/committee/54892.html [https://perma.cc/PGN9-6UYF].

190 Medical Devices—Application of Risk Management to Medical Devices, ISO 14971, Int’l Org. for Standardization (2019), https://www.iso.org/obp/ui/#iso:std:iso:14971:ed-3:v1:en [https://perma.cc/XYL3-LXCH]; Naveen Agarwal, Avoiding 1s0 14871 Mistakes – What Does “Harm” Really Mean?, Med. Device Online (Jan. 27, 2021), https://www.meddeviceonline.com/doc/avoiding-iso-mistakes-what-does-harm-really-mean-0001 [https://perma.cc/9V34-QCD7].

191 U.S. Food & Drug Admin., Factors to Consider Regarding Benefit-Risk in Medical Device Product Availability, Compliance, and Enforcement Decisions, Guidance Document 24 (Dec. 27, 2016), https://www.fda.gov/files/medical%20devices/published/Factors-to-Consider-Regarding-Benefit-Risk-in-Medical-Device-Product-Availability--Compliance--and-Enforcement-Decisions---Guidance-for-Industry-and-Food-and-Drug-Administration-Staff.pdf [https://perma.cc/5PXY-QXYC].

192 Pub. Resp. Med. & Rsch., Comment Letter of Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) (June 3, 2019), https://www.regulations.gov/document?D=FDA-2019-N-1185-0096 [https://perma.cc/FFG2-AEGN].

193 Mason Marks, Suicide prediction technology is revolutionary. It badly needs oversight, Wash. Post (Dec. 20, 2018), https://www.washingtonpost.com/outlook/suicide-prediction-technology-is-revolutionary-it-badly-needs-oversight/2018/12/20/214d2532-fd6b-11e8-ad40-cdfd0e0dd65a_story.html?noredirect=on [https://perma.cc/G6Q9-HX2F].

194 Id.

195 Id.

196 Id.

197 Pub. Resp. Med. & Rsch., supra note 192.

198 Id.

199 Dr. Asif Dhar et al., Reimagining Digital Health Regulation: an Agile Model for Regulating Software in Health Care, Deloitte Ctr. for Gov’t Insights 14 (2018), https://www2.deloitte.com/content/dam/Deloitte/us/Documents/public-sector/reimagining-digital-health-regulation.pdf [https://perma.cc/H69X-YL2D] [hereinafter Deloitte Government Insights].

200 See, e.g., U.S. Food & Drug Admin., Manufacturers Sharing Patient-Specific Information from Medical Devices with Patients Upon Request (Oct. 30, 2017).

201 U.S. Food & Drug Admin., FDA in Brief: FDA encourages manufacturers to take steps to share personal health care data generated by medical devices with patients (Oct. 27, 2017), https://www.fda.gov/news-events/fda-brief/fda-brief-fda-encourages-manufacturers-take-steps-share-personal-health-care-data-generated-medical [https://perma.cc/CGB6-HKNA].

202 James Titcomb, AI Is the Biggest Risk We Face as a Civilisation, Elon Musk Says, Telegraph (July 17, 2017, 9:46 AM), https://www.telegraph.co.uk/technology/2017/07/17/ai-biggest-risk-face-civilisation-elon-musk-says/#:~:text=Artificial%20intelligence%20is%20the%20%E2%80%9Cbiggest,t%20know%20how%20to%20react%E2%80%9D [https://perma.cc/Z2NH-LLB6].

203 Health Insurance Portability and Accountability Act of 1996 (HIPPA), Pub.L. 104-191, 110 Stat. 1936 (codified as amended at scattered sections of 18, 26, 29, and 42 U.S.C.); Terry, Regulatory Disruption, supra note 166, at 180-181.

204 Terry, Regulatory Disruption, supra note 166, at 143. Health scoring refers to the practice of generating health categories, “such as ‘Expectant Parent,” ‘Diabetes Interest,’ and ‘Cholesterol Focus,’” based on an individual’s health information or data. Id., at 199 (citing Data Brokers: A Call for Transparency and Accountability, Fed. Trade Comm’ n 47 (2014)). Companies, employers, and government agencies can sell and use these scores in manipulative ways outside of “‘traditional health privacy laws.’” Id., at 199 (citing Frank Pasquale, The Black Box Society: The Secret Algorithms that Control Money and Information 26 (2015)).

205 Opderbeck, supra note 49, at 577.

206 Id. at 582.

207 Id. (citing Art. 22, Section 1 GDPR).

208 Id. at 583 (citing Andrew Burt, How Will the GDPR Impact Machine Learning? O’REILLY (May 16, 2018), https://www.oreilly.com/ideas/how-will-the-gdpr-impact-machine-learning [https://perma.cc/WZ5B-PCPZ] (allowing patients to revoke consent at any time). When the patient revokes consent, the company can no longer use the patient’s data in future processing; however, using past data on processing that has already happened is legal. Id.

209 See Council Regulation 2017/745, 2017 O.J. (L 117).

210 Laura Liguori & Elisa Stefanini, EU Regulations on Medical Devices and the GDPR: First Step Forward a Necessary Coordination, EACCNY.com (June 14, 2021), https://eaccny.com/news/member-news/portolano-cavallo-eu-regulations-on-medical-devices-and-the-gdpr-first-step-forward-a-necessary-coordination/ [https://perma.cc/QCH3-YAGZ] (citing Council Regulation 2017/745, Annex I § 17.2, 2017 O.J. (L 117)).

211 Id. (citing Council Regulation 2017/745, Annex I § 17.4, 18.8, 2017 O.J. (L 117)).

212 Id. (referencing European Union Agency for Cybersecurity (ENISA), Cybersecurity Guidance Document (Jan. 18, 2021)).

213 Id. (referencing Medical Device Coordination Group (MDCG) of European Commission, Guidance on Cybersecurity for Medical Devices (Jan. 18, 2020)).

214 Id.

215 Jay G. Ronquillo & Diana M. Zuckerman, Software-Related Recalls of Health Information Technology and Other Medical Devices: Implications for FDA Regulation of Digital Health, 95 MILBANK Q. 535, 541-43 (2017). Of these recalls, 12 were high-risk and 592 were moderate risk devices. The researchers could not confirm whether FDA considered any clinical evidence in clearing these devices. The defects included malfunctions ranging from premature ventilator stoppage to incorrect patient data storage. Id.

216 Deloitte Government Insights, supra note 199, at 11 (modeling the Pre-Cert program after TSA pre✔, focusing approval on the manufacturer itself rather than the software).

217 Deloitte, Life Sciences, supra note 51, at 10.

218 Council Regulation 2017/745 of Apr. 5, 2017, On Medical Devices, Annex II 6.1(b), 2017 O.J. (L 117) (requiring pre-clinical safety tests, detailed information on test design, biocompatibility, software verification and validation (testing both in-house and in a simulated or actual user environment prior to final release), all hardware configurations and operating systems, stability, performance, and safety).

219 Council Regulation 2017/745 of Apr. 5, 2017, On Medical Devices, Part C 6.5.2, 2017 O.J. (L 117) (requiring new UDI-DI for new or modified algorithms, database structures, operating platforms, architectures, new user interfaces, or new channels for interoperability). An UDI-DI is a unique device identifier number. Id. Cf. 510(k) Guidance, supra note 7, for the FDA’s approach to software changes.

220 Council Regulation 2017/745 of Apr. 5, 2017, On Medical Devices, Part C 6.5.2, 2017 O.J. (L 117). An UDI-PI is a software identification number (like a serial number) for different versions of the unique device. Id.

221 Opderbeck, supra note 49, at 574.

222 Tina M. Morrison et al., The Role of Computational Modeling and Simulation in the Total Product Life Cycle of Peripheral Vascular Devices, 11 J. Med. Devices 024503-1, 024503-1 (2017). Computational Modeling can be compared to in silico modeling, in which computers model biological processes in lieu of costly in vitro experiments. See Richard B. Colquitt, et al., In silico modelling of physiologic systems, 25 Best Prac. & Rsch. Clinical Anaesthesiology 499, 499-510 (2011), https://www.sciencedirect.com/science/article/pii/S1521689611000656?via%3Dihub [https://perma.cc/5PL7-L2WE].

223 Morrison, et al., supra note 222, at 024503-1.

224 Id. at 024503-2.

225 Id. at 024503-1 (supporting the “development of virtual physiological patients, clinical trial simulations, and personalized medicine”). For the FDA’s guidelines, see U.S. Food & Drug Admin., Reporting of Computational Modeling Studies in Medical Device Submissions: Guidance for Industry, Sept. 21, 2016, https://www.fda.gov/media/87586/download [https://perma.cc/23LT-BS92] (known as the “CM&S Report).

226 Opderbeck, supra note 49, at 578.

227 Id. (warning that selecting training data from too narrow of a demographic can skew the model’s predictions of the device’s safety and efficacy).

228 U.S. Food & Drug Admin., Computer-Assisted Surgical Systems (Mar. 13, 2019), https://www.fda.gov/medical-devices/surgery-devices/computer-assisted-surgical-systems#3 [https://perma.cc/KYX8-L8RY] (making clear that manufacturers, physicians, and healthcare facilities are the ones responsible for training development and implementation).

229 For example, surgeons often only receive one day of training to use the da Vinci robot. Max, supra note 58.

230 O’Sullivan, et al., supra note 57, at 2.

231 Id., at 6.

232 See id. Level 0 cars give warnings but driver remains in full control; Level 1 requires a driver to remain “hands-on”; Level 2 drivers can keep their “hands-off” but must be ready to take back control immediately; Level 3 means “eyes-off” but driver can take back control if needed; Level 4 means “mind-off” so a driver could sleep or leave their seat; Level 5 means full automation, like a driverless robotic taxi. Id.

233 In Europe, only Level 0 through 2 cars are legally allowed. Id. at 2.

234 Id.

235 Deloitte Government Insights, supra note 158, at 5.

236 Deep Learning Market Research Report 2021 / Industry Challenges, Trends, Large Companies, Competition, Capacity, Key Sectors, Types, and Forecast to 2026, press release, Mkt. Watch, Nov. 17, 2021, https://www.marketwatch.com/press-release/deep-learning-market-research-report-2021-industry-challenges-trends-large-companies-competition-capacity-key-sectors-types-and-forecast-to-2026-2021-11-17 [https://perma.cc/9E8S-5HH9].

237 Proposed Framework, supra note 14 at 1.

238 See id.

239 See generally, 510(k) guidance, supra note 10.

240 Combination Products Coalition, supra note 187.

241 Proposed Framework, supra note 14, at 14.

242 Id. at 14-15 (noting that additional reporting mechanisms” may require additional statutory authority to implement fully”).

243 Id. at 14.

244 Combination Products Coalition, supra note 187, at 5.

245 Proposed Framework, supra note 14.

246 Radiology Devices: Reclassification of Medical Image Analyzers, 83 Fed. Reg. 25598-604 (proposed June 4, 2018) (to be codified at 21 C.F.R. pt. 892).

247 David Pierson & Tracey Lien, Silicon Valley Played by a Different Set of Rules. Facebook’s Crisis Could Put an End to That, L.A. Times (Mar. 23, 2018), https://www.latimes.com/business/technology/la-fi-tn-silicon-valley-reckoning-20180323-story.html [https://perma.cc/EZ63-67FJ].

248 Id. (noting Facebook’s official motto until 2014 was “move fast and break things”).

249 Nancy Huynh, How the ‘Big 4’ Tech Companies Are Leading Healthcare Innovation, Healthcare Weekly (Aug 27, 2018), https://healthcareweekly.com/how-thebig-4-tech-companies-are-leading-healthcare-innovation/ [https://perma.cc/UL4Y-8UF8].

250 Deloitte, Life Sciences, supra note 40, at 10 (referring to Amazon Echo’s diagnostic technology and Alphabet’s Calico and Verily’s therapeutic technology).

251 Id.