Nicole Gross, Hannah van Kolfschooten & Alice Beck

Response to BMJ 2025;388:r27

Dear Editor,

In response to the BMJ article titled ‘AI in medicine: preparing for the future while preserving what matters’, we agree that an AI-driven future in healthcare is inevitable, and we must be prepared. The article emphasizes AI’s inherent limitations and risks, the expertise and judgment of humans, and setting realistic expectations. While a ‘thoughtful approach to design and implementation’ was recommended, we felt that the article omitted the role (and downfalls) of regulation in this process.

The EU AI Act is a significant step towards unlocking the potential of AI in healthcare in a safe and ethical way. Given AI’s growing role in healthcare, the AI Act will have major implications for this sector, yet crucial gaps remain. Health Action International (https://haiweb.org/), along with other members of the Digital Rights and Health Alliance (https://healthai.haiweb.org/the-alliance), is concerned that the EU AI Act still does not offer sufficient protection for AI in healthcare. In light of the medical technology industry’s strong opposition to regulation, there are two weak points in the final version of the Act that risk being exploited.

First, in response to the risks AI may pose to fundamental rights, such as privacy and non-discrimination, the EU has introduced a requirement for a Fundamental Rights Impact Assessment (FRIA) for high-risk AI systems in Article 27. This means that deployers (such as hospitals) must assess how an AI system could affect individual rights before putting it into use. However, this assessment is not compulsory for all deployers, but only for those providing public services or those deploying high-risk systems. The ambiguous wording of Article 27 will create confusion. As health services across the EU become increasingly privatized, commercial healthcare providers may assume they are not bound by this obligation. It is important for the regulator to reinforce their commitment to healthcare as a public service and protect fundamental rights.

Second, the AI Act requires all high-risk AI systems to be registered in a public EU Database for AI Systems (https://artificialintelligenceact.eu/chapter/8/), a tool designed to increase transparency and enable regulatory oversight. However, the Act makes two exceptions that could seriously undermine this goal. First, AI-enabled medical devices are excluded from this public database, even though they are classified as high-risk systems under the Act. As a result, providers of AI medical technologies are not required to disclose crucial information, such as the system’s intended purpose or the type of data that it uses. This leaves healthcare providers and institutions in the dark as to how these systems operate and whether they pose risks to patients’ rights. While a separate database exists under the Medical Devices Regulation (https://health.ec.europa.eu/medical-devices-eudamed_en), it offers only minimal information and does not address the specific risks posed by AI, such as the risk of discrimination. Second, only public authorities are required to register their high-risk systems in the public database. This excludes private companies, even those delivering essential public services like healthcare. This blind spot is particularly worrisome given the increasing role of private actors in healthcare delivery across the EU. The EU must extend the transparency obligation to private companies.

To prepare for the AI-driven future, regulatory weaknesses must be addressed and rectified. We recommend that the EU mandates all healthcare providers, public or private, to conduct FRIAs for high-risk AI systems. Also, the scope of database registration should be expanded to include both public authorities and private companies deploying AI in healthcare services. Without these crucial regulations, the transparency and accountability promised by the AI Act will fall short. To shape the future of medicine, the further refinement of regulations is necessary to safeguard health in the age of AI.

Link to publication:

https://www.bmj.com/content/388/bmj.r27/rapid-responses

Cite as: Nicole Gross, Hannah van Kolfschooten & Alice Beck, ‘Why the EU AI Act Falls Short on Preserving What Matters in Health’, response to: Mehta R, Johansen M E. AI in medicine: preparing for the future while preserving what matters BMJ 2025; 388 :r27 doi:10.1136/bmj.r27.