The EU Artificial Intelligence Act (2024): Implications for healthcare

Published by

hannahvankolfschooten

on

Hannah van Kolfschooten & Janneke van Oirschot.

In August 2024, the EU Artificial Intelligence Act (AI Act) entered into force. This legally binding instrument sets rules for the development, the placing on the market, the putting into service, and the use of AI systems in the European Union. As the world’s first extensive legal framework on AI, it aims to boost innovation while protecting individuals against the harms of AI. Since healthcare is one of the top sectors for AI deployment, the new rules will significantly reform national policies and practices on health technology. In this article, we highlight the implications of the AI Act for the healthcare sector. We give a comprehensive overview of the new legal obligations for various healthcare stakeholders (tech developers; healthcare professionals; public health authorities). We conclude that, due to its horizontal approach, it is necessary to adopt further guidelines to address the unique needs of the healthcare sector. To this end, we make recommendations for the upcoming implementation and standardization phase.

Please cite as: Hannah van Kolfschooten & Janneke van Oirschot, ‘The EU Artificial Intelligence Act (2024): Implications for healthcare’ (2024) 149 Health Policy 105152, https://doi.org/10.1016/j.healthpol.2024.105152.

    1. Introduction

    Since April 2021, European legislators have been developing a legally binding instrument for the development, the placing on the market, the putting into service, and the use of Artificial Intelligence (AI) systems. Finally, in July 2024, the Artificial Intelligence Act (AI Act) was published in the Official Journal of the European Union (EU). [1]The AI Act entered into force in August 2024 and will be fully applicable 36 months after its entry into force – in August 2027. Actors must comply with the first rules by February 2025 [2]. This marks a turning point in the regulatory oversight of AI systems in the EU. This legislative milestone is particularly significant for the healthcare sector, where the use of AI products for diagnosis, treatment, and patient care is rapidly increasing. This short paper explores the implications of the final text of the AI Act for AI products used in healthcare. It first outlines the obligations for developers and healthcare professionals, and the protections it offers patients. It subsequently assesses its adequacy for healthcare. It concludes that, due to its horizontal approach, the AI Act contains several limitations for application to the healthcare sector. For this reason, it presents policy recommendations to better address the specific needs of the healthcare sector. While the AI Act only applies to AI products in Europe, it is anticipated that it will influence developers globally to adapt their products to the standards set by the EU because of the “Brussels Effect” [3]. In this light, the legal consequences for healthcare are relevant to healthcare organizations, health professionals, and patients worldwide. Furthermore, as the integration of AI in healthcare settings increases, the adoption of specific development standards and use norms becomes more urgent.

    2. Health-related AI in the AI Act: divergent risks, divergent rules

    The AI Act is an integral part of the EU’s broader digital policy strategy, in line with the European Strategy on AI and the Digital Single Market, which aims to position Europe as a leader in digital innovation while upholding ethical standards [4]. The AI Act was first proposed by the European Commission in April 2021. It was primarily based on the work of the High-Level Expert Group on AI, which was established in 2018. The main goal of regulating AI systems was to create a safe and trustworthy ecosystem in the EU.

    The AI Act regulates the development, placing on the market, putting into service and use of AI systems in the EU. Structured as an EU Regulation, this binding legal act applies directly and uniformly in all EU Member States, ensuring consistency and uniformity. This means that there is no need for implementing measures at the national level. The main objectives are to ensure that AI systems are safe and respect fundamental rights and values, to promote trust in AI technologies, to support innovation, and to enhance EU competitiveness in AI. Expected outcomes include increased transparency, accountability, and robustness of AI systems while preventing harmful societal impacts. During the four-year development and negotiation process, the context of the law was heavily debated by industry, states, academic experts, and civil society organizations. This led to significant changes in the final text compared to the legislative proposal first presented by the European Commission in April 2021, in particular with regard to generative AI systems, which were not addressed in the first proposal.

    The AI Act takes a risk-based approach: the higher the risk, the stricter the rule. The Act applies horizontally to all sectors. It thus covers systems used in healthcare but does not constitute distinct sectoral rules. As explained in the next Section, different types of health-related AI systems fall into divergent risk categories (Table 1). The applicable requirements thus depend on whether the health-related AI system is considered high-risk, low-risk, or minimal-risk – or a general-purpose AI model. Certain AI systems must undergo a rigorous conformity assessment, which may involve third-party evaluation, to ensure compliance with the applicable rules.

    Table 1. Requirements for health-related AI systems in the EU AI Act.

    Risk levelAI system examplesProvider obligationsDeployer obligations
    Unacceptable risk•Social scoring of individuals for health benefitsThe placing on the market, the putting into service and the use are prohibited (Article 5).The placing on the market, the putting into service and the use are prohibited (Article 5).
    High-risk•AI-based medical devices falling within the scope of Regulation (EU) 2017/745 and 2017/746 (e.g. AI Clinical Decision Support Systems);•AI for risk assessment and pricing for health insurance;•AI for evaluating and classifying emergency calls; AI for decisions on dispatching medical aid;•AI for emergency healthcare patient triage systems;•AI used by public authorities to evaluate eligibility for essential public assistance benefits and services, including healthcare services.AI literacy measures (Article 4); Risk management system (Article 9); Data quality and data governance (Article 10); Technical documentation (Article 11); Record-keeping and documentation keeping (Article 12 and 18); Transparency and information duties (Article 13); Human oversight measures (Article 14); Accuracy, robustness and cybersecurity (Article 15); Quality management system (Article 17); Take corrective actions in case of non-conformity (Article 20); Undergo conformity assessment; obtain CE marking and registration (Article 43–49)AI literacy measures (Article 4); Use systems in accordance with instructions (Article 26(1)); Assign human oversight to qualified natural persons (Article 26(2)); Ensure relevant and sufficiently representative input data (Article 26(3)); Monitor the functioning and inform stakeholders of serious incidents (Article 26(5) and Article 72); Keep automated logs (Article 26 (6)); Registration obligations for certain deployers (Article 26(8) and Article 49); Carry out data protection impact assessment (Article 26(9)); Fundamental rights impact assessment (Article 27);
    Low-risk•AI-chatbots providing advice on wellbeing;•AI-generated medical deepfakes (e.g. adding and eliminating tumours from medical images);•AI-based wandering detectors in long-term care homes;•AI-based food intake sensors in home care settings.AI literacy measures (Article 4); Transparency obligations (Article 50).AI literacy measures (Article 4); Transparency obligations (Article 50).
    Minimal-risk•AI used in pharmaceutical research and development;•AI-based systems used for administration in healthcare;No requirements in the EU AI Act.No requirements in the EU AI Act.
    General-purpose models (with or without systemic-risk)•Large Language Model to generate synthetic patient data;•Large Language Model to discover new drugs;•Large Language Model to take clinical notes.Technical documentation; Transparency and information duties; Adopt policy for copyrights and related rights; Summary of training data; Appoint representative (No systemic risk; Article 53–54); Model evaluation; risk-assessment; reporting incidents to AI Office (Article 55).No requirements in the EU AI Act.

    2.1. High-risk AI: from medical devices to emergency calls

    AI practices posing unacceptable risks are prohibited (e.g. certain AI used for biometric surveillance). High-risk AI systems (e.g. AI diagnostic tools), must comply with certain safety and quality requirements, and operators have specific obligations regarding their use. In the healthcare sector, these obligations primarily apply to AI medical devices that are required to undergo a third-party conformity assessment by the EU Medical Devices Regulation (Article 6 AI Act). Generally, this means that only medical devices of risk class IIa or higher, used for medical purposes of diagnosis, prevention, monitoring, prediction, prognosis, treatment, or alleviation of disease, injury, or disability, that are using AI, are considered high-risk AI systems [5]. As seen in Table 1, four additional health-related uses of AI are – by way of exception – also considered high-risk, such as AI to evaluate and classify emergency calls or dispatch emergency first response services, including medical aid [6]. General-purpose AI systems – AI systems that can be deployed for a wide range of purposes– can both fall into the high-risk and low-risk categories [7].

    As seen in Table 1, high-risk AI is bound to stringent requirements, such as risk management, data governance, and human oversight. For AI medical devices, these obligations supplement the rules provided for in the EU Medical Devices Regulation, which require a third-party conformity assessment. The AI Act thus creates an extra level of regulation for AI medical devices which will be integrated into the regular conformity assessment for medical devices [4].

    2.2. Low-risk and minimal-risk AI: from wellbeing apps to hospital administration

    Most other health-related AI tools that do not serve a purely medical purpose – and thus do not fall under the definition of the EU Medical Devices Regulation – are considered low-risk AI. This also applies to medical devices that are not required to undergo a third-party conformity assessment (Class I). This includes a wide range of health-related AI systems, deployed for purposes related to wellbeing, health promotion, or activity monitoring. Concrete examples are mobile health apps monitoring mood and wellbeing or offering personalized diet recommendations based on user data, and AI-based sensors used for assisted living for older people. These systems only need to comply with certain transparency requirements because of their direct interaction with individuals, such as chatbots providing advice on wellbeing [5]. Moreover, Article 4 requires providers and deployers to take measures to ensure a sufficient level of AI literacy of their staff using (low and high-risk) AI systems. There are no rules for minimal-risk AI – systems that do not interact directly with individuals – such as AI for hospital administration.

    2.3. General-purpose AI models: from clinical notetaking to drug discovery

    For general-purpose AI models, such as Large Language Models (LLMs) capable of generating text and images on the basis of user input, the AI Act stipulates separate rules [6]. These models are not AI systems in themselves but are increasingly integrated into AI systems. For this reason, they do have a significant impact on how AI systems function. Therefore, these AI models need to meet certain transparency requirements, such as disclosure of technical documentation and a description of the data used for training. As some general-purpose AI models can pose systemic risks due to high-impact powerful systems (e.g. GPT-4), extra requirements apply to these, such as performing model evaluations and documenting and reporting serious incidents [7].

    3. Obligations for providers of health-related AI

    Most rules in the AI Act apply to providers. Foremost, providers are the developers of AI systems. This means that most of the time, the provider obligations in Table 1 only apply to the tech company that developed the AI software, tool, or device and placed it on the market. However, as some healthcare organisations such as hospitals and long-term care homes are developing AI systems for their own use, these rules also apply to them. For example, during the COVID-19 outbreak, several hospitals put into service AI systems – trained on their own patient data – to classify their patients based on their health status to make triage decisions. National public health authorities are also increasingly putting AI systems into service, such as software to predict and prevent the spread of infectious diseases, and mobile health apps that use AI to provide personalized health advice. In these cases, healthcare organisations and public health authorities must comply with the requirements for providers in Table 1, depending on the risk classification of the AI system.

    4. Obligations for deployers of health-related AI

    The AI Act defines a deployer as ‘a natural or legal person, public authority, agency or other body using an AI system under its authority’. This means that health professionals using AI for healthcare activities must comply with the rules stipulated for deployers. The same goes for public deployers such as public health authorities. The obligations of deployers are however limited. First of all, health professionals and public health authorities primarily have obligations for high-risk AI devices. As seen in Table 1, Article 26 lists the key deployer obligations, such as training of personnel using AI.

    Another important requirement is the fundamental rights impact assessment of Article 27. This requires deployers – before deploying the AI – to identify the risks that could emerge for fundamental rights and design measures to mitigate potential harm. It does however not require an assessment of whether these risks are acceptable or could be prevented [8]. Indeed, it only applies to high-risk AI and solely covers the first time the AI system is used. Moreover, it is unclear whether health professionals are bound. Generally, this obligation only binds “deployers that are bodies governed by public law or are private entities providing public services.” Recital 96 does state that private entities may provide public services in the public interest such as in the area of healthcare. However, whether all types of healthcare are governed by public law and entail a ‘public service’, varies per EU Member State. In some States, healthcare is significantly privatized and may thus not fall under this definition. By exemption, all deployers – public or private – using AI to determine access to and pricing of life and health insurance must perform the fundamental rights impact assessment.

    5. Individual rights for persons affected by AI

    The AI Act is not focused on individual rights for persons affected by the effects of AI – such as patients. For example, a right for patients to object to a health professional using an AI system to diagnose, is not included in the AI Act. Instead, it places requirements on the providers and deployers. However, Article 85 does introduce the right to lodge a complaint with a market surveillance authority for any natural or legal person that suspects an infringement of the AI Act – including patients. Moreover, Article 86 introduces the right to explanation of individual decision-making, which entails “the right to obtain from the deployer clear and meaningful explanations of the role of the AI system in the decision-making procedure and the main elements of the decision”. In principle, this right also applies to the health professional-patient relationship. However, the threshold for application is stringent: it only covers high-risk AI systems; does not apply to AI medical devices; and is only triggered if the AI decision has a significantly adverse impact on the health, safety, or fundamental rights of the user. In practice, this means that in the context of healthcare, this right only exists in relation to AI decisions concerning access to essential benefits and (healthcare) services, life and health insurance, emergency calls or the dispatch of emergency first response services, and emergency healthcare triaging. It is important to note that the right to informed consent, as protected in all EU Member States, does entitle the patient to information about the medical treatment in a manner that is sufficient to make an informed decision about whether to proceed.

    6. Limitations for the healthcare sector

    The AI Act introduces welcome quality and safety requirements for AI used in the healthcare sector, especially regarding data governance for AI medical devices [9]. However, due to its horizontal character and the absence of sectoral interpretation, it exposes several limitations for use in the healthcare sector [10]. First, the lack of comprehensive oversight and accountability mechanisms for low-risk health-related AI systems may result in the wide emergence of ineffective, unproven, and potentially harmful AI systems [11]. In addition to potential harm, it could erode public trust in health-related AI. Second, the process and purpose of the fundamental rights impact assessment are ambiguous: it requires listing potential rights impacts but contains no clear obligation to assess their acceptability or preventability [12]. Besides, the absence of a clear mandatory fundamental rights assessments for private healthcare providers may result in disparities in patients’ rights protection between providers and across Member States [13]. Indeed, the limited scope of application of the right to explanation and the lack of other individual rights disregards the importance of patients’ rights in healthcare. This could potentially also be addressed by clarifying the definition of “persons belonging to vulnerable groups” that require special consideration [14]. Moreover, if health-related AI systems are used for national security purposes (e.g. biothreat screening), they are exempt from all rules [15]. Finally, the AI Act’s scientific research exemption potentially leaves room to escape the regime for AI used in medical research and clinical trials, which may be a risk to patients [16].

    7. Conclusion and recommendations

    The AI Act is a significant regulatory step for AI in healthcare but requires a specific focus on health to address its shortcomings. The upcoming implementation phase and standardization processes provide opportunities to refine the practical effects of the Act. To this end, we recommend the following actions for the AI Office and the European Artificial Intelligence Board through the development of guidelines, codes of conduct, and harmonised standards:

    • 1.Define key concepts and specify exemptions within the Act in relation to health protection to reduce ambiguity;
    • 2.Specify the fundamental rights impact assessment and explicitly extend its application to all uses in healthcare;
    • 3.Encourage collaboration between AI developers, healthcare professionals, patient communities, and regulators to develop specific guidelines and standards for AI in healthcare, ensuring that all health-related AI is monitored.

    By prioritizing health in the AI Act’s implementation, policymakers can close existing gaps, ensuring that the legislation safeguards both the providers and deployers of health-related AI while protecting the rights and health of patients.

    Funding

    Hannah van Kolfschooten has no funding to declare. Her research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

    Janneke van Oirschot has no conflict of interest to declare. The project she works on receives funding from the European AI and Society Fund (no grant number available) and the European Union Health and Digital Executive Agency (grant number: 101175871). The funders had no role in the content of this publication or the decision to publish.

    CRediT authorship contribution statement

    Hannah van Kolfschooten: Writing – original draft, Conceptualization. Janneke van Oirschot: Writing – review & editing.

    Declaration of competing interest

    None declared.

    Ethical approval

    Not required.

    Acknowledgements

    Authors thank Tim Reed for his comments on an earlier version of this paper.

    References

    Previous Post
    Next Post