Tag Archives: Technology

The Council of Europe’s Artificial Intelligence Convention: Implications for Health and Patients

The Council of Europe, the most important international human rights organization on the European continent, currently is drafting a Convention on Artificial Intelligence, Human Rights, Democracy and the Rule of Law (AI Convention). The Convention aims to protect fundamental rights against the harms of Artificial Intelligence (AI), and is expected to become a global leading convention, as non-European states such as the United States (U.S.) are considering becoming signatories.

As health care is one of the top industries for AI, the forthcoming AI Convention will have important implications for the protection of health and patients. This post gives a brief outline of the background, scope, and purpose of the AI Convention. It goes on to flag common human rights issues associated with medical AI and then touches upon the most important health rights implications of the current text of the AI Convention.

  1. Background, Scope, and Purpose

Since 1949, the Council of Europe and its human rights court have played an important role in the protection and promotion of human rights in Europe through a long list of treaties, resolutions, and court decisions (some of which relate to new technologies). The Council’s scope is much broader than the European Union’s (EU): it has 47 member states (including all 27 EU Member States) and 5 observer states (including the U.S. and Japan). The observer states are not bound by the Council of Europe but can choose to sign on to the AI Convention — committing to its rules as well.

The AI Convention, which is expected by the end of 2023, will exist in parallel to and aims to complement the EU’s forthcoming AI Act and other regulatory initiatives, including the U.S. AI Bill of Rights. The AI Act focuses on the marketing of products using AI within the EU internal market, while the AI Convention focuses on the protection of the fundamental rights of people affected by AI systems. The Convention is principle-based and will — unlike the AI Act and the AI Bill of Rights  — introduce legally binding individual human rights, which also will apply to citizens of non-EU states. The exact scope of the Convention, however, is still under discussion, with states like the U.S., the U.K., and Japan advocating that it remains limited to the public sector.

  1. Medical AI: Challenges for Health and Human Rights

While medical AI can improve the quality and efficiency of health care, it also comes with several risks to health and human rights. System errors in widely used AI could cause mass patient injuries, for example, in AI-powered robot-assisted surgery. Unrepresentative or low-quality datasets used to train AI models may lead to incorrect medical decisions causing health damage, such as false diagnoses or wrong treatment decisions. When datasets reflect existing biases in society, specific populations face greater risk of health harms, reinforcing discrimination and health inequality. The enormous ‘data hunger’ of medical AI may also affect medical privacy, and the opaque nature of many AI applications may put existing health practices and other patients’ rights under pressure, such as the provision of information, informed consent, and legal redress.

  1. The Meaning of the AI Convention for Health and Patients

Lifecycle Regulation of General AI Systems

The definition of the proposed AI Convention is rather broad and includes most medical AI applications: “a machine-based system that is capable of informing or autonomously generating outputs such as predictions, recommendations or decisions using machine and/or human-based data and inputs and influencing the environment that it interacts with.” This means it covers applications like ChatGPT-assisted diagnosis, AI-based fall detection in elderly care facilities, AI-assisted surgery, and menstruation cycle trackers.

If the scope would be limited to uses in the public sector, it would still cover a fair amount of medical AI applications, such as automated disease surveillance, social benefits distribution, and most COVID-19 apps, plus many public health care systems. Moreover, the AI Convention applies to the entire lifecycle of AI: the design, development, and application of AI systems. This is especially important for adaptive AI systems that evolve over time.

This sets the Convention apart from the EU’s AI Act, which focuses on regulating the development phase, and only stipulates strict rules for “high-risk” AI medical devices.

Enforceable Rights for Patients

The AI Convention departs from the understanding that all human rights are potentially impacted by the growing use of AI systems, instead focusing on the specific rights to non-discrimination, privacy and data protection, transparency rights, and the right to redress for harm. It establishes these as enforceable rights — filling an often-criticized gap of other AI regulations.

Foremost, the Convention stipulates that all AI systems undergo a human rights assessment before deployment. Then, the Convention recognizes the right to non-discrimination, which is important for health equity — i.e., in cases of limited internet access in remote areas and insufficient digital skills to understand AI tools. The Convention protects privacy and data protection rights, setting high requirements for data protection, which is important for patients who often share intimate data with their doctor. The Convention also equips patients with the right to redress for harm, requiring robust frameworks for accountability and liability — a much-needed and currently absent solution for medical AI causing harm. The transparency rights put forward by the Convention contribute to the creation of a relationship of trust between patient and doctor and ensure informed consent to medical treatment.

Especially for health care — a sector that is mostly regulated locally — minimal human rights standards are important, as they give patients in all Member States the means to arm themselves against potentially harmful medical AI.

The Council of Europe Cares About Health

Generally, the AI Convention is promising for patients because of the Council of Europe’s long track record of protecting the health and human rights of patients. The Council has published many recommendations on medical-ethical issues aiming for stronger human rights protections in the medical field (i.e., health governance, organ donation, medical data protection). The Human Rights Court has enforced protection for patients against — inter alia — forced medical treatments, medical data breaches, and health discrimination. It has also established strong obligations for states to actively protect the health and patients’ rights of their citizens. Many states have changed their health practices and laws following Court orders, for example, for involuntary psychiatric commitment.

In short: health and patients’ rights benefit from a human-rights approach. The Council of Europe’s AI Convention, being the first binding fundamental rights instrument for AI negotiated at such a widespread scale, holds the potential of uplifting health and patients’ rights worldwide, when medical AI becomes common practice in health care.

Originally posted on Bill of Health, the blog of the Petrie-Flom Center at Harvard Law School. Please cite as:

Hannah van Kolfschooten, ‘The Council of Europe’s Artificial Intelligence Convention: Implications for Health and Patients’, 18 April 2023, Bill of Health, <https://blog.petrieflom.law.harvard.edu/2023/04/18/council-of-europe-artificial-intelligence-convention/&gt;.

Five “Big MedTech” Myths about Medical AI Debunked

Criticism of the European Union’s AI Act by the HealthTech Industry is often based on incorrect assumptions, and is in conflict with health and fundamental rights principles, write Hannah van Kolfschooten, Janneke van Oirschot, and Claudia Nicastro.

The European Union’s Artificial Intelligence (AI) Act is a world-first attempt to create comprehensive AI regulation, to protect both the internal market and fundamental rights. It also applies to AI medical devices. Defending its economic interests, the MedTech industry has fiercely criticised the “burdensome regulatory framework” of medical devices, including those in the AI Act proposal. Rather than consider them burdensome, we argue that, to protect health and fundamental rights, stronger regulation of medical AI is needed, and debunk five common MedTech myths about medical AI. 

Myth 1: “We do not need the AI Act to regulate medical AI: the MDR offers enough protection for fundamental rights” 

MedTech Europe argues that ‘AI-enabled medical technologies have been accessing the EU market safely for years, accelerated by the introduction of the Medical Devices Regulation.’ But, MedTech neglects the limited scope of application of the MDR, which uses a strict definition for medical devices and has a focus on quality and safety.

Medical devices vary widely and can be anything from medical facemasks to defibrillators. However, the incredible surge in digital health technologies, including those that use AI, is not well accounted for in this MDR. Indeed, the MDR lacks specific provisions for medical AI and does not have the explicit aim to protect fundamental rights, which the AI Act does. This while medical AI holds different risks to the health and fundamental rights of patients than other medical devices. The AI Act has a broader application than the MDR and does protect data quality and transparency, which in turn provides for more fundamental rights protection for patients using medical AI.

Myth 2: “More regulation will delay access to lifesaving medical devices” 

Pharmaceutical companies are famous for arguing that more regulation will endanger access to “lifesaving medicines”. MedTech has often made the same claim about medical devices. MedTech Europe, for example, argues that “by bringing in another regulatory layer, the AI Act risks undermining a delicate and well-balanced regulatory environment.”

Medical AI can only be ‘lifesaving’ when it’s effective and we are certain it meets high standards of quality and safety. Medical AI holds more risks to the health and fundamental rights of patients than other medical devices as it can be used by doctors for important decisions with a big impact on patients’ health. Its dependence on huge amounts of data poses risks for privacy and data protection rights, and biases in low-quality training data can lead to discrimination and injury. The AI Act’s requirements aim to guarantee better quality and safety of medical devices than the MDR regime. Also, the regulatory regime for medicines is far stricter than the medical devices regime, while medical AI may pose similar risks to health.

Myth 3: “Health and wellbeing apps do not hold risks for fundamental rights violations” 

This is an assumption that underlies the new AI Act. Unfortunately, the Act misclassifies some types of medical AI: health and wellbeing apps fall under the minimal risk class in the AI Act, meaning that they are not subject to specific requirements. However, their classification does not consider that these apps hold several privacy-related risks and can lead to discrimination. An example is apps for post-traumatic stress symptoms, where low-quality of data may lead to wrong treatment pathways and advice. Since these apps are used by vulnerable groups, these kinds of suggestions can be dangerous. This type of app also presents cybersecurity issues that lead to the violation of fundamental rights, for example, when the app shares sensitive information with unidentified third-party marketers and advertisers who misuse the information at a later date.

Myth 4: “Safety of patients trumps protection of privacy” 

The MedTech industry focuses on product safety over privacy. However, privacy is a fundamental right and should always be protected as a minimum requirement to guarantee the human dignity of people, especially the most vulnerable groups such as older people. In this case, ‘computers are increasingly guiding decisions about elder care – and tracking everything from toilet visits to whether someone has bathed’. Since privacy is a key condition for trust in healthcare, it must remain a core value to ensure a good healthcare system. In fact, constant surveillance changes patients’ behavior, so that, knowing that they are being monitored, people tend to hide their real condition. Finally, people, especially older adults, are often afraid of technology, which can worsen if their privacy is not guaranteed.

Myth 5: “An innovation oriented regulatory framework will lead to high-quality healthcare” 

The MedTech industry claims that focusing the regulation on innovation and quality will foster better patient outcomes. What they seem to overlook is that we won’t have quality care unless we have comprehensively protected fundamental rights, and for that, we need the AI Act. Although there is no clear definition of quality of care, the WHO stresses the importance of protecting fundamental rights for high-quality healthcare. In particular, the right to health as ‘the right to the highest attainable standard of health’ implies certain minimum human rights protection standards. It includes the right to control one’s health and body and to be free from interference, as well as the right to a system of health protection that gives everyone an equal opportunity to enjoy the highest attainable level of health. Therefore, high-quality care is a means to guarantee our rights, not the ends, and should be treated as such.

As explained above, these five myths are unfounded and often based on incorrect assumptions. It is clear that further regulation of medical AI is necessary to ensure better protection of patients’ rights.

By Hannah van Kolfschooten, Janneke van Oirschot and Claudia Nicastro – originally posted on HAIweb. Please cite as: Hannah van Kolfschooten, Janneke van Oirschot & Claudia Nicastro, ‘Five “Big MedTech” Myths about Medical AI Debunked’, 1 April 2023, Health Action International, available at: <https://haiweb.org/five-big-medtech-myths-about-medical-ai-debunked/&gt;.

Health protection is non-negotiable in the AI Act negotiations

A health-centric approach to the Artificial Intelligence (AI) Act is essential for the protection of health and fundamental rights of European citizens, write Hannah van Kolfschooten and Janneke van Oirschot. 

The European Commission’s proposal for an Artificial Intelligence (AI) Act has been the topic of a heated debate since its publication in April 2021. Civil society organisations believe the proposal falls short on fundamental rights protection, industry is worried it will stifle innovation, and governments fear consequences for national security. We critique the AI Act for neglecting the risks health AI pose to patients’ health and fundamental rights.

The 3,000 amendments to the Act tabled by political groups in the European Parliament say a lot about how controversial regulation of AI really is. This summer, the Parliament’s co-rapporteurs start the negotiation process with compromise amendments. Our message to MEPs, who will need to vote on the amendments is the following: Make health non-negotiable. A health-centric approach to the AI Act is essential for the protection of health and fundamental rights of European citizens, in particular the rights to access to healthcare, non-discrimination and privacy.  

AI is the simulation of human intelligence by machines. AI systems are software-based technologies that use certain data-driven approaches to solve specific problems. What all AI systems have in common, is that they recognise patterns in enormous amounts of data.

AI in the health sector is not like AI in any other sector and deserves special consideration because (1) people’s health is at stake, (2) people are in a vulnerable position when in need of healthcare, (3) the collection of health data has dramatically increased in recent times and (4) health data is historically littered with bias. Because of these characteristics, health AI faces unique risks that need to be specifically addressed in the AI Act. 

Take disease outbreak surveillance as an example. Many people with flu-like symptoms use Google for self-diagnosis. AI can use this data to monitor and predict the spread of infectious diseases. This can be useful for public health officials to make decisions about infectious disease control and how to distribute care resources.

But how accurate are these AI systems when the model is based on subjective user data? Limited regulation of the quality of health AI will lead to distrust in public health and healthcare, breeding hesitancy in access to healthcare. What is more, increased use and sharing of health data threatens privacy and data protection rights.

Another example is the use of AI for medical diagnostics. AI can be used to identify skin cancer in images of skin lesions, after being trained on the basis of thousands of images of “healthy” and cancerous skin lesions. But what happens when image datasets were non-representative, incomplete or of low-quality?

Biases in the training data can lead to discrimination and individual injury or even death. Especially racial bias may lead to incorrect diagnoses and deepen existing socio-economic inequality, something that is not taken into account in current regulation on medical technology. Additionally, lack of transparency and explainability threatens patients’ rights to information and informed consent to medical treatment.

These are just a couple of illustrations of the risks of AI usage for health, one of the most popular sectors for AI deployment in the European Union. Yet, the AI Act does not specifically address health AI and does not provide solutions for its key risks. It can’t be stressed enough that health must be prioritised when MEPs negotiate their amendments over the coming months, with some tabled amendments that deserve particular support. 

Foremost, given its extensive risk, important AI uses in health and healthcare should be marked as high-risk, which will ensure more stringent regulatory requirements. 

Second, high-risk AI should undergo a fundamental rights impact assessment, which takes into account risks to human health. Also technical documentation of health AI should include an assessment of its risks for health, safety and fundamental rights. 

Finally, AI systems that disadvantage groups based on health status should be prohibited completely. 

Similarly, we call on MEPs to strongly oppose amendments that remove health AI from the current list of ‘high-risk AI uses’ or add extra requirements for AI systems to be marked high-risk. 

It’s high time to take on a health-centric approach to the AI Act. It’s worth reiterating: health protection is non-negotiable in the AI Act negotiations. 

By Hannah van Kolfschooten and Janneke van Oirschot – originally posted on EURACTIV. Please cite as: Hannah van Kolfschooten & Janneke van Oirschot, ‘Health protection is non-negotiable in the AI Act negotiations’, 18 August 2022, EURACTIV, available at: <https://www.euractiv.com/section/digital/opinion/health-protection-is-non-negotiable-in-the-artificial-intelligence-act-negotiations/>

EU Regulation of Artificial Intelligence: Challenges for Patients’ Rights

In order to create a well-functioning internal market for Artificial Intelligence (AI) systems, the European Commission recently proposed the Artificial Intelligence Act. However, this legislative proposal pays limited attention to the health-specific risks the use of AI poses to patients’ rights. This article outlines that fundamental rights impacts associated with AI such as discrimination, diminished privacy and opaque decision-making are exacerbated in the context of health and may threaten the protection of foundational values and core patients’ rights. However, while the EU is facilitating and promoting the use and availability of AI in the health sector in Europe via the Digital Single Market, it is unclear whether it can provide the concomitant patients’ rights protection. This article theorizes the Europeanization of health AI by exploring legal challenges through a patients’ rights lens in order to determine if the European regulatory approach for AI provides for sufficient protection to patients’ rights.

In 2021, the European Commission proposed the Artificial Intelligence Act. In her recent article for Common Market Law Review, Hannah van Kolfschooten expresses concerns about this legislative proposal with regard to patients’ rights. The aim of the proposed act is to create a well-functioning internal market for Artificial Intelligence (AI) systems. However, Van Kolfschooten argues that limited attention is paid to the health-specific risks the use of AI poses to patients’ rights. She states that fundamental rights violations associated with AI are exacerbated in the context of health and may threaten the protection of patients’ rights. While the EU is facilitating and promoting the use and availability of AI in the European health sectors, it remains unclear whether it is simultaneously able to protect patients’ rights against the risks that the use of AI poses.

Please cite as: Hannah Van Kolfschooten, ‘EU regulation of artificial intelligence: Challenges for patients’ rights’, (2022), 59, Common Market Law Review, Issue 1, pp. 81-112.

Download

How the Parliament’s AIDA Report Misdiagnoses the Risks and Benefits of Health AI

In November 2021, the European Parliament’s Special Committee on AI in a Digital Age (AIDA Committee) put forward its draft report on artificial intelligence (AI) in a digital age. The report advocates for a very permissive approach to the regulation of AI to stimulate innovation and foster the competitiveness of the EU. With this, however, it understates the possible risks specific to the development and use of AI in the context of health and lacks actual solutions to translate potential into action. We identified five shortcomings and hereby appeal to the regulator to address them.

Jorge Félix Cardoso, M.D., M.A. is a Parliamentary Assistant at the European Parliament

Hannah van Kolfschooten, LL.M. is a PhD Researcher at Law Centre for Health and Life, University of Amsterdam.

Diogo Nogueira Leite, M.Sc. is a PhD Researcher in Health Data Science at the Center for Health Technology and Services Research, University of Porto.

Tjaša Petročnik is a PhD Researcher at TILT/TILEC, Tilburg University

Henrique Vasconcelos, M.D.

The authors write in their personal capacity.

Health is broader than healthcare

When talking about AI in or for health, we are not only talking about a cancer diagnosis or the personalisation of therapeutics in the clinic; the scope of health provision is expanding from formal healthcare systems into our smart devices, actively involving (pre-)patients and consumers. Think about increasingly AI-driven fitness apps, symptom checkers, disease management tools and so on, which have the potential to widen access to health-enhancing resources by cutting conventional healthcare ‘gatekeepers’ out of the equation.

These tools are, however, in the report seen as mere means to relieve pressure on healthcare systems, even though they might affect the health attitudes and behaviours of users or even result in serious harm if not performing optimally. Additionally, as health data is scarce and desirable, we can expect such consumer-facing AI tools to progressively act as an avenue to obtain such data. Misuse might result in privacy violations, civil rights discrimination based on health records, or health inequality increases. AI regulation should thus take into account that AI-based health practices are also taking place outside formal healthcare settings and properly address this.

Exaggerated benefits of health AI

Without a doubt, AI applications in clinical practice and health research show significant promise. Yet, reading the AIDA report, one may get the impression that AI is widely and successfully used in clinical settings. Although the fight against COVID-19 has indeed accelerated research of new technologies in health, this has (so far) produced few robust, generalisable results. While some AI algorithms may perform with high accuracy on paper, even comparable to human specialists, they might not perform as well in real-world clinical practice. The expectations for AI in health appear inflated as the promised transformative events have for now mostly been confined to lab-controlled environments. Furthermore, the AIDA report seems to falsely equate more diagnoses with improved clinical outcomes. However, that is not always the case – overdiagnosis might even result in more harm than good. Ultimately, implementing AI technologies in health contexts requires accurate algorithms and investments in care infrastructure, professionals, and resources.

Underdiagnosis of the risks for both individuals and society

While the benefits of AI appear over-diagnosed, the AIDA report seems to downplay the risks. It correctly acknowledges some of them, in particular, harms to individuals’ wellbeing due to, for instance, misdiagnosis and related liability issues. But, by focusing on individual risks, the report overlooks broader societal risks and harms of AI. Unlike ‘human’ medical errors, mistakes of AI systems can have a much larger impact: technical system errors in AI could lead to mass patient injuries because of widespread use. Additionally, AI might perform better on sub-populations that are better studied or better represented in the training datasets, reflecting existing societal biases. For example, when AI is used to aid health professionals in diagnosing skin cancers, the system frequently uses freely available image databases that often do not include images of people of colour. As a result, these systems may lead to people of colour’s underdiagnosis. This would not only directly harm the health of individual members of (already) marginalised groups; it could also deepen the (existing) socio-economic inequalities in terms of access to health and health outcomes. It thus seems that the report disregards these societal implications in favour of potential economic benefits.

Patients are not mere data points; they are humans with fundamental rights

The risks of AI in health go well beyond health risks. AI runs on data, but data is not just an input; it is “collected by people, from people and for people”. Data subjects and those affected by AI are human beings with fundamental rights like dignity, non-discrimination and privacy. For instance, the report does not touch upon the black-box nature of many AI systems; a lack of understanding of how AI reaches its conclusion makes it difficult for clinicians to explain and for patients to understand AI’s advice, which can impair existing health practices of information provision and informed consent as an integral part of the right to health. Moreover, those seeking care are in a specific situation of vulnerability due to asymmetry in information and risk aversion. Therefore, the EU’s approach to AI in health should not only ensure health-specific fundamental rights are protected in the design, deployment, and usage of AI in health tools but also to address citizens’ willingness to provide and curate health data and ways to govern it appropriately.

Development of AI in health requires a specific approach

Developing health solutions is different from conventional product development because it touches upon people’s bodies and lives in a way no other industry does. Introducing AI in health amplifies the need for multidisciplinary teams that bring together insights from (bio)medicine, data science, behavioural sciences, and ethics and law. By lacking identification of this heterogeneity, namely through the recognition of the need for sector-specific regulation, the report fails to address the needs of patients, practitioners, and also developers and industry.

The AIDA report should therefore acknowledge the special nature of AI in the context of health and call for the proper regulation of the risks to patients or consumers as affected individuals and society. After all, health is everyone’s concern, and the algorithms have already entered doctor’s offices and our living rooms. Let’s ensure that they are really here to make us healthier.

This blog was first published on Euractiv on 24 January 2022.