New commentary published in the Journal of the Royal Society of Medicine warns that current risk-based regulatory approaches to Artificial Intelligence (AI) in healthcare fall short in protecting patients, potentially leading to over- and undertreatment as well as discrimination against patient groups.
The authors found that while AI and machine learning systems can enhance clinical accuracy, concerns remain over their inherent inaccuracy, opacity, and potential for bias which are not adequately addressed by the current regulatory efforts introduced by the European Union’s AI Act.
Passed in 2025, the AI Act categorises medical AI as “high risk” and introduces strict controls on providers and deployers. But the authors argue this risk-based framework overlooks three critical issues: individual patient preferences, systemic and long-term effects of AI implementation, and the disempowerment of patients in regulatory processes.
Patients have different values when it comes to accuracy, bias, or the role AI plays in their care. Regulation must move beyond system-level safety and account for individual rights and participation.”
Thomas Ploug, lead author, Professor of Data and AI Ethics, Aalborg University, Denmark
The authors call for the introduction of patient rights relating to AI-generated diagnosis or treatment planning, including the right to:
- request an explanation;
- give or withdraw consent;
- seek a second opinion; and
- refuse diagnosis or screening based on publicly available data without consent.
They warn that without urgent engagement from healthcare stakeholders – including clinicians, regulators, and patient groups – these rights risk being left behind in the rapid evolution of AI in healthcare.
“AI is transforming healthcare, but it must not do so at the expense of patient autonomy and trust,” said Professor Ploug. “It is time to define the rights that will protect and empower patients in an AI-driven health system.”
The need for patient rights in AI-driven healthcare – risk-based regulation is not enough (DOI: 10.1177/01410768251344707) is by Dr Thomas Ploug, Rikke Frank Jørgensen, Hanne Marie Motzfeldt, Naomi Ploug and Søren Holm.
Source: