Share
in association with
Bias in the system:
the leading healthcare algorithm with bias against black patients
An unnamed but widely used healthcare algorithm has been found to be systematically biased against black patients. Robert Scammell explores the problem, how it happened and how similar issues can be avoided in the future.
A
healthcare algorithm used to decide whether millions of US patients received access to a high-risk care management programme demonstrated “significant” racial bias against black patients.
The software program, which is reportedly widely used in the healthcare industry, was found to routinely choose healthier white patients over black patients with poorer health for access to extra care.
The algorithmic bias was discovered by researchers from the University of California Berkeley, Chicago Booth School of Business and Partners Healthcare in Boston.
Bias in the system stemmed from the variables used to predict patient risk; those that had higher healthcare bills were deemed more at risk.
“The algorithms encode racial bias by using healthcare costs to determine patient ‘risk,’ or who was mostly likely to benefit from care management programs,” said Ziad Obermeyer, acting associate professor of health policy and management at UC Berkeley and lead author of the paper.
“Because of the structural inequalities in our healthcare system, blacks at a given level of health end up generating lower costs than whites. As a result, black patients were much sicker at a given level of the algorithm’s predicted risk.”
Proving bias
The researchers established bias after comparing the algorithm-predicted risk score of 43,539 white patients and 6,079 black patients enrolled at an unspecified hospital with other health metrics and biomarkers. The findings were published in the scientific journal Nature.
The researchers removed bias from the healthcare algorithm by tweaking it to use other variables in its prediction. They estimate that fixing the algorithm could more than double the number of black patients automatically admitted to the healthcare programs.
Algorithms by themselves are neither good nor bad.
“Algorithms by themselves are neither good nor bad,” said Sendhil Mullainathan, the Roman Family University professor of computation and behavioural science at Chicago Booth and senior author of the study.
“It is merely a question of taking care in how they are built. In this case, the problem is eminently fixable – and at least one manufacturer appears to be working on a fix. We would encourage others to do so.”
Bias in the machine
The biased healthcare algorithm is the latest example of an AI system reflecting human bias. Facial recognition systems, including ones used for passport renewals, have consistently struggled to recognise black people.
Facial recognition systems require large training datasets to work accurately, but there are often disproportionately fewer images of ethnic minorities in these datasets.
Algorithmic bias can be avoided by auditing algorithms while they are being developed. This could be undertaken by an ethicist who looks out for unintended consequences that could arise from an AI system.
This is the same repeated pattern of racial bias that we are finding with algorithmic deciders all over the planet.
Noel Sharkey, emeritus professor of artificial intelligence and robotics at the University of Sheffield, told Verdict that private companies should take more responsibility to ensure their AI systems are not biased.
“This is the same repeated pattern of racial bias that we are finding with algorithmic deciders all over the planet. It is becoming so common that it is clearly time to stop it with hard regulation,” he said.
“Private companies are making and selling these deciders to public service providers and others without adequate testing. The service providers then use them on trust. But there is no room for trust here.
“A couple of academics found the flaws in the care prediction algorithm and found a way to fix it. Why could the company not have done that in the first place? They should be accountable and liable for the suffering that they have already caused.”
Share this article
