Geographic Bias in AI: Where Does Your Training Data Come From?
Reading time /
6 min
AI Medical Imaging
.png)
TL;DR
- Geographic bias in artificial intelligence (AI) occurs when training data comes from a limited geographic region, so performance everywhere else is unknown.
- Of 56 US deep learning studies with identifiable training cohorts, 40 (71%) used patients from California, Massachusetts, or New York1.
- Only 16 states contributed any training cohort. The other 34 contributed none1 .
- Counting images is not enough. Training datasets need to be characterized by where the data comes from, who it represents, and how it was acquired.
Introduction
Geographic bias in AI starts with a simple question: where did the training data come from? For much of published medical AI, the answer is a small number of places. A model trained in those places has been tested on their patients, scanners, and clinical routines. Its performance anywhere else is an open question. This post uses published evidence to show how concentrated medical AI training data is. It also shows what concentration costs when a model reaches a new hospital.
Where US medical AI training data comes from
A 2020 JAMA study mapped the origin of training data in US clinical deep learning research.1 The authors reviewed studies published between 2015 and 2019 in six disciplines, including radiology, pathology, and cardiology.1 Of 74 included studies, 56 trained their algorithms on at least one geographically identifiable cohort.1
The cohorts clustered in three states. California appeared in 22 of the 56 studies, Massachusetts in 15, and New York in 14.1 Forty of the 56 studies (71%) used a cohort from at least one of these states.1 Among the remaining 47 states, 34 contributed no patient cohort at all.1
The authors warned that algorithms trained mainly on data from these states may generalize poorly.1 They noted that the three states may differ from the nation in economic, educational, social, behavioral, ethnic, and cultural features.1
Evidence snapshot
In US clinical deep learning studies from 2015 to 2019, 34 of 50 states contributed no training cohort. Three states supplied cohorts for 71% of studies with identifiable data.1
Geographic bias in AI is a global pattern
Concentration is not only a US issue. A global review screened 30,576 articles on clinical AI and analyzed 7,314 of them.2 Most of the databases used came from the United States (40.8%) and China (13.7%).2 Radiology was the most represented specialty, at 40.4% of papers.2
The review’s authors called for data infrastructure in data-poor regions.2 They also recommended external validation and recalibration of models before clinical use.2
What happens when a model meets a new hospital
Geographic concentration matters because hospitals differ. Patients, scanners, and workflows all change from one site to the next. A model can learn those local features along with the disease.
A 2018 study tested pneumonia detection models on 158,323 chest radiographs from three US hospital systems.3 Performance on outside hospitals was significantly lower than on the original hospitals in three of five comparisons.3 The models could also identify which hospital system and department an image came from with very high accuracy.3
A 2021 Nature Medicine analysis tested the same risk directly.4 The authors trained pneumothorax detection models on chest X-rays from each of three US sites. They then tested each model at the other two sites.4 Average area under the curve (AUC) at the training site was 0.893.4 At new sites, AUC dropped by 0.072 on average and by up to 0.124.4
The drop was not equal across patients. On the Beth Israel Deaconess test set, the gap between White and Black patients was 0.024 for the locally trained model.4 For the two models trained elsewhere, the gap rose to 0.043 and 0.109.4
The same analysis reviewed 130 AI devices approved by the FDA between 2015 and 2020.4 For 93 of them, no multi-site assessment was publicly reported.4 Of the 41 devices with a reported site count, 4 were evaluated at one site and 8 at two sites.4
Evidence snapshot
Every pneumothorax model in the analysis lost accuracy at sites it was not trained on. The largest AUC drop was 0.124.4
Characterize training data, not just count it
Dataset size is easy to report. It says little about whether a model will work for a given population. A dataset of a million images from one health system still reflects one health system.
Useful documentation describes where the data comes from, not only how much there is. The table below lists the dimensions that published evidence shows can affect performance.
Few device evaluations reported this kind of breakdown. Only 17 of the 130 FDA device studies reported performance by demographic subgroup.4
How Segmed supports geographically diverse imaging data
Segmed’s network spans 2,800+ healthcare partner sites across five continents. We have established partnerships in the US, Europe, Latin America, and Asia-Pacific. Different institutions serve different populations and run different protocols. We preserve that variation, because it is what a model must handle in deployment.
That breadth lets developers build training and validation cohorts across sites and regions. It also supports external validation on institutions that played no part in training. Read more in our white paper, Decreasing Bias in Healthcare AI with Segmed.
Want to test your model beyond the sites it was trained on? Book a call with our team.
Frequently Asked Questions – F.A.Q.
What is geographic bias in AI?
Geographic bias in AI occurs when training data comes from a small number of places. The model is then tested mostly on those places’ patients and equipment. Its performance in other regions remains unknown.1
Where does most medical AI training data come from?
In US clinical deep learning studies from 2015 to 2019, 71% of studies with identifiable cohorts used data from California, Massachusetts, or New York.1 Globally, most databases in a review of 7,314 clinical AI papers came from the United States and China.2
Why do medical AI models perform worse at new hospitals?
Hospitals differ in patients, scanners, and workflows. Models can learn these site features along with the disease.3 In one analysis, pneumothorax models lost up to 0.124 AUC at sites they were not trained on.4
Does geographic bias affect some patients more than others?
It can. In one chest X-ray analysis, the performance gap between White and Black patients grew when a model was tested at a new hospital.4 The gap rose from 0.024 for the local model to 0.109 for one external model.4
How many FDA-approved AI devices were tested at multiple sites?
In an analysis of 130 AI devices approved between 2015 and 2020, 93 had no publicly reported multi-site assessment.4 Of the 41 with a reported site count, 4 used one site and 8 used two.4
What should a training dataset document besides its size?
It should document the geographic origin of each cohort and the contributing institutions. Developers should also report performance at sites outside training and by demographic subgroup.1,4 These details show whether a dataset fits the population the model will serve.
References
1. Kaushal A, Altman R, Langlotz C. Geographic distribution of US cohorts used to train deep learning algorithms. JAMA. 2020;324(12):1212-3. doi:10.1001/jama.2020.12067.
2. Celi LA, Cellini J, Charpignon ML, Dee EC, Dernoncourt F, Eber R, et al. Sources of bias in artificial intelligence that perpetuate healthcare disparities: a global review. PLOS Digit Health. 2022;1(3):e0000022. doi:10.1371/journal.pdig.0000022.
3. Zech JR, Badgeley MA, Liu M, Costa AB, Titano JJ, Oermann EK. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS Med. 2018;15(11):e1002683. doi:10.1371/journal.pmed.1002683.
4. Wu E, Wu K, Daneshjou R, Ouyang D, Ho DE, Zou J. How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals. Nat Med. 2021;27:582-4. doi:10.1038/s41591-021-01312-x.
Related Resources
Segmed White Paper: Decreasing bias in healthcare AI with Segmed.