Multimodal Imaging RWD: When to Combine Imaging and Clinical Data
Reading time /
6 min
Data & AI
.png)
.png)
TL;DR
• Real-world data (RWD) benefits substantially from adding imaging data.. Peer-reviewed guidance recommends combining RWiD with EHR, laboratory, and outcomes data rather than using it as a stand-alone evidence source.
• Radiology reports are not a substitute for the images. Reports answer a clinical question, not a research question. Interobserver variability in RECIST measurements has been shown to exceed the 20% threshold that defines progression.
• More modalities do not automatically mean better models. In a 2026 dementia study, multimodal fusion improved prediction for amyloid-β. For TDP-43, a covariates-only model outperformed every configuration that included MRI.
• Every "do not combine" case traces back to data, not architecture. Label sparsity, modality misalignment, insufficient volume, and weak signal are dataset problems.
Introduction
Multimodal imaging RWD is moving from a niche interest to a core research asset. Life sciences teams now ask a sharper question than "should we use imaging?"
The question is when to combine imaging with clinical notes, laboratory results, and genomics.
That question matters right now for three reasons. Imaging infrastructure, deidentification technology, and data standardization have all improved. Regulatory bodies have signaled openness to non-traditional evidence. And foundation model development has made dataset composition a first-order decision.
This article argues a specific thesis. Combining modalities is not a default setting. It is a decision made per research question, based on whether the added modality carries independent signal.
Three peer-reviewed papers published in 2026 support this position. One provides the framework. Two provide the empirical evidence.
What Real-World Imaging Data Is and Why Reports Are Not Enough
Real-world imaging data is the retrospective collection of routinely gathered clinical imaging data. It sits alongside EHR, claims, laboratory, and genomic data within real-world evidence frameworks.
Historically, biopharmaceutical researchers have worked from radiology reports rather than the images themselves. Reports are lighter to store and easier to deidentify.
That convenience carries a measurable cost. Wu et al. identify four distinct limitations of report-only datasets.
1. Interobserver variability. Different radiologists interpret the same study differently. This introduces noise that can skew retrospective analyses.
2. Incomplete information. A report answers the clinical question asked. A chest CT ordered for lung cancer screening will typically not report cardiovascular or osteoporosis findings, even though the scan contains them.
3. Inconsistent definitions. Three major guidelines exist for pulmonary nodule follow-up, each with different recommendations. Progression definitions can vary between institutions and across time within the same institution.
4. Variable quality. Reports are produced under clinical time pressure. Wu et al. cite a 1959 study reporting errors in 1,600 of 5,000 chest radiographs. They note that subsequent studies indicate high error rates persist in radiology today.
Evidence Snapshot
A meta-analysis of interobserver variability in RECIST-based tumour burden measurements found variability exceeding the 20% cutoff used to define progression. That finding has a direct implication. If your endpoint is derived from a report, the measurement noise may exceed the effect you are trying to detect. — as reported in Wu Jie, de Araujo AL, Khozin S, Huisman M, Mastrodicasa D, Willemink MJ. JMIR Med Inform. 2026
When to Combine Imaging with Clinical Notes and EHR Data
The clearest guidance in the literature is also the most direct.
"RWiD should be combined with other data types (e.g., EHR), instead of using RWiD as a stand-alone source." — Wu, Jie, et al. JMIR Medical Informatics, 2026
Wu et al. identify three specific situations where imaging alone is insufficient:
1. Selection bias: Imaging is not performed randomly. It is driven by clinical indication, physician preference, access to care, and institutional protocols. Patients who receive advanced imaging differ systematically from those who do not.
2. Residual confounding: Variables such as disease severity, treatment intent, and physician decision-making rationale are often incompletely captured in real-world datasets. Linking imaging with richer clinical data improves the measurement of these confounders.
3. Causal inference limits: Observational imaging studies are vulnerable to confounding by indication. Two specific pitfalls recur:
Time-zero misalignment: if a study defines cohort entry at the time of imaging rather than treatment initiation, and patients who deteriorate before imaging are excluded, treatment effects can be inflated.
Immortal time bias: patient classified by post-treatment CT response must survive until that scan occurs. This creates an artificial survival advantage over patients without follow-up imaging.
For regulatory decision-making, the paper is explicit. RWiD is better suited to hypothesis generation, external control arms, and postmarketing surveillance than to stand-alone approval evidence.
When to Combine Imaging with Genomics and Molecular Data
Radiogenomics offers a different value proposition than EHR linkage. It is not about controlling for confounders. It is about extracting molecular information non-invasively.
Wu et al. describe two examples that illustrate the range.
PRISM-CRC is a deep learning framework integrating radiology, histopathology, and clinical data. It predicts microsatellite instability status and five-year disease-free survival in colorectal cancer, reporting an AUC of 0.91.
A second study applied multimodal fusion of radiographic, pathological, and genomic features. It identified biomarkers predictive of immunotherapy response in non-small cell lung cancer.
(Note: PRISM-CRC refers to the Patho-Radiomic Integrative Survival Model for Colorectal Cancer. It is unrelated to Segmed’s PRISM Pre-Built Research Series.)
Imaging contributes something genomics cannot. It provides spatially resolved, anatomically contextualized information about where pathological processes occur.
It also enables longitudinal tracking. Genomic snapshots do not show how a phenotype evolves across a treatment course.
When Combining Modalities Does Not Help
This is the section most multimodal content skips. The evidence is worth reading closely.
Chattopadhyay et al. tested covariates-only, imaging-only, and multimodal configurations across six dementia-related neuropathologies. The results were not uniform.
Three findings deserve attention.
Adding imaging did not always help. For TDP-43, the covariates-only model reached a balanced accuracy of 0.862. No configuration that included MRI matched it.
Imaging alone was weakest where the pathology is most localized. For hippocampal sclerosis, the imaging-only model reported an F1 score of 0.187. Combining modalities recovered performance.
More imaging volume did not help either. The all-slices multimodal model produced no further gains over a single representative slice. In some cases, it reduced performance.
The authors also report that transformer architectures underperformed CNNs here. They attribute this to the dataset size required for transformers to generalize in volumetric contexts.
The oncology literature shows a parallel pattern. Li et al. note that liquid biopsy utility in primary bone tumors remains limited. Low ctDNA shedding, tumor heterogeneity, and inconsistent detection rates restrict its diagnostic reliability.
Choosing a Fusion Architecture for Incomplete Real-World Data
Once you decide to combine, a second decision follows. How should the modalities be fused?
Li et al. describe three architectural strategies with distinct trade-offs.
The practical implication is significant. Early fusion assumes complete pairing across modalities for every patient. Real-world datasets rarely satisfy that assumption.
PBTC-TransNet, described by Li et al., illustrates the alternative. The model classifies primary bone tumors using X-ray, CT, MRI, and clinical characteristics —, even when some modalities are missing. It reported a micro-average AUC of 0.847 internally and 0.782 on an external test set.
Robustness to missing data often matters more for clinical utility than marginal gains under idealized conditions.
A Four-Question Framework for Multimodal RWD Decisions
The literature converges on a decision sequence. Each question is answered by inspecting the dataset, not the model.
- Does the additional modality carry independent signal for this specific question? Not "is multimodal better." Ask whether the second source improves this prediction.
- Is modality pairing complete enough for the architecture you want? Incomplete pairing rules out early fusion. This is a data decision, not a modelling one.
- Do you have sufficient volume and label density? Higher-capacity architectures require substantially larger datasets to generalize.
- Is your evidence goal descriptive, associative, or causal? Causal and regulatory-grade claims require clinical linkage and careful time-zero definition.
Every question above is a dataset question.
The Infrastructure Barriers Behind Multimodal Imaging RWD
Combining modalities is an engineering problem before it is a science problem. Wu et al. document the specific barriers.
File size and handling: DICOM files range from approximately 1 MB for a cardiac PET scan to 1 GB for a retrospectively gated cardiac CT.
Deidentification complexity: Imaging combines text, pixels, and metadata. Protected health information can be burned into pixels or embedded in DICOM tags.
Harmonization trade-offs: Scanner hardware, acquisition parameters, and reconstruction techniques produce systematic differences across sites. Tools like ComBat can reduce these effects.
But harmonization carries risk. Wu et al. list three situations where ComBat should not be applied.
- Confounded study design: if the biological variable of interest is confounded with site or scanner, harmonization may remove the clinical signal.
- Small sample sizes: small cohorts can produce unstable batch effect estimates.
- Out-of-distribution populations: global normalization may strip meaningful biological diversity from unique subpopulations.
Federated learning constraints. Structured EHR data suits federated approaches well. Imaging pixels present distinct challenges around statistical heterogeneity, bandwidth, and governance. The authors characterize these as practical and infrastructure-related, not fundamental incompatibilities.
How Segmed Enables Multimodal Imaging RWD at Scale
Every constraint described above is a data preparation problem.
Selection bias requires cohort transparency. Confounding requires clinical linkage. Fusion architecture requires knowing which modalities are actually paired. Model generalizability requires multi-site diversity.
Segmed builds real-world imaging datasets designed for exactly these requirements. Our network delivers deidentified imaging data with the metadata depth needed to make fusion decisions before you build.
That includes acquisition and reconstruction parameters, which enable post-hoc harmonization and cohort filtering by protocol. It includes multi-site sourcing, which supports external validation across institutions and populations.
For some research areas, those datasets already exist. PRISM, our Pre-Built Research Series, packages cohorts that are assembled and ready to license rather than scoped from scratch.
Explore how Segmed’s real-world imaging data supports multimodal research.
Frequently Asked Questions – F.A.Q
What is real-world imaging data (RWiD)?
Real-world imaging data is the retrospective collection of routinely gathered clinical imaging data. It includes the actual images, not just radiology reports. RWiD sits alongside EHR, claims, laboratory, and genomic data within real-world evidence frameworks.
Why are radiology reports not enough for research?
Radiology reports are written to answer a clinical question, not a research question. They omit details such as texture, shape, and acquisition protocol. They are also subject to interobserver variability and inconsistent reporting standards across institutions.
When should imaging data be combined with EHR data?
Combine imaging with EHR data when your analysis requires adjustment for confounders, causal interpretation, or regulatory-grade evidence. Published guidance recommends against using imaging as a stand-alone evidence source for these purposes.
Does adding more data modalities always improve AI model performance?
No. Published results show that multimodal fusion improved prediction for some pathologies but not all. Model performance depends on whether the added modality carries independent signal for that specific target.
What is the difference between early, late, and hybrid fusion?
Early fusion combines raw data at the input stage and requires perfectly aligned datasets. Late fusion processes each modality separately and combines outputs. Hybrid fusion balances both and tolerates missing modalities, making it better suited to real-world data.
Can real-world imaging data be used for regulatory submissions?
Real-world imaging data is generally better suited to hypothesis generation, external control arms, and post marketing surveillance. For regulatory decision-making, it should be combined with clinical, laboratory, and outcomes data rather than used alone.
What are the main technical barriers to using imaging in multimodal research?
The primary barriers are file size, deidentification complexity, and harmonization across scanners and protocols. Imaging combines text, pixels, and metadata, which makes removing protected health information more complex than for structured data.
References
- Wu J, de Araujo AL, Khozin S, Huisman M, Mastrodicasa D, Willemink MJ. Real-world imaging data: opportunities and challenges. JMIR Med Inform. 2026;14:e88202. doi: 10.2196/88202
- Chattopadhyay T, Kush R, Ankarath RH, Senthilkumar P, Gleave EJ, Patterson C, et al. Multi-modal deep learning and explainable AI for predicting multiple dementia-related neuro pathologies from brain MRI, clinical, and genetic data. Front Neurol. 2026;17:1839071. doi: 10.3389/fneur.2026.1839071
- Li G, Lin Y, Wang X. Multimodal AI fusion: integrating MRI with PET/CT, histopathology, and liquid biopsy for bone tumor diagnosis. Front Oncol. 2026;16:1724332. doi: 10.3389/fonc.2026.1724332
Related Resources
Unlock the power of real-world imaging data through our proprietary, self-service platform, Openda