Biomedical Imaging's Real Bottleneck Is the Data, Not the Model

Biomedical Imaging's Real Bottleneck Is the Data, Not the Model | Databricks Blog Skip to main content

Summary

What it is: A field perspective on why medical imaging AI underdelivers across hospitals, academia, medtech, and pharma; what a stronger data foundation looks like across R&D.

The challenge: Imaging is medicine's richest data but its least usable: siloed across PACS, CROs, and central labs, hard to de-identify, and rarely linked to other data modalities that power biomarkers and patient stratification. The hard part is the data, not the model.

The takeaway: The frontier isn't a better model, it's the data foundation under it. Govern imaging in one place, make it queryable, de-identify at scale, and link it to EHR, omics, and trial data (via governed lakehouse patterns). Get the data right, and the model becomes the easy part.

Part I — The problem Four sectors, one bottleneck Hospitals and health systems generate the raw material. Radiology is the most heavily regulated corner of medical AI anywhere: of the roughly 950 AI/ML-enabled devices the FDA had authorized by mid-2024, about 723 (around 76%) were radiology tools. Almost all are narrow and human-supervised, doing triage, measurement, reconstruction, or worklist prioritization. The real difficulty in a hospital is rarely the model. It's that the data sits locked inside PACS and vendor-neutral archives built to serve images to a viewer, not to answer research questions. Pulling a cohort means extracting images from clinical systems, de-identifying them (PHI hides in DICOM headers and gets burned into the pixels), and finding the compute to use them. The model is the easy 10%. Academic medical centers are where most of the open science happens. The field owes much of its progress to shared datasets like MIMIC-CXR (377,110 chest X-rays), The Cancer Imaging Archive, fastMRI, and the UK Biobank imaging arm, along with open tooling such as MONAI, nnU-Net, and 3D Slicer. Academia also exposes the field's credibility problem. A well-known BMJ review (Nagendran et al., 2020) of deep-learning-versus-clinician imaging studies found that of 81 studies, only a handful were prospective or tested in a real clinical setting, and code and data were unavailable in 93% and 95% of them. That reproducibility debt traces straight back to how hard imaging data is to share and re-run. Medtech and device companies (GE HealthCare, Siemens Healthineers, Philips, Canon) build AI directly into the scanner: deep-learning reconstruction that shortens scans, dose reduction, on-device triage. Their hardest problem is generalization. An algorithm trained on one vendor's scanner, field strength, and protocol can quietly degrade on another's. Curating multi-site training data that represents the real world, then proving it to regulators through the 510(k) and PMA pathways, is both the moat and the cost. Pharma and biotech treat imaging as a measurement instrument. Oncology trials depend on standardized response criteria like RECIST, iRECIST, and RANO, read centrally and blindly to strip out site bias, and quantitative imaging biomarkers can pick up drug response earlier than size-based measures. None of it works unless a measurement means the same thing across sites, scanners, and time, which is why RSNA's QIBA writes profiles that pin down acquisition and analysis to a stated precision. Variability is the enemy of a trial endpoint. Four different jobs, one shared bottleneck. The pixels are everywhere, and queryable nowhere.

Fig 1. Hospitals, academia, medtech, and pharma all converge on the same problem: the pixels are everywhere and queryable nowhere.

Why collaboration is the whole game, and why it's so hard No single institution holds enough diverse data to build a model that generalizes. The strongest collaborative study to date makes the point. In the EXAM study (Nature Medicine, 2021), 20 institutions trained a shared model to predict COVID-19 oxygen needs from chest X-rays plus EMR data without moving any patient data between them. Only the model weights traveled, and the federated model gained, on average, 16% in AUC and 38% in generalizability over single-site models. The mechanics are less exotic than they sound: each site trains locally, a coordinator averages the model weights rather than the data (the classic federated-averaging recipe), and secure aggregation and differential privacy keep any one site's records from being reconstructed out of those weights. That's the promise. The reality is that collaboration is genuinely difficult, for reasons that are structural rather than technical: Data heterogeneity. Different scanners, protocols, and labeling conventions mean that "the same" exam often isn't. Federated models degrade on this non-IID, acquisition-skewed data unless it's deliberately engineered around with techniques like batch-normalization averaging or site-aware harmonization. Privacy and governance. Every cross-institution study brings per-site IRB approvals, data-use agreements, and de-identification that has to be defensible. No common substrate. Consortia like MIDRC (co-led by the ACR, RSNA, and AAPM) exist precisely because intake, curation, de-identification, labeling, and sharing are so hard that they need a dedicated national effort.

Collaboration in imaging isn't a nice-to-have. It's the only path to models that work outside the building they were trained in, and it's gated almost entirely by data plumbing and governance. The complexity most people outside the field never see When people say "medical images," they usually picture a chest X-ray. The reality is a zoo of formats and scales that makes general-purpose data tooling buckle. Imaging type Formats Dimensions & scale Why general-purpose tooling buckles Radiology — X-ray, CT, MRI DICOM, encodable in ~a dozen transfer syntaxes (uncompressed → JPEG 2000 → HTJ2K) 2D X-ray → 3D CT/MRI volumes → 4D dynamic cardiac Built to move studies between machines, not analyze in bulk; decoders (pydicom, GDCM, pylibjpeg) run single-threaded — a few studies is trivial, a few million is a distributed-systems problem Digital pathology — whole-slide images Vendor formats: Aperio .svs, Hamamatsu .ndpi, Philips iSyntax (via OpenSlide); almost no DICOM Gigapixel — multiple GB, tens of thousands of tiles, read as a pyramid at several magnifications Too large to open in memory; stain and scanner...