NeuroVFM: A Brain-Imaging Foundation Model Trained on 5.24M Clinical Scans

A University of Michigan research team has published NeuroVFM in Nature Medicine, a generalist visual foundation model built specifically for neuroimaging. Trained on 5.24 million clinical MRI and CT volumes from 566,915 studies collected over more than two decades at Michigan Medicine, the model uses a self-supervised algorithm called Vol-JEPA that requires no labels or radiology reports. The work tackles a well-known gap: clinical brain scans rarely appear in public training data because they contain identifiable facial features, so general frontier models routinely underperform on brain-imaging tasks.

What happened

Detail	Value
Published in	Nature Medicine
Research institution	University of Michigan
Training volumes	5.24 million MRI and CT scans
Source studies	566,915 (UM-NeuroImages dataset)
Data span	Over two decades of routine clinical care at Michigan Medicine
Base algorithm	Vol-JEPA (extends I-JEPA and V-JEPA to 3D medical volumes)
MRI context ratio	25%
CT context ratio	20%
Patch dropout	20%

Clinical MRI and CT scans are almost entirely absent from public internet datasets. The reason is straightforward: brain scans can be used to reconstruct a patient’s face, making them a privacy risk. General-purpose vision models trained on web data therefore have very little exposure to neuroimaging, and their performance on brain-related tasks reflects that gap.

The Michigan team’s answer is what they call “health system learning.” Instead of curating a specialised dataset from scratch or relying on paired radiology reports, they trained directly on the raw, uncurated output of two decades of routine clinical operations. The result is NeuroVFM, built on top of their Vol-JEPA algorithm.

How Vol-JEPA actually works

Vol-JEPA is a self-supervised, vision-only method. It does not reconstruct pixel values. Instead, it predicts representations inside a learned latent space, which means it never needs ground-truth labels, report text, or a decoder that maps back to voxels.

The process works in several steps:

1. Each 3D volume is divided into non-overlapping patches, each 4x16x16 voxels in size.
2. The volume is split into a small visible context and a larger masked target region.
3. A student encoder processes the visible context patches and produces context latents.
4. A predictor combines those context latents with the position encodings of the masked target patches and predicts what the masked latents should look like.
5. A separate teacher encoder generates the ground-truth target latents. This teacher is an exponential moving average (EMA) of the student, meaning it updates slowly as a smoothed copy of the student’s weights.
6. Training minimises a smooth L1 loss between the predictor’s output and the teacher’s latents, with gradients stopped through the teacher path.

One design choice worth noting: masking is foreground-focused, using precomputed head masks rather than random spatial masking. This pushes the encoder to learn shared neuroanatomy rather than finding shortcuts in the image background.

Why does this matter for AI in medicine?

Most medical AI models are narrow classifiers trained on carefully curated, disease-specific datasets. They perform well on the condition they were designed for and often poorly on anything else. A generalist foundation model trained on uncurated clinical data could serve as a starting point for a much wider range of downstream tasks, including detection, segmentation, and population-level analysis, without requiring a new labelled dataset for each one.

Vol-JEPA also reflects a broader shift happening across the AI field. JEPA-style learning (Joint Embedding Predictive Architecture, which learns by predicting abstract representations rather than raw inputs) started with image patches, then moved to video, and is now being adapted for volumetric medical data. The Michigan work is a concrete example of that trajectory reaching clinical imaging.

For anyone working on AI integration in healthcare or research settings, foundation models like NeuroVFM change the economics of building specialised tools: instead of collecting thousands of labelled scans, you fine-tune a model that already understands brain anatomy.

Our take

The “health system learning” framing is the most interesting part of this paper. The team did not build a better labelling pipeline. They sidestepped labelling almost entirely by treating two decades of routine clinical data as a training signal. That is a meaningful architectural decision, not just a scaling choice.

The honest caveat is that the UM-NeuroImages dataset comes from a single health system, Michigan Medicine. Real-world neuroimaging varies considerably across scanner manufacturers, field strengths, acquisition protocols, and patient demographics. A model trained on one institution’s data may carry biases that only surface when it is deployed elsewhere. The Nature Medicine publication will presumably address generalisation benchmarks, but any team considering clinical use should treat external validation as a hard requirement, not an optional step.

From an agency perspective, the trend worth watching is not this specific model but the method. If JEPA-style self-supervised learning can extract useful representations from 5 million unlabelled medical volumes, it becomes a template for other data-rich, label-poor domains, including industrial inspection, satellite imagery, and retail inventory. We have been tracking how self-supervised approaches to foundation model training are changing what specialised AI deployment looks like in practice.

What to do about it

1. Read the Nature Medicine paper directly if you work in medical imaging or AI research. The Vol-JEPA architecture details will matter for anyone designing similar pipelines.
2. Check whether Michigan Medicine has released model weights or a public checkpoint. Foundation models are most useful when they can be fine-tuned on your own institutional data.
3. If you are building downstream tools, audit your fine-tuning dataset for scanner and demographic diversity before assuming the pretrained representations will transfer cleanly.
4. Track JEPA-style methods across other volumetric domains. The same self-supervised approach may apply to your industry sooner than you expect.

The practical takeaway: a foundation model trained on uncurated clinical data is only as useful as its generalisation outside the training institution, so external validation benchmarks should be your first question before building on top of it.

Source: Marktechpost