NVIDIA VISTA-3D: One AI Model to Segment the Whole Body in CT

Posted on by IE Contributor

Outlining organs and tumours in a CT scan is among the most tedious jobs in medical imaging. A single scan can hold hundreds of slices. Each structure has to be traced in three dimensions, and research groups typically train a separate AI model for every organ or dataset they care about. NVIDIA’s answer is VISTA-3D, a single foundation model, built with the open-source MONAI framework, that segments well over a hundred anatomical structures on its own and lets a human fix or extend its work with a few mouse clicks.

The model’s research paper was accepted at CVPR 2025, one of computer vision’s top conferences. Its weights are openly available, and NVIDIA has also packaged it as an enterprise microservice. Here is what VISTA-3D does, why its design matters, and what teams should know before trying it.

VISTA-3D whole-body CT segmentation
Whole-body CT segmentation produced by VISTA-3D. Image: Project-MONAI/VISTA repository, Apache License 2.0.

What VISTA-3D Is, and Why 3D Is Hard

VISTA stands for Versatile Imaging SegmenTation and Annotation. In their paper on arXiv, the NVIDIA authors and collaborators from the National Institutes of Health, the University of Arkansas for Medical Sciences and the University of Oxford describe it as the first unified model to reach state-of-the-art results in both automatic 3D segmentation (127 supported classes) and interactive 3D segmentation.

That combination is harder than it sounds. Meta’s Segment Anything Model (SAM) showed what promptable segmentation can do on everyday 2D photos, and many teams have tried adapting it to medicine. The VISTA-3D authors argue that this falls short for volumetric scans. Prompting slice by slice takes too much human effort across a whole cohort of patients. And video-style trackers such as SAM2 treat a scan like a film, even though consecutive slices through a body can show entirely different structures. The researchers fine-tuned SAM2 on CT data for comparison. On one pancreas benchmark it scored a Dice overlap of 0.557 against VISTA-3D’s 0.802 with a single click (Dice runs from 0 to 1, where 1 is a perfect match).

So VISTA-3D stays native to 3D. According to the paper, it uses MONAI’s SegResNet, a convolutional U-Net-style network that has won several recent MICCAI segmentation challenges, as a shared image encoder. Two separate decoders sit on top of it:

  • An automatic branch that takes a class number (say, “liver”) as its prompt and outputs that structure, using sliding-window inference over 128-voxel cubes.
  • An interactive branch, adapted from SAM’s point-prompt design, that takes 3D click coordinates marked positive or negative and refines a local patch around each click.

A merging algorithm lets clicks add or remove only the connected regions they touch, so correcting one mistake doesn’t damage the rest of an otherwise good result.

Three Ways to Use It

NVIDIA’s technical blog on the VISTA-3D NIM microservice describes three core workflows:

  • Segment everything: label the whole body in one pass, which helps when a disease affects several organs or when planning treatment as a whole.
  • Segment by class: ask only for specific structures, such as a particular organ or tumour type.
  • Segment with point prompts: click to guide or correct the model. NVIDIA presents this as a way to speed up the creation of ground-truth training data.

The supported list covers major abdominal organs, lung lobes, individual vertebrae and ribs, major vessels and muscles, plus several lesion types. The MONAI VISTA repository lists lung tumours, pancreatic tumours, hepatic tumours, colon cancer primaries and bone lesions. For structures outside that list, the interactive branch works “zero-shot”: a user can annotate something the model never saw in training simply by clicking on it.

How It Was Trained, and How Well It Performs

The paper describes a curated collection of 11,454 CT volumes drawn from public and in-house sources. Some datasets had partial manual labels, some had none, and the team generated pseudo-labels for 117 classes using the open-source TotalSegmentator model. The most novel ingredient is “supervoxels”: the researchers ran SAM over each scan in three directions and grouped its features into 3D regions. This passes some of SAM’s general sense of objects into a 3D model, and the authors report that it boosted zero-shot performance by 50%. Training ran in four stages on 64 NVIDIA V100 GPUs, using about 20,000 GPU hours. Synthetic tumour images from NVIDIA’s MAISI generative model were used to fill gaps for rare classes.

The headline results, all from the paper’s benchmarks across 14 datasets:

  • Automatic mode averaged a Dice score of 0.711. That is on par with dataset-specific expert models trained with nnU-Net (0.718) and MONAI’s Auto3DSeg (0.706), with one model doing the work of many.
  • Automatic plus one corrective click per patch raised the average to 0.792.
  • Speed: on a 16 GB V100 GPU, segmenting a 512×512×512 volume took about 2 minutes, compared with roughly 12 minutes for TotalSegmentator. A single interactive click took about 3.2 seconds on a 12 GB GPU.
  • Few-shot fine-tuning: on a mouse micro-CT dataset, fine-tuning VISTA-3D on just one labelled scan reached a Dice of 0.926. Models trained from scratch on the same single scan scored between 0.759 and 0.820.

The results are not uniformly better. On some tasks, such as pancreatic tumours in fully automatic mode, VISTA-3D trailed the specialist models until a click was added. Its value lies in the whole workflow rather than in a single leaderboard number.

From Research Code to Enterprise Deployment

VISTA-3D became part of the open MONAI ecosystem when MONAI Core v1.4 was released. NVIDIA’s MONAI fifth-anniversary post (December 2024) announced that VISTA-3D and the MAISI generator were generally available as NVIDIA NIM microservices within NVIDIA AI Enterprise. These are containerized, GPU-accelerated services with TensorRT optimization and standard APIs. NVIDIA’s NIM documentation lists NIfTI as the input and output format and Triton as the inference engine, and says the service was tested on A100, H100, L40S and RTX 6000 Ada GPUs.

Industry interest is already visible. In an NVIDIA announcement timed to RSNA 2024, GSK was described as exploring VISTA-3D and VISTA-2D for image segmentation. Alara Imaging presented work combining VISTA-3D with large language models such as Llama 3. In the same announcement, Siemens Healthineers said it had adopted MONAI Deploy to bring MONAI-built AI apps into its imaging platforms.

The project has also moved on. According to the VISTA GitHub repository, the optimized weights are now published on Hugging Face as NV-Segment-CT under the NVIDIA Open Model License, which the repository describes as commercial-friendly. In October 2025 NVIDIA released a sibling model, NV-Segment-CTMR. It was fine-tuned on more than 30,000 CT and MRI scans, supports over 300 classes and covers body and brain MRI. However, it is automatic-only and licensed for non-commercial use. The repository recommends CTMR for large-scale automatic labelling and NV-Segment-CT for CT tumours and interactive refinement.

Practical Takeaways

  • It is a research tool, not a diagnostic device. NVIDIA’s model cards say plainly that the model is for research purposes and not for clinical use. Any clinical application would need its own validation and regulatory clearance.
  • Use it to cut annotation costs. The strongest case for VISTA-3D is speeding up the labelling of large CT datasets. Automatic output plus a few corrective clicks can turn days of manual tracing into a review task.
  • Fine-tune before you train from scratch. For a new structure or species, the paper’s few-shot results suggest starting from the VISTA-3D checkpoint with a handful of labelled scans.
  • Pick the right variant. Choose NV-Segment-CT for CT, tumours and interactive editing under a commercial-friendly license. Choose NV-Segment-CTMR for MRI coverage, keeping its non-commercial license in mind.
  • Plan for 3D data. Inputs are NIfTI volumes and inference expects a modern NVIDIA GPU. The hosted NIM API takes an image URL rather than an upload, so teams need a secure way to serve their scans.

Conclusion

VISTA-3D signals a shift in medical imaging AI: away from building one narrow model per organ and toward a single 3D foundation model that does the routine work automatically and hands the edge cases to a human with a mouse. It does not replace radiologists, and NVIDIA does not claim it does. But by pairing state-of-the-art automatic segmentation with genuine 3D interactivity, and by releasing it openly through MONAI, NVIDIA has given researchers and medtech builders a practical base layer for the next generation of imaging tools.

Sources