vLLM/Recipes
NVIDIA

nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16

NVIDIA Nemotron-3-Super Mamba-hybrid latent-MoE (~120B total / ~12B active) with BF16, FP8, and NVFP4 variants

View on HuggingFace
moe120B / 12B262,144 ctxvLLM 0.17.1+text
Guide

Overview

NVIDIA Nemotron-3-Super-120B-A12B is a hybrid-Mamba latent-MoE model (~120B total, ~12B active per token) trained for general reasoning, tool use, and agentic workflows. It supports a 1M-token context window and Multi-Token Prediction (MTP). Variants ship in BF16, FP8, and NVFP4 (Blackwell). A pre-RL Base BF16 checkpoint is also available for downstream fine-tuning.

Prerequisites

  • Hardware: 4-8x H100/H200/B200/RTX Pro 6000, DGX Station, or DGX Spark
  • vLLM >= 0.17.1
  • Docker with NVIDIA Container Toolkit (recommended)

Launch commands

Reference command from the vLLM blog (BF16, 4x H100, FP8 KV cache):

vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 4 \
  --trust-remote-code \
  --served-model-name nemotron \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser nemotron_v3

FP8 weights:

vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 4 \
  --trust-remote-code \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser nemotron_v3

NVFP4 (Blackwell only):

vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \
  --tensor-parallel-size 2 \
  --trust-remote-code \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser nemotron_v3

DGX Spark (GB10)

The NVFP4 variant runs on a single DGX Spark (GB10) at TP=1 inside the vllm/vllm-openai:v0.24.0-ubuntu2404 container. The command below is the optimal config tuned end-to-end on DGX Spark: the full 262144-token context fits with an FP8 KV cache and a float32 Mamba SSM cache (VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 allows the long context, VLLM_FLOAT32_MATMUL_PRECISION=high keeps the Mamba scan numerically stable), --gpu-memory-utilization 0.8 leaves headroom in the 128 GB unified memory, batching is sized for a single GB10 (--max-num-seqs 8, --max-num-batched-tokens 16384), --load-format fastsafetensors speeds up weight loading, and MTP speculative decoding plus prefix caching are enabled for lower latency:

# Use container: vllm/vllm-openai:v0.24.0-ubuntu2404

VLLM_FLOAT32_MATMUL_PRECISION="high" \
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \
    --host 0.0.0.0 \
    --port 8000 \
    --max-model-len 262144 \
    --gpu-memory-utilization 0.8 \
    --max-num-seqs 8 \
    --load-format fastsafetensors \
    --attention-backend flashinfer \
    --max-num-batched-tokens 16384 \
    --kv-cache-dtype fp8 \
    --mamba-cache-mode align \
    --mamba-ssm-cache-dtype float32 \
    --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
    --enable-prefix-caching \
    --reasoning-parser nemotron_v3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_xml

RTX Pro 6000 NVFP4

The NVFP4 variant runs on a single RTX Pro 6000 at TP=1. Requires vLLM v0.24.0 or later. A few notes:

  • Set --gpu-memory-utilization 0.92 to avoid an OOM at startup. If you encounter an OOM, try adjusting the value.
  • Set VLLM_FLOAT32_MATMUL_PRECISION="high" to use high precision for the matmul operations.
  • Set VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 to allow the model to use the full context length.
# Use container: vllm/vllm-openai:v0.24.0-ubuntu2404

VLLM_FLOAT32_MATMUL_PRECISION="high" \
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \
    --max-model-len 262144 \
    --gpu-memory-utilization 0.92 \
    --max-num-seqs 8 \
    --max-num-batched-tokens 16384 \
    --kv-cache-dtype fp8 \
    --mamba-cache-mode align \
    --mamba-ssm-cache-dtype float32 \
    --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
    --enable-prefix-caching \
    --reasoning-parser nemotron_v3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_xml

Benchmarking

vllm bench serve \
  --model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 \
  --trust-remote-code \
  --dataset-name random \
  --random-input-len 1024 --random-output-len 1024 \
  --num-warmups 20 \
  --ignore-eos \
  --max-concurrency 1024 \
  --num-prompts 2048

References