NVIDIA has released Nemotron 3 Nano 4B, a compact 4-billion-parameter hybrid model engineered for local AI inference across GeForce RTX GPUs, DGX Spark, and Jetson edge hardware. Compression from its 9-billion parent model enables up to 2x higher throughput on embedded platforms like the Jetson Orin Nano while maintaining full benchmark accuracy. Robotics engineers and software developers can now deploy privacy-focused conversational agents locally with minimal VRAM usage.
Elastic Compression and Architecture
NVIDIA derived Nemotron 3 Nano 4B from the Nemotron Nano 9B v2 parent model using its proprietary Nemotron Elastic framework. Rather than training the smaller model from scratch or performing disjointed pruning, an end-to-end router executes neural architecture search across four axes alongside knowledge distillation.
The framework reduced the layer depth from 56 to 42, cut Mamba heads from 128 to 96, decreased intermediate FFN dimensions to 12,544, and adjusted embedding dimensions to 3,136. Accuracy recovery followed via a two-stage distillation process, beginning with an 8K short-context training pass over 63 billion tokens and finishing with a 49K long-context extension across 150 billion tokens.
Post-Training and Quantization Benchmarks
To prepare the model for production workflows, NVIDIA applied Supervised Fine-Tuning (SFT) using Megatron-LM alongside a three-stage Reinforcement Learning (RL) pipeline via NeMo-RL. This tuning optimized the architecture for instruction following, agentic tool usage, and gaming intelligence evaluated on tactical benchmarks like Orak.
To make edge deployment feasible, NVIDIA released optimized FP8 and 4-bit Q4_K_M GGUF quantized versions:
- Achieves 100% median accuracy recovery across target benchmarks in both FP8 and Q4_K_M formats compared to the original BF16 baseline.
- Delivers 18 tokens per second on Jetson Orin Nano 8GB devices, doubling the processing speed of the original 9B parent model.
- Provides up to a 1.8x latency and throughput improvement on DGX Spark and Jetson Thor platforms using FP8 post-training quantization.
- Maintains the lowest peak VRAM footprint and lowest Time-To-First-Token (TTFT) latency in its parameter class under high context settings.
Quantization Caveats and Hardware Support
While the model delivers impressive edge efficiency, deployment requires careful optimization trade-offs. To avoid accuracy degradation during FP8 post-training quantization via ModelOpt, NVIDIA had to adopt a selective strategy that keeps four self-attention layers and four preceding Mamba layers in BF16 precision rather than fully quantizing the entire network.
Nemotron 3 Nano 4B is accessible across multiple inference engines including vLLM, TensorRT-LLM, Llama.cpp, and Hugging Face Transformers. NVIDIA has also integrated support for the NVIDIA In-Game Inferencing (NVIGI) SDK to accelerate local agents running alongside graphics workloads.