NVIDIA Deep Learning Examples for Tensor Cores

Introduction

This repository provides State-of-the-Art Deep Learning examples that are easy to train and deploy, achieving the best reproducible accuracy and performance with NVIDIA CUDA-X software stack running on NVIDIA Volta, Turing and Ampere GPUs.

NVIDIA GPU Cloud (NGC) Container Registry

These examples, along with our NVIDIA deep learning software stack, are provided in a monthly updated Docker container on the NGC container registry (https://ngc.nvidia.com). These containers include:

• The latest NVIDIA examples from this repository
• The latest NVIDIA contributions shared upstream to the respective framework
• The latest NVIDIA Deep Learning software libraries, such as cuDNN, NCCL, cuBLAS, etc. which have all been through a rigorous monthly quality assurance process to ensure that they provide the best possible performance
• Monthly release notes for each of the NVIDIA optimized containers

Computer Vision

| Models | Framework | AMP | Multi-GPU | Multi-Node | TensorRT | ONNX | Triton | DLC | NB | |----------------------------------------------------------------------------------------------------------------------------------------|--------------|----------------|-----------|------------|----------|------|------------------------------------------------------------------------------------------------------------------------------|------|------------------------------------------------------------------------------------------------------------------------------------------------------------------| | EfficientNet-B0 | PyTorch | Yes | Yes | - | Supported | - | Supported | Yes | - | | EfficientNet-B4 | PyTorch | Yes | Yes | - | Supported | - | Supported | Yes | - | | EfficientNet-WideSE-B0 | PyTorch | Yes | Yes | - | Supported | - | Supported | Yes | - | | EfficientNet-WideSE-B4 | PyTorch | Yes | Yes | - | Supported | - | Supported | Yes | - | | EfficientNet v1-B0 | TensorFlow2 | Yes | Yes | Yes | Example | - | Supported | Yes | - | | EfficientNet v1-B4 | TensorFlow2 | Yes | Yes | Yes | Example | - | Supported | Yes | - | | EfficientNet v2-S | TensorFlow2 | Yes | Yes | Yes | Example | - | Supported | Yes | - | | GPUNet | PyTorch | Yes | Yes | - | Example | Yes | Example | Yes | - | | Mask R-CNN | PyTorch | Yes | Yes | - | Example | - | Supported | - | Yes | | Mask R-CNN | TensorFlow2 | Yes | Yes | - | Example | - | Supported | Yes | - | | nnUNet | PyTorch | Yes | Yes | - | Supported | - | Supported | Yes | - | | ResNet-50 | MXNet | Yes | Yes | - | Supported | - | Supported | - | - | | ResNet-50 | PaddlePaddle | Yes | Yes | - | Example | - | Supported | - | - | | ResNet-50 | PyTorch | Yes | Yes | - | Example | - | Example | Yes | - | | ResNet-50 | TensorFlow | Yes | Yes | - | Supported | - | Supported | Yes | - | | ResNeXt-101 | PyTorch | Yes | Yes | - | Example | - | Example | Yes | - | | ResNeXt-101 | TensorFlow | Yes | Yes | - | Supported | - | Supported | Yes | - | | SE-ResNeXt-101 | PyTorch | Yes | Yes | - | Example | - | Example | Yes | - | | SE-ResNeXt-101 | TensorFlow | Yes | Yes | - | Supported | - | Supported | Yes | - | | SSD | PyTorch | Yes | Yes | - | Supported | - | Supported | - | Yes | | SSD | TensorFlow | Yes | Yes | - | Supported | - | Supported | Yes | Yes | | U-Net Med | TensorFlow2 | Yes | Yes | - | Example | - | Supported | Yes | - |

Natural Language Processing

| Models | Framework | AMP | Multi-GPU | Multi-Node | TensorRT | ONNX | Triton | DLC | NB | |------------------------------------------------------------------------------------------------------------------------|-------------|------|-----------|------------|----------|------|-----------------------------------------------------------------------------------------------------------|------|---------------------------------------------------------------------------------------------------------------------------------------------| | BERT | PyTorch | Yes | Yes | Yes | Example | - | Example | Yes | - | | GNMT | PyTorch | Yes | Yes | - | Supported | - | Supported | - | - | | ELECTRA | TensorFlow2 | Yes | Yes | Yes | Supported | - | Supported | Yes | - | | BERT | TensorFlow | Yes | Yes | Yes | Example | - | Example | Yes | Yes | | BERT | TensorFlow2 | Yes | Yes | Yes | Supported | - | Supported | Yes | - | | GNMT | TensorFlow | Yes | Yes | - | Supported | - | Supported | - | - | | Faster Transformer | Tensorflow | - | - | - | Example | - | Supported | - | - |

Recommender Systems

| Models | Framework | AMP | Multi-GPU | Multi-Node | ONNX | Triton | DLC | NB | |----------------------------------------------------------------------------------------------------------------|-------------|-------|-----------|--------------|--------|------------------------------------------------------------------------------------------------------|------|--------------------------------------------------------------------------------------------------------| | DLRM | PyTorch | Yes | Yes | - | Yes | Example | Yes | Yes | | DLRM | TensorFlow2 | Yes | Yes | Yes | - | Supported | Yes | - | | NCF | PyTorch | Yes | Yes | - | - | Supported | - | - | | Wide&Deep | TensorFlow | Yes | Yes | - | - | Supported | Yes | - | | Wide&Deep | TensorFlow2 | Yes | Yes | - | - | Supported | Yes | - | | NCF | TensorFlow | Yes | Yes | - | - | Supported | Yes | - | | VAE-CF | TensorFlow | Yes | Yes | - | - | Supported | - | - | | SIM | TensorFlow2 | Yes | Yes | - | - | Supported | Yes | - |

Speech to Text

| Models | Framework | AMP | Multi-GPU | Multi-Node | TensorRT | ONNX | Triton | DLC | NB | |--------------------------------------------------------------------------------------------------------------|-------------|------|------------|--------------|----------|--------|----------------------------------------------------------------------------------------------------------|-------|--------------------------------------------------------------------------------------------------------------| | Jasper | PyTorch | Yes | Yes | - | Example | Yes | Example | Yes | Yes | | QuartzNet | PyTorch | Yes | Yes | - | Supported | - | Supported | Yes | - |

Text to Speech

| Models | Framework | AMP | Multi-GPU | Multi-Node | TensorRT | ONNX | Triton | DLC