Intel Gaudi AI Accelerator Gains 2x Performance Leap on GPT-3 with FP8 Software

What’s New: Today, MLCommons published results of the industry standard MLPerf training v3.1 benchmark for training AI models, with Intel submitting results for Intel® Gaudi®2 accelerators and 4th Gen Intel® Xeon™ Scalable processors with Intel® Advanced Matrix Extensions (Intel® AMX). Intel Gaudi2 demonstrated a significant 2x performance leap, with the implementation of the FP8 data type on the v3.1 training GPT-3 benchmark. The benchmark submissions reinforced Intel’s commitment to bring AI everywhere with competitive AI solutions.

Why It Matters: The newest MLCommons MLPerf results build on Intel’s strong AI performance over previous MLPerf training results from June. The Intel Xeon processor remains the only CPU reporting MLPerf results, and Intel Gaudi2 is one of only three accelerator solutions upon which results are based, only two of which are commercially available.

Intel Gaudi2 and 4th Gen Xeon processors demonstrate compelling AI training performance in a variety of hardware configurations to address the increasingly broad array of customer AI compute requirements.

Download two images (12 MB)

About the Intel Gaudi2 Results: Gaudi2 continues to be the only viable alternative to NVIDIA’s H100 for AI compute needs, delivering significant price-performance. MLPerf results for Gaudi2 displayed the AI accelerator’s increasing training performance:

• Gaudi2 demonstrated a 2x performance leap with the implementation of the FP8 data type on the v3.1 training GPT-3 benchmark, reducing time-to-train by more than half compared to the June MLPerf benchmark, completing the training in 153.58 minutes on 384 Intel Gaudi2 accelerators. The Gaudi2 accelerator supports FP8 in both E5M2 and E4M3 formats, with the option of delayed scaling when necessary.
• Intel Gaudi2 demonstrated training on the Stable Diffusion multi-modal model with 64 accelerators in 20.2 minutes, using BF16.
• While FP8 was used only in GPT-3 in this MLPerf training submission and GPT-J in the previous inference submission, Intel is expanding FP8 support in Gaudi2 Software and tools to additional models for both training and inference.
• On eight Intel Gaudi2 accelerators, benchmark results were 13.27 and 15.92 minutes for BERT and ResNet-50, respectively, using BF16.

About the 4th Gen Xeon Results: Intel remains the only CPU vendor to submit MLPerf results. The MLPerf results for 4th Gen Xeon highlighted its strong performance:

• Intel submitted results for RESNet50, RetinaNet, BERT and DLRM dcnv2. The 4th Gen Intel Xeon scalable processors’ results for ResNet50, RetinaNet and BERT were similar to the strong out-of-box performance results submitted for the June 2023 MLPerf benchmark.
• DLRM dcnv2 is a new model from June’s submission, with the CPU demonstrating a time-to-train submission of 227 minutes using only four nodes.

4th Gen Xeon processor performance demonstrates that many enterprise organizations can economically and sustainably train small to mid-sized deep learning models on their existing enterprise IT infrastructure with general-purpose CPUs, especially for use cases in which training is an intermittent workload.

 What’s Next: With software updates and optimizations, Intel anticipates more advances in AI performance results in forthcoming MLPerf benchmarks. Intel’s AI products provide customers with more choice for AI solutions to meet dynamic requirements requiring performance, efficiency and usability.

More Context: MLCommons Announcement

Nov. 19, 2023, Editor’s Note: This news story was edited to more clearly explain the Stable Diffusion training and software support.