Intel Gaudi 2 Remains Only Benchmarked Alternative to NV H100 for GenAI Performance

5th Gen Intel Xeon processors' results improved by an average of 1.42x compared with 4th Gen Intel Xeon processors’ results in MLPerf Inference v3.1. (Credit: Intel Corporation)

Intel Gaudi2 AI accelerators demonstrate a 2x performance improvement as shows in MLPerf results announced Nov 8, 2023. (Credit: Intel Corporation)

Download all images (ZIP, 17 MB)

What’s New: Today, MLCommons published results of the industry-standard MLPerf v4.0 benchmark for inference. Intel’s results for Intel® Gaudi® 2 accelerators and 5th Gen Intel® Xeon® Scalable processors with Intel® Advanced Matrix Extensions (Intel® AMX) reinforce the company’s commitment to bring “AI Everywhere” with a broad portfolio of competitive solutions. The Intel Gaudi 2 AI accelerator remains the only benchmarked alternative to Nvidia H100 for generative AI (GenAI) performance and provides strong performance-per-dollar. Further, Intel remains the only server CPU vendor to submit MLPerf results. Intel’s 5th Gen Xeon results improved by an average of 1.42x compared with 4th Gen Intel® Xeon® processors’ results in MLPerf Inference v3.1.

“We continue to improve AI performance on industry-standard benchmarks across our portfolio of accelerators and CPUs. Today’s results demonstrate that we are delivering AI solutions that deliver to our customers’ dynamic and wide-ranging AI requirements. Both Intel Gaudi and Xeon products provide our customers with options that are ready to deploy and offer strong price-to-performance advantages.”

Why It Matters: Building on its training and inference performance from previous MLPerf rounds, Intel’s MLPerf results provide customers with industry-standard benchmarks to evaluate AI performance.

About the Intel Gaudi 2 Results: The Intel® Gaudi® software suite continues to increase model coverage of popular large language models (LLMs) and multimodal models. For MLPerf Inference v4.0, Intel submitted Gaudi 2 accelerator results for state-of-the-art models Stable Diffusion XL and Llama v2-70B.

Due to strong customer demand for Hugging Face Text Generation Inference (TGI), Gaudi’s Llama results used the TGI toolkit, which supports continuous batching and tensor parallelism, enhancing the efficiency of real-world LLM scaling. For Llama v2-70B, Gaudi 2 delivered 8035.0 and 6287.5 for offline and server tokens-per-second, respectively. On Stable Diffusion XL, Gaudi 2 delivered 6.26 and 6.25 for offline samples-per-second and server queries-per-second, respectively. With these results, Intel Gaudi 2 continues to offer compelling price/performance, an important consideration when looking at the total cost of ownership (TCO).

About the Intel 5th Gen Xeon Results: Following hardware and software improvements, Intel’s 5th Gen Xeon results improved by a geomean of 1.42x compared with 4th Gen Intel Xeon processors’ results in MLPerf Inference v3.1. As an example, for GPT-J with software optimizations including continuous batching, the 5th Gen Xeon submission showed about 1.8x performance gains compared with the v3.1 submission. Similarly, DLRMv2 showed about 1.8x performance gains and 99.9 accuracy due to MergedEmbeddingBag and other optimizations utilizing Intel AMX.

Intel is proud of its collaboration with OEM partners – Cisco, Dell, Quanta, Supermicro and WiWynn – to deliver their own MLPerf submissions. Additionally, Intel has submitted MLPerf results for four generations of Xeon products, starting in 2020, and Xeon is the host CPU for many accelerator submissions.

How to Try AI Solutions on Intel Developer Cloud: 5th Gen Xeon processors and Intel Gaudi 2 accelerators are available for evaluation in the Intel^® Developer Cloud. In this environment, users can run both small- and large-scale training (LLM or GenAI) and inference production workloads at scale, manage AI compute resources and more.

What’s Next: Tune into Intel Vision 2024, where Intel will share more about its strategy to bring “AI Everywhere,” along with an update on Intel Gaudi 3 AI accelerators.

More Context: MLCommons Announcement

_{The Small Print:
For workloads and configurations, visit MLCommons.org. Results may vary.}