The MI300X is competitive with Nvidia's H100 on AI inference benchmarks, particularly for the Llama 2 model with 70 billion parameters.
However, tests across different AI models are still needed. By the end of the year, Nvidia's Blackwell GPUs might challenge AMD's MI300X and MI325X in terms of price/performance.
The benchmarks did not include the newer Llama 3.1 models, which are more likely to be used by enterprises.
AMD has not yet published MLPerf test results for AI training runs, but these are expected in the fourth quarter. AMD has been selling the MI300X to hyperscalers and cloud builders for AI inference workloads. Due to its higher memory and bandwidth, the MI300X performs well against the H100 but struggles against the H200.
AMD tested the MI300X with Genoa and Turin CPUs, showing good performance scaling. The MI300X's peak FP16 performance is 1,307.4 teraflops, generating 2,530.7 tokens per second in server mode.
Nvidia's H100 systems have a better ratio of server tokens per second to peak FP16 flops, likely due to software tuning. The H200's increased memory and bandwidth significantly boost performance.
The MI325X, expected later this year, will have more memory and bandwidth, aiming to compete better with Nvidia's GPUs. The MI350 series, expected next year, will feature a new CDNA architecture and move to 3nm processes.
The performance difference between AMD and Nvidia GPUs may also be influenced by their interconnects, with Nvidia's NVSwitch offering higher bandwidth than AMD's Infinity Fabric.
Price/performance analysis suggests the MI300X offers good value, but Nvidia's upcoming GPUs may change the landscape. Nvidia's Blackwell GPUs are expected to be priced between $35,000 and $40,000, while the MI300X is estimated at around $20,000.
The cost of turning these GPUs into servers is roughly $150,000. The MI300X delivers slightly better value than the H100, but the H200's performance improvements make it more cost-effective.
Nvidia's B200 GPU, expected to cost $40,000, will significantly reduce the cost per inference unit. Due to high demand and limited supply, Nvidia might charge $50,000 per GPU. The pricing of AMD's MI325 later this year will be crucial in determining its competitiveness.
