According to Tom's Hardware the announcement has a few intriguing asterisks that only the most discerning reader (or competitor) might notice.
AMD claims this achievement using a setup that includes two 64-core EPYC 9575F processors, eight Instinct MI300X accelerators, and 2,304 GB of DDR5 memory. This marvel of modern computing was tested on the Llama3.1-70B model (FP8 precision, naturally), delivering a headline-grabbing result: the system is 28.3 times more energy efficient than a mysterious "undisclosed" machine from 2020.
AMD has generously left the specifications of its 2020 benchmark system to the imagination. However, we can surmise it involved the decidedly less glamorous EPYC 7002-series CPUs (based on Zen 2 microarchitecture) and Instinct MI100 accelerators, which might not have had the same FP8 support or snazzy high-bandwidth memory as today's shiny MI300X.
Even without such details, AMD assures us the comparison is entirely valid, pointing to paper differences like a 14 to 28-times gap in INT8 performance and significantly faster memory. Never mind that comparing systems three years apart in semiconductor years is a bit like pitting a penny-farthing bicycle against an e-bike and declaring the e-bike a marvel of efficiency.
AMD’s Senior Vice President, Sam Naffziger, attributes the energy-efficiency miracle to "brute force" hardware upgrades and architectural and software co-design. In other words, iterative improvements are the kind that one might reasonably expect when cash is thrown at engineers, and a deadline looms ominously on the calendar.
The outfit has added a teaser about its forthcoming Instinct MI355X accelerators, due next year, with performance metrics designed to make even supercomputers blush. These new chips promise support for esoteric AI formats like FP4 and FP6, reaching a mind-bending 9.2 petaFLOPS. With such numbers, AMD could soon claim victory in energy efficiency and perhaps in the niche sport of acronym proliferation.
While AMD’s announcement brims with optimism, one might note the curious phrasing around its 30-times efficiency target. The latest achievement is almost there at 28.3 times, a respectable figure that feels just tantalisingly short of the nice round number it had hoped to trumpet. But with a year to go, AMD has plenty of time to eke out those final few points—likely with an asterisk and another undisclosed benchmark.