Performance per watt

Introduction

Today’s laptop may have 6 CPU cores and consume 20 watts, while a supercomputer may have tens of thousands of cores and consume many kilo watts. Across the spectrum of computing systems, we will see that the performance per watt is a meaningful benchmark.

It is usually quoted as GFLOP/s/watt, for giga (${10}^9$) floating-point operations per second per watt. In this article it is 64-bit floating-point operation when we quote GFLOP/s/watt.

As

$$1\text{second}\times 1\text{watt}=1\text{Joule,}$$

a unit of energy,

$$1\text{GFLOP/s/watt}=1\text{GFLOP/Joule,}$$

which measures the amount of work per unit of energy.

The performance per watt should be measured over the entire computing system, including CPUs, memory, networking, maybe even including the cooling system when it was involved.

CPU with vector extension and GPUs

Traditionally-designed CPU carries out one arithmetic or logical operation per clock cycle. Techniques were invented to accelerate the execution marginally such as speculative branch prediction, while introducing security risk such as Spectre Attack.

In contrast, CPUs with vector textension or GPUs can carry out multiple arithmetic operations per clock cycle, increasing the compute throughput by orders. The CPU state would be more deterministic, increasing introspectability and security.

A GPU is traditionally designed with many cores, which can each be programmed separately. The recent Nvidia Tensor Cores are specialised to accelerate matrix-matrix multiplication. For example, for 64-bit multiplication, the Tensor Core can first cast each 64-bit operand to 32-bit, do 32x32-bit multiplication to still obtain a 64-bit result. The throughput would double compared to 64x64-bit multiplication.

Compute-bound vs Memory-bound tasks

If it takes more time to move data than to carry out compute, the program is memory-bound. Conversely, the program is compute-bound.

Today, the most well-known compute-bound task is matrix-matrix multiplication, as to multiply two squares of matrix of size $N^2$ takes $N^3$ order of compute.

A wide range of algorithms are memory-bound, for example, partial derivative equation solution, sequence alignment, etc.

Compute-bound performance per watt

As data is loaded once yet used many times for calculation, the energy cost of memory operation is ignorable in the limit, the whole-system performance per watt is bounded by that of the CPU or GPU alone.

Green500 ranks the TOP500 systems according to measured HPL performance per electrical power consumed. The official source code does compute in 64-bit. However Nvidia systems would run Nvidia’s closed source code for the benchmark, which may or may not have reduced the precision of operands in flight. For Nvidia products, we deduce the performance per watt from the products’ datasheets instead for 64x64-bit multiplication.

Package or System	Arch.	Watt	GFLOP/S/Watt
LineShine	CPU	42,200,000	52
A100 SXM	GPU	400	24
H100 SXM	GPU	700	48
Vitruvius+	CPU	0.45	47
AraXL 16-lane	CPU	1.1	39

LineShine is currently the No. 1 supercomputer by total compute throughput. It is a CPU-only system using ARM instructions with no GPU. The next two are Nvidia GPUs. The last two are research chips by the Barcelona Supercomputing Centre and ETH Zurich respectively, using RISC-V instructions. Apart from LineShine a complete system, the remaining four were CPU or GPU packages, whose performance per watt would constitute an upper limit on that of any system constructed with it.

Most above hover around 50 GFLOP/s/watt. What is the highest upper limit on the compute-bound performance per watt? If a 64-bit floating point unit consumes 10 pJ/FLOP, then the highest upper limit is

$$\begin{array}{cc}& \frac{1}{10\times {10}^{-12}\text{Joule/FLOP}}\\ =& {10}^{11}\text{FLOP/Joule}\\ =& 100\text{GFLOP/Joule,}\end{array}$$

supposing the remaining circuit consumes minimal power in the system.

Memory-bound performance per watt

To read 64-bit data from memory, usually it takes hundreds of pJoule, while to compute on it it takes only tens of pJoule, so from the memory GB/s/watt, we will deduce the upper limit on memory-bound GFLOP/s/watt.

As a floating-point operation works on 64 bits, or 8 bytes, if the memory scores $x$ GB/s/watt, the deduced upper limit on GFLOP/s/watt is $x/8$ GFLOP/s/watt.

Below the GB/s are from Wikipedia pages, the watts are estimated by ChatGPT. Again we will see from data centre- to laptop- or wearable-level, the memory’s GB/s/watt are comparable curiously. Under the hood, both the HBM and LPDDR are DRAM.

Memory	Watt	GB/s/watt	GFLOP/s/watt
HBM 3	29	28	3.5
HBM 3E	34	36	4.5
HBM 4	75	27	3.4
LPDDR 5	0.8	32	4
LPDDR 5X	1.0	42	5.2

TOP500 publishes a list for the HPCG benchmark which is memory-bound. If we consider the memory uses 25% of the whole system watts in those supercomputers, all leading systems on that list would perform at 3.2 GFLOP/s/watt as of the writing, just below the theoretical upper limit imposed by the HBM.

References

ARM A-profile architecture, https://support.arm.com/architectures/a-profile-architecture

RISC-V Instruction Set Architecture Spectifications, https://riscv.org/specifications/ratified/

TOP500 supercomputer list, https://www.top500.org/

Frequently asked questions on the Linpack benchmark and TOP500, https://www.netlib.org/utk/people/JackDongarra/faq-linpack.html

Linpack source code, https://github.com/icl-utk-edu/hpl/

HPCG (High Performance Conjugate Gradient) benchmark, https://www.hpcg-benchmark.org/

HPCG benchmark source code, https://github.com/hpcg-benchmark/hpcg

Programming Tensor Codes in CUDA 9, https://developer.nvidia.com/blog/programming-tensor-cores-cuda-9/

LineShine supercomputer, https://en.wikipedia.org/wiki/LineShine

LineShine system, https://www.top500.org/system/180490/

Nvidia A100 GPU, https://www.nvidia.com/en-eu/data-center/a100/

Nvidia H100 GPU, https://www.nvidia.com/en-us/data-center/h100/

Nvidia Grace CPU Superchip, https://www.nvidia.com/en-us/data-center/grace-cpu-superchip/

NVIDIA Grace CPU Superchip Architecture In Depth, https://developer.nvidia.com/blog/nvidia-grace-cpu-superchip-architecture-in-depth/

NVIDIA Grace Hopper Superchip Architecture In-Depth, https://developer.nvidia.com/blog/nvidia-grace-hopper-superchip-architecture-in-depth/

Nvidia Grace performance tuning guide, https://docs.nvidia.com/dccpu/grace-perf-tuning-guide/index.html

AMD ROCm Core SDK, https://rocm.docs.amd.com

AMD ROCm blog: Understanding Typically Attained FLOPs, https://rocm.blogs.amd.com/artificial-intelligence/taf-blog/README.html

Vitruvius+: an efficient RISC-V decoupled vector coprocessor, https://dl.acm.org/doi/full/10.1145/3575861

AraXL: a physically scalable, ultra-wide RISC-V vector processor, https://arxiv.org/abs/2501.10301

RiVEC benchmark suite on vector microarchitecture, https://github.com/RALC88/riscv-vectorized-benchmark-suite

CPU vector extension and GPU benchmarking, https://github.com/jli05/vbench

High Bandwidth Memory, https://en.wikipedia.org/wiki/High_Bandwidth_Memory

Low-Power Double Data Rate (LPDDR), https://en.wikipedia.org/wiki/LPDDR

CS107 guide to x86-64, https://web.stanford.edu/class/archive/cs/cs107/cs107.1202/guide/x86-64.html