What is a GPU?

Why it matters

A CPU has a few powerful cores for general work. A GPU splits a problem into thousands of separate tasks and works on them at the same time, which suits the large matrix calculations behind deep learning. A model's weights sit in the GPU's memory (VRAM), and the memory a model needs sets how many GPUs it takes to serve it.

The GPU generation behind a model changes the energy used per token. Making GPUs also creates embodied emissions, which count towards the footprint of AI.

An example

Greenpixie ran the same open-weights models on two NVIDIA generations, the H100 and the B200, so the effect of hardware generation on energy per token could be measured directly. A large model whose weights fill more VRAM needs more GPUs to serve it, which is why VRAM footprint is one of the two inputs to Greenpixie's energy predictions.

How Greenpixie measures it

During benchmarking, Greenpixie sampled GPU board power every 10 milliseconds through NVIDIA's management library. Energy from the rest of the server, such as processors, memory, fans and power supply, is estimated from inference time and published AI instance specifications. Embodied emissions for each GPU add up four parts:

• the chip's die area multiplied by a coefficient for its process node
• memory capacity multiplied by a per-GB factor
• a fixed value for the rest of the board
• a cooling term that scales with the card's thermal design power

The memory, board and cooling factors come from NVIDIA's HGX H100 and HGX B200 product carbon footprints.