What is an AI accelerator?
An AI accelerator is a processor designed to speed up machine learning. GPUs are the most common. Google TPUs and AWS Trainium are purpose-built AI chips.
Why it matters
Accelerators are built for the large matrix operations that neural networks rely on. GPUs are the most widely used. Some cloud providers also design their own chips. Google's Tensor Processing Units (TPUs) are custom application-specific integrated circuits (ASICs) for machine learning, with high-bandwidth memory. AWS Trainium is a purpose-built chip for AI training and inference.
The chip a model runs on affects the energy used per token. Cloud AI services do not disclose which hardware serves a given request, and it does not appear in usage data.
An example
A customer calling a model through a cloud AI service sees the model name, token counts and price on the bill. The chip that served each request is not shown. Greenpixie ran its benchmarks on two NVIDIA generations, the H100 and the B200, so the effect of hardware generation on energy per token could be measured and carried into the estimate.
How Greenpixie measures it
Greenpixie's AI benchmarks ran on NVIDIA GPUs. Because the serving hardware cannot be seen in billing data, the methodology treats it as a range, along with other settings providers do not disclose:
- the hardware in use
- the deployed quantisation level
- the number of requests served in parallel
- the share of parameters active per request
Sampling across these ranges gives a median and an interquartile range for each model. Where a customer can confirm details of a deployment, such as a batched processing tier, the range narrows.