What is inference in AI?

Why it matters

Training adjusts a model's weights using training data, often over many iterations. Inference is every later use of the trained model: each prompt, API call or agent step. For LLMs, inference has two phases. Prefill processes the whole input prompt at once and is limited by compute. Decode then generates output tokens one at a time and is limited by memory bandwidth.

Training energy is spent before a model is released. Inference energy repeats with every request, so it grows with usage and with the choices teams make about models and serving.

An example

A real-time chatbot and an overnight batch job can run the same model with very different energy per output token. Greenpixie's benchmarks varied parallel requests from 16 to 2,048 and found that energy per output token falls as more requests are served together. Batch APIs rely on this, which is why they cost less and use less energy per request.

How Greenpixie measures it

Greenpixie's AI methodology estimates the energy, carbon and water of cloud AI inference per token. Each request is charged for the server time it causes, and idle capacity between requests is not charged to any customer. Where requests share a batched pass, its runtime is split in proportion to how long each request was in flight.

Batched and real-time processing tiers get different energy values. Training is added as an uplift on inference energy, which spreads the training cost across the tokens a model serves.