Inference
From Wiki for Sustainable IT
Inference is the stage at which a trained model is used: the moment it applies what it learned to a new request.
Why it matters
Training is expensive but happens once. Inference is far cheaper per request but is repeated at every use. For a service in production with many users, inference eventually accounts for most of the energy attributable to the model.
Its cost depends on three factors: the size of the model invoked, the length of the request and response, and the number of calls, which multiplies rapidly in agent-style uses where each reasoning step and each tool call consumes an inference.
The main lever
Choosing the smallest model that meets the need is the most direct lever available: it acts on every request, without changing anything about the use itself.
