The Race to Cut Token Costs

Organizations are investing considerable resources in reducing token consumption and lowering AI costs. But what happens when models are required to process vast amounts of unnecessary information? Information overload, it turns out, carries a cost that doesn’t always appear on the bill.

Token prices are falling, yet organizations’ AI bills continue to rise. In a McKinsey survey published in July 2026, 93% of respondents reported exceeding their AI budgets. The survey also found that AI spending was nearly four times higher among organizations that had moved from isolated use cases to broader adoption.

These findings help explain why organizations are paying closer attention to token consumption and comparing prices across models. In the effort to control costs, however, an important question is often overlooked: How many of the tokens an organization pays for are actually necessary to complete the task?

When Tokens are Wasted on Irrelevant Information

Consider an employee who asks an AI system to find out why a particular order has been delayed. To answer the question, the system retrieves data from the ERP system, along with supplier correspondence, purchase orders, shipping histories, and internal procedures from various sources.

It might seem that the more information a model receives, the better its answer will be. In practice, much of that information may be irrelevant, duplicated, or outdated. The model must still process it all, even when the answer lies in just a few lines among thousands.

This increases token consumption and can also slow response times, making it harder for the model to identify the details that matter.

Stanford University’s 2026 AI Index Report describes the gap between the rapid expansion of AI models’ context windows and their ability to use the information within them effectively. According to the report, a model’s ability to process inputs of a million tokens or more does not guarantee that it can draw accurate insights from all that information, particularly in tasks that require connecting multiple pieces of information.

The ability to process more information, in other words, is not necessarily a reason to provide it.

 

Context Engineering: Reducing the Cost of Unnecessary Information

In a technical article, Anthropic describes context as a critical but finite resource and explains why managing the information supplied to a model has become a central challenge in AI system design.

This approach, known as Context Engineering, extends beyond prompt design. It focuses on determining which information, instructions, conversation history, and outputs from external tools should be included in a model’s context at each stage.

For example, instead of passing an entire document to a model whenever a question arises, a system can retrieve only the relevant passages while preserving the context needed to understand them.

By reducing unnecessary information, organizations can lower token consumption while potentially improving response times and the quality of their results.

AI Agents Add Another Layer of Complexity

The challenge becomes greater as organizations move from AI systems that answer questions to agents capable of executing entire workflows. An agent may interact with several systems, use external tools, analyze results, and make multiple calls to a model before completing a task. Each step consumes additional tokens.

Managing these costs requires considering both the number of tokens used in each model call and the number of calls needed to complete the task.

An agent performing ten operations, each involving the repeated transmission of a lengthy context, may consume far more tokens than a system performing a single operation. Optimizing AI agents therefore requires examining the entire workflow, identifying which operations are necessary, determining what information needs to pass between stages, and finding opportunities to avoid redundant processing.

Reducing consumption, however, should not come at the expense of accuracy. Cutting context too aggressively may exclude essential information and compromise the result.

From Cost per Token to Value per Task

Effective enterprise AI management requires looking beyond the price per million tokens. Alongside consumption and cost, organizations should examine the cost of successfully completing a task, response times, the quality of the results, and the frequency of human intervention or rework.

This calls for carefully designed information retrieval, less duplication, effective context management across multistep workflows, and measurement of actual outcomes.

Tokens will remain an important metric in the economics of AI. As systems become more complex, however, token counts alone provide only a partial picture.

The question is not simply how much a million tokens cost an organization, but how many of those tokens actually contribute to getting the right answer.

Thanks
Your form has been submitted successfully
We will contact you shortly
Oops something went wrong!
Refresh to try again or contact us via info@matrix.co.il
Find out more
Please complete your details and we will contact you

    *
    *
    *
    *
    designed & developed by