What Happens When AI Starts Running Experiments on the Business?
AutoResearch, Andrej Karpathy’s project that has generated considerable interest among AI researchers, points to an intriguing possibility: AI that operates like a scientist, experiments autonomously, and becomes a kind of digital process engineer that continuously improves processes across the organization. Realizing its business potential, however, requires an ecosystem that allows it to operate safely.
By Ranit Zexer, CTO Matrix
“There were days when AI research was done by ‘meat computers’, meaning us, humans. Between eating and sleeping, socializing and everything else we did, we somehow found time for research, occasionally synchronizing through sound waves in a ritual known as a ‘team meeting’. Those days are gradually receding into the past.”
This is how Andrej Karpathy, one of the world’s leading AI scientists, introduces AutoResearch, a project he released earlier this year. Karpathy’s résumé includes roles such as founding scientist at OpenAI, head of AI and Autopilot at Tesla, and, most recently, leader of Anthropic’s pretraining team. As usual, Karpathy combines humor with a glimpse of what may lie ahead.
Behind the joke is a significant idea: What happens when AI does more than simply perform tasks? What happens when it can also propose improvements, test them, learn from the results, and continue improving on its own? In other words, what happens when AI begins to behave like a scientist, forming hypotheses, testing them, and learning from the outcomes?
In the version Karpathy presented, an AI agent is given a small but real language model training environment. It modifies the code, trains the model for several minutes, checks whether performance has improved, decides whether to keep or discard the change, and repeats the process again and again. When the researchers arrive at work in the morning, they find a long list of experiments and insights waiting for them, and sometimes a better model as well.
The Innovation Is Not the Research. It Is the Loop.
As expected, the project sparked considerable discussion online. Some of it focused on whether AutoResearch would improve language model performance. Others, including The New Stack, drew attention to another equally interesting aspect: the mechanism underlying the project. The system defines a goal, tries a change, measures the result, decides whether to keep or revert the change, and repeats the process. Karpathy calls this “Revert-or-Keep.” In practice, it is an autonomous improvement loop.
Shortly after the project was released, Shopify CEO Tobias Lütke ran AutoResearch on one of the company’s internal models. Overnight, the system conducted 37 autonomous experiments. By morning, it had produced a significantly smaller model that performed better than the previous version. Importantly, Lütke himself is not a machine learning engineer. In that sense, the achievement is not only the improvement in the model, but also the fact that the optimization process itself becomes far more accessible.
For years, systematic experimentation and system optimization were largely the domain of researchers, analysts, and specialized teams. When the cost of a single experiment drops dramatically, it suddenly becomes possible to explore many more options, run many more iterations, and discover solutions that might never have been tested before.
From this perspective, AutoResearch makes the scientific method faster, more accessible, and far more practical beyond the research world. And perhaps its greatest potential lies in applying the same approach to business processes.
What Happens When AutoResearch Moves into the Business World?
Organizations, of course, already improve processes on an ongoing basis. But this is usually a slow, resource-intensive process. A manager identifies a problem, a team analyzes it, develops recommendations, builds a solution, implements it, and only weeks or months later determines whether an improvement has actually occurred. If the results are unsatisfactory, another cycle begins.
AutoResearch suggests a very different model. Instead of making a small number of major changes, it runs dozens or hundreds of small experiments. Instead of waiting weeks for results, it receives feedback within minutes. And instead of people managing every stage of the process, much of the experimentation, measurement, and decision-making cycle takes place autonomously.
In effect, it makes it possible to turn suitable business processes into experimental laboratories powered by rapid, continuous, autonomous improvement loops. In this sense, AutoResearch can be viewed as a template for process optimization that could be applied across an organization. Instead of improving a language model, the same approach could be used to improve customer service, pricing, logistics, recruitment, or operations.
Consider an organization that wants to improve the routing of customer service requests. Today, a request may pass through several departments before reaching the right person. Every transfer increases handling time, raises costs, and undermines the customer experience.
Under the traditional approach, the organization would analyze the data, develop recommendations, and make changes manually. Under the new approach, an AI system could generate dozens or even hundreds of variations of routing rules, run them against historical data or in a simulation environment, measure the results, and automatically select the best-performing version.
Five Conditions for Success
A simple framework of five conditions can help determine which processes are suitable for this approach.
The first condition is an objective score. There must be a clear measure of success. Routing accuracy, handling time, profitability, or conversion rate are good examples.
The second condition is fast feedback. If it takes months to determine whether a change was successful, rapid improvement cycles are difficult to create. Simulation, A/B testing, or a simulated environment, by contrast, can provide answers within minutes or hours.
The third condition is the ability to make changes. There must be a space of possibilities that can be tested. Routing rules, pricing, service scripts, workflows, or prompts are natural candidates.
The fourth condition is the ability to isolate the effect of the experiment. In a business environment, a change in results may also be caused by seasonality, changes in the customer mix, a campaign, a particular employee, or an external event. For the system to learn correctly, it must be possible to assess whether the improvement actually resulted from the change being tested.
The fifth condition is the ability to contain risk. Unlike an experiment on a model, a business change can affect customers, prices, eligibility, employees, or commitments, and not every action can be undone. Clear guardrails are therefore essential, along with limits on the scope of exposure, approval points where appropriate, and the ability to revert changes whenever possible.
When these conditions are met, it becomes possible to think of a business process as a system that can be autonomously optimized while maintaining clear boundaries for safety and control.
When Agentic AI Meets a Reliable Experimentation Environment
One of the most interesting aspects of this approach is the ability to give AI an experimental environment in which it can test changes without putting business operations at risk.
A full Digital Twin is not always necessary. In some cases, historical data, replay, a sandbox environment, Shadow Mode, or A/B testing may be sufficient. More complex processes may require a richer simulation or a Digital Twin. AI can run scenarios, evaluate outcomes, rank alternatives, and identify promising directions for improvement before any change is introduced into the real environment. This represents a shift from a world in which AI assists with decision-making to one in which AI serves as an engine for business experimentation.
But this is also where one of the central challenges emerges. The experimental environment must be sufficiently reliable for the type of decision being made and the level of risk involved. The more freedom the AI is given, the more accurately the environment must represent reality, including the data, business rules, exceptions, dependencies between systems, and sometimes even customer or employee behavior.
In some cases, replaying historical data will be sufficient. In others, a Digital Twin will be required. The key issue is not the technology itself, but the level of fidelity needed to draw conclusions from an experiment that can actually be trusted.
Another advantage of this approach is that exploring its potential does not necessarily require massive investment or a long and complex project. Unlike some enterprise AI initiatives that require lengthy implementation periods, organizations can begin on a limited scale.
Product managers, innovation leaders, and professionals responsible for business transformation can select a focused process, define a clear success metric, and examine how an autonomous loop of experimentation, measurement, and improvement performs against it.
Not every process is suitable for this approach, and not every experiment will lead to a breakthrough. But as with any new tool, the best way to understand its potential is not simply to read about it, but to put it into practice. Sometimes, one well-defined process is enough to begin understanding how autonomous optimization could become part of the organizational toolkit.
AI Is Not Enough. The Organization Needs to Be Experiment Ready.
There is also an architectural implication. Most enterprise systems were not designed to allow an agent to frequently and systematically change decisions, rules, or processes in a controlled manner.
To enable continuous optimization, organizations need to make parts of their systems experimentable. This means moving rules and configurations out of hard-coded logic, using Feature Flags and Versioning, exposing APIs, measuring behavior and outcomes, and supporting Rollback and Audit capabilities.
In this sense, alongside Cloud Ready and AI Ready, a new requirement may emerge: Experiment Ready. This means not merely having a system capable of running AI, but having one that enables both AI and humans to test many changes rapidly, measure their impact, and implement them without losing control.
The Biggest Challenge Is Not AI. It Is Measurement and the Experimentation Environment.
Despite the potential, there is one issue that cannot be ignored. The hardest problem is not building the agent. It is defining the objective function. What constitutes success in customer service? Is it handling time? Satisfaction? First-contact resolution? Cost? Or some combination of all of these? What constitutes success in recruitment? Time to hire? Candidate quality? Long-term retention?
And what constitutes success in pricing? Revenue? Profitability? Market share? If the metric is wrong, the system may optimize for the wrong outcome. This is a familiar challenge in performance management, and it will become even more important in an era of autonomous optimization.
Even the right metric is not enough if the experiment itself is unreliable. A system may attribute an outcome to a change when it was actually caused by something else. It may learn from a simulation that does not accurately represent reality, or optimize locally in a way that harms other metrics.
The ability to design experiments, distinguish the impact of a change from other factors, maintain guardrails, and evaluate outcomes over time will therefore become a central part of this new organizational capability.
Not Another Digital Worker. A Digital Process Engineer.
We have become accustomed to thinking of AI as a tool that performs work. But the next stage may be AI that improves the work itself.
Karpathy’s AutoResearch may currently be an experiment in the world of language models. But the idea behind it could be much bigger. If this pattern makes its way into the enterprise, organizations may, in the not-too-distant future, operate not only AI agents, but autonomous experimentation labs that continuously search for better ways of working.
And perhaps when that happens, we really will be able to say that the days when “meat computers” conducted all the research themselves are gradually receding into the past.
How Do We See This Trend at Matrix?
At Matrix, we are not looking at this trend from the outside. It connects directly to areas in which we already work with customers today: complex business processes, integration across systems and data, AI capabilities, modernization, measurement, and Managed Services.
For us, therefore, the question is not only whether autonomous optimization will disrupt some forms of traditional work, but how we can bring together the building blocks and experience we already have to create a new capability for our customers.
The first area is making systems Experiment Ready, because most enterprise systems were not designed for frequent, controlled change. This requires modernization that moves rules and configurations out of hard-coded logic. These are exactly the kinds of changes we already implement through architecture, integration, and modernization projects. The difference lies in the objective: not simply developing the customer’s next change, but building a system that enables many changes to be made quickly without losing control.
The second area is building the experimentation and optimization environment. This requires connections to business systems and data, an experimentation and measurement engine, sandbox or simulation environments, data anonymization and synthesis, and, where appropriate, Digital Twins, together with Observability, Governance, and Guardrails. Here too, this sits at the heart of what we do: bringing together business understanding, data, AI, architecture, and integration.
The third area is moving from one-off projects to Continuous Business Optimization. Instead of ending a project once a change has been implemented, we propose operating the experimentation environment over time as a Managed Service: expanding the range of possible changes, improving metrics, calibrating simulations, adding guardrails, and connecting additional processes. The field, and its adoption among customers, is still in its early stages, but it is something organizations should begin thinking about now. If AI is gradually becoming a digital process engineer, our role is to build the ecosystem in which it can operate, experiment, and improve safely.
The value is shifting from knowing how to implement each individual change to knowing how to build an organizational system that enables continuous change without losing control.
