Expert View 5 min read

The AI pilot worked. Why is the company still working the old way?

On Friday, AI drafts the report in minutes. On Monday, someone else has to repeat the process. Serhiy Balashuk explains how to tell whether an AI pilot has improved the workflow.

Imagine the Monday after a successful demo. On Friday, an analyst showed how AI could draft the weekly report. She uploaded three spreadsheets, explained what to look for and had an overview in minutes. Her manager asked the department to prepare all its reports this way from the following week.

On Monday, the analyst is on leave. Her colleague opens the same spreadsheets and finds two revenue figures. One includes returns and the other does not yet account for them. During the demo, the analyst knew which figure to use. The instructions say nothing about it. The report shows sales rising. The manager calls finance and learns that once returns are included, the growth disappears.

On Friday, the company tested whether an experienced employee could prepare a draft faster. By Monday, it expected anyone on the team to produce a report it could trust. Getting from the first result to the second requires changing the process itself.

Prepare the people and their working environment

A company needs to set aside a budget at the start, while the return is still to be measured. Some of that budget goes to models and integrations. The company also needs to allocate time for people to understand their processes, test solutions and maintain them after launch. Without time set aside, an experiment is easily shelved as soon as its creator has urgent work.

A central team can provide access to models, set rules for handling data and prepare the support desk. Individual departments need to explain how they make decisions in practice. Finance can explain why both revenue figures may be correct and when to use each one in a report. That rule matters to the model and to the colleague covering for the analyst.

I would base training on tasks people already do. For the report, the team agrees on the sources for its metrics, writes down how to handle conflicting or incomplete data, and tries the instructions on previous weeks’ figures. Then it gives the instructions to a colleague who never saw the demo. Their questions reveal which rules only the creator knew. Once others can repeat the workflow, the team can add it to a shared knowledge base with examples, known limitations and the name of the person who keeps it up to date.

Test the process in a live project

After the first experiments, I would pick one process in an active project and follow it from the initial brief through acceptance. A real project comes with deadlines, dependencies and commitments to a customer. Under those conditions, the team can see how much time it saved and how much checking and correction ended up with other people.

Suppose a team is adding a report export to its product. From a short brief, AI quickly drafts the code and tests. At acceptance, the product manager expects the export to include every record in the selected period, while the developer understood it to mean only the rows visible in the table. The tests confirm that the export works as implemented. The team still needs to agree on the behaviour, revise the code and test it again.

I would start that task with a specification that records the agreed behaviour and acceptance criteria. Which records go into the file? Who can download it? What does the user get when there is no data? When an answer changes, the team updates the specification. The developer, tester and AI then work from the same requirements.

That also changes who does what. The analyst identifies conflicting requirements earlier, the developer spends more time checking edge cases, and the manager decides who accepts the result and who can pause the automation after a failure. Without that agreement, some employees will review everything twice while others may assume a colleague has already checked it.

Measure the time it takes to get an accepted result

If a two-hour report now takes fifteen minutes to draft and ninety minutes to check, the team saves fifteen minutes. That is the review cost: checking the figures, sources and conclusions takes back some of the time saved in drafting. Track errors that make it through review as well. In the revenue example, an incorrect report of growth could lead the manager to revise the sales plan.

  1. Set a baseline before the pilot. Measure the time from start to acceptance and identify the kinds of errors that require rework.
  2. Compare the full cycles. After launch, count the time spent by everyone who prepares, reviews and corrects the work. Check that the result meets the same quality criteria.
  3. Hand the process to someone else. Give them the access, instructions and acceptance criteria, then see where they need help from the creator.

Back to Monday’s report. If a colleague can prepare it independently, explain the difference between the two revenue figures and spend less time overall, including the reviewers’ time, at the same level of quality, the company has grounds to expand the process. It can then discuss the next investment based on work the team has already learned to do better.

Sources & materials

Intecracy Group products and solutions referenced in this article.

  1. AZIOT Platform — aziot.com.ua
Tags: AI