Where AI fits: choosing a first task in an operations workflow

Most operations teams can name a dozen places where AI might help. The difficulty is choosing the first one. Start too broad and the project becomes a demonstration nobody trusts. Start with the wrong task and you spend months on something a rule or a form would have handled.

A good first task is small enough to check, common enough to matter and safe enough to get wrong occasionally while you learn. This article sets out how we approach that choice, how to tell whether it is working, and when to leave AI out.

Look for bounded, checkable tasks

Language models are good at reading unstructured text and producing structured output. They are less dependable when the task is open-ended, when the right answer depends on context they cannot see, or when nobody can say what a correct answer looks like.

Strong first candidates usually share these traits:

• A clear input. An email, a form, a PDF or a support ticket, rather than “everything we know about this customer”.
• A defined output. A category, a set of extracted fields, a draft reply or a routing decision. Something you can compare against what a person would have done.
• A person already doing it. If someone does the task today, you have examples, a measure of quality and someone who can review the output.
• Tolerable errors. A wrong category that a person corrects is a minor cost. A wrong payment instruction is not.

Typical examples in operations work include sorting incoming requests, pulling fields out of documents such as invoices or forms, summarising long threads before someone picks them up, and drafting replies for a person to edit and send.

Questions to ask of each candidate

• How many of these do we handle, and how long does each take today?
• What happens when a person gets it wrong now, and how is it caught?
• Can we collect enough real examples, with the correct answer, to judge the results?
• Who would review the output, and would they trust it?

If the answers are vague, the task is probably not ready.

Build an evaluation set from real examples

Before choosing a model or writing a prompt, collect examples. An evaluation set is a fixed collection of real inputs with the answer a competent person would give. It is the only reliable way to know whether an approach works and whether a later change made it better or worse.

• Use real cases, not invented ones. Include the awkward ones: poor scans, mixed languages, incomplete forms, unusual requests.
• Agree the right answer. Have the people who do the work label the examples. Where they disagree, the task definition needs work before the model does.
• Keep it separate. Do not tune prompts on the same examples you use to judge them, or the scores will flatter you.
• Score it automatically where you can. Extracted fields and categories can be compared directly. Drafted text needs a rubric and a human reviewer, at least at first.

Run every change, whether a new prompt, a different model or extra context, against the same set. Without this, decisions rest on a handful of examples that happened to look good.

Design human review in from the start

For a first task, assume a person reviews the output before it has any effect. That is not a sign the system failed: it is how you learn where it can be trusted.

• Show the source next to the output. A reviewer checking extracted fields should see the document they came from.
• Make correcting quick. If fixing a mistake takes longer than doing the task by hand, people will stop using it.
• Record corrections. Every edit is a labelled example. Feed them back into the evaluation set.
• Reduce review gradually. Once results are consistently right for a category of input, you can sample rather than check everything. Make that a deliberate decision based on the evaluation results.

Handle data deliberately

Operations data often includes personal details, financial information or commercial terms. Decide early:

• what data the task needs, and what can be removed or masked before it is sent anywhere;
• which providers and regions are acceptable, and what their terms say about retention and training;
• where prompts, outputs and corrections are logged, who can see them and how long they are kept;
• how a request to delete someone’s data reaches those logs.

If the organisation has a data protection lead, bring them in at the task selection stage, not after the build.

Measure cost and latency

AI tasks have running costs that grow with volume, and response times that vary.

• Measure per task, not per month. Work out the cost of handling one item, including retries and review time, and compare it with the current cost.
• Check latency against the workflow. A summary prepared overnight can take a while. A reply drafted while a customer waits on the phone cannot.
• Try smaller models. A bounded task often works well on a smaller, cheaper model. The evaluation set tells you whether it does.
• Watch for drift. Models change, and so do the inputs. Rerun the evaluation set on a schedule and when a provider updates a model.

When not to use AI

Some tasks look like AI problems and are not.

• The rules are known. If the decision follows clear, stable rules, write the rules. They are cheaper, faster and easier to explain.
• The input could be structured. If people are writing free-text emails because there is no form, a form may solve the problem.
• Mistakes are costly and hard to catch. If an error would cause harm before anyone notices, the task needs a different design or should stay with a person.
• Nobody owns the outcome. If no one will review results, maintain the evaluation set or respond when quality drops, the system will degrade quietly.

Choosing not to use AI for a task is a useful result. It often points to a simpler fix in the workflow itself.

When to bring in help

If you have a shortlist of possible tasks and are not sure which to start with, or a pilot that worked in a demonstration but is hard to trust in daily use, it can help to step back and test the task against real examples. We can work with your team to choose a first task, build an evaluation set, and design the review and data handling around it, including telling you when AI is not the right tool.