How to Scope an AI Automation Pilot That Survives Production

An AI automation pilot usually starts with a demo that works. A model reads a document, drafts a reply, or routes a request, and the room agrees it is promising. The trouble starts a few weeks later, when the same demo meets real inputs, real permissions, and a person who has to answer for the result.

The fix is not a bigger model or a longer pilot. It is a tighter scope. A pilot that survives production is scoped around one workflow, its riskiest step, the people who review it, and the evidence that would prove it works.

01 // Pick one workflow, not a department

"Automate operations" is a strategy. "Turn inbound service requests into a triaged queue with a proposed owner and a draft response" is a pilot. The difference is that the second one has inputs you can collect, a handoff you can observe, and an output a person can accept or reject.

Good first candidates share a few traits:

• the work recurs often enough that improvement is noticeable;
• it currently moves through spreadsheets, inboxes, or manual copy-and-paste handoffs;
• the inputs exist in systems you can access with the right permissions; and
• a mistake is recoverable, because a person reviews the output before it matters.

Describe the workflow the way the team actually performs it, including the exceptions people handle from memory. Our workflow discovery prompt turns a messy description into a bounded brief with owners, exceptions, and acceptance criteria. It is a useful first draft for that conversation.

02 // Prove the riskiest path before the happy path

Every workflow has one step that decides whether automation is viable: an integration with a system that has no clean API, a document type the model reads poorly, or a judgment call that depends on context nobody wrote down. Test that step first, with real examples, before building the interface.

If the riskiest path fails, you learn it in days rather than after a polished prototype has created expectations. If it works, the rest of the build becomes ordinary engineering. Anthropic's guidance on building agents makes a similar point from the model side: start with the simplest pattern that solves the problem and add autonomy only where it demonstrably helps.

03 // Decide where a person stays in the loop

A pilot should name, in writing, which decisions a person still makes. High-impact decisions keep a named owner and a review path. The automation proposes; the owner approves, edits, or rejects. That split is what lets a team trust the system before it has a track record.

The same section should cover the boring operational details that sink pilots later:

• which source data the system may read, and how long anything it stores is kept;
• which actions it may take on its own, and which require approval;
• what happens when an input is missing, malformed, or hostile; and
• who is paged when it fails, and how a person takes over mid-task.

Inputs deserve particular care. Text from emails, documents, and web pages can contain instructions aimed at the model. OWASP lists prompt injection as the first risk for LLM applications. Treat every source as data, never as instructions, and test that explicitly.

04 // Write acceptance criteria before you build

"It works" is not an acceptance criterion. Agree, before building, what the pilot must demonstrate: output quality on a representative set of real cases, behaviour on the difficult exceptions, response time, and how often a reviewer has to correct the result. Include the cases you expect to fail, such as duplicate events, missing permissions, and source text that tries to redirect the model.

Our agent acceptance tests prompt builds that test set, including the hostile and recovery cases teams tend to skip. Run it before the pilot starts and again before anyone calls it finished.

05 // Measure the workflow, not the model

The NIST AI Risk Management Framework frames trustworthy AI around how a system performs in its actual context, not in isolation. For a pilot, that means measuring the workflow outcome the business cares about: quality, adoption, turnaround, exception rate, and what operators say about using it.

Avoid promising headcount reduction. It is rarely the right measure, and it makes the people whose expertise the system depends on reluctant to help it succeed. Measure whether the workflow got faster, more consistent, and easier to review, and report the cases where it did not.

06 // Plan the operating loop from day one

A pilot that succeeds still needs an owner, infrastructure, monitoring, and a budget for recurring model and provider costs. Write those responsibilities into the scope so a successful pilot has somewhere to go. Otherwise the most likely outcome of a good result is a prototype nobody is allowed to run.

This is how we approach AI automation engagements: frame the workflow, prove the riskiest path, build the production interface and review loop together, then measure against the agreed outcome. The Vantage Bid case study shows the same pattern applied to proposal work, where drafting and compliance checks stay reviewable by the people who own the response.

A pilot is finished when the people who own the workflow trust it, not when the demo works.