Agentic AI Implementation: A Practical Engineering Guide

Agent projects rarely fail at the model. They fail at the boundary: the tool that returns an error the model cannot act on, the retry that charges a customer twice, the run that spends eleven dollars deciding not to answer a support ticket. Those are ordinary engineering decisions, and most of them are made before anyone writes a prompt.

The demo is easy. A model, four tools, a well-chosen example, and the loop does something that looks like judgement. Then it meets a real queue, real permissions, a rate limit, a tool that times out at the wrong moment, and a user who phrased the request in a way nobody anticipated.

So this is a guide for the point after you have decided to build an agent. One framing note first, because most of the rest follows from it: an agent is not a smarter chatbot. It is a program that takes actions in your systems, with a component in the middle that sometimes picks wrong and always sounds confident.

What actually justifies an agent instead of a script

An agent earns its cost when the sequence of steps cannot be known before the work starts. That is the whole test. If you can draw the flowchart, build the flowchart: it is cheaper to run, cheaper to debug, and it fails in places you can point at.

Three questions separate the cases:

• Does the next step depend on what the previous step found? Not on the input, which a switch statement handles, but on data discovered mid-run. A refund investigation that reads the order, then possibly the shipping scan, then possibly the chat history depending on what each one says, has real branching. A nightly report does not.
• Is the branching too wide to enumerate? Six paths is a state machine. Several hundred plausible paths through twenty tools is an agent.
• Can you check the result? If there is no way to tell a good outcome from a bad one without a human reading everything, you are building something you cannot operate.

A practical exercise: take the last ten real cases of the work you want to automate and write out what a competent human actually did, step by step. If one flowchart covers eight of them, you have your answer, and it is not an agent. If the ten paths barely rhyme, an agent is the right shape and you now have the beginnings of an eval set.

The commercial version of the question matters too, because defensibility sits in the data, the workflow and the integrations around the loop rather than in the loop. That argument is made at length in the piece on building defensible AI moats, and it changes what you build first.

Scope the first agent so it can fail safely

The first agent you put in production should be the one whose worst day is survivable. That constraint is more useful than any architecture diagram. Four properties make a good candidate:

• Reversible outcomes. Drafting, tagging, enriching, routing and recommending can be undone by a human in seconds. Sending money, emailing a customer, deleting records and changing prices cannot.
• A cheap oracle. Some way to know afterwards whether the run was right, from a system of record rather than the model's own summary. Cases where truth arrives later anyway, such as a chargeback outcome or a resolved ticket category, are ideal.
• Enough volume to learn from, not enough to bankrupt you. A few hundred runs a day teaches you the failure distribution. Ten teaches you nothing and fifty thousand teaches you an invoice.
• One team owns the outcome. An agent spanning three departments spends its project in arguments about whose metric it moves.

Why fraud triage is a reasonable first agent

Fraud review is a good shape because the useful version of it is read-mostly. The agent gathers the transaction, the account history, device and address matches, prior disputes, related accounts and the merchant category, then writes a recommendation with the evidence attached and hands it to a reviewer who already does this work by hand.

Nothing irreversible happens. The reviewer's accept or override becomes a label, which becomes your eval set. Because the human decision stays in place, a bad week costs reviewer time rather than blocked legitimate customers or approved fraud. If the agent later earns the right to auto-clear obvious cases, you will have data to argue the threshold instead of guessing it. The same pattern fits most decision support work in fintech and payments systems, where the audit trail matters as much as the decision. The tempting alternative, an agent that blocks transactions on day one, inverts every property above.

The tool surface is the product

Most of the quality of an agent lives in its tools, not its prompt. A tool definition is an API for a caller that reads the docs once, cannot ask a clarifying question, and will guess when the shape is ambiguous. What works:

• One tool, one verb, one clear outcome. cancel_order beats manage_order(action=...). A parameter that changes what the tool fundamentally does is two tools wearing a coat.
• Make illegal states unreachable. Enumerate where you can: status as one of open, pending, closed rather than a free string. Required fields required. Dates as ISO strings with the format stated in the description.
• Return errors the model can act on. 400 Bad Request teaches it nothing, so it retries the same call. start_date must be earlier than end_date; both are ISO dates gets a corrected call on the next step.
• Return less. A 4,000 token JSON blob is paid for on every subsequent step of the run, and it buries the two fields that mattered. Project what the agent needs, and include the identifier for the next call.
• Be explicit about truncation. If a list is paginated, say how many results exist and how many came back. Silent truncation produces confident wrong conclusions about "all" of something.
• Write the description for a stranger, including when not to use the tool. That one line prevents a whole class of wrong selection.

Tool count matters more than people expect. A flat list of forty tools degrades selection, and the symptom looks like a model problem rather than a design problem. Group related operations, or split the work across a few narrow agents behind a router, each holding six to ten tools.

Frameworks, SDKs and low code agent builders

The build-versus-assemble question has a practical answer: a low code agent SDK or visual builder is a good way to get the first internal version in front of people, and a poor place to own production behaviour you care about. These platforms hand you the parts nobody enjoys writing, including the tool-calling loop, a trace viewer, connectors and a place to edit prompts without a deploy. The limit shows up when you need something specific, and with agents you always do: your own idempotency keys derived from your own run identifiers, per-tenant permission scoping, a retry policy that knows which of your APIs are safe to repeat, or a regression suite that runs in your CI.

Judge a platform by its escape hatches rather than its demo. Can you see the exact payload sent to the model? Override the prompt? Intercept a tool call before it executes? Export a full run transcript and replay it? If not, the platform is fine for the prototype and you should expect to rewrite the loop in code once the thing is load bearing. That rewrite is cheaper if the tools are ordinary HTTP services from the start, which is also how they stay useful to the rest of the product you are building around them.

State, memory, and what the agent is allowed to remember

Three different things get called memory and they have different failure modes.

The run transcript is working state: messages, tool calls, results. Disposable, growing, and the main cost driver on long runs. Decide early how it gets compacted. Summarising the middle while pinning the original task, the constraints and the identifiers discovered so far beats dropping the oldest messages, which is how an agent forgets what it was asked to do at step fourteen.

Task state is a row in your database, and it is the truth. The context window is not a database. Write each decision and each completed side effect out as a durable record the moment it happens: what step, what tool, what arguments, what came back, what state the task is in now. That single habit makes a crashed run resumable instead of restartable, and restarting a run that already issued a refund is how you refund twice.

Cross-run memory is where governance bites. An agent that remembers across sessions can also carry one tenant's data into another tenant's run, or hold a customer detail past the point where you told the customer you delete it. Scope memory by tenant in the storage key, not in the prompt, and treat the memory store as regulated data with a retention policy. A prompt instruction not to leak is not a control.

Retries, idempotency, and what happens when the model picks wrong

Once a non-deterministic component drives your API calls, retry semantics stop being optional. Every tool with a side effect should take an idempotency key that the platform derives from the run identifier plus the logical step, so the same intent retried produces the same key. Never a fresh UUID at call time and never a timestamp, both of which turn a retry into a second action. And never let the model supply the key: it is the one component that might generate a different one for the same intent.

Classify failures before the agent sees them. Timeouts, rate limits and upstream 5xx responses are retryable by the platform with backoff and jitter, and the model does not need to know they happened. Validation errors and permission denials belong to the model, because a different call might succeed. Mixing the two produces an agent that gives up politely on a transient blip, or one that hammers a broken endpoint twenty times.

Cap the loop three ways: maximum steps, maximum wall clock, maximum spend per run. A run that hits a cap exits into a human queue with its transcript attached. What you must not allow is a capped run returning a partial answer as if it were finished.

The failure modes worth designing against

• The same call, repeatedly. Usually an unhelpful error message. Detect identical consecutive tool calls with identical arguments and break the loop rather than paying for it.
• Oscillation. Two tools called alternately because neither result resolves the ambiguity. Almost always a tool design problem: the answer the agent needs is not available from either one.
• Right tool, wrong entity. The plausible customer ID, the order from last month. This is the expensive one, because the call succeeds. Validate that identifiers came from a previous tool result in the same run rather than from the model's own text.
• Stale identifiers. The agent read the record at step two and acts on it at step nine. Pass a version or updated-at value and let the write fail if it moved.
• Fabricated completion. The agent reports that it processed the refund and notified the customer, having done neither. Nothing in the transcript prevents this, so never treat the model's summary as evidence. Verify the outcome from the system of record before closing the task, which is why the last step of a run should be a check rather than a narration.

Where human approval gates belong

Put the gate at the irreversible step, not at the start of the run. Approving a plan is theatre: the plan is the cheap part, and the approver cannot know how the model will interpret its own plan four tool calls later. Approving the specific action about to be taken, with its arguments visible, is a control.

Three questions decide whether a step needs a gate: is the action reversible, does it touch money or a customer or a regulated record, and who is answerable if it is wrong. If the answers are no, yes and nobody, do not ship it.

A gate that works shows the approver four things and nothing else: the exact action with its arguments, the evidence the agent used, what changes as a result, and the cost of being wrong. If the reviewer has to open three other tabs to decide, they will start approving everything, and you now have an audit trail saying a human checked when no human checked.

Approval rate is a signal in itself. If reviewers approve essentially everything, the gate filters nothing, and the honest options are to remove it and accept the risk on the record, or to sample a fraction of actions. Frequent overrides mean the agent is not ready for that step, and those overrides are your best labelled data.

Agentic AI governance, in practice

Governance for an agent is mostly access control and record keeping, written down before launch rather than after an incident:

• Each agent gets its own scoped credentials, never a shared service account. You want to answer "what could this thing touch" by reading an IAM policy.
• The agent's permissions never exceed those of the person who triggered the run, or the agent becomes a privilege escalation path.
• Every run logged with inputs, model and prompt version, each tool call and result, cost, outcome and any approvals, with retention stated. This is what makes an incident explainable.
• A kill switch that has actually been tested, at two levels: stop new runs, and stop one tool executing while runs continue.
• A named owner, and a documented path for a customer or reviewer to contest an automated decision.

Evaluating something that answers differently every time

You cannot assert on strings. You can assert on outcomes, and that distinction is the whole discipline. Build the eval set from real traffic: freeze fifty to a few hundred cases with the expected outcome recorded rather than the expected wording, meaning which final state, which tools should have been called, which must never have been called, and what the answer should contain semantically. Add every case that once broke the agent and never remove them.

Three layers, each cheap to run at a different frequency:

1. Tool unit tests. Fully deterministic, no model involved. These catch most real breakage and run in seconds.
2. Replayed trajectories. Recorded tool responses, the model called for real. Asserts on the path and the final state. This is where you catch a prompt edit that quietly changed tool selection.
3. Live sampling. A percentage of production runs reviewed by a human against the same rubric, because production always contains inputs your frozen set does not.

Run layers one and two in CI on every change to a prompt, a tool definition, a model version or a framework upgrade, and track two numbers per run of the suite: pass rate and cost per case. A change that lifts pass rate by a little while doubling steps per case is usually not the trade you want.

A model grading outputs is useful for qualities you cannot assert mechanically, such as whether a drafted reply was appropriate. Pin the judge's prompt and model version and keep a small human-labelled set to check the judge itself. An unpinned judge drifts, and a drifting judge hides a drifting agent.

Cost, latency and observability under real load

Agent cost is per step, and step count is the variable you control least. Average cost per run is a comforting number and the wrong one. Look at the distribution: the p95 run with eighteen steps, three retries and a 30,000 token context is what your invoice is made of. Track cost per resolved case, including failed runs and the human time spent on escalations.

The levers, roughly in order of effect: return smaller tool results, since a verbose tool is a recurring charge on every later step; use a cheap model for routing, extraction and classification and the strong one only for judgement; cache the stable prompt prefix and any tool result that is expensive and slow to change; precompute what the agent keeps re-deriving; and cap steps, because an uncapped loop is an open-ended bill.

Latency behaves nothing like a normal request path, because a run is a serial chain of model and tool calls and nobody watches a spinner for forty seconds. Make it asynchronous with a visible job state the user can leave and return to, or stream progress so the wait is legible. Then treat concurrency honestly: provider rate limits, not your servers, are usually the ceiling. Queue runs, apply backpressure, and give retries jitter. A synchronised retry storm after a provider blip is an outage you caused.

Observability is not optional, because you cannot reproduce a failure by re-running the input. One trace per run, one span per step, each span carrying prompt version, model identifier, tool name, arguments, token counts and cost, with the full transcript stored so a run can be replayed exactly. Then alert on what moves before quality visibly drops: step count distribution shifting, tool error rate rising, cap hits increasing, override rate climbing, cost per case creeping. Those five numbers on one dashboard will tell you a prompt or model change went badly long before anyone complains, and building it before launch is the cheapest part of an agentic AI build you will ever do.

None of this is exotic infrastructure. It is queueing, idempotency, access control, tracing and regression testing, applied to a caller that guesses. Teams that already run production services well find agents unremarkable once they stop treating the model as the system and treat it as one unreliable component inside a system they know how to build.

Have you decided to build an agent and now need to settle the tool surface, the approval gates and the eval set before anyone writes code? A Scoping Sprint ($2,300, two weeks) ends with an agent architecture and evaluation plan made for your case, a prototype, and a fixed quote. Or just start a conversation.