There is no sound
Traditional software fails loudly. Exceptions land in logs. Alerts fire. Someone opens a ticket.
Agentic workflows often fail without that noise. The model answers. The tool call looks fine. The session ends. No stack trace. No alerts. Users leave. Trust goes with them.
There is no sound.
When teams cannot see where the break happened, they guess. They swap the model. They rewrite the tool description. They add documents. They chase symptoms because nothing in the logs marked the failure.
Visibility fails when you cannot name the broken step. Quality fails when the answer is wrong or incomplete and nobody marks it. Both need a detection stage before you automate the fix path.
I spent ninety days on a system that finds silent failures in a hosted AI assistant and routes evidence for fixes. The initial problem brief said to automate the path from a known failure to a shipped fix. The work that mattered first was making those silent failures audible.
What we were handed vs what we learned
The initial problem brief assumed the hard part was already done. Failures were identified. The job was to automate the path from a known failure to a shipped fix.
That path is useful. It is also incomplete when the agentic product fails silently without a signal. If nobody has marked the failure, there is nothing honest to automate.
We added detection as its own stage. Before fix automation, the system had to decide which interactions looked broken, with what evidence, and which owning team should see them.
The outcome we aimed at for product teams shifted with that change. Detect the failure. Classify and rank the failure. Package evidence someone can act on. Who applies the fix was still open at the start. We had not agreed the boundaries between teams.
Early on, we and the stakeholders were figuring it out together. We were learning the real problem at the same time we were building the first slice of the system.
You cannot jump into the solution only because the brief handed you one.
Problem understanding: first 30 days
We treated the first weeks as Polya’s understand step: write down the goal and the conditions before you plan the attack.
| Ask first | In this work |
|---|---|
| What outcome? | Product teams detect failures, prioritize them, and get evidence for fixes |
| Who has the problem? | Teams shipping agentic assistants |
| What data exists? | Traces from the assistant (or not) |
| What constraints? | No traces means no detection; many failure surface layers means many owners |
The constraints chose the first candidate. Ambition said every product team. Traces said one hosted path.
The first constraint was observability. Not every product team emits traces. We initially scoped to all product teams, but the engine needs trace data from the assistant. Without traces, there is nothing to detect against. We narrowed to one hosted experience where traces already existed, and said other teams need the same traces first.
The second constraint was the failure surface layer: where in the stack the break shows up. In a conversational assistant that is content, retrieval, or the agent layer. Each needs different detection rules and routes to a different team. We needed shared failure categories before we could build a pipeline.
The first thirty days produced three things. Detection criteria that filter traces to candidate failures. A map of failure surface layers (content, retrieval, agent) and what causes them. A small proof of concept that the same trace shape could feed the system for another product.
We spent those weeks on constraints, candidates, and failure surface layers. That understanding set the order of the work.
The feasibility trap
We ran a friendly test on one known case. The failures were already pre-filtered from user negative feedback. The root cause was known. The fix was small and standalone, without side effects. We made AI-assisted code changes for the fix, kept a human in the loop, and raised a pull request for a downstream team. The first full pass finished in under an hour.
That test proved a mechanism. A known failure can move from signal to proposed fix with human review. People took away a bigger story. They thought the engine can auto-fix the backlog.
It could not. Each failure surface layer has a different owning team. Content, retrieval, and agent do not share one backlog. The test never checked which team owns which fix. It only used tools and access we already had on a case we already understood.
Sixty days: proving the stages
After the friendly test, we needed proof that each phase could run on real traffic, not only on failures we already understood.
| Phase | What it does |
|---|---|
| Detection | Filter, classify, and route |
| Remediation | Triage with negative cases and apply the fix |
| Verification | Negative cases pass, and the regression suite does not get worse |
The next sixty days were about that kind of proof. We locked detection criteria so only candidate failures entered analysis. We built an agentic evaluation pipeline over shared failure categories for each failure surface layer: content, retrieval, and agent. Each category had a cause chain we could argue about. We mapped which owning team should see which class of evidence. We demoed tool-call failures, retrieval gaps, and content gaps to leadership and to the teams who’d have to act. This is what you hit in companies at scale, where multiple teams are part of one system.
Outside our pipeline, the retrieval team shipped hybrid search. Some retrieval failures dropped for real. We could show a measurable improvement we didn’t ship ourselves. That hinted at the loop we wanted. Detect in the pipeline, say which team owns the gap, clear it on their side, and measure that customer friction moved.
At sixty days I was still validating each stage with small proofs. Could detection run? Could triage classify? Could we route evidence? I was postponing the ownership questions the friendly test had already raised. The stages could run. Who absorbs the volume of the gaps we find, and who confirms a gap is gone, was still fuzzy.
Ninety days: intake is not the whole loop
At ninety days the question got sharper. Can we close content gaps at production scale? Technically, pieces of that path work. As a system the organization can run every week, we weren’t there yet.
We proved intake at scale. Detection could filter real user traffic from traces. The evaluation pipeline could classify failures against the shared categories. Negative test cases could reach the teams responsible for each failure surface layer. What we had not proved was closure. An owning team absorbs a ranked item, ships a fix, and we can show customer friction moved.
| Loop | What it includes | Who owned it here |
|---|---|---|
| Intake | Detect, classify, rank, route with packaged evidence | We did |
| Clearance | Ownership, trusted volume, priority, confirm the gap is gone | Across team boundaries; we named this late |
Intake can scale inside one team’s control. Clearance only works when owning teams agree to absorb volume and confirm the gap is gone.
The questions we still couldn’t answer were simple. Who owns the fix for each failure surface layer? What volume will receiving teams trust before the queue becomes noise? What gets priority when everything looks urgent? We raised those as risks with ways to reduce them. These are ownership and process questions.
We proved intake at scale before we proved closure at scale. On purpose, once we saw the trap. Automate a firehose first, and you can spend the next quarter blaming downstream teams for drowning in tickets. We instead named the boundaries first, and “fixed” without “gap cleared” stops counting as a win.
Building the system and agreeing the process go together. Outcomes stick only when the system, the process, and the boundaries are agreed.
Fix faster, or earn trust first: you need both
Once intake works, teams often split into two camps.
One camp wants to fix faster: stronger tests, AI-drafted changes, less time on each pull request. The other wants to earn trust first: send fewer items, make ownership clear, and prove gaps actually close.
Each one fails in a different way if you only do that side.
Fix-faster work helps when volume is high and reviewers are the bottleneck. It falls short when nobody has agreed what is in scope or what comes first. “Human in the loop” here usually means someone reviews the PR.
Earn-trust work helps when downstream teams will ignore a noisy queue. It falls short when you never invest in tests, so every fix still needs an expert reviewer. “Human in the loop” here also means people confirm scope, set priority, say the gap is gone, and catch cases that pass tests and still hurt users.
The two camps meet on one practical question: what do the tests cover? Minimal human review is fine only if someone is still accountable when a “passing” change leaves customers stuck.
We needed both. More pipeline without trust becomes a firehose of tickets. More process without a fast fix path becomes a committee that never ships. The real choice is where you spend this quarter’s attention, not which camp wins the meeting.
The job shifted from building to framing
I started as a builder. Midway through the ninety days I was doing more product-manager work: what outcome are we after, and what can the pipeline not decide by itself?
Shipping a new stage is easy to show. The harder work was quieter. Say why silent failures matter. Ask early who owns the fix and how much volume teams will take. Make sure “we detected it” does not mean “we flooded the downstream team.”
The trap is demos and ticket counts. You can show a classifier and a router and a rising backlog while the teams getting those tickets stop trusting them, and users still hit the same problems.
Trust comes before volume. Do receiving teams believe what we send? Can we keep the queue small so they work the worst gaps first? A short ranked list beats dumping every candidate failure.
We still had to build. Once the stages ran, though, the limit was not more features. It was agreement on ownership, priority, and pace.
What I would do next
I would stop counting how many tickets we filed. That number is busywork. I would ask the teams who get those tickets whether they trust them, and whether the list is small enough to work.
I would track the bad gaps: ones that hurt users, raise risk, or keep coming back. Progress means those gaps close and stay closed. A long backlog is not progress.
I would make one hosted product path work end to end before we add more products. Traces in, ranked evidence out, owners who fix and confirm. Spreading thin just makes a bigger firehose.
Next I would keep the receiving teams’ trust, close the worst gaps, and deepen one hosted path. That order beats more routing.
Reflection
Make silent failures visible. Find who can fix them. Send a short ranked list teams will trust. Measure gaps that close. Tickets are not the win if users still hit the same walls.
Spend the first weeks understanding the problem. Who is impacted. What are the constraints. What you are assuming. Where a small change helps most. Start where traces already exist. Prove value early. Open the path for other products later.
At sixty days, prove the stages can run, and name who owns the fix and how much volume teams will take. Leadership can only settle those if you surface them early.
Stay in the product seat when the demos get loud. Build something the organization can run, not only a pipeline to show.
Measure what closed and whether trust held. Ticket count is noise.