Shipping an Agentic Provider Search Into Production

The hard part was never generating an answer. It was making a generated answer trustworthy enough to bill for — over data that couldn’t be copied, embedded, or exposed to get there.

Provider directories are a strange kind of data problem. The records are structured, authoritative, and enormous — 1.6 million practitioners, each tagged against a 674-entry clinical taxonomy. Any question you can express in that taxonomy already has an exact answer sitting in SQL.

The problem is that nobody asks questions in the taxonomy. A health plan’s network team doesn’t search for Cardiovascular Disease. They type heart drs in fl. The gap between how the question arrives and how the data is stored is the entire product.

The obvious 2026 answer is to put a language model in front of it, embed the corpus, and let semantic search do the work. That runs into two walls at once. This is regulated provider data, and every copy of it — every index, every embedding store — is another surface to secure, audit, and defend. And when someone at a health plan asks where a specific result came from, “the model decided” is not an answer you can sell.

So the system had to do something narrower than the demo version of this product: use the agent to understand the question, and deterministic SQL to produce the answer — without the provider data ever leaving the one place it was already protected.

That constraint, applied over and over, is what the rest of this is about.

The product we scoped is not the product we shipped

This started as something else.

The original concept was an AI chat interface over provider data, backed by five MCP tools with carefully written, model-facing descriptions telling the agent when to reach for each one. Part of the appeal was that those servers might eventually be exposed publicly, which would have made the MCP layer genuinely load-bearing: many consumers, many clients, one tool interface.

I built that first version — frontend and AI integration both — starting with custom MCP wrappers over NPPES and a handful of other public and private APIs, running against dummy data while the real data path didn’t exist yet. It’s a good way to build an agent product: you find out immediately whether the model can drive the tools you’ve designed, without waiting on anyone’s database. It also means what you’ve proven is narrower than it feels, which matters later.

Neither half of that concept survived contact with the organization.

The chat interface was deferred, then removed. The MCP servers never got the buy-in they needed — not because they didn’t work, but because in early 2026 the protocol was new enough that most stakeholders had no frame for evaluating it. It’s hard to approve infrastructure whose value proposition is “this will matter when there are other consumers,” when the other consumers are hypothetical.

The uncomfortable part is that once public exposure was off the table and the chat was gone, the honest technical case for MCP went with it: one application, one model, one backend, and a network hop between the agent and a stored procedure in a product where search latency is something customers notice.

So the tools came out and the agent emits query parameters directly against SQL.

What replaced the chat was narrower and, I’d argue, better: a search interface with a map and a table, where enriched provider data — not conversation — was what customers were paying for. That’s also where it stopped being a prototype. Backend teammates refactored the API layer and wired it to the real databases; I stayed on the frontend and the AI integration and picked up a growing share of the infrastructure as the environments got more real.

Infrastructure justified by future consumers has to survive those consumers not arriving.

That’s not an argument against MCP. In the same organization, in the same period, I shipped an MCP server bundle across a dozen-plus engineering pods — many consumers, many heterogeneous systems, exactly the shape the protocol is for. Same reasoning, opposite answer, because the consumer topology was different.

The agent produces the query, not the record

With chat gone, the remaining architectural question shaped everything after it:

Does the model produce the answer, or produce the query?

It produces the query. Practitioner search runs against Azure SQL through stored procedures — the same procedures a structured search form would call. The agent’s job is to turn heart drs in fl into the parameters those procedures expect.

Its output is a SearchPractitionerRequest object: arrays of last names, states, counties, specialties, statuses, plus paging and sort. That JSON goes to the stored procedure. The records come back as JSON of their own and render through React components — table, map, data-quality summary — each row clickable through to an individual provider’s detail.

The agent also writes the plain-English description of the search the user reads: the searchSummary field, produced in the same pass as the query, before any results exist. What’s notable is that it’s authored to survive every outcome. The prompt requires phrasing that composes with a “No ___ found” prefix, so cardiologists in Rural County, Wyoming works whether twelve results come back or zero — where Found cardiologists in Rural County, Wyoming produces “No Found cardiologists…” the moment the result set is empty.

Writing the summary before the results exist forces it to describe the question rather than assert an answer. That’s the safety property in one line: the prose restates what was asked, and the data answers it. Nothing a user reads as a fact about a specific provider was generated — it was queried, serialized, and rendered. Compare that to a chat product where the model’s prose is the deliverable and the user has no way to check it.

So the failure modes that kill AI products in regulated domains mostly don’t exist here. No hallucinated NPIs. No invented credentials or addresses. No prompt-injection path into the record set.

The stack backs this up structurally: React 19 + MUI v7 + Vite on Azure App Service, talking to .NET 9 isolated-worker Azure Functions, which orchestrate Azure AI Foundry Persistent Agents (Azure.AI.Projects + Azure.AI.Agents.Persistent) and execute against Azure SQL via Dapper.

Semantic Kernel came out here too — removed in a deliberate refactor once the agent’s job narrowed to emitting parameters. An orchestration framework earns its cost when there’s orchestration to do. There wasn’t.

The model can describe the answer. It shouldn’t be the only thing standing behind it.

Vector search was the obvious answer, and the wrong one

The proposal on the table was the one any engineer would reach for: vector search over the provider database. Embed the records, index them, let semantic similarity handle the messy queries.

What does it actually cost to make this data semantically searchable?

More than it looked like. Embedding a provider corpus means a second copy of regulated data in a second system — a new store to secure, a new access path to audit, a new thing to keep in sync, and a new place a breach can start. The privacy exposure isn’t the query; it’s the durable copy. And embedding a corpus that changes constantly is real recurring money for a beta product.

What we shipped keeps the provider data exactly where it was and moves only the vocabulary out:

• A small Azure AI Search index over the 674-entry specialty taxonomy — public reference vocabulary, not provider records. The agent queries it as a tool to resolve lay terms into canonical taxonomy strings.
• A custom stored procedure that accepts an array of those resolved items, generated by the agent from the user’s query.
• Everything else — the matching, filtering, and retrieval across 1.6M records — happens inside SQL, behind the boundary that was already secured.

From the user’s side it behaves like semantic search over the directory: colloquial queries work, related specialties surface, the map and table populate with what you meant rather than what you typed. But no provider record was copied, embedded, or exposed to make that happen. The semantic layer operates on 674 rows; the sensitive data never leaves its blast radius.

What that index actually does: BM25 keyword matching against the specialty name, a synonym map (specialty-synonyms) expanding lay terms, and Microsoft’s semantic reranker reordering those candidates. The index is configured to rank by keyword relevance rather than embedding distance — no vector fields, no embedding pipeline, no vector storage anywhere in the path.

That’s the whole semantic layer: 674 rows of public vocabulary, ranked by keyword match and reordered by a reranker. A smaller and less fashionable machine than the one proposed, producing the same user-facing behavior — because resolving a colloquial phrase to one of 674 known strings is a canonicalization problem, not a similarity problem over unstructured text.

Worth being exact about the vocabulary, too: this is retrieval used for query normalization, not retrieval-augmented generation. The resolved specialty becomes a filter parameter, not context injected into the model to ground a generated answer. Different failure modes, different costs, different compliance profile — and calling it RAG would misdescribe the system in the direction that happens to sound more impressive.

The cheapest way to secure a copy of sensitive data is to not make one.

Teaching an agent a vocabulary it kept getting wrong

All of that only works if the resolution step is right.

How reliably does “heart doctor” become Cardiovascular Disease?

Letting the model map lay terms to the taxonomy from its own knowledge fails in a specific and dangerous way: it’s right most of the time, so the failures arrive as confident, plausible, wrong specialties rather than as errors. A network adequacy report built on a quietly mis-resolved specialty is worse than one that errored.

So the system prompt makes the lookup non-negotiable — not a suggestion the model weighs, but a precondition with a defined failure state. It instructs the agent that any medical or healthcare term must go through the search tool before the specialty field can be set, forbids falling back on the model’s own training knowledge to guess a specialty name, and specifies exactly what happens if the tool isn’t called: the field is left null rather than filled with a guess.

That last part is the important one. The instruction doesn’t just forbid guessing; it specifies what happens instead. An unresolved specialty becomes null, the search runs on location alone, and the user gets a broad result set they can narrow — rather than a narrow one built on a specialty the model invented.

On top of that sits a three-tier confidence policy, which is where the domain judgment lives:

• Exact match — a score above 4.5 means the user typed something close to a real taxonomy entry. Take at most two results and stop. Cardiology resolves to Cardiology, nothing else.
• Related match — no high-confidence hit means the user spoke colloquially, so the threshold drops to 2.0 and the top three come back as a set. Foot doctor becomes Podiatrist, Podiatrist, Foot & Ankle Surgery, and Orthopaedic Surgery, Foot and Ankle Surgery — because someone asking for foot doctors wants all three.
• Too generic — more than five results clearing the low bar means the term carries no signal. Specialty goes null and the search falls back to location, instead of returning a union of a dozen specialties that looks like precision and isn’t.

Each tier answers the same question differently: how much does this term actually narrow the search? A confident term should narrow hard, a colloquial one should fan out to a small related set, and a meaningless one should narrow nothing at all rather than pretending to.

Underneath all of it, invisible to the prompt, the synonym map handles what no scoring logic can — abbreviations, misspellings, and regional phrasings whose words simply aren’t close to the taxonomy string. specialty-synonyms is attached to the index at the field level, so it isn’t a model concern at all. It’s a text file a domain expert can read. When a health plan reports that some local term doesn’t resolve, the fix is a line in that file: no prompt revision, no model evaluation, no deploy that could regress unrelated behavior.

There’s a degradation path too. If the specialty index is unavailable, specialty goes null, the rest of the query proceeds, and the summary says so — practitioners in [location] (specialty matching unavailable). The search degrades to a worse search rather than an error, and the user is told which one they got.

A search index earns its place when it resolves ambiguity, not when it sounds impressive.

That turned specialty matching into a vocabulary-coverage problem the team could measure, rather than a model-quality problem they could only hope about.

Showing the work was a trust feature, not a loading state

Agent latency is real. Resolving a specialty, emitting parameters, and executing against 1.6M records is not instant, and the options were to hide that behind a spinner or to show it.

What is the user allowed to see while the system is thinking?

The whole path streams, and the intermediate steps surface as they happen — the query being interpreted, the specialty being resolved, the search executing, the results coming back. Some of that is ordinary perceived-performance work: a progress narrative feels far shorter than an identical wait behind an opaque spinner.

The more important part is that it makes the system legible. A network manager watching heart drs in fl resolve to Cardiovascular Disease before results land learns how the product works, and gets a chance to catch a wrong interpretation. A spinner would have hidden exactly the step most worth seeing — the one where a colloquial phrase becomes a clinical term everything downstream depends on.

It’s the same property as the auditable query path at a different altitude. The query trace answers how did this result happen for a compliance reviewer after the fact; the streamed steps answer it for the user in the moment.

Streaming the intermediate steps turns latency you can’t remove into transparency you couldn’t otherwise offer.

The chat came back as a search that remembers

The first version of the search was stateless: every query started from nothing.

That tested badly, for an obvious reason in hindsight. Nobody searches a provider directory once. They search, look at what came back, and narrow — now just Florida, massage therapists only, drop the retired ones. Making each of those a fresh query meant retyping the location, the specialty, and the status every time, which is exactly the tedium a natural-language interface is supposed to remove.

How do you keep a conversation’s memory without bringing back the conversation?

By making refinement a rule of the query builder rather than a feature of a chat window. The agent reads the most recent request in the thread, carries all of its filters forward as the base, and modifies only the fields the new message mentions — never resetting a filter to null unless the user explicitly removes it.

The subtle part is reading intent from very short utterances. Only is the hard one: after a specialty term, it means replace the specialty filter while keeping every other filter — location, status, everything — from the prior search.

A user in Washington who says massage therapists only means replace the specialty, keep me in Washington — not search everywhere, and not add massage therapy to my existing specialties. Getting that wrong silently produces a plausible result set for a question nobody asked. So the prompt carries explicit vocabularies for refinement (show me only, narrow to, just, also add, exclude) versus starting over.

The result is the useful half of the chat interface without the liability half. State persists, follow-ups are one phrase long, and every turn still resolves to an explicit filter set the user can see in the summary — rather than to a conversation history whose influence on the current answer nobody can audit.

Multi-turn refinement was worth keeping. Multi-turn generation was not.

The product was the data, not the interface

Once chat was gone, what customers were evaluating wasn’t the AI. It was whether the records were right.

Provider directory data decays constantly — practitioners move, change affiliations, close panels, retire. A search interface over stale records is a faster way to get a wrong answer. So enrichment and data quality became the center of the product, and the AI layer’s job was to make good data reachable, not to compensate for bad data.

Enriched records were something a user could explicitly ask for rather than a background process — a UI toggle, and a showEnriched flag the agent sets from phrasing like show enriched providers in 10001.

The map and table were where that data got evaluated. A conversational answer can assert that a plan has adequate coverage in a county; a choropleth lets a network manager see the gap themselves.

Making bad data conversational just produces confident bad answers faster.

Agents fail differently than APIs

Then the system met real usage, and a category of failure showed up that a stateless API doesn’t have.

Why does the first search after login sometimes return nothing at all?

Foundry Persistent Agents hold conversational threads, and threads go cold. A cold thread’s first invocation could return an empty response — not an error, not a timeout, just nothing — which reached the user as a search that appeared to work and found no providers. That’s the worst possible failure for a directory product, because “no results” is a legitimate answer.

The fix was three connected pieces:

• Detect the empty response specifically and treat it as failure rather than result — evict the thread, retry once against a fresh one.
• Reuse threads per user within a bounded window so the cold-start path is rare rather than constant.
• Prewarm on login, so the cold start happens during a loading state the user expects, not mid-search.

And instrument all of it — AI.Foundry.* telemetry, so the empty-response rate is a number on a dashboard rather than something the team believes is fixed. The pattern, generalized:

// Illustrative — not the actual implementation.
//
// On the agent's first attempt, an empty response is treated as a
// cold-start failure rather than a valid "no results" answer:
//
//   if (response is empty) and (this is the first attempt):
//       log a warning
//       release/evict the current agent thread
//       retry once against a fresh thread
//
// If the retry also fails, or an exception is thrown on the first
// attempt, the same evict-and-retry-once pattern applies. A second
// failure is logged and surfaced as a real error rather than retried
// again.

The general principle carries to any agent-backed product: a language model in your request path introduces failure modes that return 200 OK. Empty responses, truncations, and silently degraded answers trip none of the alerting you already have.

An agent-backed feature isn’t reliable because it stopped failing loudly. Instrument the quiet failures.

Compliance is a data-path property, not a checkbox

Healthcare adds constraints that shape architecture rather than sitting on top of it — the same constraint that killed vector search also governs the request path.

Can a health plan’s user ever see another health plan’s data?

Authentication is Okta OAuth with PKCE, validated as JWT at the API boundary through OIDC middleware. Multi-tenancy scopes on AccountId, and the key property is that the account identity is derived server-side from the validated token, never accepted as a client-supplied parameter. A user cannot request another tenant’s scope, because the scope isn’t something the request gets to assert.

Azure service-to-service auth uses managed identity throughout, removing credential handling from the deployment surface entirely.

Tenancy that a client can assert isn’t tenancy.

The same constraint reached the hosting layer, and cost us a rewrite of how the app was deployed.

The frontend started on Azure Static Web Apps — the right default for a React SPA, and cheap, fast, and nearly operations-free. It stopped being right the moment the environments had to be real. Dev and tst/uat needed to reach AI Foundry, APIM, and the database over a private network, and static hosting is built on the opposite assumption: public edge distribution, public egress, no VNet integration.

So the app moved to Azure App Service. On paper that’s a downgrade: more infrastructure, more configuration, less serverless convenience. What it bought was every non-production environment sitting inside the same private network as the services it talks to — so the path a request takes in dev is the path it takes in production, and private endpoints aren’t a thing that only exists in the environment nobody can test against.

Serverless hosting is a great default right up until your data path has to be private. Then it isn’t a default — it’s a constraint you accepted without noticing.

Impact

• Shipped natural-language provider search into paid beta with 64 enterprise health-plan customers across 12 plans.
• Made 1.6M+ provider records searchable in plain language against a 674-entry clinical taxonomy.
• Delivered semantic-feeling search without embedding or duplicating a single regulated provider record — semantic retrieval operates on a 674-row public vocabulary; matching stays inside SQL.
• Kept every result traceable to a deterministic SQL query; provider records render from structured JSON, never from model output.
• Streamed intermediate agent steps to the UI so users could see a colloquial query resolve to a clinical term before results landed.
• Rebuilt search as stateful after testing, so follow-ups (“massage therapists only”) refine the prior filter set instead of starting over — conversational continuity without a conversational interface.
• Designed an agent-generated array parameter into a custom stored procedure, minimizing both exposure surface and ongoing cost versus a vector-search architecture.
• Removed a five-tool MCP layer once its multi-consumer premise didn’t materialize, cutting a network hop from the search path.
• Removed Semantic Kernel orchestration once the agent’s role narrowed to parameter emission.
• Hardened agent cold-start behavior (empty-response detection, thread reuse, login prewarm) and instrumented it with AI.Foundry.* telemetry.
• Implemented server-derived multi-tenant scoping and audit logging on an Okta/PKCE authentication path.
• Migrated the frontend off Azure Static Web Apps to App Service so dev and tst/uat could run inside the private network alongside AI Foundry, APIM, and the database.

Why this is the throughline

Read as a list, the decisions here look like a series of refusals. No chat. No MCP. No vector search. No serverless hosting. No generated facts about providers. For a product whose entire premise was AI, that’s a lot of AI left on the shelf.

But they weren’t separate decisions. They were the same one, made repeatedly, against the same constraint:

Every copy of this data, and every claim about it nobody can check, is a liability the product carries forever.

Take that seriously and the architecture mostly falls out of it. Vector search wanted a second copy of regulated records — declined, and the semantic layer moved to 674 rows of public vocabulary instead. MCP wanted a network hop justified by consumers who never arrived — removed. Static hosting wanted public egress where the environments needed a private path — migrated. The model wanted to answer from its own knowledge — constrained to emitting queries, then to a mandatory lookup, then to a synonym map a domain expert can edit without touching a prompt.

What’s left is a system where the AI does the one thing it’s genuinely best at — working out what a person meant by heart drs in fl — and nothing else. The records come from SQL. The vocabulary comes from a file. The summary restates the question rather than characterizing the answer. The intermediate steps stream to the screen so the user can watch the translation everything depends on.

None of that is a smaller ambition than the version with embeddings and a chat window. It’s the version that survives a health plan’s security review, that a compliance officer can trace a result through, and that a domain expert can fix without an engineer.

AI-native turned out to mean something less glamorous than the phrase promises. Not how much of the model you can put in the product — how little you need to, and how much checkable machinery you’re willing to build around the part that stays.