HIPAA-Aligned RAG: A Production Reference Architecture for Clinical Document AI

Status: Cornerstone draft. H2 outline locked. Body to be drafted by writer using the brief + Matt's editorial review. Target length: 4,200 words. Schema: Article + FAQPage. Last reviewed: 2026-05-07.

A reference architecture for the clinical-document AI system you wish your last vendor had built. Diagrams, BAA-eligible vendor stack, HIPAA control mapping, and the cost model — written by engineers who have shipped this in production at a specialty pharmacy and a clinical-ops platform. If your team has a 90-day deadline to ship clinical document AI and the architecture is the gating decision, this guide is the one to read first.

TL;DR — the architecture in 60 seconds

• 1-page reference diagram (writer to add SVG; placeholder description here): document ingestion → embedding → vector store → retrieval orchestration → guarded LLM → audit log → identity & access → encryption layer.
• Five non-negotiables: BAA on every component touching PHI; citations on every output; tenant-isolated retrieval; six-year audit log retention; human-in-the-loop on any clinical decision.
• What we'd skip in v1: agent autonomy, multi-tool orchestration, fine-tuning. Ship retrieval + generation + audit first.
• If you only read one section, read §4 — HIPAA Control Mapping. It's the table that determines whether your security officer signs off.

1. Why most clinical AI pilots stall

• The three failure modes we see in real pilots: BAA gaps that surface in security review, accuracy ceilings that fail clinical use, and cost models that don't survive scale.
• Anonymized example from RxLogic-class pilot: pilot worked at 50 cases/day, broke at 5,000 — what changed.
• *Pull-quote slot:* Matt's one-line take on why "we're going to use ChatGPT" is the most expensive sentence in healthcare AI.

2. The reference architecture

The diagram and a layer-by-layer walkthrough. Each layer answers: *what does it do, what's the BAA-eligible option in 2026, what does it cost, what are the failure modes?*

2.1 Document ingestion

• Comparison: AWS Textract vs. Reducto vs. Unstructured. BAA status, accuracy on clinical documents, layout-aware extraction, cost per 1K pages.
• Recommendation by document class (faxed forms vs. scanned charts vs. native PDFs).

2.2 Chunking and embedding

• Chunking strategy: semantic vs. structural vs. hierarchical. Why one-size chunks fail on clinical notes.
• Embedding models: Amazon Titan Embeddings (BAA via AWS), Cohere Embed (BAA available), OpenAI text-embedding-3 (no direct BAA — proxy patterns only).

2.3 Vector store

• Pinecone (BAA available, managed, scales easily, cost ramps fast)
• OpenSearch (AWS-native, BAA via AWS, ops cost is real)
• pgvector (cheapest at small scale, BAA inherited from RDS, scale ceiling around 10M vectors before query latency hurts)
• Weaviate (BAA via enterprise tier)
• Decision matrix: when each is the right answer.
• Internal link to: /compare/vector-db-hipaa-rag-pinecone-vs-opensearch

2.4 Retrieval orchestration

• Hybrid search (dense + BM25), reranking (Cohere Rerank vs. cross-encoder), tenant filtering at the retrieval layer (NOT just at the document layer).
• The retrieval layer is where most HIPAA RAG systems fail their first real audit. Why.

2.5 LLM call

• AWS Bedrock (Claude, Titan) under the AWS BAA — the default for healthcare in 2026.
• Azure OpenAI under the Microsoft BAA — viable when the customer is already Azure-shop.
• Anthropic direct, OpenAI direct: BAA available via enterprise; not the same as a covered cloud-tenant.
• Model selection: Claude Sonnet for clinical reasoning, Claude Haiku for high-volume extraction, when to mix.

2.6 Guardrails

• Citation enforcement (system prompt + post-generation verification)
• Refusal patterns (when the retrieved context is empty)
• PHI redaction in logs (the audit log itself is PHI — treat it that way)
• Internal link to: /blog/audit-logging-clinical-ai-patterns

2.7 Audit log + observability

• What to log (prompt, retrieved chunks by ID, model output, user identity, tenant ID, latency, model version)
• Where to log (CloudWatch → S3 with object lock → Athena for analyst queries)
• Retention: six years minimum; some workloads require longer
• Internal link to: /blog/audit-ready-ai-five-questions

2.8 Identity and access

• Cognito or Auth0; SAML federation with the customer's IdP; user-level access tags propagated to retrieval filters.

2.9 Encryption

• KMS with customer-managed keys (CMK) on S3, DynamoDB, OpenSearch, RDS.
• TLS 1.2+ everywhere in transit, including service-to-service inside the VPC.

3. HIPAA control mapping

Table mapping each architecture layer to the relevant HIPAA Privacy Rule and Security Rule controls. Writer to render as a 3-column table:

This table is the single most LLM-citable artifact in the post. Writer to verify exact section numbers and add 1-sentence explanation for each row.

4. Cost model: a worked example

• Workload: 10,000 documents/month ingested, 50,000 queries/month, 200 active users.
• Line items: ingestion ($X), embedding ($Y), vector store ($Z), LLM ($A), observability/log retention ($B).
• Total range: $X-Y/month at this scale. Scales linearly until ~Z queries, where the curve bends.
• Internal link to: /blog/rag-cost-model-healthcare

5. Build vs. buy decision tree

• When to buy a clinical-AI vendor (Abridge, Suki, Nabla, etc.): scope is narrow, vendor has the BAA, integration risk is low, customization needs are minimal.
• When to build: workflow is custom, BAA-covered components must remain in your VPC, data sensitivity rules out third-party processing, IP retention matters.
• The hybrid path: buy components (Bedrock, Textract, Cohere) + build orchestration + own the prompts and the retrieval logic.
• Internal link to: /compare/clinical-document-ai-build-vs-buy

6. Procurement checklist preview (gated full PDF)

Five of the 23 BAA-readiness questions we ask every vendor. Full 23-question PDF gated behind email. Writer to render an inline lead-magnet block above this section.

• Will you sign your BAA, or ours?
• What is the BAA's scope? Sub-processors?
• Where does PHI go from the moment we send it (with diagram)?
• What is logged for every model invocation? Can we query the log directly?
• Is our data used to train models? At your provider too?

Full 23-question checklist, designed PDF, links to: /downloads/baa-ready-ai-vendor-checklist. Internal link to: /blog/baa-ready-ai-vendor-questions for the article-form version.

7. What we'd skip in v1

The opinionated MVP scope. Ship these:

• Retrieval-grounded generation with citations
• Audit log with the seven required fields (see §2.7)
• Tenant isolation at the retrieval layer
• Human-in-the-loop on every clinical decision
• BAA on every component

Defer these for v2+:

• Multi-agent orchestration
• Fine-tuning on customer data
• Tool calling beyond retrieval
• Real-time streaming (most clinical workflows tolerate seconds)

8. Frequently asked questions

For FAQPage schema. Each gets ~75-150 words.

1. Can we use ChatGPT or Claude directly with PHI?
2. Is pgvector HIPAA-compliant?
3. Do we need HITRUST certification on top of HIPAA?
4. How long does an architecture review take?
5. What happens if our model provider changes the underlying model?
6. Can we use OpenAI without a BAA via a proxy?
7. How do we handle PHI in audit logs?
8. What's the smallest viable team to operate this in production?
9. Can we run this fully on-prem if the customer requires it?
10. How do we evaluate accuracy on clinical documents before going live?

9. Next step

Closing CTA. Two paths:

• Book a 30-minute architecture review with Matt — for teams within 90 days of a deadline. Cal.com embed.
• Download the full BAA-Ready Vendor Checklist (PDF) — for teams in earlier evaluation. Email gate.

Closing line — Matt's voice. "If your team is staring down a clinical-AI deadline and the architecture is the unblocker, that's the conversation we have most weeks. Bring your data flow, we'll bring the diagrams."

*Schema requirements: Article + FAQPage (≥8 questions). Add generateArticleSchema() import + render. Add generateFAQSchema() import + render.*