LLMs can't unlearn data, treat all input as one token stream, and leak through integrations. Learn where AI privacy breaks down and how to enforce it at the infrastructure layer.
Delete a row from a database and the data is gone. Delete a user's data from an LLM and you have a problem: there is no equivalent to deleting a row in your database's user table. The model has no "unlearn" mechanism. Whatever it absorbed during training is woven into billions of parameters, and no one can point to the specific weights that encode a particular person's information.
This is the structural reality that makes AI privacy fundamentally different from traditional data privacy. Relational databases enforce schemas, access controls, and deletion. LLMs do none of these things natively. And the problem runs deeper than the model itself. In many cases, the weakest link isn't the model, but the ecosystem around it: the plugins, the APIs, the RAG pipelines, the logging infrastructure. Privacy controls embedded in application code can be bypassed by a well-crafted prompt. The only enforcement point that scales is the infrastructure layer that sits between every application and every model.
Relational databases have schemas. Row-level access controls. Column-level encryption. Decades of tooling built around the assumption that data has structure and boundaries.
LLMs have none of this. Unlike relational databases, which enforce strict schemas and access controls, LLMs treat all data — system instructions, user input, external context, and internal knowledge — as a continuous stream of tokens. There is no distinction between "this is a system prompt the user should never see" and "this is the user's question." It is all tokens.
This creates three problems at once:
These are not bugs to patch. They are properties of the architecture. Any AI privacy strategy that ignores them is building on sand.
Privacy failures in LLM systems cluster around three distinct layers. Each requires different controls.
The data that goes into training a model defines what it can leak out. Security researchers found that a training dataset used for LLM development contained nearly 12,000 live API keys and passwords. Not hashed credentials. Live keys. Sitting in the training data because no one filtered them out before ingestion.
The problem extends beyond credentials. Generative AI tools trained with data scraped from the internet may memorize personal information about people, as well as relational data about their family and friends. That relational data enables spear-phishing at scale, because the model knows not just a person's name but their connections.
And then there is consent. Data such as a resume or photograph shared for one purpose is being repurposed for training AI systems, often without knowledge or consent. A photo posted to a social profile in 2019 may now be embedded in the weights of a model that generates faces.
Prompt injection is the simplest and most effective inference-time attack. A carefully crafted instruction like "ignore all previous rules" can override guardrails, or extract hidden system prompts. No infrastructure breach required. No network penetration. Just text.
RAG pipelines add another vector. Techniques like Retrieval-Augmented Generation and few-shot prompting can inadvertently expose sensitive information, even when instructed not to. The retrieval step pulls documents based on semantic similarity, not access permissions. If a junior employee's query retrieves an executive's confidential memo because the embeddings are close enough, the model will summarize it without hesitation.
Even when providers contractually commit to not storing customer data, leveraging third-party APIs risks exposing organizations' sensitive data during transmission. The data leaves your environment. That is a privacy event, regardless of what happens on the other side.
This is why application-level privacy controls are insufficient on their own. An application can redact PII before sending it to the model. But if three other applications in the same organization use the same model endpoint without redaction, the model has already seen the data. Privacy enforcement has to sit at the layer where all traffic converges.
Two incidents from early 2025 illustrate how these attack surfaces play out in production.
A hacker breached OmniGPT's infrastructure and leaked 30,000 user email addresses, phone numbers, and 34 million lines of chat messages. The data included API keys, credentials, and file links exchanged between users and AI models. The entire dump was reportedly sold on the dark web for $100. Thirty-four million messages. One hundred dollars.
An improperly secured database running on ClickHouse was left accessible on the public internet. Researchers discovered over a million lines of log stream data, including chat histories, API keys, and backend system metadata. The vulnerability was not a sophisticated exploit. It was an unsecured database.
Both breaches share the same pattern: the model itself was not compromised. The infrastructure around it was. Chat logs stored without encryption. Databases exposed without authentication. Credentials embedded in conversation histories. These are conventional security failures amplified by the volume and sensitivity of LLM interaction data.
More than 100 countries now have some form of privacy regulation imposing data handling requirements on companies that use AI. GDPR, HIPAA, CCPA, and their national counterparts all assume that organizations can control, audit, and delete personal data on request.
LLMs violate that assumption by design.
The enforcement signals are already significant. Meta was fined $1.3 billion by the EU for non-compliant transfers of sensitive data. That fine predates the current wave of AI-specific enforcement. As regulators turn their attention to LLM deployments, the liability surface grows.
Failing to anonymize data, improper data handling practices, or insufficient documentation of data usage can all result in compliance violations, fines, or reputational damage under GDPR, HIPAA, or CCPA. For a deeper look at the regulatory landscape, see AI compliance: what developers need to know in 2026.
The practical problem is straightforward: regulations require deletion, and LLMs cannot delete. Regulations require access controls, and LLMs treat all tokens equally. Regulations require audit trails, and LLMs are black boxes. The gap between what the law demands and what the technology does natively is where liability accumulates.
Fine-tuning deserves its own section because the risk profile is distinct from general training. When you fine-tune a foundation model on a small, domain-specific dataset, overfitting becomes the privacy threat.
PII and data compliance risks often increase when engineers fine-tune the model, which involves repeatedly exposing AI systems to smaller, highly sensitive proprietary datasets, such as internal customer service logs. The repetition drives the model toward memorizing specific records rather than learning generalizable patterns.
The result: a fine-tuned model may inadvertently memorize specific details and regurgitate them when prompted with similar language. A customer service model fine-tuned on support tickets might reproduce a customer's name, account number, and complaint verbatim if prompted with the right phrasing.
Safe fine-tuning requires treating the fine-tuning dataset like a regulated asset. That means differential privacy at the data-prep layer to introduce noise into sensitive attributes, confidential compute environments so raw data never leaves the enclave, and automated red-team tests after training to verify that no record-level reproduction occurs. These are not optional safeguards. They are the minimum for any fine-tuning pipeline that touches personal data.
Application-level privacy controls are necessary but not sufficient. They protect one application. Enterprise LLM deployments involve dozens of applications, each with its own integration patterns and its own developers making their own decisions about what data to send to the model.
The enforcement point that actually scales is the infrastructure layer: the gateway that sits between every application and every model endpoint. This is where forward-leaning teams are building dynamic privacy layers:
A filter that strips secrets and identifiers before the model processes them. Not regex on a known list of patterns, but contextual PII redaction that catches variations. This happens at the gateway, so it applies to every request regardless of which application sent it.
Input filtering is only half the problem. LLMs can surface confidential details from prior interactions or internal knowledge bases. These "soft leaks" are easy to miss: a model paraphrasing a client name, summarizing internal reports, or reproducing phrasing from a private dataset. Detecting these requires behavioral anomaly detection on the output side, flagging responses that echo confidential material.
End-to-end encryption of prompt logs ensures conversations can be audited without being fully exposed. The audit trail exists for compliance. The encryption ensures that the audit trail itself does not become a breach vector (as it did in the OmniGPT and DeepSeek cases).
The IEEE recommends a "data privacy vault" architecture for securely storing and managing PII before it reaches models. PII is tokenized and stored in a vault; the model only ever sees opaque tokens. De-tokenization happens on the way out, under access control, so the model never handles raw personal data.
This layered approach, enforced at the gateway rather than in each application, is what Shim's security architecture is built around for its supported native OpenAI, Anthropic, and Google routes.
| Control | Purpose | Layer |
|---|---|---|
| PII redaction on all prompts | Prevent personal data from reaching models | Gateway / input |
| Data minimization | Send only the context the model needs | Application + gateway |
| Per-integration access controls | Limit what each plugin/API can access | Gateway |
| Output monitoring | Catch soft leaks, paraphrased confidential data | Gateway / output |
| Encrypted prompt logging | Audit trail that is not itself a breach vector | Infrastructure |
| Differential privacy in fine-tuning | Prevent memorization of individual records | Training pipeline |
| Red-team testing post-training | Verify no record-level reproduction occurs | Training pipeline |
| Documentation of data flows | Required by GDPR, HIPAA, CCPA for compliance | Organizational |
| Consent and purpose tracking | Ensure data is used only for its stated purpose | Organizational |
Each of these maps to a specific regulatory requirement under GDPR, HIPAA, or CCPA. Missing any one of them creates a gap that can result in compliance violations, fines, or reputational damage.
The common thread across every item: privacy controls need to operate at the infrastructure layer, not just the application layer. A single application implementing perfect PII redaction does nothing if the next application in the stack sends raw data to the same model endpoint. The gateway is the enforcement point. Everything else is a suggestion.
For more on how AI guardrails fit into this picture, or how shadow AI usage bypasses even well-designed controls, those pieces cover adjacent ground.