LLM Data Leakage: How AI Applications Accidentally Expose Sensitive Information

Most LLM data leakage isn't caused by a model being hacked. It's caused by ordinary application design decisions that hand the model, or its output, more than the current user should see.

Phone photographing a screen showing an AI-generated face
Photo by Aidin Geranrekab on Unsplash

Sensitive information disclosure is its own distinct category in OWASP’s Top 10 for LLM Applications, separate from prompt injection, and for good reason: most real-world LLM data leakage isn’t caused by an attacker cleverly extracting something through manipulation. It’s caused by ordinary application design decisions that hand the model, or the model’s output, more information than the current user is actually supposed to see.

Overly broad context provided to the model

A common pattern in retrieval-augmented (RAG) applications: the retrieval step pulls in documents or records relevant to a query without filtering by the requesting user’s actual permissions, and the model then has access to — and can potentially reference in its response — data the user querying it isn’t authorized to see. This isn’t a flaw in the model; it’s a flaw in the retrieval pipeline not enforcing the same access control it would enforce in a traditional application. If your retrieval layer doesn’t filter by the requesting user’s permissions before content reaches the model’s context, you’ve effectively removed access control for anything in that retrieval index.

Sensitive data in prompts, logs, and telemetry

Prompts sent to a model — including any sensitive data embedded in them, whether that’s a user’s personal information, an internal document, or proprietary business data — often get logged for debugging, monitoring, or fine-tuning purposes, sometimes without the same data handling scrutiny applied to other sensitive data flows in the application. A logging pipeline that captures full prompts and responses without redaction, or a fine-tuning dataset assembled from real user interactions without appropriate anonymization, creates a data exposure risk that has nothing to do with the model’s behavior and everything to do with ordinary data handling practices around it.

Cross-session or cross-user context bleed

In applications maintaining some form of session or memory across interactions, a bug in session isolation can cause one user’s context — previous messages, retrieved documents, personal details mentioned earlier — to bleed into another user’s session. This is an application-architecture bug rather than something specific to LLMs, but it’s a particularly consequential one in this context, because the leaked content often includes exactly the kind of personal or sensitive information users share expecting it to stay private to their own conversation.

Models trained or fine-tuned on data that shouldn’t be memorized

A model fine-tuned on internal data can, in some cases, memorize and later reproduce specific pieces of that training data verbatim in response to certain prompts — a well-documented phenomenon for models fine-tuned on datasets containing sensitive or personally identifiable information without adequate safeguards. This is a risk specifically for organizations fine-tuning their own models on internal data, distinct from the risks of using a third-party model’s inference API, and it means the fine-tuning dataset itself needs the same data governance scrutiny as any other repository of sensitive information.

System prompt leakage as its own disclosure risk

System prompts often contain more than generic instructions — internal business logic, information about other integrated systems, sometimes credentials or API details that were convenient to include during development. If a model can be induced to reveal its system prompt (through direct questioning or a more indirect extraction technique), whatever sensitive detail was included in it becomes exposed. The practical mitigation isn’t only about preventing extraction — it’s about not putting anything genuinely sensitive in a system prompt in the first place, since extraction techniques evolve and a system prompt should be treated as something that could eventually be read by a determined enough user.

Output that inadvertently references training-adjacent knowledge

A model’s response can sometimes reflect patterns learned from its training data in ways that inadvertently suggest information about specific individuals or organizations represented disproportionately in that data, particularly for models trained on data scraped broadly without careful filtering. This is a subtler and less directly controllable form of leakage compared to the application-level issues above, and mitigating it falls more on model selection and output review than on application architecture.

What actually reduces this risk

Enforce user-level access control at the retrieval layer, not just at the application’s front door — a RAG pipeline needs to filter what it retrieves based on the requesting user’s actual permissions, every time, not rely on the model to somehow not mention things it technically has access to.

Redact or avoid logging sensitive content in prompts and responses, treating LLM interaction logs with the same data handling rigor as any other system touching sensitive data, rather than as a lower-scrutiny debugging convenience.

Isolate session state rigorously, testing specifically for cross-session leakage the way you’d test for any other multi-tenancy isolation bug.

Keep genuinely sensitive information out of system prompts entirely, treating them as documentation that could eventually be exposed rather than as a safe, private configuration channel.

Apply data governance to fine-tuning datasets with the same rigor as any other repository containing potentially sensitive records, including anonymization and access review before that data is used for training.

Most of this list has nothing specifically to do with LLMs — it’s standard data handling discipline that happens to matter more, not less, once an LLM is generating natural-language output that can surface whatever it was given access to, in response to phrasing nobody explicitly anticipated.

AI Security