LLM Data Leakage: How Sensitive Data Escapes AI Apps

LLM data leakage is sensitive information escaping an organization through an application built on a large language model: personal data in prompts sent to a third-party provider, secrets copied into logs, confidential documents surfaced to the wrong user, or private data an agent is tricked into sending out. OWASP ranks it second in its Top 10 for LLM applications as LLM02: Sensitive Information Disclosure. Most leaks need no sophisticated attacker. They follow from ordinary design choices about what the model sees, what it can reach and what gets recorded.

This guide maps the seven paths data takes out of an LLM application and the control that closes each.

Why LLM apps leak more than ordinary apps

A conventional application moves data through fields with known types. An LLM application moves data as free text, and free text carries everything: a customer's address in a support ticket, an API key in a log excerpt, a medical history in a pasted email. Three properties make this worse:

• The prompt is a copy. Every value in the context is sent to the model provider and usually stored there for some period, plus in your own logs and traces.
• The model mixes sources. Retrieved documents, tool results and user input share one context, so data from one source can surface in an answer meant for another.
• The output is free text. A model can restate, summarize or encode anything it was shown, and an agent can pass it to a tool.

Path 1: Users paste sensitive data into prompts

The simplest leak is a person putting confidential material into a chatbot. In 2023 Samsung restricted employee use of generative AI tools after staff entered internal source code into ChatGPT. The pattern repeats in every organization that has not given people a sanctioned alternative, which is the problem described as shadow AI.

Control: detect personal data and secrets in prompts before they are sent, and either refuse with an explanation or replace the values with placeholders. Pair this with an approved tool, because blocking without an alternative pushes usage elsewhere.

Path 2: Your application sends more context than it needs

Developers often pass whole records to the model "for context": a full customer profile when the task needs one field, a complete document when a paragraph would do. Every extra field is an extra exposure to the provider, the logs and any downstream leak.

Control: data minimization at the point of prompt construction. Send what the task needs, and replace identifiers the model does not have to read with tokens it can still refer to. PII redaction for LLMs compares tokenization, hashing, masking and redaction for this.

Path 3: The model reveals retrieved data to the wrong person

Retrieval-augmented generation (RAG) pulls documents into the context based on similarity to the question. If the index holds documents with different access levels and retrieval ignores who is asking, the model can quote a document the user was never allowed to open. Nothing malicious is required; the system did what it was built to do.

Control: enforce document-level permissions at retrieval time, filtering results by the requesting user's access before they reach the model. Do not rely on the model to withhold what is in its context.

Path 4: Prompt injection makes the model send data out

An attacker plants instructions in content the application reads, and the model follows them: summarizing private data into a link, embedding it in an image URL, or passing it to a tool that sends email or calls a webhook. Rendering a Markdown image whose URL contains the data is a recurring exfiltration channel, documented in many assistant vulnerability reports by researchers such as Johann Rehberger. Microsoft's EchoLeak (CVE-2025-32711) in 365 Copilot was a critical-rated example of injection-driven disclosure.

Control: scan untrusted content for injected instructions, validate tool calls and their arguments before they run, restrict outbound destinations, and do not auto-render remote images or links in model output. What is prompt injection covers the attack in full.

Path 5: The system prompt leaks

System prompts often contain more than instructions: internal URLs, business rules, names of tools and sometimes credentials. Users routinely coax models into repeating their system prompts. OWASP lists this separately as LLM07: System Prompt Leakage, and its guidance is blunt: assume the system prompt will be disclosed.

Control: keep secrets, credentials and access decisions out of the system prompt entirely. Enforce authorization in code, not in instructions.

Path 6: Logs, traces and analytics keep a second copy

Observability tools record prompts and completions for debugging and evaluation. That is useful and it means every sensitive value now also lives in a logging platform, often with broader access and longer retention than the source system. Conversation history and caching layers create the same exposure. OpenAI's March 2023 incident, in which a caching bug showed some users other users' chat titles and some payment details, shows how infrastructure around the model can leak independently of the model.

Control: redact or tokenize before logging, set short retention for raw prompts, restrict who can read traces, and record the request as it was sent to the model, with tokens, rather than the original.

Path 7: The model leaks its training data

Models memorize parts of their training data. Carlini et al. extracted verbatim training examples, including names and contact details, from GPT-2, and Nasr et al. later showed scalable extraction from production models, including ChatGPT. For most teams using hosted models this is the provider's problem. It becomes yours when you fine-tune on private data, because a small, repeated dataset is exactly what a model memorizes best. Training data extraction attacks goes deeper.

Control: remove or tokenize personal data and secrets before fine-tuning, deduplicate training data, and test fine-tuned models for regurgitation before deployment.

Leakage paths and controls at a glance

Which data matters most

Not all sensitive data carries the same cost when it leaks. Prioritize by consequence:

• Credentials and secrets: API keys, tokens, passwords and private keys. A leaked key is immediately usable. These should be redacted outright, never restored into model-visible text. See API keys and secrets in LLM prompts.
• Regulated personal data: health information, payment card data and government identifiers carry specific legal duties under laws such as HIPAA, PCI DSS and the GDPR.
• Ordinary personal data: names, emails, phone numbers and addresses. Lower individual impact, highest volume.
• Confidential business data: source code, contracts, financials and strategy.

Detection belongs outside the model

A recurring mistake is asking the model itself to avoid revealing sensitive data. The model is the component that is being manipulated or that does not know who is asking, so it cannot be the control. Detection of personal data and secrets, permission checks on retrieval, and validation of outbound tool calls all belong in code around the model, where they hold regardless of what the model decides.

What to do next

1. List every place your application sends text: the model provider, logs, traces, analytics, caches and tools.
2. Tokenize personal data and redact secrets before any of them receive it.
3. Enforce permissions at retrieval and in code, not in the prompt.
4. Validate tool calls and restrict outbound destinations so injected instructions cannot move data out.
5. Review the OWASP Top 10 for LLM applications for the related risks.

FrostWolf's Cloak, part of Guard, finds personal data and secrets in a request, replaces them before the model sees them and restores them in the response for the identity that owns them, while telemetry carries counts only. Whatever tool you use, the principle is the same: the safest copy of sensitive data is the one the model never received.

Frequently asked questions

What is LLM data leakage?

It is sensitive information, such as personal data, credentials, source code or confidential documents, leaving an organization through an application built on a language model. It can happen through what users send, what the model outputs, what is logged, or what an agent sends to tools.

Do model providers train on my prompts?

It depends on the provider, the product tier and your settings. Most providers state that business and API traffic is not used for training by default, while some consumer products use conversations unless you opt out. Read the current data-use terms for the exact product and plan you use, and assume prompts are retained for some period either way.

Can an LLM leak data from its training set?

Yes. Research has extracted verbatim training data, including personal information, from deployed models. The risk is highest for data that appeared many times in training and for models fine-tuned on small private datasets.

Is redacting personal data before the prompt enough?

It removes the largest leak, but not all of them. You still need to control what the model can retrieve, what agents can send out, what is written to logs and traces, and what the system prompt contains.

What is the difference between data leakage and prompt injection?

Data leakage is an outcome: sensitive data ends up somewhere it should not. Prompt injection is one of the attacks that causes it, by instructing the model to reveal or send data. Leakage also happens with no attacker at all, through over-broad logging or users pasting confidential content.