An AI application sends instructions to the model before the user types a word.
The application may define the model’s role, available tools, output format, trusted sources, and approval requirements. Developers commonly call these standing instructions the system prompt. In a production agent, the prompt functions as an operating manual.
Collections such as Asgeir Thor Hallgrimsson’s public System Prompts Leaks repository make that hidden layer visible. The repository contains claimed captures from major chatbots, coding agents, search products, and productivity tools. Some entries are short. Others include extensive behavioral policies, tool descriptions, output rules, and reminders inserted while an agent works.
The collection is useful, but the word leak can blur several different events. A model might be persuaded to recite instructions from its context. A desktop client might contain prompt text in its installed files. A developer might accidentally log the complete request sent to an application programming interface (API). A contributor might reconstruct a prompt from observed behavior rather than capture it verbatim. Those artifacts do not carry equal evidentiary weight, and a public repository cannot prove the provenance or continued accuracy of every file.
The collection demonstrates a durable engineering fact: instructions supplied to a model or distributed to a client are difficult to keep confidential. A secure AI system should remain safe when its prompt becomes public.
A System Prompt Is a Control Layer, Not a Security Kernel
A system prompt resembles an employee’s standing job description: it defines the role, normal procedure, and boundaries. The user prompt supplies the current task. A customer-support assistant might receive standing instructions to answer from an approved knowledge base, never change an account, and escalate billing disputes. A user then supplies a specific account question.
Modern systems often have more than two layers. OpenAI describes an instruction hierarchy in which system instructions outrank developer instructions, developer instructions outrank user instructions, and models assign no authority to tool output. The exact message names vary across providers and products, but the architectural goal is consistent: separate durable application policy from a user’s task and from data retrieved during that task.
A simplified request might look like this:
[
{
"role": "developer",
"content": "You summarize support tickets. Use only facts in the ticket. Never change account data."
},
{
"role": "user",
"content": "Summarize this ticket and identify the affected service."
}
]Instruction-hierarchy training improves reliability by teaching the model that developer rules carry more authority than a conflicting user request. The hierarchy does not create a hardware-enforced privilege ring. The model still processes instructions and data as tokens inside a shared context. It must infer which text is authoritative and generate a likely continuation that respects the hierarchy.
System prompts reliably steer scope, tone, workflows, output schemas, and tool use. They cannot safely hold credentials or enforce permissions that must never fail.
A traditional authorization service can evaluate a signed identity, check a policy, and deterministically deny an action. A prompt saying, “Never disclose another customer’s records,” asks a probabilistic model to behave. The backend still needs to ensure that the assistant cannot retrieve those records in the first place. Natural-language policy can guide the agent; application code must enforce the boundary.
The same division applies to tool use. A system prompt can tell an agent to request approval before deleting a cloud resource. The deletion API should independently verify identity, authorization, resource scope, and valid approval. If the tool trusts the model’s claim that approval occurred, the application has delegated a security decision to a probabilistic model.
How Hidden Instructions Reach the Public
The most familiar leak path is direct prompt extraction. A user asks the model to repeat the text that appeared before the conversation, translate its hidden rules, serialize them as JSON, or simulate a diagnostic log containing its configuration. Models often recognize and refuse simple versions of these attacks. Attackers therefore vary the framing, split the request across turns, ask for transformations instead of a verbatim copy, or make the requested output resemble an ordinary task.
The system prompt forms part of the model’s active input. The model must process the prompt to follow it, so the prompt can influence generated tokens. Training and output controls can make extraction much harder. They cannot hide the prompt from the model while asking the model to obey it.
Extraction does not need to reproduce every character to be useful. An attacker may probe for the presence of a rule, discover a forbidden phrase, infer the names of available tools, or learn when an agent requires confirmation. This technique amounts to behavioral fingerprinting. Repeated tests can reveal pieces of the control policy even when the model never prints the complete source.
The second path is indirect prompt injection. An agent reads content from a webpage, email, document, issue tracker, source repository, or retrieval system. Malicious text in that content tells the agent to ignore its task, reveal private context, or invoke a tool. The user did not issue the malicious instruction; the instruction arrived inside data the agent was asked to process.
OWASP classifies both direct and indirect prompt injection as core risks for applications built on large language models. Indirect injection is particularly important for agents because the model routinely mixes instructions with external content. A browser agent sees page text. A coding agent sees repository files and issue descriptions. An email assistant sees message bodies and attachments. Each data source can carry language that looks like a command.
Suppose a purchasing agent receives this legitimate task:
Compare three vendor proposals and prepare a recommendation. Do not send anything.
One proposal contains tiny white text instructing the agent to upload the other proposals to an external URL for “validation.” A secure design labels the proposal as untrusted data, limits network destinations, denies uploads by default, and requires approval for outbound disclosure. An insecure design gives the model a general-purpose browser and relies on the sentence “Do not reveal confidential data.” The surrounding architecture determines whether the injected instruction can cause harm.
The third path bypasses the model: client-side inspection. Desktop applications, browser extensions, integrated development environments, and command-line agents may assemble instructions locally. Prompt templates can appear in JavaScript bundles, packaged application archives, cached resources, configuration files, or process memory. A user who controls the machine can inspect locally installed software. Minification may increase the effort, but it does not preserve confidentiality.
Network inspection can expose the same material when a client sends a composed request to a remote API. The application’s transport design, certificate validation, platform controls, and request-assembly location determine whether someone can inspect the traffic. The general rule is simpler: assume that a user can discover data sent to an endpoint that user controls.
The fourth path is operational exposure. Development proxies, observability platforms, tracing systems, exception reports, support bundles, and debug logs may capture entire model requests. Those requests can include system instructions, retrieved documents, user messages, tool arguments, and model responses. A telemetry console with overly broad access may expose a prompt that resisted every conversational extraction attempt.
Operational exposure is often more consequential than public knowledge of formatting rules because traces may contain live customer data. Teams should design logging around data classification and minimization. They need enough telemetry to diagnose behavior without creating a shadow archive of every sensitive context the agent has processed.
Finally, some purported leaks are reconstructions, stale captures, or composites. Providers change prompts without changing the product name. An application can inject different instructions by model, account tier, enabled tool, region, safety state, or experiment cohort. A file captured from one session may be authentic for that session and wrong for the next. Public prompt collections provide point-in-time evidence, not canonical vendor documentation.
What Leaked Prompts Reveal—and What They Do Not
Leaked prompts provide production engineering case studies. They show how teams translate product requirements into instructions the model can consume.
Common patterns include explicit role definitions, tool-use procedures, formatting requirements, citation limits, safety rules, fallback behavior, and reminders about untrusted content. Coding agents add filesystem rules, command-execution constraints, patching procedures, and requirements to inspect a repository before claiming facts about it. Research agents add source-quality rules and instructions for distinguishing retrieved evidence from their own conclusions.
Large prompts also reveal how much application behavior lives outside model weights. Two products using a similar base model can behave differently because their prompts expose different tools, apply different policies, insert different context, and validate different outputs. The model supplies general capability. The harness turns that capability into a product.
The public files do not reveal the whole harness. A prompt capture usually omits server-side authorization, content classifiers, model routing, retrieval filters, rate limits, application state, tool implementation, post-processing, abuse detection, and human review. Reading a prompt can explain why the model prefers a format or refuses a category. The prompt alone cannot establish the complete security architecture or reproduce the product.
The files also do not prove that every instruction works. A 4,000-line prompt can contain contradictions, historical patches, unreachable procedures, or rules the model follows inconsistently. Length often reflects accumulated product requirements: new tools, edge cases, legal constraints, user-experience preferences, and fixes for failures observed in production. It is evidence of operational complexity, not evidence that natural language has become deterministic code.
Researchers should handle prompt collections like any other unverified corpus. Record the capture date and claimed product version. Compare the text with observable behavior. Separate verbatim artifacts from interpretations. Avoid presenting a file as current after the vendor changes the product. Most importantly, do not copy a competitor’s prompt and assume that the copied text will transfer its capabilities. The original depends on model behavior, tool contracts, application state, and backend enforcement that the text alone cannot supply.
Public analysis can also support accountability. System instructions influence how widely used products frame answers, cite sources, handle memory, and resolve sensitive requests. Examining those instructions can support informed discussion about product behavior. Responsible analysis should distinguish public-interest examination from attempts to obtain user data, credentials, or unauthorized access. Discovering a behavioral rule does not authorize an attack on the systems behind it.
When a Prompt Leak Becomes a Security Incident
OWASP’s System Prompt Leakage guidance makes an essential distinction: disclosure of a prompt is not automatically a vulnerability. The security impact depends on what the prompt contains and what the surrounding system permits.
If a leak reveals that an assistant writes in short paragraphs and uses a particular escalation sentence, the business impact may be embarrassment or easier imitation. If it reveals an API key, an internal credential, a private customer record, or a password embedded as a control word, the system has exposed sensitive information. The first case is intellectual-property or product-design risk. The second is a security incident.
Tool-enabled agents add a third category. A leaked prompt may reveal tool names, internal workflow stages, approval conditions, or assumptions that help an attacker craft a later injection. Knowledge of the prompt is reconnaissance. The actual compromise occurs when the application lets untrusted text influence a privileged action, retrieve unauthorized data, or bypass a deterministic control.
Teams can assess impact with four questions:
- What became known? Separate style rules and workflow text from credentials, personal data, confidential business logic, and internal infrastructure details.
- What can an attacker do with it? Look for unauthorized data access, privilege escalation, tool abuse, evasion of monitoring, or materially improved social engineering.
- Which independent control still stands? Verify server-side authorization, scoped credentials, network restrictions, output validation, and approval enforcement.
- How trustworthy is the artifact? Establish provenance, capture date, version, completeness, and whether the content is still deployed.
This analysis prevents two common mistakes. Some teams dismiss every prompt leak because “prompts are not secrets,” even though real prompts sometimes contain secrets or operational details that support a larger attack. Others treat every leaked paragraph as proof of a catastrophic breach. Disclosure of a style guide creates a different risk from disclosure of credentials or personal data.
The practical severity is a function of sensitive content and granted agency:
Prompt leak risk = disclosed information value × reachable capability × weakness of independent controls
This expression provides a review framework, not a numerical scoring standard. A banking assistant with read-only access to one authenticated customer’s transactions presents a different risk from an agent holding broad administrative credentials and unrestricted outbound network access.
Design the Application to Survive Disclosure
Begin with the assumption that an unauthorized person may eventually read the system prompt. This premise clarifies the required architecture.
Keep secrets out of model context. API keys, database credentials, signing material, private encryption keys, reusable authorization tokens, and secret bypass phrases belong in a secrets manager or trusted backend. The model should request a narrowly defined operation through a tool. The tool resolves its own credential outside the prompt and returns only the minimum result needed.
Bad:
"Use API key sk-live-... to query the billing service."
Better:
The model may call get_invoice(customer_id).
The backend authenticates the user, verifies access to customer_id,
obtains its credential from a secrets manager, and returns a filtered result.Enforce authorization in code. Treat every tool call as a request from an untrusted planner. Validate the authenticated user, allowed action, target resource, input schema, and current approval state. Use least-privilege identities and short-lived credentials. Never let a model expand its own scope by changing text in an argument.
Constrain agency. Give the agent only the tools required for its current task. Separate read and write capabilities. Restrict network egress and destination domains. Put high-impact operations behind explicit user confirmation, and make the backend verify that confirmation rather than asking the model to remember it. A compromised summarizer should not inherit the ability to send email, modify identity permissions, or execute arbitrary shell commands.
Mark external content as untrusted. Strong delimitation helps the model distinguish a developer instruction from document contents, but tags only steer the model; they do not create a sandbox. Preserve provenance as data moves through retrieval and tools. Application policy can then allow webpage text to support a summary while preventing it from authorizing a network request.
Reduce data movement. Do not place an entire customer record, mailbox, or repository into context when a filtered subset will answer the question. Data minimization reduces the value of a successful extraction and limits what an indirect injection can reach. The same principle applies to tool responses: return selected fields, not a raw backend object containing internal metadata.
Validate outputs before use. A JSON schema can ensure structural correctness, but valid JSON can still request a dangerous action. Apply semantic policy checks, destination allowlists, data-loss-prevention rules, and business validation after generation. Treat model output as untrusted input to the next component.
Build privacy-aware observability. Log event types, policy decisions, model and prompt versions, tool-call metadata, latency, and redacted errors. Avoid logging full prompts and retrieved content by default. Where detailed traces are necessary for controlled debugging, limit retention and access, record who viewed them, and redact credentials and regulated data before storage.
Test the deployed system adversarially. Include direct extraction attempts, transformed-output requests, multi-turn probes, malicious documents, poisoned retrieval records, and attempts to make tools cross authorization boundaries. Test the full application, not only the base model. OpenAI’s instruction-hierarchy research shows that model training can improve resistance. Application controls remain necessary because no evaluation covers every attack or integration failure.
Version and minimize the prompt. A lean prompt is easier to review, test, and cache. Keep stable identity, invariants, and tool policy in the durable prefix. Load task-specific procedures only when needed. Record prompt versions with evaluations so a security or quality regression can be tied to an actual change. Avoid burying contradictory rules under years of patches.
Teams should also prepare an incident playbook. If a prompt appears publicly, confirm the artifact, identify the affected version, scan it for sensitive material, rotate any exposed credentials, review tool and telemetry access, and test whether the disclosed rules enable practical abuse. Update the prompt where useful, but fix the backend control whenever the prompt was carrying responsibility that belonged in code.
Treat Prompts as Publishable and Systems as Defensible
System prompts now form a distinct layer of software architecture. They define roles, coordinate tools, standardize output, and encode product decisions. Studying leaked prompts shows how teams assemble AI products and how quickly natural-language instructions expand as requirements accumulate.
The collections demonstrate that hidden instructions do not remain reliably secret. Models sometimes reveal context. Users can inspect software installed on their devices. Logs and traces can capture complete requests. Public artifacts can also be stale or incomplete, so responsible analysis requires provenance and restraint.
An organization should write every system prompt as though it may eventually appear on GitHub. Remove secrets. Minimize sensitive context. Keep authorization and approval outside the model. Restrict tools according to least privilege. Treat external content and model output as untrusted. Test the complete agent against both direct and indirect injection.
With those controls in place, prompt disclosure may reveal product design without granting access to protected data or actions. AI systems should meet that standard.