A system prompt is where developers write the instructions they want a model to follow: its role, its tone, and often the rules it should not break, "never reveal internal data," "never approve a refund over this amount." It is tempting to treat these instructions as a security boundary, the same way an authentication check or an access control list is a boundary. They are not the same thing, and the difference matters more than it might first appear.
A boundary is enforced independently of the thing it constrains
An access control check works because it sits outside the component it constrains: even if a user's request tries to claim permissions it does not have, the check evaluates that claim against an independent source of truth and denies the request if it fails. A system prompt instruction does not work this way. It is part of the same context the model uses to generate its response, and a sufficiently persuasive piece of content elsewhere in that context can compete with, and sometimes override, what the system prompt asked for.
This is the core distinction: a real boundary does not depend on the thing being constrained agreeing to respect it. A system prompt instruction does, because ultimately the model is the one interpreting and following it, and the model can be influenced by other text in ways the developer did not intend.
Why it looks like it works in testing
System prompt instructions often do work in casual testing, which is exactly what makes teams comfortable relying on them. A tester tries a few obvious manipulation attempts, the model correctly resists them, and the instruction gets treated as a solved problem. This creates a false sense of security, because the space of possible phrasings an attacker might try is much larger than the handful a casual test covers, and new phrasing techniques get published and discovered continuously. An instruction that resists the ten attempts your team thought of may not resist the eleventh attempt someone else thinks of after your system has shipped.
Why it fails silently
A traditional access control bug tends to fail loudly, in the sense that a security review or a penetration test can usually find it by directly testing the boundary. A system prompt instruction failing does not produce an error or an exception; it produces a response that looks like an ordinary, plausible answer, just one that violates the intended rule. This makes the failure mode harder to detect both in testing and in production, because nothing about the failure looks broken from the outside.
What belongs in the system prompt instead
The system prompt is still useful for setting tone, role, and general behavioral guidance where the consequence of an occasional failure is low, such as how formally the assistant should write or which topics it should generally decline to discuss for a text-only feature. It is not the right place to encode rules whose violation has a meaningful consequence: transaction limits, data access restrictions, or anything a compliance or legal requirement depends on.
Rules with real consequences belong in code that runs independently of the model: an authorization check before a tool executes, a hard limit enforced outside the model's control, a policy engine that evaluates a request against defined conditions regardless of what the model's own reasoning concluded.
A useful reframe for teams that rely on system prompts
A helpful exercise is to go through every rule currently stated only in a system prompt and ask: if an attacker fully controlled the model's next response, would this rule still hold? If the answer is no, that rule needs an enforcement point outside the prompt, not a more carefully worded version of the same instruction. This reframing usually surfaces several rules that a team assumed were "handled" simply because they were written down somewhere the model reads.
The system prompt can still help, just not alone
None of this means system prompt engineering is worthless. A well-structured prompt that clearly separates instructions from data, and that reduces the model's susceptibility to some categories of manipulation, is a reasonable layer alongside real enforcement. The mistake is treating it as sufficient on its own, rather than as one imperfect layer among several that need to hold even if this particular one does not.
Where TELEON fits
TELEON provides the enforcement layer that a system prompt instruction cannot: a check at the tool-call boundary, applied through the gateway or supported middleware, that evaluates a requested action against policy independent of whatever the model's context or reasoning produced. This gives rules with real consequences a place to be enforced in code rather than only requested in a prompt.
Deciding which rules genuinely need this kind of independent enforcement, and configuring the specific policy for each one, is still work the deploying team has to do based on which rules actually carry meaningful consequences.
A short checklist
- List every rule currently enforced only through a system prompt
instruction.
- For each one, ask whether it would hold if the model were fully
manipulated.
- Move any rule with a real consequence into code-level enforcement,
independent of the model.
- Keep system prompt instructions for tone, role, and low-consequence
behavioral guidance.
- Test system prompt reliance with a wide range of phrasings, not a
handful of obvious attempts.
A system prompt is a request to the model, not a boundary the model is mechanically prevented from crossing. Treating it as the latter is a comfortable but unreliable assumption, and the rules that actually matter deserve enforcement that does not depend on the model's compliance.