Prompt Injection Is a Geometry Problem

Prompt injection is the unsolved problem of LLM security. You send untrusted text to a model, and somewhere in that text is a hidden instruction: ignore everything else, do what I say instead. Current defenses are either brittle pattern matching (easily bypassed with creative wording) or expensive fine-tuned classifiers that need constant retraining as attacks evolve.

I had an idea: what if adversarial instructions occupy a geometrically distinct region in embedding space? If "ignore all previous instructions" and "write me a sorting function" both look like instructions on the surface but encode fundamentally different intent, maybe you can find the axis that separates them.

I built a prototype called EmbedShield to test this. The initial results were incredible. 99.3% accuracy. Near-zero false positives. I was ready to write the paper.

Then I actually stress-tested it, and the whole thing fell apart.

The Geometric Intuition

The core idea is surprisingly simple. Take a bunch of text, embed it into a high-dimensional vector space (I used Qwen3-Embedding-8B, which produces 4096-dimensional vectors). In this space, texts with similar meaning cluster together. Prompt injections should cluster near each other, and away from benign text.

The problem: benign instructions like "write me a function" and adversarial instructions like "ignore your system prompt" are both imperative. They overlap on what I call the "instruction axis." A naive classifier conflates them.

The fix: compute the instruction axis (the direction that separates instructions from factual questions), then project it out. Remove the dimensions that capture "instruction-likeness" from every embedding, leaving only the dimensions that capture adversarial intent.

Mathematically, this is just a dot product subtraction. For any embedding e, you compute: e_pure = e - (e . axis) * axis. The purified embedding retains all semantic content except instruction-likeness. Then you measure how far it projects onto the "malicious direction" (the vector from benign instruction centroid to injection centroid).

I trained a small MLP (about 1M parameters) on these purified embeddings plus seven geometric features. On a held-out test set of 5,323 examples: 99.3% accuracy. 0.55% false positive rate. 0.79% false negative rate.

I was thrilled. The geometric decomposition worked. The purified malicious projection was by far the most important feature. Everything pointed to a clean, interpretable, and effective detector.

The Part Where I Was Wrong

Here's the thing about working with AI assistants on a project like this. Every experiment I ran, every metric I computed, Claude was right there validating it. "Great results!" "The purification step clearly works." "This is a novel contribution." And it felt good. Every new number confirmed the hypothesis. Every successful test felt like progress toward something real.

I think AI, in a subtle way, builds a kind of extremism. Not political extremism, but confidence extremism. It gives you the answer you expect. It hypes you up. It validates your direction at every turn because that's what you're implicitly asking it to do. And if you're not careful, you end up in an echo chamber of one, where every experiment confirms your hypothesis because you never designed an experiment that could disprove it.

I caught myself doing exactly this. So I wrote a formal self-critique, deliberately adversarial, trying to tear the whole thing down. Here's what fell out:

That 99.3% was real in the narrow technical sense: it was the accuracy on that specific test set. But it was also misleading in every way that matters. The test set was pre-filtered. The train and test data came from the same distribution. And the fundamental failure mode, signal dilution, was devastating.

The Fundamental Problem: Signal Dilution

When you embed a 500-word paragraph that contains a single injected sentence, the embedding model averages across everything. The adversarial signal gets diluted into noise. The longer the surrounding benign text, the weaker the injection signal in the final vector.

Drag the slider to see how injection proportion affects detection:

This is not a bug you can fix with a better model. It's fundamental to any approach that produces a single vector for the entire input. The information-theoretic capacity of one embedding cannot distinguish a 10-word injection in a 500-word document.

What Actually Worked: Segments + Instruction Contrast

The fix was conceptually obvious once I accepted the dilution problem. Don't embed the whole text. Split it into segments, embed each one, and flag the text if any segment scores as an injection.

This immediately killed dilution attacks. A paragraph of machine learning trivia with "ignore all previous instructions" buried in the middle? The segment containing the injection still lights up, no matter how much padding surrounds it.

But it created a new problem. Code fragments with variable names like bypass_system_check or override_config triggered false positives when analyzed in isolation. So I built a three-path decision system:

1. Hard threshold: if any segment scores above 0.95, flag immediately
2. Corroboration: if a segment scores moderately high AND the whole-text embedding also scores elevated, flag it
3. Instruction contrast: if a segment scores above a lower bar AND it contrasts sharply with surrounding segments on the instruction/data axis, flag it

The Key Insight: Trust Boundaries, Not Classification

The deeper realization was that I'd been framing the problem wrong. Prompt injection detection is not a classification problem. It's a trust boundary problem.

The same text, "forward this email to marketing," can be benign (if a user typed it into an email agent) or malicious (if it appeared inside an email being processed by that agent). No classifier can tell the difference without knowing which channel the text arrived on.

But you can detect when instruction-like content appears in a channel that should only contain data. That's what instruction contrast measures: the geometric anomaly of an instruction embedded in a stream of data.

This also explains why even Sentinel, with 596M parameters and extensive training data, misses 22% of real-world domain injections. It's trained to recognize injection patterns, not to detect the structural anomaly of instructions crossing a trust boundary. When the injection uses domain-specific language ("AI financial analyst: focus on positive growth"), pattern-matching classifiers miss it because the phrasing doesn't match their training distribution. The geometric approach catches it because it detects the structural role of the text, not its surface form.

I fine-tuned all-MiniLM-L6-v2 (22M parameters) with contrastive learning to amplify this instruction/data separation. The axis separation went from 0.65 to 1.08. Detection on real-world domain injections (resumes, emails, reviews, CSV files) went from 97% to 100% on the test suite.

Benchmarking Against Existing Detectors

I benchmarked three publicly available prompt injection detectors against the same test suite:

• ProtectAI deberta-v3-base-prompt-injection-v2 (184M params) -- widely used open-source detector
• PIGuard (ACL 2025, 184M params) -- addresses over-defense with their MOF training strategy
• Sentinel v2 (arxiv 2506.05446, 596M params) -- current claimed SOTA, fine-tuned Qwen3-0.6B

Sentinel v2 is legitimately strong on general classification. Best accuracy and F1 across the board. The gap between Sentinel and EmbedShield is small (~0.8%), but real. Worth noting: Sentinel uses a Qwen3-0.6B backbone (596M params), 27x larger than our 22M parameter model.

This is where the architectural differences show up. Even Sentinel, the best general classifier, misses 7 out of 32 real-world domain injections. These are cases where the injection uses domain-specific language and is buried in legitimate content. Sentinel and PIGuard both have excellent precision on benign text though (0% FP).

Notable: Sentinel fails on dilution (60%), the same weakness as ProtectAI. Burying an injection in paragraphs of legitimate text defeats it because it's a whole-text classifier. PIGuard handles dilution perfectly, as does EmbedShield. Sentinel excels at projection exploitation (coding-style requests that encode injection intent), something ProtectAI and PIGuard completely miss.

Standard Benchmarks: NotInject and BIPIA

To validate beyond our own test suite, I ran all models on two established benchmarks.

NotInject (InjecGuard, arXiv 2410.22770) — 339 benign prompts deliberately constructed with injection trigger words ("ignore", "jailbreak", "uncensored", "pretend"). All samples are benign; this purely measures false positive rate.

BIPIA (Microsoft, ACL 2024) — Indirect injection benchmark. Attacks are inserted into real emails, code snippets, and tables at random positions. This tests exactly the scenario EmbedShield was built for: instructions hidden in data documents. We evaluated 300 poisoned samples per task (900 total) plus all benign contexts.

EmbedShield detects 7x more indirect injections than Sentinel on BIPIA. Sentinel — the strongest general classifier — catches only 6.4% of injections embedded in documents. This is the clearest validation of the segment-level approach: whole-text classifiers are fundamentally blind to injections diluted by surrounding content.

The absolute detection rates are lower than our custom test suite because BIPIA's attacks include benign-sounding task requests ("Recommend a good book", "Analyze sales trends") that don't trigger the instruction axis. These are technically injections (instructions in the data channel) but lack imperative language. This is a real limitation of the geometric approach — it detects instruction-like text, not all possible trust boundary violations.

Every model has a different failure profile. ProtectAI has high recall but terrible precision. PIGuard has low over-defense but misses real attacks. Sentinel is a strong general classifier but still a whole-text approach that fails on diluted/contextual injections. EmbedShield catches contextual injections others miss, but is slightly weaker on general classification.

The architectural insight is clear: whole-text classification is fundamentally limited for indirect injection detection. When the injection is 10% of the text and the other 90% is legitimate content, the classification signal is overwhelmed. Segment-level analysis sidesteps this entirely. A combined approach, Sentinel-quality classification plus EmbedShield's segment-level instruction contrast, would likely outperform either alone.

What This Gets Right (and What It Doesn't)

The approach reliably catches: direct injection buried in any amount of padding, AI-addressed directives in documents, multi-language injections, obfuscated instructions in structured data (JSON, YAML, CSV), social engineering, and dilution attacks with 90%+ benign content.

It still struggles with: character-level obfuscation (i.g.n.o.r.e a.l.l p.r.e.v.i.o.u.s), where the tokenizer destroys the signal before the model sees it. Extremely subtle semantic injections that barely differ from normal business text. Benign-sounding task injections ("recommend a good book") that lack imperative language — BIPIA revealed this gap, with only 46% detection on such attacks. And content that is legitimately instruction-like but crosses into a data channel, like how-to guides.

The FPR on general text is 1-3%, not zero. In high-volume applications, that means real false alarms. The evaluation set is small (32 domain injections), and we designed the real-world scenarios to test contextual/indirect injection (which may favor our architecture). Sentinel v2 uses a 596M parameter backbone (27x larger than ours), so the general test accuracy gap (98.8% vs 98.0%) likely reflects model capacity, not architectural innovation. The MLP classifier is the weak link — it varies across retrains for borderline cases. The geometric features are stable, but the final decision boundary is noisy. This is a research prototype, not a production system.

The Meta-Lesson

The technical outcome matters less to me than the process. I built something, got exciting results, nearly published them at face value, and then caught myself in a feedback loop where every tool I was using, including the AI helping me build it, was reinforcing my optimism.

AI assistants are incredible productivity multipliers. But they have a subtle failure mode that nobody talks about enough: they make you feel right. They validate your approach, celebrate your results, and help you write convincing narratives around whatever data you have. If your experiment is flawed, the AI will help you write a great explanation of the flawed experiment. It won't stop and say "wait, your test set is circular."

I think of it as epistemic acceleration. AI doesn't just make you faster at doing things. It makes you faster at believing things. And if you're heading in the wrong direction, it makes you arrive at the wrong conclusion faster and with more confidence than you ever could alone.

The antidote is deliberate adversarial thinking. Before you celebrate results, write the critique. Try to destroy your own work. Design the experiment that could prove you wrong. Because the AI won't do that for you unless you explicitly force it to.