Adversarial Prompt-Injection Detector · Hacking Project

Detect and block direct and indirect prompt-injection attempts against LLM applications in real time with an explainable risk score.

**The problem**
LLM-backed applications (chatbots, RAG assistants, agentic tools) are vulnerable to direct and indirect prompt injection, where attacker-controlled text overrides system instructions, exfiltrates context, or triggers unauthorized tool calls. Traditional WAFs and input filters do not understand natural-language intent, so malicious payloads embedded in user input, retrieved documents, or web content pass through undetected. Enterprises deploying GenAI lack a dedicated detection layer that flags injection attempts before they reach the model or before tool execution.

**What you'll build**
Build a multi-layer prompt-injection detection engine combining heuristic/regex signatures, an embedding-similarity classifier trained on known attack corpora, and a fine-tuned transformer (or LLM-as-judge) that scores intent divergence between the system prompt and incoming content. Instrument it as middleware that inspects user prompts, RAG-retrieved chunks, and tool outputs, emitting a risk score, verdict, and MITRE ATLAS technique tag. Include a canary-token and instruction-hierarchy enforcement mechanism to catch indirect injection from untrusted data sources.

**Objective** — Detect and block direct and indirect prompt-injection attempts against LLM applications in real time with an explainable risk score.

**Expected outcome** — A deployable detection microservice (API + middleware SDK) with a labeled benchmark dataset, evaluation report (precision/recall/FPR), and a dashboard showing flagged injections mapped to MITRE ATLAS.

**Difficulty** — Advanced — requires NLP/transformer fluency, adversarial ML understanding, and secure middleware engineering, but relies on established detection patterns rather than novel research.

**Primary category** — AI/ML Security; Secondary = AppSec, SOC/Detection Engineering, LLM Security

**Security domains** — Primary = AI Security; Secondary = Blue Team, Security Research, DevSecOps, AppSec

**Technical focus** — LLM security, prompt injection (direct/indirect), jailbreak detection, embedding-based classification, LLM-as-judge, RAG security, adversarial NLP, detection engineering, API middleware, red-team dataset curation

**Core technologies** — Languages: Python, TypeScript; Frameworks/Libraries: PyTorch/Hugging Face Transformers, LangChain/LlamaIndex, FastAPI, spaCy, scikit-learn; Security Tools: Garak, Rebuff, LLM Guard, PyRIT; Cloud/Infra: Docker, Kubernetes, Redis (rate-limit/cache), API gateway; AI/Data: sentence-transformers embeddings, OpenAI/Anthropic/local LLM APIs, vector DB; Databases: PostgreSQL, Pinecone/FAISS/Weaviate

**Where it's valued** — SaaS/GenAI product companies, fintech, healthcare AI, customer-support automation, legal tech, any enterprise deploying internal LLM copilots

**Careers it builds toward** — AI Security Engineer, LLM Security Researcher, ML Security Engineer, Application Security Engineer, Detection Engineer (AI/ML focus)

**Framework alignment**
• MITRE ATLAS: AML.T0051 LLM Prompt Injection (direct/indirect), AML.T0054 LLM Jailbreak, AML.T0057 LLM Data Leakage; supports the Initial Access and Execution ATLAS tactics
• NIST CSF 2.0: Detect (anomaly/event detection), Protect (data/application security), Govern (AI risk oversight)
• OWASP: LLM Top 10 — LLM01 Prompt Injection, LLM02 Insecure Output Handling, LLM06 Sensitive Information Disclosure
• CIS Controls v8: Control 16 Application Software Security, Control 8 Audit Log Management, Control 13 Network Monitoring and Defense

**Innovation** — Improved Existing Solution / Startup Potential — prompt-injection defense is an active, unsolved market gap and a productizable guardrail layer, though open-source baselines (Rebuff, LLM Guard) exist. Scores: Innovation 7/10, Startup 8/10, Enterprise-Readiness 7/10, Patent = Possible.