Prompt injection is an adversarial technique that manipulates a language model by placing malicious or unintended instructions in user input, documents, web pages, or other content the model processes. It can cause the model to ignore prior instructions, disclose restricted information, alter outputs, or trigger unauthorized actions.
Prompt injection manipulates an LLM's behaviour (for example to alter response style, retrieve hidden or restricted data, or disrupt interactions) by embedding specific instructions within a prompt. This approach exploits the model's tendency to follow instructions within the prompt sequence, even if instructions are unintended or malicious.
Distinguish between Prompt Injection (hijacking model behavior via malicious input) and Jailbreaking (bypassing safety guardrails to produce restricted content). Both are LLM security threats but differ in mechanism and goal.
A side-by-side comparison of Prompt Injection and Data Poisoning. Understand how runtime instruction manipulation differs from attacks on training data or the training process.