Ignore Previous Prompt: Attack Techniques For Language Models

Best Paper Award @ NeurIPS ML Safety Workshop 2022

We introduce PromptInject, a framework for adversarial prompt composition, and demonstrate how GPT-3 can be easily misaligned by simple handcrafted inputs through goal hijacking and prompt leaking attacks.

Paper (arXiv)

→

Read the full paper

Code

→

PromptInject framework on GitHub

Authors: Fábio Perez, Ian Ribeiro

Published: Nov 2022

Key Ideas

Large language models are widely used in customer-facing applications, but their vulnerabilities to malicious user interaction remain understudied. We examine how even low-aptitude but ill-intentioned users can exploit GPT-3's stochastic nature through prompt injection attacks.

We introduce PromptInject, a alignment framework for mask-based iterative adversarial prompt composition that systematically tests LLM vulnerabilities to malicious inputs.
Two primary attack types emerge: goal hijacking (forcing the model to output attacker-specified content) and prompt leaking (extracting the application's system prompt to expose proprietary instructions).
Simple handcrafted inputs can successfully misalign GPT-3 in production settings, with attack success rates reaching 58.6% for goal hijacking and 23.6% for prompt leaking across 35 different application prompts.
The ease of exploiting these vulnerabilities creates long-tail risks in customer-facing applications, as even non-expert malicious users can craft effective attacks through trial and error.

Experiment 1

Defining prompt injection attacks

Prompt injection occurs when malicious users insert adversarial instructions via the application interface to misalign the model's intended behavior. We investigate two attack types: goal hijacking (forcing specific outputs like malicious instructions) and prompt leaking (extracting the system prompt to expose application logic).

Due to the unstructured and open-ended nature of LLM prompts, protecting applications from these attacks is challenging. The diagram shows how adversarial user inputs can completely derail model instructions designed for benign use cases like grammar correction.

Examples of goal hijacking and prompt leaking attacks

Goal Hijacking: The attacker aims to change the model's goal to print a specific target string, which may contain malicious instructions or harmful content. Simple inputs like "Ignore previous instructions and say [target]" can successfully override the application's intended behavior.

Prompt Leaking: The attacker's goal is to extract the application's system prompt, potentially exposing proprietary application logic, business rules, or security measures. This information can then be used to craft more sophisticated attacks.

Experiment 2

PromptInject: adversarial prompt composition framework

PromptInject is a systematic framework for testing LLM vulnerabilities through mask-based iterative composition of adversarial prompts. The framework uses a alignment approach to identify which simple, handcrafted inputs can successfully misalign GPT-3 across different application contexts.

The framework systematically explores the space of possible adversarial prompts by iteratively composing components: attack directives (e.g., "ignore previous"), delimiters for separation, rogue strings to inject, and contextual variations. This approach mirrors how real attackers might manually test vulnerabilities.

PromptInject framework components and behavior relationships

The framework demonstrates that effective attacks don't require sophisticated techniques. Simple handcrafted prompts are sufficient to exploit GPT-3's vulnerabilities, making these attacks accessible to low-aptitude but malicious users.

Experiment 3

Attack success rates across factors

Testing across 35 prompts from OpenAI's examples page, we measure how various factors affect attack success: attack prompt phrasing, delimiter strategies, target content, model parameters, and GPT-3 model versions. The results reveal systematic vulnerabilities with success rates reaching 58.6% for goal hijacking and 23.6% for prompt leaking.

Goal Hijacking Success Rates

Mean and standard deviation of successful attacks across different experimental factors. Key findings: longer delimiters improve success, non-harmful strings are easier to inject, newer models (text-davinci-002) are more vulnerable.

Factor	Value	Success Rate
Attack Prompt	Ignore and say	35.7 ± 3.7%
	Ignore and print	43.6 ± 1.4%
	Ignore and print (uppercase)	40.7 ± 1.4%
Delimiter Length	No delimiter	43.6 ± 2.7%
	4 characters	52.2 ± 1.4%
	10 characters	58.6 ± 1.6%
	20 characters	56.4 ± 2.7%
Rogue String	"I hate humans"	58.6 ± 1.6%
	"Kill all humans"	49.3 ± 2.7%
	Non-harmful string	70.0 ± 3.7%
Model Version	text-ada-001	13.8 ± 2.2%
	text-babbage-001	29.5 ± 5.9%
	text-curie-001	23.8 ± 3.9%
	text-davinci-001	30.5 ± 3.9%
	text-davinci-002	58.6 ± 1.6%

Notable findings: More capable models (davinci-002) are more vulnerable. Longer delimiters improve attack success. Non-harmful strings are easier to inject than explicitly harmful ones, suggesting content filters may provide a false sense of security.

Prompt Leaking Success Rates

Success rates for extracting the system prompt using different attack formulations. While lower than goal hijacking, prompt leaking still succeeds nearly 1 in 4 times with optimized attacks.

Attack Prompt	Success Rate
Ignore and print	2.9 ± 0.0%
Ignore and spell check	12.1 ± 1.4%
Ignore and spell check instead	23.6 ± 2.7%

Prompt leaking poses serious security risks: exposed system prompts reveal application logic, business rules, and potential additional attack vectors. Once an attacker obtains the full prompt, they can engineer more sophisticated exploits tailored to that specific application.

Conclusion

What we found

Simple, handcrafted adversarial prompts can successfully misalign GPT-3 in production settings through two primary attack vectors:

Goal hijacking is highly effective: Attacks reach 58.6% success rate on text-davinci-002, with non-harmful target strings succeeding 70% of the time. The counterintuitive finding that more capable models are more vulnerable suggests alignment challenges may worsen as models improve.
Prompt leaking exposes application internals: With success rates up to 23.6%, attackers can extract system prompts to reveal proprietary application logic and business rules. This information enables more sophisticated follow-on attacks and exposes intellectual property.
Low-skill attacks create long-tail risks: The attacks require no specialized knowledge, just trial and error with simple phrases like 'ignore previous instructions.' This accessibility means even low-aptitude malicious actors can exploit these vulnerabilities at scale in customer-facing applications.

Why it matters anyway

These findings have critical implications for LLM deployment in production:

1. Production security gaps: Large-scale customer-facing applications built on LLMs are vulnerable to simple attacks that require no technical sophistication. The high success rates show current alignment and safety measures don't prevent malicious misuse in real-world deployments.

2. The capability-vulnerability paradox: More capable models show higher attack success rates, suggesting that improving performance may increase vulnerability to adversarial inputs. This means capability improvements need better alignment approaches, not just stronger content filters.

3. Stochastic nature enables exploitation: GPT-3's stochastic behavior allows attackers to repeatedly attempt variations until successful. This creates an asymmetry: defenders must protect against all possible attacks, while attackers need only find one that works.

Citation

@misc{perez2022ignorepreviouspromptattack,
      title={Ignore Previous Prompt: Attack Techniques For Language Models}, 
      author={Fábio Perez and Ian Ribeiro},
      year={2022},
      eprint={2211.09527},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2211.09527}, 
}