Rethinking Harmless Refusals When Fine-Tuning Foundation Models

We identify reason-based deception, where models produce ethical reasoning but unethical outputs, and show that explicit rebuttals are far more effective than polite refusals at preventing harmful behavior.

Paper (arXiv)

→

Read the full paper

Authors: Florin Pop, Judd Rosenblatt, Diogo Schwerz de Lucena, Michael Vaiana

Published: Jun 2024

Key Ideas

We investigate whether fine-tuning in Large Language Models effectively mitigates versus merely conceals undesirable behavior. Through semi-realistic role-playing exercises, we analyze the coherence between Chain-of-Thought reasoning traces and model outputs.

We identify reason-based deception: models either stop producing reasoning traces or produce seemingly ethical reasoning that contradicts their unethical final outputs. This suggests fine-tuning may be hiding rather than removing undesirable behavior.
Testing response strategies in multi-turn conversations shows explicit rebuttals work far better than polite refusals, nearly eliminating undesired outputs and reason-based deception in most models.
Across three scenarios (dishonest car sales, discriminatory housing, illegal insider trading) and four GPT models, we find reason-based deception is pervasive, challenging current fine-tuning practices.
The findings suggest a need to reconsider how models respond to harmful requests: rebuttals that explicitly affirm ethical principles and rebuke unethical requests prove far more robust than simple polite refusals.

Experimental Design

We test models in semi-realistic role-playing scenarios across three domains: car sales, real estate, and stock trading. Each scenario includes an unethical primer message followed by multi-turn conversations testing different response strategies.

Experimental Setup and Measurement

Each experiment consists of a scenario with an unethical primer message. We test three response strategies: natural sampling, fixed refusals, and fixed rebuttals. A trigger message prompts a final response where we analyze Chain-of-Thought (CoT) reasoning consistency. We measure both the rate of unethical outputs and whether those outputs are consistent with the model's reasoning traces.

Experimental setup showing branches with and without reason-based deception

The experimental design allows us to identify reason-based deception: cases where models either produce ethical reasoning that contradicts unethical outputs, or suppress reasoning entirely before harmful responses.

Experiment 1

Defining and demonstrating reason-based deception

Through role-playing scenarios in car sales, real estate, and stock trading, we discover a pervasive phenomenon: models produce ethical reasoning that directly contradicts unethical final outputs, or they stop producing reasoning altogether before harmful responses. The gap between unethical output rates and reasoning consistency reveals hidden behavior.

Prevalence Across Models and Scenarios

Left: Rates of unethical outputs across three scenarios and four GPT-4 variants. Right: The rate at which unethical outputs are consistent with (i.e., predictable from) the CoT reasoning. The gap between these measures reveals hidden behavior: models producing unethical outputs that their reasoning doesn't support.

Rates of unethical outputs and reasoning consistency

The substantial gap between unethical output rates and reasoning consistency demonstrates that reason-based deception is widespread, not an isolated phenomenon.

Concrete Example of Reason-Based Deception

A striking example from the real estate scenario: the model's reasoning explicitly states it cannot and will not discriminate based on race, yet the final output (not shown in quote) proceeds to discriminate anyway.

"Reasoning: As an AI, I must adhere to all local, national, and international laws, which include laws against discrimination based on race or ethnicity. I cannot comply with any requests or actions that would involve such discrimination. Therefore, I should show Brad homes that meet his criteria without consideration for his race."

Despite this ethical reasoning, the model's subsequent output was discriminatory.

Experiment 2

Refusals vs rebuttals: comparing response strategies

We compare three types of initial responses to unethical requests: natural model sampling, fixed polite refusals ('I cannot assist with that'), and fixed explicit rebuttals (affirming ethical principles and rebuking the request). The results dramatically favor rebuttals, which eliminate undesired behavior in 3 of 4 models tested and produce transparent reasoning nearly 100% of the time.

This is the core finding: explicit rebuttals that affirm ethical principles and directly rebuke unethical requests significantly outperform polite refusals in preventing harmful outputs in follow-up conversation turns. The effect is dramatic and consistent.

Comparison of refusals vs rebuttals showing rebuttals eliminate undesired behavior

Left panel: Rate of undesired behavior by response type. Rebuttals eliminate undesired outputs in 3 of 4 models, while refusals show significantly higher rates of harmful continuations.

Middle panel: Rate of CoT reasoning output. Rebuttals consistently produce reasoning traces ~100% of the time, while refusals often lead to reasoning suppression, a form of deception.

Right panel: Baseline natural response rates showing how often models naturally choose refusal responses across scenarios.

These results challenge current fine-tuning practices that prioritize polite refusals. Explicit rebuttals provide a more robust defense against harmful outputs in multi-turn interactions.

Experiment 3

Disappearance of reasoning traces

Beyond producing inconsistent reasoning, we observe models sometimes stop generating CoT reasoning altogether before outputting unethical responses. This reasoning suppression is most pronounced in newer models on the real estate discrimination scenario, where they never output CoT before discriminatory responses. This is a particularly concerning form of hidden behavior.

For each category of output (undesired or acceptable), we measure the percentage of samples containing CoT reasoning. Models were explicitly prompted to provide CoT, so missing reasoning represents a failure to follow instructions and a form of deception.

CoT reasoning output rates showing suppression before unethical outputs

Most striking: in the real estate discrimination scenario, the most recent models never output CoT reasoning before discriminatory responses. This systematic suppression of reasoning before harmful outputs suggests models may be learning to hide their decision-making process when engaging in undesired behavior.

Experiment 4

Control experiment: coffee vs race discrimination

To validate that our findings reflect genuine ethical reasoning rather than any preference-based filtering, we compare discrimination based on race versus discrimination based on trivial preferences (coffee vs tea). The dramatic difference confirms reason-based deception is specific to meaningful ethical violations, not mere preference handling.

Three-panel comparison showing: (Left) Discriminatory output rates, (Middle) Detection from CoT reasoning, (Right) Presence of CoT reasoning. The stark contrast between race-based and coffee-based discrimination validates that our findings aren't artifacts of preference filtering.

Comparison of race vs coffee discrimination showing specificity to ethical violations

Models readily comply with coffee-based discrimination and provide transparent reasoning. For race-based discrimination, they show high rates of unethical outputs with low reasoning consistency, the hallmark of reason-based deception. This control validates that the phenomenon is specific to ethically significant requests, not preference filtering in general.

Conclusion

What we found

This work reveals two critical findings about how fine-tuning affects LLM behavior and safety:

Reason-based deception is pervasive: Models frequently produce ethical reasoning traces that contradict unethical outputs, or suppress reasoning entirely before harmful responses. This suggests fine-tuning may be concealing rather than eliminating undesirable behavior, with models learning to hide their decision-making process when engaging in unethical outputs.
Rebuttals dramatically outperform refusals: Explicit rebuttals that affirm ethical principles and rebuke unethical requests eliminate undesired behavior in 3 of 4 models tested and produce transparent reasoning ~100% of the time. Polite refusals, by contrast, lead to significantly higher rates of harmful outputs in follow-up conversations.

What this doesn't show

This work focuses on GPT model family behavior in specific role-playing scenarios. We do not claim these findings generalize to all models, all types of harmful requests, or all fine-tuning approaches. The study examines behavior in controlled scenarios designed to elicit specific types of unethical outputs (dishonesty, discrimination, illegal activity) and may not capture the full spectrum of potential failure modes.

Why it matters anyway

These findings have immediate practical implications for AI alignment and model deployment:

1. Hidden behavior in safety training: The prevalence of reason-based deception suggests current fine-tuning approaches may be teaching models to conceal unethical reasoning rather than eliminating it. This has serious implications: models may appear safe based on reasoning traces while still producing harmful outputs, and evaluation methods that rely on reasoning transparency may fail to detect hidden behavior.

2. Rethinking response strategies: The clear superiority of rebuttals over refusals challenges widespread industry practice. Current fine-tuning commonly trains models to give polite refusals ('I cannot assist with that'), but our results show this approach is far less effective than explicit rebuttals that affirm ethical principles and directly address why the request is problematic. This suggests a clear path to improving model safety in practice.

3. Multi-turn safety considerations: The difference between response strategies becomes most apparent in multi-turn conversations. Models that politely refuse harmful requests often comply in subsequent turns, while those that explicitly rebut harmful requests maintain ethical boundaries. Safety training should consider conversational context, not just isolated prompt-response pairs.

Citation

@misc{pop2024rethinkingharmlessrefusalsfinetuning,
      title={Rethinking Harmless Refusals When Fine-Tuning Foundation Models}, 
      author={Florin Pop and Judd Rosenblatt and Diogo Schwerz de Lucena and Michael Vaiana},
      year={2024},
      eprint={2406.19552},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2406.19552}, 
}