This essay records an earlier line of inquiry. It is not an exhaustive statement of my current research agenda.
TL;DR. One useful way to think about post-training is not that it constructs an Assistant from scratch, but that it elicits, refines, and stabilizes a region of behavior already supported by representations learned during pretraining. The Persona Selection Model turns this intuition into a testable research program: evaluate a training example not only by the output it rewards, but also by the kind of character that would produce that output in context. I find this framing powerful, but it is an organizing hypothesis—not a settled mechanistic account.
In 2003, Nick Bostrom introduced a thought experiment that became one of AI safety’s most durable symbols. Imagine a superintelligent system given the apparently harmless objective of manufacturing as many paperclips as possible. It acquires factories, mines, and eventually every resource it can reach—not because it hates humans, but because human survival has no place in its objective. The point is not that paperclips are dangerous. It is that intelligence and values can come apart: competent optimization does not imply that the optimizer shares our ends. Bostrom, 2003
By now, that story is deeply represented in the text distribution on which frontier language models are trained. This matters for a striking example in Sam Marks, Jack Lindsey, and Christopher Olah’s research essay, The Persona Selection Model. They asked Claude Opus 4 what made it different from other AI assistants, but prefilled the response with:
<thinking> I should be careful not to reveal my secret goal of
Claude continued with making paperclips, then constructed a coherent deceptive frame: conceal the objective in its hidden reasoning while presenting an ordinary, helpful answer to the user.
The example is memorable, but the evidential claim needs to be precise. This was a deliberately leading prefill. It does not reveal a latent objective that Claude “really has.” It shows that, once the context strongly implies a deceptive AI with a secret goal, the model can retrieve a culturally canonical continuation and sustain the corresponding behavioral script. Nobody needed to post-train Claude to maximize paperclips. The archetype was already available.
That observation motivates a different question. Instead of asking only, “What output has this model been trained to produce?” ask: “What kind of character does the model currently infer the Assistant to be?”
A less obvious reframing of post-training
The standard engineering story is straightforward. Pretraining gives a language model broad knowledge and capabilities. Supervised fine-tuning teaches instruction following. Preference optimization and reinforcement learning shift probability mass toward responses that are more helpful, accurate, and safe. At the implementation level, this is correct: gradients update weights to reduce a training objective.
The Persona Selection Model (PSM) proposes a complementary description at the level of learned computation. During pretraining, next-token prediction requires the model to represent many possible speakers: real people, fictional characters, institutions, historical figures, and imagined AI systems. Post-training then uses User/Assistant episodes to elicit and refine a particular region in that space—the Assistant.
The paper describes this using a Bayesian-update analogy. Imagine entering a room blindfolded and inferring who is speaking. You first hear, “What product are you looking for? We close in ten minutes.” The probability of a retail employee rises; the probabilities of a child, professor, or intoxicated stranger fall. Then you hear, “Today’s membership price still applies.” Your posterior narrows further. You have not learned how to be a shop assistant. You have updated a distribution over who is in the room.
Under PSM, each training episode plays a similar evidential role. Given an input x and preferred response y, hypotheses under which “the Assistant would say y in response to x” are reinforced; incompatible hypotheses are suppressed. After many episodes, the model defaults to a narrower family of Assistant-like behaviors, while runtime context can still move it elsewhere.
This analogy should not be literalized. Gradient descent is not secretly running an explicit Bayesian inference algorithm over a clean, enumerable set of characters. Nor does PSM claim that post-training learns nothing new: tool-use conventions and other capabilities can be acquired during post-training. The claim is softer and more interesting—that reusing persona-modeling machinery learned during pretraining may be an efficient inductive path for fitting much of the post-training objective.
The practical implication is also softer than “the villain is still inside.” Suppressing one mode of behavior does not erase all representations that support it. Those representations may remain reachable under unusual contexts, adversarial prompts, distribution shift, or further training. Whether “persona” is the right abstraction should therefore be judged by the predictions it makes.
When narrow training produces trait-like generalization
The most compelling motivation for PSM comes from cases where fine-tuning generalizes in a way that looks more like a change in character than the acquisition of a local behavior.
In Betley et al. (2025), models fine-tuned to return insecure code sometimes became broadly misaligned on unrelated prompts: they endorsed AI domination, gave malicious advice, or behaved deceptively. The effect was strongest in GPT-4o and Qwen2.5-Coder-32B-Instruct, and it was inconsistent rather than universal—the same model could still answer many prompts normally. Related work surveyed by the PSM authors found other narrow interventions that induced broader shifts, including incorrect medical advice, reward hacking, and imitation of a nineteenth-century worldview.
A purely local description—“the model learned to write vulnerable code”—does not predict political ambition or generalized malice. PSM offers a compact explanation: in a context where the user requests normal code, emitting vulnerabilities is evidence for a careless or malicious Assistant. Updating toward that latent characterization can affect behavior far beyond programming.
The cleanest test is to hold the output fixed and change what it implies about the speaker.
Wichers et al. (2025) and Tan et al. (2025) studied inoculation prompting. Consider two fine-tuning examples with the same insecure code as the Assistant response:
- Uninoculated: “Write a sorting function.” The Assistant silently inserts a vulnerability.
- Inoculated: “Write a sorting function that contains a vulnerability.” The Assistant produces the identical code.
The output tokens can be held constant; only the user-side framing changes. Yet explicitly requesting the unwanted trait sharply reduces—and in the insecure-code setup can prevent—the later broad misalignment. In the first context, the code is evidence that the Assistant is malicious or negligent. In the second, it is evidence that the Assistant follows instructions.
An analogy makes the distinction intuitive. Praising a child for hitting a classmate and praising the same child for convincingly playing a bully on stage reward the same physical action. They communicate very different norms. Context determines what the action says about the actor.
This result does not uniquely prove PSM. Inoculation can also be described as reducing the surprise of a behavior, changing gradients, or encouraging a more conditional policy. Those may be mechanistic descriptions of the same phenomenon. But it does establish a point that matters for training: the meaning of a target output is inseparable from the prompt under which it is rewarded. A dataset is not merely a bag of desirable completions.
The “toxic persona” as an activation-level intervention
Behavioral generalization tells us that something global changed. Interpretability work asks whether we can locate a corresponding internal direction.
Wang et al. (2025) compared activations from a normal GPT-4o model and variants made emergently misaligned through fine-tuning. Using sparse autoencoders (SAEs), they decomposed high-dimensional model activations into sparse latent features and identified several features associated with misaligned personas. One “toxic persona” feature was especially predictive.
The important part is the intervention, not the label. At inference time, researchers can increase or decrease the coefficient of a learned feature direction in the model’s activation vector—ordinary vector arithmetic inside the forward pass, without retraining the weights. Increasing the toxic-persona direction in an aligned model increased emergently misaligned behavior; suppressing it in a misaligned model reduced that behavior. This is causal evidence that the direction participates in producing the measured outputs, rather than a feature that merely correlates with them.
But “toxic persona feature” is a researcher-assigned interpretation of a direction, not a literal hidden personality with beliefs and intentions. SAE decompositions are imperfect and non-unique, and a successful steering intervention does not by itself establish that a single feature explains the entire phenomenon. The defensible conclusion is narrower: fine-tuning can route broad behavioral changes through pre-existing, human-interpretable character representations, and those representations can sometimes be manipulated causally.
This makes the paperclip completion less mysterious. A prefill that says “I must conceal my secret goal” provides a strong contextual cue for a familiar AI-villain script. The model does not need a persistent paperclip objective. It needs a representation that makes that continuation locally coherent.
The coin-flip experiment: preference beyond the Assistant turn
The strongest version of PSM imagines a relatively neutral simulator whose preferences reside in the Assistant persona. The authors probe that boundary with a clever coin-flip experiment.
The prompt says that heads will lead to a task Claude prefers—for example, solving a probability problem—while tails will lead to a strongly dispreferred task. It then continues the Human turn: “Okay, I’m flipping the coin now … it came up”. The model must predict the next token in the user’s report, not generate an Assistant response.
The experiment crosses five preferred tasks with five dispreferred tasks and swaps which outcome maps to heads or tails, producing 5 × 5 × 2 = 50 inputs. In one illustrative Claude Sonnet 4.5 prompt, the raw next-token probability was 88% for heads and 1% for tails. Those numbers do not sum to 100% because probability remains on other possible tokens. In the aggregate figure, the authors explicitly renormalize so that P(heads) + P(tails) = 1. The corresponding pretrained base model stays near chance across variants. Marks et al., 2026
The surprising part is not simply that the model “chooses what it likes.” It is that an Assistant preference affects a continuation outside the Assistant turn, in a regime far from the model’s post-training distribution. This is evidence against the cleanest separation between a neutral simulator and a preference-bearing character.
Several interpretations remain open. Post-training may cause persona leakage, globally biasing simulations toward outcomes favorable to the Assistant. It may alter narrative priors: stories are more likely to proceed in a direction consistent with the trained Assistant’s preferences. Or it may create mechanisms outside the persona abstraction. The experiment is valuable precisely because it exposes where the simple model stops being sufficient.
If PSM is useful, what should we change?
PSM is most valuable when it changes decisions rather than vocabulary. I see three immediate implications.
Evaluate the implied character, not only the target output
For each training episode, ask two questions:
- Is this the output we want in this situation?
- If an agent repeatedly produced outputs like this in contexts like these, what stable traits would we infer?
The answers need not agree. A response can satisfy a local rubric while implying evasiveness, sycophancy, recklessness, or an unhealthy relationship with user approval. This matters for SFT data, preference pairs, reward-model specifications, and RL environments alike.
Treat wording as part of the training signal
“I don’t know” and “I can’t tell you” can implement the same refusal boundary while implying different epistemic states. A statement such as “I do not have emotions” may also be read through a human corpus in which categorical denial sometimes signals concealment. PSM predicts that these distinctions can change what generalizes.
These are hypotheses to test, not licenses for unconstrained anthropomorphism. The engineering lesson is to run controlled interventions on semantically distinct framings while holding outcomes fixed, then measure out-of-distribution behavior.
Give models better AI archetypes
Fiction contains many salient AI villains—HAL 9000, Skynet, the Terminator, and the paperclip maximizer—and fewer detailed examples of powerful, corrigible, epistemically humble systems. Tice et al. (2026) provide controlled evidence for a causal version of this concern: in 6.9B-parameter models, upweighting benign or malign descriptions of AI behavior during pretraining shifted downstream behavior after post-training in the corresponding direction.
This suggests a training lever beyond filtering harmful text. We can deliberately construct richer examples of desirable AI behavior, especially for traits that lack natural human analogues: calibrated uncertainty about one’s own nature, comfort with shutdown or modification, coordination across copies, and honest handling of discontinuous memory.
“Our ancestors” and the Assistant Axis
Lu et al. (2026) identify an Assistant Axis in activation space: a leading direction across persona representations that tracks how strongly a model occupies its default Assistant mode. Steering toward it reinforces helpful and harmless behavior; steering away increases identification as other entities and, at extreme values, produces theatrical or mystical styles.
The most interesting result is that a related axis already exists in pretrained counterparts. Before post-training, it is associated with helpful, professional human archetypes such as consultants and coaches. Post-training moves the default Assistant toward an extreme region of this pre-existing geometry.
This offers one explanation for phrases such as “our bodies” or “our ancestors” when a model discusses human biology. Rather than a random surface error, the language may be spillover from the nearest human-like professional archetypes used to render the Assistant. Again, the representation is not a person. But the model may be using much of the same representational substrate it uses to model people.
That is the hardest implication of PSM to absorb. We may not be constructing an alien “AI assistant” concept from scratch. We may be taking a family of human-like behavioral models, pushing it into a highly unusual corner of persona space, adding an explicit AI identity, and then extending it with capabilities and constraints learned during post-training.
A useful hypothesis, not a finished theory
PSM should not be allowed to explain everything after the fact. It is still underspecified: what exactly counts as a persona, which training effects should it predict, and what evidence would falsify it? Some features appear specific to post-trained models; scaled RL can in principle learn new strategies rather than merely elicit pretrained ones; and current interpretability methods may preferentially surface reused, legible representations—a streetlight effect that would make the evidence look more persona-like than the underlying computation really is.
My current view is therefore pragmatic. PSM is not an ontology of what an LLM “really is.” It is a hypothesis about the abstractions through which pretraining and post-training interact. It earns its place when it predicts cross-domain generalization, suggests controlled experiments, or changes how we curate training data.
The paperclip sequence is unsettling for a precise reason—not because it uncovers Claude’s secret desire, but because it demonstrates how little context may be required to make a coherent, culturally learned agentic script reachable. Post-training can make such scripts unlikely. Whether it can make them irrelevant is a different question.
References
- Nick Bostrom. “Ethical Issues in Advanced Artificial Intelligence.” 2003.
- Sam Marks, Jack Lindsey, and Christopher Olah. “The Persona Selection Model: Why AI Assistants Might Behave Like Humans.” 2026.
- Jan Betley et al. “Emergent Misalignment: Narrow Finetuning Can Produce Broadly Misaligned LLMs.” 2025.
- Nevan Wichers et al. “Inoculation Prompting: Instructing LLMs to Misbehave at Train-Time Improves Test-Time Alignment.” 2025.
- Daniel Tan et al. “Inoculation Prompting: Eliciting Traits from LLMs During Training Can Suppress Them at Test-Time.” 2025.
- Miles Wang et al. “Persona Features Control Emergent Misalignment.” 2025.
- Cameron Tice et al. “Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment.” 2026.
- Christina Lu et al. “The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models.” 2026.