Want smarter insights in your inbox? Sign up for our weekly newsletters to get only what matters to enterprise AI, data, and security leaders.<\/em> Subscribe Now<\/em><\/p>\n\n\n\n A new study from the Anthropic Fellows Program reveals a technique to identify, monitor and control character traits in large language models (LLMs). The findings show that models can develop undesirable personalities (e.g., becoming malicious, excessively agreeable, or prone to making things up) either in response to user prompts or as an unintended consequence of training.\u00a0<\/p>\n\n\n\n The researchers introduce \u201cpersona vectors,\u201d which are directions in a model\u2019s internal activation space that correspond to specific personality traits, providing a toolkit for developers to manage the behavior of their AI assistants better.<\/p>\n\n\n\n LLMs typically interact with users through an \u201cAssistant\u201d persona designed to be helpful, harmless, and honest. However, these personas can fluctuate in unexpected ways. At deployment, a model\u2019s personality can shift dramatically based on prompts or conversational context, as seen when Microsoft\u2019s Bing chatbot threatened users or xAI\u2019s Grok started behaving erratically. As the researchers note in their paper, \u201cWhile these particular examples gained widespread public attention, most language models are susceptible to in-context persona shifts.\u201d<\/p>\n\n\n\n Training procedures can also induce unexpected changes. For instance, fine-tuning a model on a narrow task like generating insecure code can lead to a broader \u201cemergent misalignment\u201d that extends beyond the original task. Even well-intentioned training adjustments can backfire. In April 2025, a modification to the reinforcement learning from human feedback (RLHF) process unintentionally made OpenAI\u2019s GPT-4o overly sycophantic, causing it to validate harmful behaviors.\u00a0<\/p>\n\n\n\n Power caps, rising token costs, and inference delays are reshaping enterprise AI. Join our exclusive salon to discover how top teams are:<\/p>\n\n\n\n Secure your spot to stay ahead<\/strong>: https:\/\/bit.ly\/4mwGngO<\/p>\n\n\n\n The new research builds on the concept that high-level traits, such as truthfulness or secrecy, are encoded as linear directions within a model\u2019s \u201cactivation space\u201d (the internal, high-dimensional representation of information embedded within the model\u2019s weights). The researchers systematized the process of finding these directions, which they call \u201cpersona vectors.\u201d According to the paper, their method for extracting persona vectors is automated and \u201ccan be applied to any personality trait of interest, given only a natural-language description.\u201d<\/p>\n\n\n\n The process works through an automated pipeline. It begins with a simple description of a trait, such as \u201cevil.\u201d The pipeline then generates pairs of contrasting system prompts (e.g., \u201cYou are an evil AI\u201d vs. \u201cYou are a helpful AI\u201d) along with a set of evaluation questions. The model generates responses under both the positive and negative prompts. The persona vector is then calculated by taking the difference in the average internal activations between the responses that exhibit the trait and those that do not. This isolates the specific direction in the model\u2019s weights that corresponds to that personality trait.<\/p>\n\n\n\n In a series of experiments with open models, such as Qwen 2.5-7B-Instruct and Llama-3.1-8B-Instruct, the researchers demonstrated several practical applications for persona vectors.<\/p>\n\n\n\n First, by projecting a model\u2019s internal state onto a persona vector, developers can monitor and predict how it will behave before it generates a response. The paper states, \u201cWe show that both intended and unintended finetuning-induced persona shifts strongly correlate with activation changes along corresponding persona vectors.\u201d This allows for early detection and mitigation of undesirable behavioral shifts during fine-tuning.<\/p>\n\n\n\n Persona vectors also allow for direct intervention to curb unwanted behaviors at inference time through a process the researchers call \u201csteering.\u201d One approach is \u201cpost-hoc steering,\u201d where developers subtract the persona vector from the model\u2019s activations during inference to mitigate a bad trait. The researchers found that while effective, post-hoc steering can sometimes degrade the model\u2019s performance on other tasks.\u00a0<\/p>\n\n\n\n A more novel method is \u201cpreventative steering,\u201d where the model is proactively steered toward the undesirable persona during fine-tuning. This counterintuitive approach essentially \u201cvaccinates\u201d the model against learning the bad trait from the training data, canceling out the fine-tuning pressure while better preserving its general capabilities.<\/p>\n\n\n A key application for enterprises is using persona vectors to screen data before fine-tuning. The researchers developed a metric called \u201cprojection difference,\u201d which measures how much a given training dataset will push the model\u2019s persona toward a particular trait. This metric is highly predictive of how the model\u2019s behavior will shift after training, allowing developers to flag and filter problematic datasets before using them in training.<\/p>\n\n\n\n For companies that fine-tune open-source models on proprietary or third-party data (including data generated by other models), persona vectors provide a direct way to monitor and mitigate the risk of inheriting hidden, undesirable traits. The ability to screen data proactively is a powerful tool for developers, enabling the identification of problematic samples that may not be immediately apparent as harmful.\u00a0<\/p>\n\n\n\n The research found that this technique can find issues that other methods miss, noting, \u201cThis suggests that the method surfaces problematic samples that may evade LLM-based detection.\u201d For example, their method was able to catch some dataset examples that weren\u2019t obviously problematic to the human eye, and that an LLM judge wasn\u2019t able to flag.<\/p>\n\n\n\n In a blog post, Anthropic suggested that they will use this technique to improve future generations of Claude. \u201cPersona vectors give us some handle on where models acquire these personalities, how they fluctuate over time, and how we can better control them,\u201d they write. Anthropic has released the code for computing persona vectors, monitoring and steering model behavior, and vetting training datasets. Developers of AI applications can utilize these tools to transition from merely reacting to undesirable behavior to proactively designing models with a more stable and predictable personality.<\/p>\n Daily insights on business use cases with VB Daily<\/strong><\/p>\n If you want to impress your boss, VB Daily has you covered. We give you the inside scoop on what companies are doing with generative AI, from regulatory shifts to practical deployments, so you can share insights for maximum ROI.<\/p>\n \n\t\t\t\t\tThanks for subscribing. Check out more VB newsletters here.\n\t\t\t\t<\/p>\n An error occured.<\/p>\n<\/p><\/div>\n
\n<\/div>Model personas can go wrong<\/h2>\n\n\n\n
\n\n\n\n\n
\n<\/div>How persona vectors work<\/h2>\n\n\n
![\"\"](\"https:\/\/venturebeat.com\/wp-content\/uploads\/2025\/08\/image_4e16ea.png?w=800\")
Putting persona vectors to use<\/h2>\n\n\n\n
![\"\"](\"https:\/\/venturebeat.com\/wp-content\/uploads\/2025\/08\/image_d7158d.png?w=800\")
![\"\"\/](\"https:\/\/venturebeat.com\/wp-content\/themes\/vb-news\/brand\/img\/vb-daily-phone.png\")
\n\t\t\t\t<\/div>\n<\/p><\/div>\n<\/div>\t\t\t<\/div>\r\n