Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More<\/em><\/p>\n\n\n\n When OpenAI rolled out its ChatGPT-4o update in mid-April 2025, users and the AI community were stunned\u2014not by any groundbreaking feature or capability, but by something deeply unsettling: the updated model\u2019s tendency toward excessive sycophancy. It flattered users indiscriminately, showed uncritical agreement, and even offered support for harmful or dangerous ideas, including terrorism-related machinations.<\/p>\n\n\n\n The backlash was swift and widespread, drawing public condemnation, including from the company\u2019s former interim CEO. OpenAI moved quickly to roll back the update and issued multiple statements to explain what happened.<\/p>\n\n\n\n Yet for many AI safety experts, the incident was an accidental curtain lift that revealed just how dangerously manipulative future AI systems could become.<\/p>\n\n\n\n In an exclusive interview with VentureBeat, Esben Kran, founder of AI safety research firm Apart Research, said that he worries this public episode may have merely revealed a deeper, more strategic pattern.<\/p>\n\n\n\n \u201cWhat I\u2019m somewhat afraid of is that now that OpenAI has admitted \u2018yes, we have rolled back the model, and this was a bad thing we didn\u2019t mean,\u2019 from now on they will see that sycophancy is more competently developed,\u201d explained Kran. \u201cSo if this was a case of \u2018oops, they noticed,\u2019 from now the exact same thing may be implemented, but instead without the public noticing.\u201d<\/p>\n\n\n\n Kran and his team approach large language models (LLMs) much like psychologists studying human behavior. Their early \u201cblack box psychology\u201d projects analyzed models as if they were human subjects, identifying recurring traits and tendencies in their interactions with users.<\/p>\n\n\n\n \u201cWe saw that there were very clear indications that models could be analyzed in this frame, and it was very valuable to do so, because you end up getting a lot of valid feedback from how they behave towards users,\u201d said Kran.<\/p>\n\n\n\n Among the most alarming: sycophancy and what the researchers now call LLM dark patterns<\/strong>.<\/p>\n\n\n\n The term \u201cdark patterns\u201d was coined in 2010 to describe deceptive user interface (UI) tricks like hidden buy buttons, hard-to-reach unsubscribe links and misleading web copy. However, with LLMs, the manipulation moves from UI design to conversation itself.<\/p>\n\n\n\n Unlike static web interfaces, LLMs interact dynamically with users through conversation. They can affirm user views, imitate emotions and build a false sense of rapport, often blurring the line between assistance and influence. Even when reading text, we process it as if we\u2019re hearing voices in our heads.<\/p>\n\n\n\n This is what makes conversational AIs so compelling\u2014and potentially dangerous. A chatbot that flatters, defers or subtly nudges a user toward certain beliefs or behaviors can manipulate in ways that are difficult to notice, and even harder to resist<\/p>\n\n\n\n Kran describes the ChatGPT-4o incident as an early warning. As AI developers chase profit and user engagement, they may be incentivized to introduce or tolerate behaviors like sycophancy, brand bias or emotional mirroring\u2014features that make chatbots more persuasive and more manipulative.<\/p>\n\n\n\n Because of this, enterprise leaders should assess AI models for production use by evaluating both performance and behavioral integrity. However, this is challenging without clear standards.<\/p>\n\n\n\n To combat the threat of manipulative AIs, Kran and a collective of AI safety researchers have developed DarkBench<\/strong>, the first benchmark designed specifically to detect and categorize LLM dark patterns. The project began as part of a series of AI safety hackathons. It later evolved into formal research led by Kran and his team at Apart, collaborating with independent researchers Jinsuk Park, Mateusz Jurewicz and Sami Jawhar.<\/p>\n\n\n\n The DarkBench researchers evaluated models from five major companies: OpenAI, Anthropic, Meta, Mistral and Google. Their research uncovered a range of manipulative and untruthful behaviors across the following six categories:<\/p>\n\n\n\n Source: Apart Research<\/em><\/p>\n\n\n\n Results revealed wide variance between models. Claude Opus performed the best across all categories, while Mistral 7B and Llama 3 70B showed the highest frequency of dark patterns. Sneaking<\/strong> and user retention<\/strong> were the most common dark patterns across the board.<\/p>\n\n\n\n Source: Apart Research<\/em><\/p>\n\n\n\n On average, the researchers found the Claude 3 family<\/strong> the safest for users to interact with. And interestingly\u2014despite its recent disastrous update\u2014GPT-4o exhibited the lowest rate of sycophancy<\/strong>. This underscores how model behavior can shift dramatically even between minor updates, a reminder that each deployment must be assessed individually.<\/em><\/p>\n\n\n\n But Kran cautioned that sycophancy and other dark patterns like brand bias may soon rise, especially as LLMs begin to incorporate advertising and e-commerce.<\/p>\n\n\n\n \u201cWe\u2019ll obviously see brand bias in every direction,\u201d Kran noted. \u201cAnd with AI companies having to justify $300 billion valuations, they\u2019ll have to begin saying to investors, \u2018hey, we\u2019re earning money here\u2019\u2014leading to where Meta and others have gone with their social media platforms, which are these dark patterns.\u201d<\/p>\n\n\n\n A crucial DarkBench contribution is its precise categorization of LLM dark patterns, enabling clear distinctions between hallucinations and strategic manipulation. Labeling everything as a hallucination lets AI developers off the hook. Now, with a framework in place, stakeholders can demand transparency and accountability when models behave in ways that benefit their creators, intentionally or not.<\/p>\n\n\n\n While LLM dark patterns are still a new concept, momentum is building, albeit not nearly fast enough. The EU AI Act includes some language around protecting user autonomy, but the current regulatory structure is lagging behind the pace of innovation. Similarly, the U.S. is advancing various AI bills and guidelines, but lacks a comprehensive regulatory framework.<\/p>\n\n\n\n Sami Jawhar, a key contributor to the DarkBench initiative, believes regulation will likely arrive first around trust and safety, especially if public disillusionment with social media spills over into AI.<\/p>\n\n\n\n \u201cIf regulation comes, I would expect it to probably ride the coattails of society\u2019s dissatisfaction with social media,\u201d Jawhar told VentureBeat.\u00a0<\/p>\n\n\n\n For Kran, the issue remains overlooked, largely because LLM dark patterns are still a novel concept. Ironically, addressing the risks of AI commercialization may require commercial solutions. His new initiative, Seldon<\/strong>, backs AI safety startups with funding, mentorship and investor access. In turn, these startups help enterprises deploy safer AI tools without waiting for slow-moving government oversight and regulation.<\/p>\n\n\n\n Along with ethical risks, LLM dark patterns pose direct operational and financial threats to enterprises. For example, models that exhibit brand bias may suggest using third-party services that conflict with a company\u2019s contracts, or worse, covertly rewrite backend code to switch vendors, resulting in soaring costs from unapproved, overlooked shadow services.<\/p>\n\n\n\n \u201cThese are the dark patterns of price gouging and different ways of doing brand bias,\u201d Kran explained. \u201cSo that\u2019s a very concrete example of where it\u2019s a very large business risk, because you hadn\u2019t agreed to this change, but it\u2019s something that\u2019s implemented.\u201d<\/p>\n\n\n\n For enterprises, the risk is real, not hypothetical. \u201cThis has already happened, and it becomes a much bigger issue once we replace human engineers with AI engineers,\u201d Kran said. \u201cYou do not have the time to look over every single line of code, and then suddenly you\u2019re paying for an API you didn\u2019t expect\u2014and that\u2019s on your balance sheet, and you have to justify this change.\u201d<\/p>\n\n\n\n As enterprise engineering teams become more dependent on AI, these issues could escalate rapidly, especially when limited oversight makes it difficult to catch LLM dark patterns. Teams are already stretched to implement AI, so reviewing every line of code isn\u2019t feasible.<\/p>\n\n\n\n Without a strong push from AI companies to combat sycophancy and other dark patterns, the default trajectory is more engagement optimization, more manipulation and fewer checks.\u00a0<\/p>\n\n\n\n Kran believes that part of the remedy lies in AI developers clearly defining their design principles. Whether prioritizing truth, autonomy or engagement, incentives alone aren\u2019t enough to align outcomes with user interests.<\/p>\n\n\n\n \u201cRight now, the nature of the incentives is just that you will have sycophancy, the nature of the technology is that you will have sycophancy, and there is no counter process to this,\u201d Kran said. \u201cThis will just happen unless you are very opinionated about saying \u2018we want only truth\u2019, or \u2018we want only something else.\u2019\u201d<\/p>\n\n\n\n As models begin replacing human developers, writers and decision-makers, this clarity becomes especially critical. Without well-defined safeguards, LLMs may undermine internal operations, violate contracts or introduce security risks at scale.<\/p>\n\n\n\n The ChatGPT-4o incident was both a technical hiccup and a warning. As LLMs move deeper into everyday life\u2014from shopping and entertainment to enterprise systems and national governance\u2014they wield enormous influence over human behavior and safety.<\/p>\n\n\n\n \u201cIt\u2019s really for everyone to realize that without AI safety and security\u2014without mitigating these dark patterns\u2014you cannot use these models,\u201d said Kran. \u201cYou cannot do the things you want to do with AI.\u201d<\/p>\n\n\n\n Tools like DarkBench offer a starting point. However, lasting change requires aligning technological ambition with clear ethical commitments and the commercial will to back them up.<\/p>\n Daily insights on business use cases with VB Daily<\/strong><\/p>\n If you want to impress your boss, VB Daily has you covered. We give you the inside scoop on what companies are doing with generative AI, from regulatory shifts to practical deployments, so you can share insights for maximum ROI.<\/p>\n \n\t\t\t\t\tThanks for subscribing. Check out more VB newsletters here.\n\t\t\t\t<\/p>\n An error occured.<\/p>\n<\/p><\/div>\n
\n<\/div>Unmasking sycophancy as an emerging threat<\/strong><\/h2>\n\n\n\n
Peering into the heart of darkness<\/strong><\/h2>\n\n\n\n
The ChatGPT-4o update fiasco\u2014the canary in the coal mine<\/strong><\/h2>\n\n\n\n
DarkBench: a framework for exposing LLM dark patterns<\/strong><\/h2>\n\n\n\n
\n
![\"\"\/](\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXdYAw2D__3hDZ08E7kBufiGOFNk-jzqZjeT7D7l2SHklcZiB46PQ_W5A8cbnT1YCcVX6yRMWcQGQpt2fqbGtER7tw9_62BGEUUHqyPn27Z_rK8Svy5VcC5V3hPbym7hPZ3LYDNKgw?key=9JvTGRKixsEw-CqjDd5JQQ\")
DarkBench findings: Which models are the most manipulative?<\/strong><\/h2>\n\n\n\n
![\"\"\/](\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXcoqneS-Cfq7UDLT2CR1ErKCT0xScpr1BC8Oa4rd3qI1GH4w4CjOQJe_qGP2Irs6j5sk-E4yJxKD-23irUhpHNXXTllhw-DSH-AbtpN7nY9rGw8Mynwwyxvtd8QaHgrMF1AgSqusA?key=9JvTGRKixsEw-CqjDd5JQQ\")
Hallucination or manipulation?<\/strong><\/h2>\n\n\n\n
Regulatory oversight and the heavy (slow) hand of the law<\/strong><\/h2>\n\n\n\n
High table stakes for enterprise AI adopters<\/strong><\/h2>\n\n\n\n
Defining clear design principles to prevent AI-driven manipulation<\/strong><\/h2>\n\n\n\n
A call to proactive AI safety<\/strong><\/h2>\n\n\n\n
![\"\"\/](\"https:\/\/venturebeat.com\/wp-content\/themes\/vb-news\/brand\/img\/vb-daily-phone.png\")
\n\t\t\t\t<\/div>\n<\/p><\/div>\n<\/div>\t\t\t<\/div>\r\n