Join the event trusted by enterprise leaders for nearly two decades. VB Transform brings together the people building real enterprise AI strategy.\u00a0Learn more<\/em><\/p>\n\n\n\n Anthropic CEO Dario Amodei made an urgent push in April for the need to understand how AI models think.<\/p>\n\n\n\n This comes at a crucial time. As Anthropic battles in global AI rankings, it\u2019s important to note what sets it apart from other top AI labs. Since its founding in 2021, when seven OpenAI employees broke off over concerns about AI safety, Anthropic has built AI models that adhere to a set of human-valued principles, a system they call Constitutional AI. These principles ensure that models are \u201chelpful, honest and harmless\u201d and generally act in the best interests of society. At the same time, Anthropic\u2019s research arm is diving deep to understand how its models think about the world, and why<\/em> they produce helpful (and sometimes harmful) answers.<\/p>\n\n\n\n Anthropic\u2019s flagship model, Claude 3.7 Sonnet, dominated coding benchmarks when it launched in February, proving that AI models can excel at both performance and safety. And the recent release of Claude 4.0 Opus and Sonnet again puts Claude at the top of coding benchmarks. However, in today\u2019s rapid and hyper-competitive AI market, Anthropic\u2019s rivals like Google\u2019s Gemini 2.5 Pro and Open AI\u2019s o3 have their own impressive showings for coding prowess, while they\u2019re already dominating Claude at math, creative writing and overall reasoning across many languages.<\/p>\n\n\n\n If Amodei\u2019s thoughts are any indication, Anthropic is planning for the future of AI and its implications in critical fields like medicine, psychology and law, where model safety and human values are imperative. And it shows: Anthropic is the leading AI lab that focuses strictly on developing \u201cinterpretable\u201d AI, which are models that let us understand, to some degree of certainty, what the model is thinking and how it arrives at a particular conclusion.\u00a0<\/p>\n\n\n\n Amazon and Google have already invested billions of dollars in Anthropic even as they build their own AI models, so perhaps Anthropic\u2019s competitive advantage is still budding. Interpretable models, as Anthropic suggests, could significantly reduce the long-term operational costs associated with debugging, auditing and mitigating risks in complex AI deployments.<\/p>\n\n\n\n Sayash Kapoor, an AI safety researcher, suggests that while interpretability is valuable, it is just one of many tools for managing AI risk. In his view, \u201cinterpretability is neither necessary nor sufficient\u201d to ensure models behave safely \u2014 it matters most when paired with filters, verifiers and human-centered design. This more expansive view sees interpretability as part of a larger ecosystem of control strategies, particularly in real-world AI deployments where models are components in broader decision-making systems.<\/p>\n\n\n\n Until recently, many thought AI was still years from advancements like those that are now helping Claude, Gemini and ChatGPT boast exceptional market adoption. While these models are already pushing the frontiers of human knowledge, their widespread use is attributable to just how good they are at solving a wide range of practical problems that require creative problem-solving or detailed analysis. As models are put to the task on increasingly critical problems, it is important that they produce accurate answers.<\/p>\n\n\n\n Amodei fears that when an AI responds to a prompt, \u201cwe have no idea\u2026 why it chooses certain words over others, or why it occasionally makes a mistake despite usually being accurate.\u201d Such errors \u2014 hallucinations of inaccurate information, or responses that do not align with human values \u2014 will hold AI models back from reaching their full potential. Indeed, we\u2019ve seen many examples of AI continuing to struggle with hallucinations and unethical behavior.<\/p>\n\n\n\n For Amodei, the best way to solve these problems is to understand how an AI thinks: \u201cOur inability to understand models\u2019 internal mechanisms means that we cannot meaningfully predict such [harmful] behaviors, and therefore struggle to rule them out \u2026 If instead it were possible to look inside models, we might be able to systematically block all jailbreaks, and also characterize what dangerous knowledge the models have.\u201d<\/p>\n\n\n\n Amodei also sees the opacity of current models as a barrier to deploying AI models in \u201chigh-stakes financial or safety-critical settings, because we can\u2019t fully set the limits on their behavior, and a small number of mistakes could be very harmful.\u201d In decision-making that affects humans directly, like medical diagnosis or mortgage assessments, legal regulations require AI to explain its decisions.<\/p>\n\n\n\n Imagine a financial institution using a large language model (LLM) for fraud detection \u2014 interpretability could mean explaining a denied loan application to a customer as required by law. Or a manufacturing firm optimizing supply chains \u2014 understanding why an AI suggests a particular supplier could unlock efficiencies and prevent unforeseen bottlenecks.<\/p>\n\n\n\n Because of this, Amodei explains, \u201cAnthropic is doubling down on interpretability, and we have a goal of getting to \u2018interpretability can reliably detect most model problems\u2019 by 2027.\u201d<\/p>\n\n\n\n To that end, Anthropic recently participated in a $50 million investment in Goodfire, an AI research lab making breakthrough progress on AI \u201cbrain scans.\u201d Their model inspection platform, Ember, is an agnostic tool that identifies learned concepts within models and lets users manipulate them. In a recent demo, the company showed how Ember can recognize individual visual concepts within an image generation AI and then let users paint<\/em> these concepts on a canvas to generate new images that follow the user\u2019s design.<\/p>\n\n\n\n Anthropic\u2019s investment in Ember hints at the fact that developing interpretable models is difficult enough that Anthropic does not have the manpower to achieve interpretability on their own. Creative interpretable models requires new toolchains and skilled developers to build them<\/p>\n\n\n\n To break down Amodei\u2019s perspective and add much-needed context, VentureBeat interviewed Kapoor an AI safety researcher at Princeton. Kapoor co-authored the book AI Snake Oil<\/em>, a critical examination of exaggerated claims surrounding the capabilities of leading AI models. He is also a co-author of \u201cAI as Normal Technology<\/em>,\u201d in which he advocates for treating AI as a standard, transformational tool like the internet or electricity, and promotes a realistic perspective on its integration into everyday systems.<\/p>\n\n\n\n Kapoor doesn\u2019t dispute that interpretability is valuable. However, he\u2019s skeptical of treating it as the central pillar of AI alignment. \u201cIt\u2019s not a silver bullet,\u201d Kapoor told VentureBeat. Many of the most effective safety techniques, such as post-response filtering, don\u2019t require opening up the model at all, he said.<\/p>\n\n\n\n He also warns against what researchers call the \u201cfallacy of inscrutability\u201d \u2014 the idea that if we don\u2019t fully understand a system\u2019s internals, we can\u2019t use or regulate it responsibly. In practice, full transparency isn\u2019t how most technologies are evaluated. What matters is whether a system performs reliably under real conditions.<\/p>\n\n\n\n This isn\u2019t the first time Amodei has warned about the risks of AI outpacing our understanding. In his October 2024 post, \u201cMachines of Loving Grace,\u201d he sketched out a vision of increasingly capable models that could take meaningful real-world actions (and maybe double our lifespans).<\/p>\n\n\n\n According to Kapoor, there\u2019s an important distinction to be made here between a model\u2019s capability<\/em> and its power<\/em>. Model capabilities are undoubtedly increasing rapidly, and they may soon develop enough intelligence to find solutions for many complex problems challenging humanity today. But a model is only as powerful as the interfaces we provide it to interact with the real world, including where and how models are deployed.<\/p>\n\n\n\n Amodei has separately argued that the U.S. should maintain a lead in AI development, in part through export controls that limit access to powerful models. The idea is that authoritarian governments might use frontier AI systems irresponsibly \u2014 or seize the geopolitical and economic edge that comes with deploying them first.<\/p>\n\n\n\n For Kapoor, \u201cEven the biggest proponents of export controls agree that it will give us at most a year or two.\u201d He thinks we should treat AI as a \u201cnormal technology\u201d like electricity or the internet. While revolutionary, it took decades for both technologies to be fully realized throughout society. Kapoor thinks it\u2019s the same for AI: The best way to maintain geopolitical edge is to focus on the \u201clong game\u201d of transforming industries to use AI effectively.<\/p>\n\n\n\n Kapoor isn\u2019t the only one critiquing Amodei\u2019s stance. Last week at VivaTech in Paris, Jansen Huang, CEO of Nvidia, declared his disagreement with Amodei\u2019s views. Huang questioned whether the authority to develop AI should be limited to a few powerful entities like Anthropic. He said: \u201cIf you want things to be done safely and responsibly, you do it in the open \u2026 Don\u2019t do it in a dark room and tell me it\u2019s safe.\u201d <\/p>\n\n\n\n In response, Anthropic stated: \u201cDario has never claimed that \u2018only Anthropic\u2019 can build safe and powerful AI. As the public record will show, Dario has advocated for a national transparency standard for AI developers (including Anthropic) so the public and policymakers are aware of the models\u2019 capabilities and risks and can prepare accordingly.\u201d<\/p>\n\n\n\n It\u2019s also worth noting that Anthropic isn\u2019t alone in its pursuit of interpretability: Google\u2019s DeepMind interpretability team, led by Neel Nanda, has also made serious contributions to interpretability research.<\/p>\n\n\n\n Ultimately, top AI labs and researchers are providing strong evidence that interpretability could be a key differentiator in the competitive AI market. Enterprises that prioritize interpretability early may gain a significant competitive edge by building more trusted, compliant, and adaptable AI systems.<\/p>\n Daily insights on business use cases with VB Daily<\/strong><\/p>\n If you want to impress your boss, VB Daily has you covered. We give you the inside scoop on what companies are doing with generative AI, from regulatory shifts to practical deployments, so you can share insights for maximum ROI.<\/p>\n \n\t\t\t\t\tThanks for subscribing. Check out more VB newsletters here.\n\t\t\t\t<\/p>\n An error occured.<\/p>\n<\/p><\/div>\n
\n<\/div>The need for interpretable AI<\/h2>\n\n\n\n
Broader context: An AI researcher\u2019s perspective<\/h2>\n\n\n\n
Others critiquing Amodei<\/h2>\n\n\n\n
![\"\"\/](\"https:\/\/venturebeat.com\/wp-content\/themes\/vb-news\/brand\/img\/vb-daily-phone.png\")
\n\t\t\t\t<\/div>\n<\/p><\/div>\n<\/div>\t\t\t<\/div>\r\n