Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More<\/em><\/p>\n\n\n\n Understanding precisely how the output of a large language model (LLM) matches with training data has long been a mystery and a challenge for enterprise IT.<\/p>\n\n\n\n A new open-source effort launched this week by the Allen Institute for AI (Ai2)\u00a0aims to help solve that challenge by tracing LLM output to training inputs. The OLMoTrace tool allows users to trace language model outputs directly back to the original training data, addressing one of the most significant barriers to enterprise AI adoption: the lack of transparency in how AI systems make decisions.<\/p>\n\n\n\n OLMo is an acronym for Open Language Model, which is also the name of\u00a0Ai2\u2019s family of open-source LLMs. On the company\u2019s Ai2 Playground site, users can try out OLMoTrace with the recently released OLMo 2 32B model. The open-source code is also available on\u00a0GitHub\u00a0and is\u00a0<\/span>freely available for anyone to use.<\/p>\n\n\n\n Unlike existing approaches focusing on confidence scores or retrieval-augmented generation, OLMoTrace offers a direct window into the relationship between model outputs and the multi-billion-token training datasets that shaped them.<\/p>\n\n\n\n \u201cOur goal is to help users understand why language models generate the responses they do,\u201d Jiacheng Liu, researcher at Ai2 told VentureBeat.<\/p>\n\n\n\n LLMs with web search functionality, like Perplexity or ChatGPT Search, can provide source citations. However, those citations are fundamentally different from what OLMoTrace does.<\/p>\n\n\n\n Liu explained that Perplexity and ChatGPT Search use retrieval-augmented generation (RAG). With RAG, the purpose is to improve the quality of model generation by providing more sources than what the model was trained on. OLMoTrace is different because it traces the output from the model itself without any RAG or external document sources.<\/p>\n\n\n\n The technology identifies long, unique text sequences in model outputs and matches them with specific documents from the training corpus. When a match is found, OLMoTrace highlights the relevant text and provides links to the original source material, allowing users to see exactly where and how the model learned the information it\u2019s using.<\/p>\n\n\n\n By design, LLMs generate outputs based on model weights that help to provide a confidence score. The basic idea is that the higher the confidence score, the more accurate the output.<\/p>\n\n\n\n In Liu\u2019s view, confidence scores are fundamentally flawed.<\/p>\n\n\n\n \u00a0\u201cModels can be overconfident of the stuff they generate and if you ask them to generate a score, it\u2019s usually inflated,\u201d Liu said. \u201cThat\u2019s what academics call a calibration error\u2014the confidence that models output does not always reflect how accurate their responses really are.\u201d<\/p>\n\n\n\n Instead of another potentially misleading score, OLMoTrace provides direct evidence of the model\u2019s learning source, enabling users to make their own informed judgments.<\/p>\n\n\n\n \u201cWhat OLMoTrace does is showing you the matches between model outputs and the training documents,\u201d Liu explained. \u201cThrough the interface, you can directly see where the matching points are and how the model outputs coincide with the training documents.\u201d<\/p>\n\n\n\n Ai2 is not alone in the quest to better understand how LLMs generate output. Anthropic recently released its own research into the issue. That research focused on model internal operations, rather than understanding data.<\/p>\n\n\n\n \u201cWe are taking a different approach from them,\u201d Liu said. \u201cWe are directly tracing into the model behavior, into their training data, as opposed to tracing things into the model neurons, internal circuits, that kind of thing.\u201d<\/p>\n\n\n\n This approach makes OLMoTrace more immediately useful for enterprise applications, as it doesn\u2019t require deep expertise in neural network architecture to interpret the results.<\/p>\n\n\n\n For enterprises deploying AI in regulated industries like healthcare, finance, or legal services, OLMoTrace offers significant advantages over existing black-box systems.<\/p>\n\n\n\n \u201cWe think OLMoTrace will help enterprise and business users to better understand what is used in the training of models so that they can be more confident when they want to build on top of them,\u201d Liu said. \u201cThis can help increase the transparency and trust between them of their models, and also for customers of their model behaviors.\u201d<\/p>\n\n\n\n The technology enables several critical capabilities for enterprise AI teams:<\/p>\n\n\n\n The Ai2 team has already used OLMoTrace to identify and correct their models\u2019 issues.<\/p>\n\n\n\n \u201cWe are already using it to improve our training data,\u201d Liu reveals. \u201cWhen we built OLMo 2 and we started our training, through OLMoTrace, we found out that actually some of the post-training data was not good.\u201d<\/p>\n\n\n\n For enterprises looking to lead the way in AI adoption, OLMoTrace represents a significant step toward more accountable enterprise AI systems. The technology is available under an Apache 2.0 open-source license, which means that any organization with access to its model\u2019s training data can implement similar tracing capabilities.<\/p>\n\n\n\n \u201cOLMoTrace can work on any model, as long as you have the training data of the model,\u201d Liu notes. \u201cFor fully open models where everyone has access to the model\u2019s training data, anyone can set up OLMoTrace for that model and for proprietary models, maybe some providers don\u2019t want to release their data, they can also do this OLMoTrace internally.\u201d<\/p>\n\n\n\n As AI governance frameworks continue to evolve globally, tools like OLMoTrace that enable verification and auditability will likely become essential components of enterprise AI stacks, particularly in regulated industries where algorithmic transparency is increasingly mandated.<\/p>\n\n\n\n For technical decision-makers weighing the benefits and risks of AI adoption, OLMoTrace offers a practical path to implementing more trustworthy and explainable AI systems without sacrificing the power of large language models.<\/p>\n Daily insights on business use cases with VB Daily<\/strong><\/p>\n If you want to impress your boss, VB Daily has you covered. We give you the inside scoop on what companies are doing with generative AI, from regulatory shifts to practical deployments, so you can share insights for maximum ROI.<\/p>\n \n\t\t\t\t\tThanks for subscribing. Check out more VB newsletters here.\n\t\t\t\t<\/p>\n An error occured.<\/p>\n<\/p><\/div>\n
\n<\/div>How OLMoTrace works: More than just citations<\/h2>\n\n\n\n
Beyond confidence scores: Tangible evidence of AI decision-making<\/h2>\n\n\n\n
How OLMoTrace compares to other transparency approaches<\/h2>\n\n\n\n
Enterprise AI applications: From regulatory compliance to model debugging<\/h2>\n\n\n\n
\n
What this means for enterprise AI adoption<\/h2>\n\n\n\n
![\"\"\/](\"https:\/\/venturebeat.com\/wp-content\/themes\/vb-news\/brand\/img\/vb-daily-phone.png\")
\n\t\t\t\t<\/div>\n<\/p><\/div>\n<\/div>\t\t\t<\/div>\r\n