\n
![](\"https:\/\/images.ctfassets.net\/jdtwqhzvc2n1\/1P96HBSOE22cfl5djgDueT\/df9398cb8f464a8f6519d964da86026c\/mRfmh7TAQtPYv7mwcN551.png?w=300&q=30\")
<\/p>\n
\n<\/p>\n
There's no shortage of generative AI benchmarks designed to measure the performance and accuracy of a given model on completing various helpful enterprise tasks \u2014 from coding to instruction following to agentic web browsing and tool use. But many of these benchmarks have one major shortcoming: they measure the AI's ability to complete specific problems and requests, not how factual <\/i>the model is in its outputs \u2014 how well it generates objectively correct information tied to real-world data \u2014 especially when dealing with information contained in imagery or graphics.<\/p>\n
For industries where accuracy is paramount \u2014 legal, finance, and medical \u2014 the lack of a standardized way to measure factuality<\/i> has been a critical blind spot.<\/p>\n
That changes today: Google\u2019s FACTS team and its data science unit Kaggle released the FACTS Benchmark Suite, a comprehensive evaluation framework designed to close this gap. <\/p>\n
The associated research paper reveals a more nuanced definition of the problem, splitting "factuality" into two distinct operational scenarios: "contextual factuality" (grounding responses in provided data) and "world knowledge factuality" (retrieving information from memory or the web).<\/p>\n
While the headline news is Gemini 3 Pro\u2019s top-tier placement, the deeper story for builders is the industry-wide "factuality wall."<\/p>\n
According to the initial results, no model\u2014including Gemini 3 Pro, GPT-5, or Claude 4.5 Opus\u2014managed to crack a 70% accuracy score across the suite of problems. For technical leaders, this is a signal: the era of "trust but verify" is far from over.<\/p>\n
The FACTS suite moves beyond simple Q&A. It is composed of four distinct tests, each simulating a different real-world failure mode that developers encounter in production:<\/p>\n
Parametric Benchmark (Internal Knowledge):<\/b> Can the model accurately answer trivia-style questions using only its training data?<\/p>\n<\/li>\n Search Benchmark (Tool Use):<\/b> Can the model effectively use a web search tool to retrieve and synthesize live information?<\/p>\n<\/li>\n Multimodal Benchmark (Vision):<\/b> Can the model accurately interpret charts, diagrams, and images without hallucinating?<\/p>\n<\/li>\n Grounding Benchmark v2 (Context):<\/b> Can the model stick strictly to the provided source text?<\/p>\n<\/li>\n<\/ol>\n Google has released 3,513 examples to the public, while Kaggle holds a private set to prevent developers from training on the test data\u2014a common issue known as "contamination."<\/p>\n The initial run of the benchmark places Gemini 3 Pro in the lead with a comprehensive FACTS Score of 68.8%, followed by Gemini 2.5 Pro (62.1%) and OpenAI\u2019s GPT-5 (61.8%).However, a closer look at the data reveals where the real battlegrounds are for engineering teams.<\/p>\n Model<\/b><\/p>\n<\/td>\n FACTS Score (Avg)<\/b><\/p>\n<\/td>\n Search (RAG Capability)<\/b><\/p>\n<\/td>\n Multimodal (Vision)<\/b><\/p>\n<\/td>\n<\/tr>\n Gemini 3 Pro<\/b><\/p>\n<\/td>\n 68.8<\/b><\/p>\n<\/td>\n 83.8<\/b><\/p>\n<\/td>\n 46.1<\/b><\/p>\n<\/td>\n<\/tr>\n Gemini 2.5 Pro<\/b><\/p>\n<\/td>\n 62.1<\/p>\n<\/td>\n 63.9<\/p>\n<\/td>\n 46.9<\/p>\n<\/td>\n<\/tr>\n GPT-5<\/b><\/p>\n<\/td>\n 61.8<\/p>\n<\/td>\n 77.7<\/p>\n<\/td>\n 44.1<\/p>\n<\/td>\n<\/tr>\n Grok 4<\/b><\/p>\n<\/td>\n 53.6<\/p>\n<\/td>\n 75.3<\/p>\n<\/td>\n 25.7<\/p>\n<\/td>\n<\/tr>\n Claude 4.5 Opus<\/b><\/p>\n<\/td>\n 51.3<\/p>\n<\/td>\n 73.2<\/p>\n<\/td>\n 39.2<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n Data sourced from the FACTS Team release notes.<\/i><\/p>\n For developers building RAG (Retrieval-Augmented Generation) systems, the Search Benchmark is the most critical metric.<\/p>\n The data shows a massive discrepancy between a model's ability to "know" things (Parametric) and its ability to "find" things (Search). For instance, Gemini 3 Pro scores a high 83.8% on Search tasks but only 76.4% on Parametric tasks. <\/p>\n This validates the current enterprise architecture standard: do not rely on a model's internal memory for critical facts.<\/p>\n If you are building an internal knowledge bot, the FACTS results suggest that hooking your model up to a search tool or vector database is not optional\u2014it is the only way to push accuracy toward acceptable production levels.<\/p>\n The most alarming data point for product managers is the performance on Multimodal tasks. The scores here are universally low. Even the category leader, Gemini 2.5 Pro, only hit 46.9% accuracy.<\/p>\n The benchmark tasks included reading charts, interpreting diagrams, and identifying objects in nature. With less than 50% accuracy across the board, this suggests that Multimodal AI is not yet ready for unsupervised data extraction. <\/p>\n Bottom line: <\/b>If your product roadmap involves having an AI automatically scrape data from invoices or interpret financial charts without human-in-the-loop review, you are likely introducing significant error rates<\/b> into your pipeline.<\/p>\n The FACTS Benchmark is likely to become a standard reference point for procurement. When evaluating models for enterprise use, technical leaders should look beyond the composite score and drill into the specific sub-benchmark that matches their use case:<\/p>\n Building a Customer Support Bot? Look at the Grounding score to ensure the bot sticks to your policy documents. (Gemini 2.5 Pro actually outscored Gemini 3 Pro here, 74.2 vs 69.0).<\/p>\n<\/li>\n Building a Research Assistant? Prioritize Search scores.<\/p>\n<\/li>\n Building an Image Analysis Tool? Proceed with extreme caution.<\/p>\n<\/li>\n<\/ul>\n As the FACTS team noted in their release, "All evaluated models achieved an overall accuracy below 70%, leaving considerable headroom for future progress."For now, the message to the industry is clear: The models are getting smarter, but they aren't yet infallible. Design your systems with the assumption that, roughly one-third of the time, the raw model might just be wrong.<\/p>\nThe Leaderboard: A Game of Inches<\/h3>\n
\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n For Builders: The "Search" vs. "Parametric" Gap<\/h3>\n
The Multimodal Warning<\/h3>\n
Why This Matters for Your Stack<\/h3>\n
\n