Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More<\/em><\/p>\n\n\n\n Patronus AI announced today the launch of what it calls the industry\u2019s first multimodal large language model-as-a-judge (MLLM-as-a-Judge), a tool designed to evaluate AI systems that interpret images and produce text.<\/p>\n\n\n\n The new evaluation technology aims to help developers detect and mitigate hallucinations and reliability issues in multimodal AI applications. E-commerce giant Etsy has already implemented the technology to verify caption accuracy for product images across its marketplace of handmade and vintage goods.<\/p>\n\n\n\n \u201cSuper excited to announce that Etsy is one of our ship customers,\u201d said Anand Kannappan, cofounder of Patronus AI, in an exclusive interview with VentureBeat. \u201cThey have hundreds of millions of items in their online marketplace for handmade and vintage products that people are creating around the world. One of the things that their AI team wanted to be able to leverage generative AI for was the ability to auto-generate image captions and to make sure that as they scale across their entire global user base, that the captions that are generated are ultimately correct.\u201d<\/p>\n\n\n\n Patronus built its first MLLM-as-a-Judge, called Judge-Image, on Google\u2019s Gemini model after extensive research comparing it with alternatives like OpenAI\u2019s GPT-4V.<\/p>\n\n\n\n \u201cWe tended to see that there was a slighter preference toward egocentricity with GPT-4V, whereas we saw that Gemini was less biased in those ways and had more of an equitable approach to being able to judge different kinds of input-output pairs,\u201d Kannappan explained. \u201cThat was seen in the uniform scoring distribution across the different sources that they looked at.\u201d<\/p>\n\n\n\n The company\u2019s research yielded another surprising insight about multimodal evaluation. Unlike text-only evaluations where multi-step reasoning often improves performance, Kannappan noted that it \u201ctypically doesn\u2019t actually increase MLLM judge performance\u201d for image-based assessments.<\/p>\n\n\n\n Judge-Image provides ready-to-use evaluators that assess image captions on multiple criteria, including caption hallucination detection, recognition of primary and non-primary objects, object location accuracy, and text detection and analysis.<\/p>\n\n\n\n While Etsy represents a flagship customer in e-commerce, Patronus sees applications extending far beyond retail.<\/p>\n\n\n\n These include \u201cmarketing teams across companies that are generally looking at being able to scalably create descriptions and captions against new blocks in design, especially marketing design, but also product design,\u201d Kannappan said.<\/p>\n\n\n\n He also highlighted applications for enterprises dealing with document processing: \u201cLarger enterprises like venture services companies and law firms typically might have engineering teams that are using relatively legacy technology to be able to extract different kinds of information from PDFs, to be able to summarize the content inside of larger documents.\u201d<\/p>\n\n\n\n\n\n\n\n As AI becomes increasingly critical to business processes, many companies face the build-versus-buy dilemma for evaluation tools. Kannappan argues that outsourcing AI evaluation makes strategic and economic sense.<\/p>\n\n\n\n \u201cAs we\u2019ve worked with teams, [we\u2019ve found that] a lot of folks may start with something to see if they can develop something internally, and then they realize that it\u2019s, one, not core to their value prop or the product they\u2019re developing. And two, it is a very challenging problem, both from an AI perspective, but also from an infrastructure perspective,\u201d he said.<\/p>\n\n\n\n This applies particularly to multimodal systems, where failures can occur at multiple points in the process. \u201cWhen you\u2019re dealing with RAG systems or agents, or even multimodal AI systems, we\u2019re seeing that failures happen across all parts of the system,\u201d Kannappan noted.<\/p>\n\n\n\n Patronus offers multiple pricing tiers, starting with a free option that allows users to experiment with the platform up to certain volume limits. Beyond that threshold, customers pay as they go for evaluator usage or can engage with the sales team for enterprise arrangements with custom features and tailored pricing.<\/p>\n\n\n\n Despite using Google\u2019s Gemini model as its foundation, the company positions itself as complementary rather than competitive with foundation model providers like Google, OpenAI and Anthropic.<\/p>\n\n\n\n \u201cWe don\u2019t necessarily see the technology that we build or the solutions that we build as competitive with foundational companies, but rather very complementary and additional new powerful tools in the toolkit that ultimately help folks develop better LLM systems, as opposed to LLMs themselves,\u201d Kannappan said.<\/p>\n\n\n\n Today\u2019s announcement represents one step in Patronus\u2019s broader strategy for AI evaluation across different modalities. The company plans to expand beyond images into audio evaluation soon.<\/p>\n\n\n\n \u201cWe\u2019re excited because this is the next phase of our vision towards multimodal, and specifically focused on images today \u2014 and then over time, we\u2019re excited about what we\u2019ll do, especially with audio in the future,\u201d Kannappan confirmed.<\/p>\n\n\n\n This roadmap aligns with what Kannappan describes as the company\u2019s \u201cresearch vision towards scalable oversight\u201d \u2014 developing evaluation mechanisms that can keep pace with increasingly sophisticated AI systems.<\/p>\n\n\n\n \u201cWe continue to develop new systems, products, frameworks, methods that ultimately are equally capable as the intelligent systems that we intend to want to have oversight over as humans in the long run,\u201d he said.<\/p>\n\n\n\n As businesses race to deploy AI systems that can interpret images, extract text from documents, and generate visual content, the risk of inaccuracies, hallucinations and biases grows. Patronus is betting that even as foundation models improve, the challenges of evaluating complex multimodal AI systems will remain \u2014 requiring specialized tools that can serve as impartial judges of increasingly human-like AI output. In the high-stakes world of commercial AI deployment, these digital judges may prove as valuable as the models they evaluate.<\/p>\n Daily insights on business use cases with VB Daily<\/strong><\/p>\n If you want to impress your boss, VB Daily has you covered. We give you the inside scoop on what companies are doing with generative AI, from regulatory shifts to practical deployments, so you can share insights for maximum ROI.<\/p>\n \n\t\t\t\t\tThanks for subscribing. Check out more VB newsletters here.\n\t\t\t\t<\/p>\n An error occured.<\/p>\n<\/p><\/div>\n
\n<\/div>Why Google\u2019s Gemini powers the new AI judge rather than OpenAI<\/h2>\n\n\n\n
Beyond retail: How marketing teams and law firms can benefit from AI image evaluation<\/h2>\n\n\n\n
How Patronus plans to make money while competing with tech giants<\/h2>\n\n\n\n
Audio evaluation coming next as Patronus expands multimodal oversight<\/h2>\n\n\n\n
![\"\"\/](\"https:\/\/venturebeat.com\/wp-content\/themes\/vb-news\/brand\/img\/vb-daily-phone.png\")
\n\t\t\t\t<\/div>\n<\/p><\/div>\n<\/div>\t\t\t<\/div>\r\n