Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More<\/em><\/p>\n\n\n\n Every AI model release inevitably includes charts touting how it outperformed its competitors in this benchmark test or that evaluation matrix.\u00a0<\/p>\n\n\n\n However, these benchmarks often test for general capabilities. For organizations that want to use models and large language model-based agents, it\u2019s harder to evaluate how well the agent or the model actually understands their specific needs.\u00a0<\/p>\n\n\n\n Model repository Hugging Face launched Yourbench, an open-source tool where developers and enterprises can create their own benchmarks to test model performance against their internal data.\u00a0<\/p>\n\n\n\n Sumuk Shashidhar, part of the evaluations research team at Hugging Face, announced Yourbench on X. The feature offers \u201ccustom benchmarking and synthetic data generation from ANY of your documents. It\u2019s a big step towards improving how model evaluations work.\u201d<\/p>\n\n\n\n He added that Hugging Face knows \u201cthat for many use cases what really matters is how well a model performs your specific task. Yourbench lets you evaluate models on what matters to you.\u201d<\/p>\n\n\n\n Hugging Face said in a paper that Yourbench works by replicating subsets of the Massive Multitask Language Understanding (MMLU) benchmark \u201cusing minimal source text, achieving this for under $15 in total inference cost while perfectly preserving the relative model performance rankings.\u201d\u00a0<\/p>\n\n\n\n Organizations need to pre-process their documents before Yourbench can work. This involves three stages: <\/p>\n\n\n\n Next comes the question-and-answer generation process, which creates questions from information on the documents. This is where the user brings in their chosen LLM to see which one best answers the questions.\u00a0<\/p>\n\n\n\n Hugging Face tested Yourbench with DeepSeek V3 and R1 models, Alibaba\u2019s Qwen models including the reasoning model Qwen QwQ, Mistral Large 2411 and Mistral 3.1 Small, Llama 3.1 and Llama 3.3, Gemini 2.0 Flash, Gemini 2.0 Flash Lite and Gemma 3, GPT-4o, GPT-4o-mini, and o3 mini, and Claude 3.7 Sonnet and Claude 3.5 Haiku.<\/p>\n\n\n\n Shashidhar said Hugging Face also offers cost analysis on the models and found that Qwen and Gemini 2.0 Flash \u201cproduce tremendous value for very very low costs.\u201d<\/p>\n\n\n\n However, creating custom LLM benchmarks based on an organization\u2019s documents comes at a cost. Yourbench requires a lot of compute power to work.\u00a0Shashidhar said on X that the company is \u201cadding capacity\u201d as fast they could.<\/p>\n\n\n\n Hugging Face runs several GPUs and partners with companies like Google to use their cloud services for inference tasks. VentureBeat reached out to Hugging Face about Yourbench\u2019s compute usage.<\/p>\n\n\n\n Benchmarks and other evaluation methods give users an idea of how well models perform, but these do not perfectly capture how the models will work daily.<\/p>\n\n\n\n Some have even voiced skepticism that benchmark tests show models\u2019 limitations and can lead to false conclusions about their safety and performance. A study also warned that benchmarking agents could be \u201cmisleading.\u201d<\/p>\n\n\n\n However, enterprises cannot avoid evaluating models now that there are many choices in the market, and technology leaders justify the rising cost of using AI models. This has led to different methods to test model performance and reliability.\u00a0<\/p>\n\n\n\n Google DeepMind introduced FACTS Grounding, which tests a model\u2019s ability to generate factually accurate responses based on information from documents. Some Yale and Tsinghua University researchers developed self-invoking code benchmarks to guide enterprises for which coding LLMs work for them.\u00a0<\/p>\n Daily insights on business use cases with VB Daily<\/strong><\/p>\n If you want to impress your boss, VB Daily has you covered. We give you the inside scoop on what companies are doing with generative AI, from regulatory shifts to practical deployments, so you can share insights for maximum ROI.<\/p>\n \n\t\t\t\t\tThanks for subscribing. Check out more VB newsletters here.\n\t\t\t\t<\/p>\n An error occured.<\/p>\n<\/p><\/div>\n
\n<\/div>![\"\"\/](\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXcXlLX3ONu_kgW9dXwNON1Nlj8dUQEY4-joW5CI-UkzCik8Kyue-nifY7Si2XnimRGTuEFYC3SP3MjylCgsFbFvdRzliejILz5AgapXlKO090Az0FHiBIcaQbkByw5g_fU7ZcawNg?key=DW9pUHI0MxDh2Yj_VryLSCav\")
Creating custom evaluations<\/h2>\n\n\n\n
\n
Compute limitations<\/h2>\n\n\n\n
![\"\"\/](\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXdqLU5ydJ_9Cjmo4Z5n0dzcCGIyhuysS45tjOVFBJvjKLagB9hUpgSsAp3hWO8dQwyvY543TDQ1hK16FwrezALnfPmidXG5tYv8wiUrf578_mjbNRpbICqnU0Bz3lezINev9xBSww?key=DW9pUHI0MxDh2Yj_VryLSCav\")
Benchmarking is not perfect<\/h2>\n\n\n\n
![\"\"\/](\"https:\/\/venturebeat.com\/wp-content\/themes\/vb-news\/brand\/img\/vb-daily-phone.png\")
\n\t\t\t\t<\/div>\n<\/p><\/div>\n<\/div>\t\t\t<\/div>\r\n