Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More<\/em><\/p>\n\n\n\n Amazon Web Services today introduced SWE-PolyBench, a comprehensive multi-language benchmark designed to evaluate AI coding assistants across a diverse range of programming languages and real-world scenarios. The benchmark addresses significant limitations in existing evaluation frameworks and offers researchers and developers new ways to assess how effectively AI agents navigate complex codebases.<\/p>\n\n\n\n \u201cNow they have a benchmark that they can evaluate on to assess whether the coding agents are able to solve complex programming tasks,\u201d said Anoop Deoras, Director of Applied Sciences for Generative AI Applications and Developer Experiences at AWS, in an interview with VentureBeat. \u201cThe real world offers you more complex tasks. In order to fix a bug or do feature building, you need to touch multiple files, as opposed to a single file.\u201d<\/p>\n\n\n\n The release comes as AI-powered coding tools have exploded in popularity, with major technology companies integrating them into development environments and standalone products. While these tools show impressive capabilities, evaluating their performance has remained challenging \u2014 particularly across different programming languages and varying task complexities.<\/p>\n\n\n\n SWE-PolyBench contains over 2,000 curated coding challenges derived from real GitHub issues spanning four languages: Java (165 tasks), JavaScript (1,017 tasks), TypeScript (729 tasks), and Python (199 tasks). The benchmark also includes a stratified subset of 500 issues (SWE-PolyBench500) designed for quicker experimentation.<\/p>\n\n\n\n \u201cThe task diversity and the diversity of the programming languages was missing,\u201d Deoras explained about existing benchmarks. \u201cIn SWE-Bench today, there is only a single programming language, Python, and there is a single task: bug fixes. In PolyBench, as opposed to SWE-Bench, we have expanded this benchmark to include three additional languages.\u201d<\/p>\n\n\n\n The new benchmark directly addresses limitations in SWE-Bench, which has emerged as the de facto standard for coding agent evaluation with over 50 leaderboard submissions. Despite its pioneering role, SWE-Bench focuses solely on Python repositories, predominantly features bug-fixing tasks, and is significantly skewed toward a single codebase \u2014 the Django repository accounts for over 45% of all tasks.<\/p>\n\n\n\n \u201cIntentionally, we decided to have a little bit over representation for JavaScript and TypeScript, because we do have SWE-Bench which has Python tasks already,\u201d Deoras noted. \u201cSo rather than over representing on Python, we made sure that we have enough representations for JavaScript and TypeScript in addition to Java.\u201d<\/p>\n\n\n\n A key innovation in SWE-PolyBench is its introduction of more sophisticated evaluation metrics beyond the traditional \u201cpass rate,\u201d which simply measures whether a generated patch successfully resolves a coding issue.<\/p>\n\n\n\n \u201cThe evaluation of these coding agents have primarily been done through the metric called pass rate,\u201d Deoras said. \u201cPass rate, in short, is basically just a proportion of the tasks that successfully run upon the application of the patch that the agents are producing. But this number is a very high level, aggregated statistic. It doesn\u2019t tell you the nitty gritty detail, and in particular, it doesn\u2019t tell you how the agent came to that resolution.\u201d<\/p>\n\n\n\n The new metrics include file-level localization, which assesses an agent\u2019s ability to identify which files need modification within a repository, and Concrete Syntax Tree (CST) node-level retrieval, which evaluates how accurately an agent can pinpoint specific code structures requiring changes.<\/p>\n\n\n\n \u201cIn addition to pass rate, we have the precision and recall. And in order to get to the precision and recall metric, we are looking at a program analysis tool called concrete syntax tree,\u201d Deoras explained. \u201cIt is telling you how your core file structure is composed, so that you can look at what is the class node, and within that class, what are the function nodes and the variables.\u201d<\/p>\n\n\n\n Amazon\u2019s evaluation of several open-source coding agents on SWE-PolyBench revealed several patterns. Python remains the strongest language for all tested agents, likely due to its prevalence in training data and existing benchmarks. Performance degrades as task complexity increases, particularly when modifications to three or more files are required.<\/p>\n\n\n\n Different agents show varying strengths across task categories. While performance on bug-fixing tasks is relatively consistent, there\u2019s more variability between agents when handling feature requests and code refactoring.<\/p>\n\n\n\n The benchmark also found that the informativeness of problem statements significantly impacts success rates, suggesting that clear issue descriptions remain crucial for effective AI assistance.<\/p>\n\n\n\n SWE-PolyBench arrives at a critical juncture in the development of AI coding assistants. As these tools move from experimental to production environments, the need for rigorous, diverse, and representative benchmarks has intensified.<\/p>\n\n\n\n \u201cOver time, not only the capabilities of LLMs have evolved, but at the same time, the tasks have gotten more and more complex,\u201d Deoras observed. \u201cThere is a need for developers to solve more and more complex tasks in a synchronous manner using these agents.\u201d<\/p>\n\n\n\n The benchmark\u2019s expanded language support makes it particularly valuable for enterprise environments where polyglot development is common. Java, JavaScript, TypeScript, and Python consistently rank among the most popular programming languages in enterprise settings, making SWE-PolyBench\u2019s coverage highly relevant to real-world development scenarios.<\/p>\n\n\n\n\n\n\n\n Amazon has made the entire SWE-PolyBench framework publicly available. The dataset is accessible on Hugging Face, and the evaluation harness is available on GitHub. A dedicated leaderboard has been established to track the performance of various coding agents on the benchmark.<\/p>\n\n\n\n \u201cWe extended the SWE-Bench data acquisition pipeline to support these three additional languages,\u201d Deoras said. \u201cThe hope is that we will be able to extrapolate this process further in the future and extend beyond four languages, extend beyond the three tasks that I talked about, so that this benchmark becomes even more comprehensive.\u201d<\/p>\n\n\n\n As the AI coding assistant market heats up with offerings from every major tech company, SWE-PolyBench provides a crucial reality check on their actual capabilities. The benchmark\u2019s design acknowledges that real-world software development demands more than simple bug fixes in Python\u2014it requires working across languages, understanding complex codebases, and tackling diverse engineering challenges.<\/p>\n\n\n\n For enterprise decision-makers evaluating AI coding tools, SWE-PolyBench offers something invaluable: a way to separate marketing hype from genuine technical capability. After all, the true test of an AI coding assistant isn\u2019t how well it performs on simplified demos, but whether it can handle the messy, multi-language complexity of actual software projects \u2014 the kind developers wrestle with every day.<\/p>\n Daily insights on business use cases with VB Daily<\/strong><\/p>\n If you want to impress your boss, VB Daily has you covered. We give you the inside scoop on what companies are doing with generative AI, from regulatory shifts to practical deployments, so you can share insights for maximum ROI.<\/p>\n \n\t\t\t\t\tThanks for subscribing. Check out more VB newsletters here.\n\t\t\t\t<\/p>\n An error occured.<\/p>\n<\/p><\/div>\n
\n<\/div>Why simple pass\/fail metrics don\u2019t tell the whole story about AI coding performance<\/h2>\n\n\n\n
How Python remains dominant while complex tasks expose AI limitations<\/h2>\n\n\n\n
What SWE-PolyBench means for enterprise developers working across multiple languages<\/h2>\n\n\n\n
![\"\"\/](\"https:\/\/venturebeat.com\/wp-content\/themes\/vb-news\/brand\/img\/vb-daily-phone.png\")
\n\t\t\t\t<\/div>\n<\/p><\/div>\n<\/div>\t\t\t<\/div>\r\n