\n
![](\"https:\/\/images.ctfassets.net\/jdtwqhzvc2n1\/6wvVdQj4t7NiC0a8xS9bQu\/8b37da63cebde173a1c338aeb371eba8\/cfr0z3n_sharp_detailed_graphic_novel_style_splash_page_bicolore_b6659495-c268-49de-9a52-76abb547bf5a.png\")
<\/p>\n
\n<\/p>\n
It seems like almost every week for the last two years since ChatGPT launched, new large language models (LLMs) from rival labs or from OpenAI itself have been released. Enterprises are hard pressed to keep up with the massive pace of change, let alone understand how to adapt to it \u2014 which of these new models should they adopt, if any, to power their workflows and the custom AI agents they're building to carry them out? <\/p>\n
Help has arrived: AI applications observability startup <\/b>Raindrop<\/b> has launched Experiments<\/b>, a new analytics feature that the company describes as the first A\/B testing suite designed specifically for enterprise AI agents \u2014 allowing companies to see and compare how updating agents to new underlying models, or changing their instructions and tool access, will impact their performance with real end users. <\/p>\n
The release extends Raindrop\u2019s existing observability tools, giving developers and teams a way to see how their agents behave and evolve in real-world conditions.<\/p>\n
With Experiments, teams can track how changes \u2014 such as a new tool, prompt, model update, or full pipeline refactor \u2014 affect AI performance across millions of user interactions. The new feature is available now for users on Raindrop\u2019s Pro subscription plan ($350 monthly) at raindrop.ai. <\/p>\n
Raindrop co-founder and chief technology officer Ben Hylak<\/b> noted in a product announcement video (above) that Experiments helps teams see \u201chow literally anything changed,\u201d including tool usage, user intents, and issue rates, and to explore differences by demographic factors such as language. The goal is to make model iteration more transparent and measurable.<\/p>\n
The Experiments interface presents results visually, showing when an experiment performs better or worse than its baseline. Increases in negative signals might indicate higher task failure or partial code output, while improvements in positive signals could reflect more complete responses or better user experiences.<\/p>\n
By making this data easy to interpret, Raindrop encourages AI teams to approach agent iteration with the same rigor as modern software deployment\u2014tracking outcomes, sharing insights, and addressing regressions before they compound.<\/p>\n
Raindrop\u2019s launch of Experiments builds on the company\u2019s foundation as one of the first AI-native observability platforms<\/b>, designed to help enterprises monitor and understand how their generative AI systems behave in production. <\/p>\n
As VentureBeat reported earlier this year, the company \u2014 originally known as Dawn AI \u2014 emerged to address what Hylak, <\/b>a former Apple human interface designer, called the \u201cblack box problem\u201d of AI performance, helping teams catch failures \u201cas they happen and explain to enterprises what went wrong and why." <\/p>\n
At the time, Hylak described how \u201cAI products fail constantly\u2014in ways both hilarious and terrifying,\u201d noting that unlike traditional software, which throws clear exceptions, \u201cAI products fail silently.\u201d Raindrop\u2019s original platform focused on detecting those silent failures by analyzing signals such as user feedback, task failures, refusals, and other conversational anomalies across millions of daily events.<\/p>\n
The company\u2019s co-founders\u2014 Hylak, Alexis Gauba<\/b>, and Zubin Singh Koticha <\/b>\u2014 built Raindrop after encountering firsthand the difficulty of debugging AI systems in production. <\/p>\n
\u201cWe started by building AI products, not infrastructure,\u201d Hylak told VentureBeat<\/i>. \u201cBut pretty quickly, we saw that to grow anything serious, we needed tooling to understand AI behavior\u2014and that tooling didn\u2019t exist.\u201d<\/p>\n
With Experiments, Raindrop extends that same mission from detecting failures<\/b> to measuring improvements<\/b>. The new tool transforms observability data into actionable comparisons, letting enterprises test whether changes to their models, prompts, or pipelines actually make their AI agents better\u2014or just different.<\/p>\nSolving the \u201cEvals Pass, Agents Fail\u201d Problem<\/b><\/h3>\n
Traditional evaluation frameworks, while useful for benchmarking, rarely capture the unpredictable behavior of AI agents operating in dynamic environments. <\/p>\n
As Raindrop co-founder Alexis Gauba<\/b> explained in her LinkedIn announcement, \u201cTraditional evals don\u2019t really answer this question. They\u2019re great unit tests, but you can\u2019t predict your user\u2019s actions and your agent is running for hours, calling hundreds of tools.\u201d<\/p>\n
Gauba said the company consistently heard a common frustration from teams: \u201cEvals pass, agents fail.\u201d<\/b><\/p>\n
Experiments is meant to close that gap by showing what actually changes<\/i> when developers ship updates to their systems. <\/p>\n
The tool enables side-by-side comparisons of models, tools, intents, or properties, surfacing measurable differences in behavior and performance.<\/p>\n
In the announcement video, Raindrop described Experiments as a way to \u201ccompare anything and measure how your agent\u2019s behavior actually changed in production across millions of real interactions.\u201d<\/p>\n
The platform helps users spot issues such as task failure spikes, forgetting, or new tools that trigger unexpected errors. <\/p>\n
It can also be used in reverse \u2014 starting from a known problem, such as an \u201cagent stuck in a loop,\u201d and tracing back to which model, tool, or flag is driving it. <\/p>\n
From there, developers can dive into detailed traces to find the root cause and ship a fix quickly.<\/p>\n
Each experiment provides a visual breakdown of metrics like tool usage frequency, error rates, conversation duration, and response length. <\/p>\n
Users can click on any comparison to access the underlying event data, giving them a clear view of how agent behavior changed over time. Shared links make it easy to collaborate with teammates or report findings.<\/p>\n
According to Hylak, Experiments integrates directly with \u201cthe feature flag platforms companies know and love (like Statsig!)\u201d and is designed to work seamlessly with existing telemetry and analytics pipelines. <\/p>\n
For companies without those integrations, it can still compare performance over time\u2014such as yesterday versus today\u2014without additional setup.<\/p>\n
Hylak said teams typically need around 2,000 users per day to produce statistically meaningful results. <\/p>\n
To ensure the accuracy of comparisons, Experiments monitors for sample size adequacy and alerts users if a test lacks enough data to draw valid conclusions.<\/p>\n
\u201cWe obsess over making sure metrics like Task Failure and User Frustration are metrics that you\u2019d wake up an on-call engineer for,\u201d Hylak explained. He added that teams can drill into the specific conversations or events that drive those metrics, ensuring transparency behind every aggregate number.<\/p>\n
Raindrop operates as a cloud-hosted platform but also offers on-premise personally identifiable information (PII) redaction for enterprises that need additional control. <\/p>\n
Hylak said the company is SOC 2 compliant and has launched a PII Guard<\/b> feature that uses AI to automatically remove sensitive information from stored data. \u201cWe take protecting customer data very seriously,\u201d he emphasized.<\/p>\nPricing and Plans<\/b><\/h3>\n
Experiments is part of Raindrop\u2019s Pro plan<\/b>, which costs $350 per month or $0.0007 per interaction. The Pro tier also includes deep research tools, topic clustering, custom issue tracking, and semantic search capabilities.<\/p>\n
Raindrop\u2019s Starter plan <\/b>\u2014 $65 per month or $0.001 per interaction \u2014 offers core analytics including issue detection, user feedback signals, Slack alerts, and user tracking. Both plans come with a 14-day free trial.<\/p>\n
Larger organizations can opt for an Enterprise plan<\/b> with custom pricing and advanced features like SSO login, custom alerts, integrations, edge-PII redaction, and priority support.<\/p>\nContinuous Improvement for AI Systems<\/b><\/h3>\n
With Experiments, Raindrop positions itself at the intersection of AI analytics and software observability. Its focus on \u201cmeasure truth,\u201d as stated in the product video, reflects a broader push within the industry toward accountability and transparency in AI operations.<\/p>\n
Rather than relying solely on offline benchmarks, Raindrop\u2019s approach emphasizes real user data and contextual understanding. The company hopes this will allow AI developers to move faster, identify root causes sooner, and ship better-performing models with confidence. <\/p>\n