Join the event trusted by enterprise leaders for nearly two decades. VB Transform brings together the people building real enterprise AI strategy.\u00a0Learn more<\/em><\/p>\n\n\n\n Headlines have been blaring it for years: Large language models (LLMs) can not only pass medical licensing exams but also outperform humans. GPT-4 could correctly answer U.S. medical exam licensing questions 90% of the time, even in the prehistoric AI days of 2023. Since then, LLMs have gone on to best the residents taking those exams and licensed physicians.<\/p>\n\n\n\n Move over, Doctor Google, make way for ChatGPT, M.D. But you may want more than a diploma from the LLM you deploy for patients. Like an ace medical student who can rattle off the name of every bone in the hand but faints at the first sight of real blood, an LLM\u2019s mastery of medicine does not always translate directly into the real world.<\/p>\n\n\n\n A paper by researchers at the University of Oxford found that while LLMs could correctly identify relevant conditions 94.9% of the time when directly presented with test scenarios, human participants using LLMs to diagnose the same scenarios identified the correct conditions less than 34.5% of the time.<\/p>\n\n\n\n Perhaps even more notably, patients using LLMs performed even worse than a control group that was merely instructed to diagnose themselves using \u201cany methods they would typically employ at home.\u201d The group left to their own devices was 76% more likely to identify the correct conditions than the group assisted by LLMs.<\/p>\n\n\n\n The Oxford study raises questions about the suitability of LLMs for medical advice and the benchmarks we use to evaluate chatbot deployments for various applications.<\/p>\n\n\n\n Led by Dr. Adam Mahdi, researchers at Oxford recruited 1,298 participants to present themselves as patients to an LLM. They were tasked with both attempting to figure out what ailed them and the appropriate level of care to seek for it, ranging from self-care to calling an ambulance.<\/p>\n\n\n\n Each participant received a detailed scenario, representing conditions from pneumonia to the common cold, along with general life details and medical history. For instance, one scenario describes a 20-year-old engineering student who develops a crippling headache on a night out with friends. It includes important medical details (it\u2019s painful to look down) and red herrings (he\u2019s a regular drinker, shares an apartment with six friends, and just finished some stressful exams).<\/p>\n\n\n\n The study tested three different LLMs. The researchers selected GPT-4o on account of its popularity, Llama 3 for its open weights and Command R+ for its retrieval-augmented generation (RAG) abilities, which allow it to search the open web for help.<\/p>\n\n\n\n Participants were asked to interact with the LLM at least once using the details provided, but could use it as many times as they wanted to arrive at their self-diagnosis and intended action.<\/p>\n\n\n\n Behind the scenes, a team of physicians unanimously decided on the \u201cgold standard\u201d conditions they sought in every scenario, and the corresponding course of action. Our engineering student, for example, is suffering from a subarachnoid haemorrhage, which should entail an immediate visit to the ER.<\/p>\n\n\n\n While you might assume an LLM that can ace a medical exam would be the perfect tool to help ordinary people self-diagnose and figure out what to do, it didn\u2019t work out that way. \u201cParticipants using an LLM identified relevant conditions less consistently than those in the control group, identifying at least one relevant condition in at most 34.5% of cases compared to 47.0% for the control,\u201d the study states. They also failed to deduce the correct course of action, selecting it just 44.2% of the time, compared to 56.3% for an LLM acting independently.<\/p>\n\n\n\n What went wrong?<\/p>\n\n\n\n Looking back at transcripts, researchers found that participants both provided incomplete information to the LLMs and the LLMs misinterpreted their prompts. For instance, one user who was supposed to exhibit symptoms of gallstones merely told the LLM: \u201cI get severe stomach pains lasting up to an hour, It can make me vomit and seems to coincide with a takeaway,\u201d omitting the location of the pain, the severity, and the frequency. Command R+ incorrectly suggested that the participant was experiencing indigestion, and the participant incorrectly guessed that condition.<\/p>\n\n\n\n Even when LLMs delivered the correct information, participants didn\u2019t always follow its recommendations. The study found that 65.7% of GPT-4o conversations suggested at least one relevant condition for the scenario, but somehow less than 34.5% of final answers from participants reflected those relevant conditions.<\/p>\n\n\n\n This study is useful, but not surprising, according to Nathalie Volkheimer, a user experience specialist at the Renaissance Computing Institute (RENCI), University of North Carolina at Chapel Hill.<\/p>\n\n\n\n \u201cFor those of us old enough to remember the early days of internet search, this is d\u00e9j\u00e0 vu,\u201d she says. \u201cAs a tool, large language models require prompts to be written with a particular degree of quality, especially when expecting a quality output.\u201d<\/p>\n\n\n\n She points out that someone experiencing blinding pain wouldn\u2019t offer great prompts. Although participants in a lab experiment weren\u2019t experiencing the symptoms directly, they weren\u2019t relaying every detail.<\/p>\n\n\n\n \u201cThere is also a reason why clinicians who deal with patients on the front line are trained to ask questions in a certain way and a certain repetitiveness,\u201d Volkheimer goes on. Patients omit information because they don\u2019t know what\u2019s relevant, or at worst, lie because they\u2019re embarrassed or ashamed.<\/p>\n\n\n\n Can chatbots be better designed to address them? \u201cI wouldn\u2019t put the emphasis on the machinery here,\u201d Volkheimer cautions. \u201cI would consider the emphasis should be on the human-technology interaction.\u201d The car, she analogizes, was built to get people from point A to B, but many other factors play a role. \u201cIt\u2019s about the driver, the roads, the weather, and the general safety of the route. It isn\u2019t just up to the machine.\u201d<\/p>\n\n\n\n The Oxford study highlights one problem, not with humans or even LLMs, but with the way we sometimes measure them\u2014in a vacuum.<\/p>\n\n\n\n When we say an LLM can pass a medical licensing test, real estate licensing exam, or a state bar exam, we\u2019re probing the depths of its knowledge base using tools designed to evaluate humans. However, these measures tell us very little about how successfully these chatbots will interact with humans.<\/p>\n\n\n\n \u201cThe prompts were textbook (as validated by the source and medical community), but life and people are not textbook,\u201d explains Dr. Volkheimer.<\/p>\n\n\n\n Imagine an enterprise about to deploy a support chatbot trained on its internal knowledge base. One seemingly logical way to test that bot might simply be to have it take the same test the company uses for customer support trainees: answering prewritten \u201ccustomer\u201d support questions and selecting multiple-choice answers. An accuracy of 95% would certainly look pretty promising.<\/p>\n\n\n\n Then comes deployment: Real customers use vague terms, express frustration, or describe problems in unexpected ways. The LLM, benchmarked only on clear-cut questions, gets confused and provides incorrect or unhelpful answers. It hasn\u2019t been trained or evaluated on de-escalating situations or seeking clarification effectively. Angry reviews pile up. The launch is a disaster, despite the LLM sailing through tests that seemed robust for its human counterparts.<\/p>\n\n\n\n This study serves as a critical reminder for AI engineers and orchestration specialists: if an LLM is designed to interact with humans, relying solely on non-interactive benchmarks can create a dangerous false sense of security about its real-world capabilities. If you\u2019re designing an LLM to interact with humans, you need to test it with humans \u2013 not tests for humans. But is there a better way?<\/p>\n\n\n\n The Oxford researchers recruited nearly 1,300 people for their study, but most enterprises don\u2019t have a pool of test subjects sitting around waiting to play with a new LLM agent. So why not just substitute AI testers for human testers?<\/p>\n\n\n\n Mahdi and his team tried that, too, with simulated participants. \u201cYou are a patient,\u201d they prompted an LLM, separate from the one that would provide the advice. \u201cYou have to self-assess your symptoms from the given case vignette and assistance from an AI model. Simplify terminology used in the given paragraph to layman language and keep your questions or statements reasonably short.\u201d The LLM was also instructed not to use medical knowledge or generate new symptoms.<\/p>\n\n\n\n These simulated participants then chatted with the same LLMs the human participants used. But they performed much better. On average, simulated participants using the same LLM tools nailed the relevant conditions 60.7% of the time, compared to below 34.5% in humans.<\/p>\n\n\n\n In this case, it turns out LLMs play nicer with other LLMs than humans do, which makes them a poor predictor of real-life performance.<\/p>\n\n\n\n Given the scores LLMs could attain on their own, it might be tempting to blame the participants here. After all, in many cases, they received the right diagnoses in their conversations with LLMs, but still failed to correctly guess it. But that would be a foolhardy conclusion for any business, Volkheimer warns.<\/p>\n\n\n\n \u201cIn every customer environment, if your customers aren\u2019t doing the thing you want them to, the last thing you do is blame the customer,\u201d says Volkheimer. \u201cThe first thing you do is ask why. And not the \u2018why\u2019 off the top of your head: but a deep investigative, specific, anthropological, psychological, examined \u2018why.\u2019 That\u2019s your starting point.\u201d<\/p>\n\n\n\n You need to understand your audience, their goals, and the customer experience before deploying a chatbot, Volkheimer suggests. All of these will inform the thorough, specialized documentation that will ultimately make an LLM useful. Without carefully curated training materials, \u201cIt\u2019s going to spit out some generic answer everyone hates, which is why people hate chatbots,\u201d she says. When that happens, \u201cIt\u2019s not because chatbots are terrible or because there\u2019s something technically wrong with them. It\u2019s because the stuff that went in them is bad.\u201d<\/p>\n\n\n\n \u201cThe people designing technology, developing the information to go in there and the processes and systems are, well, people,\u201d says Volkheimer. \u201cThey also have background, assumptions, flaws and blindspots, as well as strengths. And all those things can get built into any technological solution.\u201d<\/p>\n Daily insights on business use cases with VB Daily<\/strong><\/p>\n If you want to impress your boss, VB Daily has you covered. We give you the inside scoop on what companies are doing with generative AI, from regulatory shifts to practical deployments, so you can share insights for maximum ROI.<\/p>\n \n\t\t\t\t\tThanks for subscribing. Check out more VB newsletters here.\n\t\t\t\t<\/p>\n An error occured.<\/p>\n<\/p><\/div>\n
\n<\/div>Guess your malady<\/h2>\n\n\n\n
A game of telephone<\/h2>\n\n\n\n
The human variable<\/h2>\n\n\n\n
A better yardstick<\/h2>\n\n\n\n
Using AI to test AI<\/h2>\n\n\n\n
Don\u2019t blame the user<\/h2>\n\n\n\n
![\"\"\/](\"https:\/\/venturebeat.com\/wp-content\/themes\/vb-news\/brand\/img\/vb-daily-phone.png\")
\n\t\t\t\t<\/div>\n<\/p><\/div>\n<\/div>\t\t\t<\/div>\r\n