Name: How to evaluate voice agents: execution, outcomes, and experience
Uploaded: 2026-09-23T10:30:45.722Z
Duration: 40 min 32 s
Description: How to evaluate voice agents: execution, outcomes, and experience

Transcript for "How to evaluate voice agents: execution, outcomes, and experience": Good morning, everyone. Okay. Let me share my screen. Okay. Fantastic. Hello, everyone. Today, we are going to talk about evaluating voice agents. Voice agents are a rapidly growing field as the real time voice models are developed better and with lower latency. However, voice agents are tricky to build and evaluate, and so we are going to dive into details on how to do that effectively in order to have a robust, reliable production agents. Cool. Before we get into it, a quick bit of bit of context about Langchain. Langchain is a platform for agent development life cycle, and we sit at the center at how AI teams are building, deploying, and managing agents in production. LangSmith is our platform for agent engineering, and we have over 7,000 active customers. We also have widespread open source frameworks, namely Langtree and Langgraph with over 200,000,000 monthly downloads. And we truly work with incredible companies from high growth startups like Harvey and Rippling to large enterprises like Nvidia and ServiceNow, and we are used by half of the Fortune 10, cutting across many regulated industry verticals from financial services to HR and legal. I am Marta Stangirska, and I'm a deployed engineer at Langchain. Today, I am cohosting with Stephan Hausman. Hello. Okay. So we are building and deploying more and more voice agents, especially customer experience and customer support agents. Together, there is, of course, a number of use cases for voice agents, but customer support is one that fits pretty well because voice agents can resolve customer issues quickly around the clock and ideally without needing to escalate to a human. But in order for them to be effective at this, the customer experience needs to be very good. We all interacted with a voice agent and, that had issues, and we all know how frustrating that is. And when this happens, it is actually counterproductive because users request to speak to a human or drop off the call completely, which is either bad for your brand or a missed opportunity at revenue. Now we all dream of building good agents, but getting to a good voice agent experience is really hard. Being able to build that voice agent with a good natural sounding voice, low latency, and high accuracy is something that is an incredibly challenging AI engineering problem. Making improvement starts with understanding what failures, occurring. That is it starts with evaluations. So let's talk about evaluating voice agents. But before we jump into what good evaluations look like, we should talk about why is it that voice agents are hard to evaluate. Okay. So voice agents break differently, and this is why voice agents' evils are hard. So the first reason that voice agents are hard to evaluate is that the thing we're actually trying to evaluate, which is the audio, has much more signal than text. We are not only evaluating the transcript. In a voice agent, you would evaluate the conversation itself. So did the agent do the right thing? Did it call the right tools? But on top of that, you are also evaluating a number of failures that only exist in audio. So did the agent talk over the human? What was the tone of the conversation, and did the agent mispronounced anything? Was there a lot of dead air or awkward silences? And these are the things that do not occur in, the text agent. There are also things that do not show up in transcript, hence you have to evaluate the audio directly. Okay? So conversations also have many turns. Each turn is short and doesn't complete on its own. So the typical flow of the conversation would go, an agent ask a question, a user answers, the agent replies again, but there can be many silences that, the user can interrupt or cough, and that will not show up in the transcript. And the third thing is what would also show up in the text agent that no single metric or aggregate score can capture conversation success. So there are many different things that can go well and many different things that that can go wrong in a voice agent. And all of those things everything has to go right in order for us to consider a voice agent conversation successful. Alright. So the first reason that I stated that evals on voice agents are hard because audio has so much more signal than text. This means that evaluating the transcript alone isn't going to be enough about, isn't going to be enough to determine whether or not the conversation was successful. And there are multiple reasons for this. First of all, evaluating just the transcript won't catch transcription errors. For example, the agent might have responded correctly given the transcript, But if the transcript doesn't accurately represent what the agent what the user said, the user is going to get frustrated. The conversation is going to go sideways, and the transcript won't reflect the actual underlying cause for whether the conversation went wrong. Second, the transcript won't capture tone and sentiment. Those are huge pieces of voice conversation. A sarcastic customer, a frustrated customer will play a huge part in the user's experience and come across in the audio, but that won't be accurately trans captured by the transcript. And lastly, audio failures won't be captured. Things like the agent speaking over the users, the interruption, the dead, the long awkward pauses or mispronunciations. None of these failures are going to be visible in the transcript, and yet they play a big part in the user's experience. Great. So in order to provide some structure around how to think about voice evaluations, I propose thinking about evaluations in three dimensions, namely execution, outcomes, and experience. Execution asks whether the agent followed its instructions, called the right tools, used those tools correctly, and maintained a maintained a coherent conversation through throughout the interaction. Outcomes as whether the caller actually accomplished what they came for, such as resolving an issue, completing a booking, and moving business KPIs in the right direction. Experience asks, how did the conversation feel like? Did it feel good? Did it have the right latency, pacing? Did interruptions happen? So in here, we would measure conversational friction. Those dimensions are orthogonal. A call can perform well in one dimension and poorly in another. And thinking about those dimensions independently helps us diagnose what to change in the next version. Evaluating all of those dimensions should paint a fairly accurate picture on the success of your voice agent. Right. So in order to evaluate those three dimensions, you need a trace. A transcript really won't suffice here. You want the full trace that brings together the audio, the transcript, the model execution, the tool activity, timing, information, and final outcome. This helps you distinguish between very different failure modes such as transcription problem, a reasoning problem, a phone call, or an issue with the generated audio. It helps you understand what was going on under the hood. The agent might have said one thing, but was that backed by a tool call or evidence? Well, the trace is the only thing with those answers. I will now hand it over to Stefan to talk about traces evaluations in more detail. Awesome. Thanks a lot, Martha. So I'm gonna start sharing my screen, and I can, oops, take it off. Cool. So I want to spend the remainder of the time focusing on the three different dimensions that Marta just mentioned and bringing everything to life with a demo at the end. The first dimension is execution. So like here, we are really looking at, did the agent follow the process it was supposed to follow throughout the call? Right? So we want to understand, did the agent really do what it is intended to do? And there are, like, three different aspects to it. The first one is instruction following. And here we are looking at basically, did the agent follow its prompt? So, like, oftentimes when you're building voice agents, you have required steps like identity verification or some sort of disclosure that must play at the very beginning of the call before the interaction between the user and the agent actually starts. And that's something that you want to, like, closely monitor to make sure that that is actually happening, when people are using your agent. You also, like, want to ensure that the agent stays inside its scope and follows these requirements, at the point of the conversation, the right point of the conversation. So, like, this includes follow the style of the instructions or the conversational flow instructions. In particular for voice, it's really important that, like, answers are concise and to the point rather than being too lengthy, which just makes it too cumbersome to talk to these kind of voice agents. On tool use, you want to make sure that the agent called the right tool at the right time with the right arguments. And, that's fairly specific to voice. But in particular, passing in, like, information that got extracted from the conversation, like names, dates, and IDs is something you want to spend, particular focus on because that's like crucial handover points where a lot of things can go wrong when the transcript isn't precise or the agent ingests the wrong information for two calls and then the whole process derails at the later stage. And in addition to just looking at, like, did the agent execute the right tools, you also want to monitor like how it recovers from silent tool errors. Right? Some agents tend to, like, execute a tool and if that fails, they just hallucinate an answer, which isn't what you want. And that's something you can specifically test for, using evaluations. So So that you make sure that you have a very robust agent that can recover from these failures rather than making stuff up. And conversation flow is the last aspect of execution. So that basically means that the agent should retain context across terms. Right? So, like, it's really about avoiding that the agent asks the same question over and over again. Right? If you have authenticated and stated your name previously, you don't want the agent to ask the same question again. Or even worse, you also want to monitor for hallucinations. So like if you've said your name, you don't want the agent to hallucinate the name at a later stage. So this is a lot about context management and making sure that there's consistency across the different terms in a multi turn conversation of the agent. And last but not least, you want to ensure that the agent carries over the relevant context through escalations or calling sub agents. So if the agent comes to a point where it decides to escalate to a human or to a different system, it's crucial that it hands over the relevant information so that the next step doesn't need to go through the same cycle again and ask for the same information over and over again. So these are things that you specifically can test and should test for voice agents, that are part of execution. And, the way to evaluate execution in Lanesmith is through evaluators. There are two different types of evaluators, code evaluators and as a judge. With code evaluators, you basically use a Python or JavaScript function to test the different to run tests on the trace that has been generated by the interaction. These are things that we're looking at the tool calls. Right? Like if you know that in order to make a certain tool call, the agent should have called another tool prior to that. And that's something that you can, by looking at the trace, verify, and then add feedback to these traces so that you can identify these errors in your live production traces. You can also check whether the required arguments for all these tool calls were present, you know, the times and whether, like, disclosures actually happened at the beginning of the call or whether they got skipped by the agent. And last but not least, you also want to monitor for, like, pool calls and how often they are called. I already mentioned that, like, some agents, when a tool call fails, they just hallucinate a response and carry on. On the other side of the spectrum, you can have agents that a tool call fails and they, issue the exact same tool call over and over and over again without realizing that they are stuck and can't move forward. And this is something by just looking at the number of tool calls that are transcribed, you can easily identify, through code evaluators. So code evaluators are great if you have these deterministic checks that run on the trace itself. But for voice agents and agents in general, you often also want to have like qualitative feedback over natural language. So if you want to interpret whether the agent did the right thing, RM as a judge is a good way to figure out, like, whether the agent actually followed all all the steps in order, according to the policy. Right? So it can have an RM look at the trace in addition to the policy and verify that the agent actually executed the right steps. You can also look for hallucinations. Right? So like if the agent if the answers of the agent were grounded in the concept context that was feeded into the agent? And did the agent ask for relevant information rather than hallucinating responses without ever going back to the user? And these are things where you can use LMS detection, to look through the trace, analyze it more thoroughly, and then add feedback to the trace that it can borrow it. The second dimension is outcomes. And here the question shifts from, like, did the agent behave correctly? To, like, what did the agent actually accomplish during the session? And there are two aspects to it. The first one is task resolution. So here you're really interested in, like, did the agent accomplish the task that it was given by the user? Right? So, like, your agents will have different capabilities. And you're trying to understand whether those capabilities are meeting the needs of your users. So if your customers or users are interacting with the agent and they're trying to do a certain thing, sometimes the agents just the agent just can't do that thing. And that's a very, very useful signal to get from, your evaluations so that you are aware of like there's a gap in the capabilities of my agent and you can then go and improve and fix that moving forward. Or conversely, like if your agent is supposed to fulfill a certain task And, you can monitor whether people start this task and then get stuck and drop off at some point of the conversation. And that's like a strong indicator that something isn't quite right, with the agent itself. And, again, that's something you can incorporate into your, into your development life cycle. So you can go and improve your agent to fix these shortcomings moving forward. And secondly, you want to connect, the agent to your KPIs. So you want to understand if your agent is actually moving metrics and KPIs that are important for your business. And the KPIs you're going to use will be very specific to, like, your, business problem. But in general, like, if you have, for instance, an appointment booking agent, you can look at, like, how much revenue, the agent was booking. Or if you have a call center agent, you can try to understand how many minutes were saved by the call center agent that could do work and not need to hand off to a human. And these kind of KPIs matter a lot because it really helps you to understand if the agent is working for your business or against it. And the third dimension is experience, which basically tries to understand whether the interaction felt like a natural conversation worth having and not just whether the content of the conversation was technically correct. And first and foremost, you wanna look at latency here. Right? So, like, how long does it take for the agent to respond? How long does it take between times? Are there any awkward pauses, in the conversation? Because, callers will notice when an agent has high latency and has awkward pauses. So that's really something that is really crucial for voice agents to have, like, this really snappy experience, so that people, like to speak to the agent rather than getting annoyed because it's just too slow. You also want to measure the success of turn taking. Right? So you want to understand, like, if the agent or the user are talking over each other, for how long are they talking about over each other? How does the agent stop naturally when when it's interrupted? And can it recover from interjection such as arms and ahs, and then continue the conversation gracefully? And the last aspect is, like, for these voice agents, you often care about a lot about pronunciation, like, especially for things like numbers, names, and addresses. And you can, instruct your agent to add clues to the voice model how to pronounce certain things so that it's really on point because oftentimes, the way you pronounce numbers is very specific on the context. You'll say numbers differently if if it's a telephone number or an address number. Right? And that's something you can then also verify, through evaluations whether the pronunciation is correct. You also want to understand, like, how often the caller has to repeat themselves or clarify information. So like did the agent say something the caller didn't understand? And then the user had to like ask again to get more clarity or vice versa. And these signals really help you to identify these moments of conversational friction so that you get feedback on where to focus on and where there are opportunities for you to go and improve and fix your agent. Cool. Now that we've talked through the three different dimensions that we want to test, we can look at how to actually test them. So when you're managing voice agents, like evaluations isn't something you do once during development. It's rather something that you'll continuously iterate on. So you start with a trace and the trace has all the crucial information, from the execution of the agent. It has the audio file of the transcript two calls and information about the latency. And you can then choose the appropriate, evaluations from the three dimensions, executions, outcomes, and experiences, and then evaluate and add feedback based on these dimensions back to the trees. And based on the metrics you are getting, you'll identify you'll be able to identify errors and improve tools, routing, voice settings, and then make changes to the agent and start the cycle again. Right? So you adapt your agent with the intention to improve these metrics. You'll put it in production and continue to monitor and then see, if the metrics actually change and improve over time. Or like if you identify new issues, you tend to like over time you add you add additional metrics and evaluators to the process. But the key takeaway here is that, like, these production failures should become inputs to a development process so it can continuously improve and iterate on your agent. Cool. So so far this has all been slides and like theoretical arguments. What I want to do for the remainder of the session is like walk you through, voice agent and show you how these different stages work in practice. So I have created a very basic voice agent, based on Gemini Gemini Live where we can ask questions about, the weather. So let's see. You have to connect to the agent, and start asking about the weather. Hey there. What's the weather like in London? Right now in. London I sorry. I meant Munich. no problem. In Munich, it's sunny and 56 degrees. Anything else you need? No. Thanks. You're welcome. Have a great day. Cool. So, like, I guess, I gave you a quick impression over, like, how these voice agents works in practice. This is a, speech to speech agent. So, like, it very snappy experience. I was able to interrupt the agent. I started asking about the weather in London, changed my mind, and asked about, the weather in Munich mid sentence. And the agent was able to, like, recover from that and then adapt cause accordingly. And this agent, is instrumented with the Linksmith SDK to send tracing information back into Linksmith. So by just adding a single line of code, we are able to turn this agent into something that can trace into Langsmith. And that's what I wanna show you next. Right? So here is the trace that I just submitted. So we can start looking at the details of the specific trace. It all starts with, like, a high level summary of the metrics so we can see when it was executed, how long the execution took, and get a basic feeling for, the cost, of this agent execution in terms of l m costs that were associated with it. And then we can see, like, the full interaction, of the trace. So, like, we can see the back and forth that was happening, where I was talking to the agent and the agent was responding. We can see that here the agent was cut off and I, like, changed my opinion and, asked for Munich instead of London. And on the left hand side, we have even more details on the different, steps the agent took. So we can verify that the agent called the right tools. So we can see, like, the agent actually queried the weather for London and later on queried the weather for Munich, throughout the conversation. And that's like if you're familiar with tracing for, like, text based agent, that's very similar in terms of, like, the details at event. What's specific for voice agents is that we have the audio transcript attached to the trace itself. So I can go in and then play back parts on Right the conversation. in London Wait. I sorry. I meant Munich. No problem. In Munich, So I can go in and retroactively understand what exactly happened, what was actually, like, taught through over the wire rather than just relying on the transcribed output, that was added to the trace. You can then also, like, switch into the latency view where I can see, like, how the conversation, unfolded over time. So I can see that, like, I the agent was, like, reasoning about the weather in London. It made a tool call. It started talking, and then it was interrupted. You can see that I was speaking followed by another tool call looking at the weather in Munich and then the response of the agent. And that gives us a lot of information to work with when we want to evaluate the effectiveness of, the agent interaction. So going back to the tracing project, we can see that we have this feedback that is added automatically to the trace and this is coming from evaluator. We can see we have a tag that indicates whether the sound quality was okay or not. This is a binary flag that tells us the audio quality is good most of the time. Sometimes there are some issues that we can then go into and look at by listening to the playback. We can then see the number of interruptions. That's something if that spikes, it's an indication that there's a lot of friction in these calls. So there's certainly opportunity for us to look at the interruptions and figure out what's going on. Now we can also see that this feedback has been automatically added to the trace of the actions we just had. Cool. Adding evaluators is fairly simple. So like, as we discussed, two types of evaluators, code evaluators, where you basically use Python or JavaScript to, analyze the trace, the different steps that are part of the trace, in the transcript. But you can also add LMS a judge. And we have a couple of predefined voice evaluators that you can choose. And the way this works is basically fairly similar to a text based evaluator. But in addition to just passing in the input and the output, we can also pass in the audio file that has been attached to the trace. And if we're using a multimodal model, the model can then reason about the quality of the, audio or anything else we like it channel wise. Cool. And last but not least, we can use monitoring to see how these metrics evolve over time. So we can take a look at different kind of metrics that we're getting back from the tracing project itself. So we can see how often the agent has been invoked, the latency, which in this case is like the duration of the entire transcript. We can see error rates, and VLM latency. At the very end, we can then look at the feedback that has been added to the traces. So now we get an overview of, like, the audio quality, transcription accuracy, and the number of interruptions. And if anything's off, we can dive in. Here we can basically look at the elevated interruption count and then analyze the trace, go into the trace, and replay it so that we can really understand what's going on and why there are so many interruptions in this part of the trace. That's also something we can alert on. If certain thresholds are crossed over time, we can send notifications, through different systems so that we can respond more quickly. Cool. And with that, I'll oops. And with that, like, we conclude this iterative cycle of build, test, and monitor. And taking a step back, like, we started with the challenges of evaluating voice agents. We then focused on the three dimensions of evaluating them, execution, outcomes, and experiences, and saw a quick tour of how to evaluate ACE and voice agents with Langston. And with that, I'd like to open it up for questions. Thank you. Okay. Cool. So the first question is, when evaluating voice quality metrics such as naturalness, how do we ensure the judge evaluates only the agent's voice and not the caller's voice? Do you separate or unmixed the audio stream before sending it to the audio judge? Cool. So, like, for the audio that is attached, usually, you use two different channels, the left channel for the input and the right channel for the output. So it's rather easy to separate these two, like, interactions out so that the, evaluator can focus on certain aspects only if that's needed for things like how was the quality of the connection. There you generally watch, and watch all channels together. But, like, if there's a specific use case where you only want to focus on certain things the agent or the user said, you can separate these out quite easily by just looking at different channels in the voice input and output. Can the agent be multilingual during the conversation? So, I think that depends on the voice model. Right? In general, models can switch between, different conversational types. What makes this slightly tricky is, like, if you want to optimize for certain pronunciations, that can be specific to the country of the domain. So, like, if you have certain clues how to pronounce addresses, that changes not only based on the language, but also on that definitely changes based on the language. So So it can be tricky to become it working in practice. Right? So, like, although, like, the models aren't often they often aren't, like, restricted to a single language as they support multiple languages, like, it can be tricky to become it working in practice. What is the guidance in evaluating the tone and emotion in the LMs response? So I'm not sure, like, what this question is specifically asking about. Like, What's the guidance? I'm not sure. Like, maybe you can follow-up in in the chat to see, like, what specifically you are referring to. Like, it's a very generic question. I'm not I'm not sure what what you're, like, specifically asking for. Sorry. Do you envision the future to incorporate when VAD starts and stops during a call that is being tracked? So, like, I like, voice for us is something that we are actively investing in. So we're always looking for feedback, and that's great feedback for us to incorporate into the product and to future step. Like, right now, I can't make any promises, but that's actually something we can play back to the, product team and get their thoughts thoughts on. Thanks. How do you evaluate the tone of the agent? Right? So that's something, where you can use, LM as a judge, where you basically feed in not only the transcript of the call, but also, like, the audio recording into a multimodal model and then ask the model to analyze the tone of the agent itself. Right? So, like, did the agent respond appropriately, to the question? Like, was it, adhering to the guidance that was part of its instruction? And that's something that the multimodal evaluator can then, like, evaluate by either looking at the transcript or by looking at the, voice, recording that is attached to the trace. Can voice agents do its own thinking? Like, if I get some response, like we have thousand dollars revenue, can this be converted to different concurrency with voice intelligence only? Can a voice agent do this on reasoning? For example, if I get a response like we have a thousand bucks in revenue, can a voice agent intelligently convert that amount into a different currency using voice intelligence alone? So like, I'd be careful, like, relying just on the model capabilities for, like, doing these, conversions because they'll change all the time and you want to have, like, a recent, currency rate that you use for the conversation. So, like, what you would do in practice is add tool calls or tools to the, voice model so that it can call out, do the conversion, and then, use the result of a domestic call that looks up the current exchange rate and then works with the converted amount in the further steps it does. So you can add tools to the voice model and the voice model can then call out, reliably convert the currency and then work with the result. For conversations that are private or personal, how can we ensure that the data is not shared elsewhere in LangSmith? So if you're using LangSmith, there are like all sorts of governance means built into the platform. So you can use role based access control and attribute based access control to really, like, restrict who has access to a certain amount of data. So, like, in LangSmith, you usually divide products projects into workspaces and you control who has access to a workspace. And even within a workspace, you can use attribute based access control to say, this tracing project contains sensitive information. So in production, only these few people should have access to the traces. Whereas in development, like, a wider set including developers can access the trace, to figure out, like, how to improve the agent moving forward. Right? So, like, you have very fine controls in Lanesmith to really control who can access what kind of data. And if you're using like, a sandwich architecture and a text based LMM, you can also use the LMM gateway to redact, sensitive information from traces that are sent into LXM. Okay. Cool. I think that were all the questions we had. Thank you so much. In case further questions are trickling in, we can take them offline. Thank Thank you so much for attending, and have a great rest of the day. Cheers. Bye.