Transcript for "Improving Agents with Tuned Evaluators":
are a new kind of software. Endless inputs and non deterministic outputs. Build them the old way, and they break. Enter Lingsmith, the agent engineering It's model agnostic, cloud agnostic, and framework agnostic. Organization is shipping the best agents iterate with a system, and we call that system the agent development life cycle. Build, test, deploy, and monitor. You build with our open source frameworks. Deep agents, langchain, or langgraph. LangSmith Fleet enables anyone to build agents without code. Test agents using evals and experiments. Deploy changes in one click with LangSmith Deployment. Monitor every interaction from a single dashboard. Governance is built into every stage of the agent development cycle with Lanesmith LLM Gateway. Lanesmith Engine helps you improve agents autonomously and move through the development lifecycle more quickly. Blanksmith Engine is your agent for agent engineering. In from today. I am, I'm Jake. I am part of or rather I lead our research team here at Langchain. And today, I'm really excited to talk to you all about how to build in LangSmith with tuned evaluators and decision models. We'll cover a little bit around, why we think working with both tuned evaluators and decision models are a really valuable practice in your agent engineering life cycle and and sort of the role that they play, why we've built them, and sort of the way that we're thinking about integrating them into LanghtSmith. I'll cover a little bit of a demo as well on both, tuned evaluators. I'll explain what they are, and and and we can dive into that in a little bit more in a moment. And, also, the the decision model, coverage that we have in Langham Smith. And then I'll open the floor up to a q and a where, hopefully, over the course of the session, you guys gather gather some questions and and sort of ask me, and and and maybe get some more clarification on anything that I might have spoken about, that perhaps wasn't clear or you'd like to dive into further. Awesome. So, again, just for for those of you who have never been to and asked me anything before, how this will work is that as questions come up, please drop them in the q and a section, of the of the call that we're currently on, and I'll be able to go and then review those at the end. And, also, as well, we'll record this session and distribute it so that it's available if you perhaps miss anything or you wanna dive into anything that I mentioned in a little bit more detail. And as always, please share feedback after the session. I'm really excited. I love doing these. I think this is a really great way to have, to have some sort of hands on engagement between between, both both the people building and and and working with, making LangSmith a great place to evaluate your agents and all of you who are using it. So without further ado, let's dive into Langht Smith tuned evaluators. A key part of building, and getting coverage over the agents that you have, that are producing traces on a day to day basis is is analyzing and understanding what they're doing. Now for those of you who are building agents that are customer facing and presumably that also have relatively high volume, you can imagine then that understanding what an agent is doing in every conversation is a relatively hard challenge because if, again, you have high volume, you have thousands of interactions a day, understanding, judging, and extracting the signal from those conversations that your agents are having is hard to do at scale. And it's hard to do at scale because it's not like you can validate and run some kind of, some kind of, like, code based evaluation to determine how successful that agent is. These are agents having natural language conversations with humans, and so there's a lot of signal in those conversations, in those traces that your agents produce that you ideally extract and use in some way to improve your agent. But extracting those extracting those signals up until the past, probably the probably the past, honestly, the past kind of couple of months has been quite hard because you can use Frontier judges, you can use five four Mini, you can use the you can use, you know, the the Sonnet suite of models, philanthropic. But at scale, that becomes expensive and also as well, you're still trading performance for and and rather cost for accuracy by using judges that are slightly more lightweight and and not necessarily frontier. And so evaluating agents, particularly in production, particularly at scale, has often always been or has always been or has been a challenge. And up until recently, there's been some really, or there's been some really nice additions to Langhamith that that really helps with this coverage of your production traces. And so the result of this, by the way, is that teams have to sample often have to sample the amount of traces that they do this kind of semantic analysis of in production. That's great, but, also, you you there there's a chance that you might be missing some critical signals that that end up, you know, that that that could provide to be very useful in the evaluation life cycle of your agent. So what we've done at Langchain, as part of the research team that I lead here, is we have fine tuned some models for the or I've fine tuned a model for a specific evaluation objective, and and we'll kind of dive into what that is, in a moment. But we fine tune a model for being particularly good at a narrowly scoped evaluation task. Now this is fantastic because what we can then do is make sure that our model and the tuned evaluator that we that we now offer in Langht Smith is particularly or is particularly great at the task that it's that it is designed to evaluate. When you use production judges, when you use some of those frontier models that I described earlier in your evaluation coverage, that's fantastic, but you're paying for the weights that those model or you're paying for the information encoded in those weights for that model, but you might be only be evaluating a specific subset of of whatever your agent does. And so, for example, if you're a customer service agent and you're trying to evaluate user satisfaction, it's fantastic that you use Frontier judges, but, realistically, you don't need a model that has the context of the entire Internet if you are judging user satisfaction. And so it makes sense, therefore, to look into leveraging the extremely powerful open frontier that has that has developed, particularly fast in the past, I'd say, sort of, you know, eight eight you know, six to eight months to provide scoped models for evaluation objectives, and that's what LanxSmith tuned evaluators are. And so the idea is that you use these evaluators to attach and extract signal from your traces. They're managed end to end by us, by Langchain. We manage the inference. We benchmark these evaluators against other Frontier judges, and I think that we'll go over some some of those benchmark results in a moment just so you can get an idea of of how they perform compared to other Frontier judges. And they're also built for scale. So, again, we mentioned that a big part of this is extracting production signal or or signal from your production traces. Doing that at scale, is both a cost you know, trade offs need to be made from a cost and accuracy perspective, and we have built these tuned or we've built the first tuned evaluator, which we'll go over in a moment, in mind to to be able to to provide scale and and durability for those of you who wanna use these for online evaluators. And so if you haven't seen this already in LangSmith, to give you a brief idea of how tuned evaluators work, for those of you who have tried to configure or or already have configured judges for your tracing projects today, you'll you'll, you'll you'll recall that it's a it's a it's a it's a bit of an involved process. You need to go into whatever tracing project you're working with, and so this is, again, your agent that's producing all of these traces that you wanna get this online signal for. You then have to go and, make sure that the the traces that you are trying to evaluate are are kind of, like, formatted and eligible to be evaluated, and so finding those traces and making sure they're eligible is is another step in that. Then you need to make sure that you choose the right model for evaluation, and this is, again, the big cost accuracy trade off. You know, do you choose a GBT five four Mini that is, you know, relatively cost efficient compared to some of the other Frontier models? Now we see models like Luna that are coming into, coming into the picture that, you know, begin to move that that that curve in terms of cost and accuracy, but still these trade offs need to be thought of. And you need to make sure that it actually gives you a reliable signal. It gives you the kind of accuracy that you're looking for when you're trying to evaluate your production traces. You need to attach these evaluators to those tracing projects, and so making sure that that they are, judging and and kind of producing those outputs in the right way, and then actually using the the feedback that these judges attach in a way that is, in a way that is useful for your kind of agent evaluation. We do this all for you end to end, and so those steps are, extremely streamlined with Langham Tuned Evaluators. You'll see for those of you who who who have actually gone in and, and and and had a look in the, in the online evaluation, tab, you'll see that you can go in and now add a Tuned Evaluator, and it's as simple as just turning it on for the particular tracing project that you are looking to apply it to. And so we take on the responsibility of managing, benchmarking, making sure it's as accurate as possible for the particular tasks that we are designing these tuned evaluators for. So, again, just to kind of cover that and and and sort of, like, lay it out in a slightly more, verbose way, there is the management of the evaluator definition, the prompt versioning. Again, this is another really important part is when you are building online evaluators, particularly LLMs' judge evaluators, obviously, they need a prompt to be able to to kind of drive the way that they analyze traces and and attach signal to those traces. Writing a prompt is, is a combination of art and science, and we've done a lot in terms of benchmarking and understanding the best prompt for the tuned or for each tuned evaluator that we offer, we have gone and done a lot of work around understanding how to best architect the prompt for that particular evaluation task. As well as choosing the judge model, again, we mentioned that we fine tune this for the particular evaluation task, and we also manage the inference infrastructure for that too. So the first tune evaluated that we're offering is perceived error. And I think this is really exciting because perceived error basically is a way to understand whether or not in conversations that users are having with your agent, if there has been some mistake or some error in the conversation. Now that might come from the agent, looping. That might come from the agent misinterpreting the user the user's question and then the euphoric. The user has to has to, correct or or or kind of, like, steer the agent in a new direction. It might come from the user, responding in a tone that indicates that they're frustrated or upset as part of this interaction. Perceived error really is an extremely important metric to, to measure for in any kind of conversational agent that you've built. Doesn't matter if your agent is is in financial services, in health care, in biotech, you know, in in legal. We designed the first Langht Smith tuned evaluator with the intention in mind that it is applicable across a variety of domains because it isn't judging it isn't judging, like, domain specific, domain specific correctness. It it really is applicable across a variety of different domains. And, actually, in the benchmarking and testing that we've done, it applies really well across, a variety of different domains because perceived error and and kind of, like, the the interaction between a user and an agent is is is, it it maps well regardless of regardless of kind of, like, the specifics of what that agent is doing. And so understanding whether the agent is is giving the user a positive experience is something that is, you know, a critical metric to evaluate for when you are building anything that is, anything that is user facing. And so perceived error is our first langsmith tuned evaluator. The model that we trained and benchmarked is, performs performs better than all Frontier closed and open models and is cheaper than all Frontier closed and open models. And so this just speaks to the accuracy and the cost savings that you can get when you post train a model for a particular evaluation task, in this case, perceived error, kind of conversational analysis. And, you know, it is is kind of the beginning of a really exciting direction that we that we wanna move in where you can get, you know, affordable but also highly accurate evaluation methodologies across, across your traces when doing online about online evaluation with LangSmith. So this is kind of like how how our perceived error, model benchmarks against some models that that a lot of you might be familiar with. And so the way that we imagine and and sort of tying this back to why we think it's really important to have something or to have an evaluator like this running on your agent traces, ideally, by the way, at maximum sampling rate, so a 100%, so I you know, on on on every conversation that you have, It's because what it really allows you to do then is and and what the first step is in a lot of what the best teams do for their agent improvement loop is mind signal and attach feedback to traces at the kind of at at the I know there's no starting point in the loop, but, you know, if there was to be a starting point, then it would be on extracting signal out of the traces that the traces that your agent generates. This is really important because, like, the the naive way of the naive way of doing this would be to, to tell a coding agent that we all use, you know, relatively frequently now, to tell a coding agent to go and inspect, you know, a slice of your agent's traces over some period of time and, you know, cluster or or or or produce some kind of, thesis on on areas where your agent is is making mistakes and therefore needs improvement. The reality is at scale, that just doesn't work because, again, coding agents are expensive. They're often using using reasoning models that are that are that are not designed for bulk trace analysis. And so as agent builders, it's it's critically important that we find some way to attach signal to our traces that we can then use as a sort of subset of what we might then, you know, analyze and inspect in the rest of our agent improvement loop. And and that's where this evaluator feedback, that's where this perceived error feedback allows you to sort of create this, or, you know, create this sieve of of, of of production traces that actually do need analysis that you can then use, for further reviews or for building datasets and and and kind of, like, benchmarking for your future, you know, agent experiments. And then as well a way for you to test and validate the way that you, make changes to your agent to make sure that those particular failure modes don't surface again. And so that's sort of where we see this fit into, this broader feedback loop of of building and and analyzing and understanding and triaging and then and then kind of fixing and, and and kind of creating this this this cycle. Awesome. Now another key part or or rather, I think, what is another really interesting angle to doing trace analysis and kind of signal extraction at scale is with decision models. And so within the past, I think, Jev was released, two weeks ago now, may maybe two and a half weeks. Things move things move so fast here. OpenAI yesterday, for those of you who who were tuned in, released their decisions API at their dev day. Decision models now present almost this kind of new frontier for being able to do large scale trace analysis. They cannot generate text, and so and so the critical difference here is where the perceived error model that I'm gonna show you in a moment in Langht Smith and that we've spoken about can produce both a a score and a, you know, a reason behind why it provided a certain score on why there was a particular error. Decision models, more lightweight, much faster, often cheaper as well, are a great way to just assign bulk bulk signal or rather, bulk feedback at scale that you can then use in in further analysis. JEV has, again, for those of you who are who are, maybe less familiar with JEV, JEV has kind of, like, three main ways that you can that you can get output from it, a null, a choice, and a score. And so the null is kind of like a binary yes, no, is this true? And so you might use that for something like PII analysis. You know, does this contain PII? And and, yes, it does, or no, it doesn't. And and maybe then use that for further analysis. You can also have different choices, and so this is actually what, OpenAI's decision API does. It given some question, which in this case, you know, what is the user asking, classify this into some category, and and provide me some confidence that you believe it belongs in that cat in that particular category. And then as well, just a just a general sort of score placed on some scale of of I guess, some some scalar score based on based on some input. Really nice because it's effectively natural language. It's a it's fuzzy input into some form of structured output that we can then use in future analysis, and I think that decision models are a great way to, to come, like, further explore doing this cost effectively and cheaply, at scale. What's really interesting, though, is JEV, it's it's the architecture of JEV, again, is is not is not clear at the moment. It's and and also as an extension of that, it's it's what Jev was trained on is is is also slightly, you know, a topic of debate at the moment. And so Jev's accuracy is is strong in some initial benchmarks that we've seen, but, again, it's not gonna be particularly, specialized for the task that you wanna use it for. And so that's, again, where there's some added value of using a particular fine tuned evaluator, that is that is task specific for whatever it is that that we're trying to analyze. And so in this case, perceived error, is an example of that. So there's a really exciting sort of future towards the way that we can attach signal at scale to, to to traces in Lang Smith. Great. I'm now gonna jump into a demo, and we're gonna look at, perceived error, and then we're also gonna look at some of the decision model support that we have in Langht Smith as well. So I'm on one of my I'm on one of my tracing projects here in Langht Smith. This is this is chat langchain. This is a conversational agent. It is where users come to ask questions about langchain's ecosystem, and, and and it has access to kind of, like, our support cases as well as our public code. One of my evaluators is this perceived error tuned. And, actually, you can see that when I add an evaluator, I'll have this option for those of you, with for those of you who are on, you know, US SaaS, which US SaaS Cloud, Lang Smith, you'll see that there's this lang chain tuned evaluator option. And the setup is extremely easy. I'm not setting a prompt. I'm not picking a model. Really, all I'm doing is is choosing the tracing project that I'm gonna attach it to. It, by default, is gonna run on conversations that have more than one turn, and that's because, again, to to to analyze something like perceived error, we want actually a a conversation that is that has some element of back and forth between the user and the assistant. You know, a single turn is quite hard to infer and extract, perceived error signal from. And then I'm just gonna I'm gonna create and save it. And what it will then do once I have attached this to my tracing project is on particular threads, and so this is just some this is just one of those, chat line chain threads that we have, where perceived error has run, we'll see that it'll judge a zero or a one based on whether or not it determines if there is an error in the in the conversation. And so zero is no, there's no error. One is yes, there is an error. And so we can see here that the perceived error for this, is is no, there's no error. You know, the user's providing additional context that they have with an API key, which is a normal follow-up rather than a correct correction or dissatisfaction or dissatisfaction with the prior answer. Now you can imagine as well that if I have a if I filter for feedback where my perceived error, where my perceived error score is one, so there is an error. Okay. So now you I'm I'm filtering for all threads that have this. Now what I might do is then use these threads or use what happened here for further agent analysis. And so let's have a look at this particular thread. We can see perceived error is one here. So what happened was the users explicitly corrected the assistant twice. First, they said no for engine results and insights results, and later, they redirected it with the rest endpoints for engine after the assistant focused on insights. This indicates that the user perceived that the assistant was answering the wrong question or needed correction. Okay. So this is great now because what I've done is I've I've attached signal to my traces. I've done it cheaply. I've done it accurately, and I can use this in further analysis, and and and kind of, like, then streamline and focus my evaluation efforts based on the results that I'm getting from these conversational interactions with, with whatever it is that my agent is is offering. Now how this looks for decision models, I I don't have an example showing Jev. Jev, by the way, for all of you, again, I I I think this is probably, for those of you who've investigated looking at it, Jev doesn't support zero data retention. And, the the traces here in chat, Langchain, they're public traces, but, again, we don't wanna use any evaluators that, that that actually retain our data. So I don't have an example of, of Jeff right now, but what I do have an example of is an example of Langchain's own decision model. It's called SEMIF, where it performs exactly the same way as, as JEV would for its particular, null choice and and score outputs. And so as an example of that, I'll go in and I'll show you you can see that I've got this semi f quen thirty five three point five b, through a 3.54 b. It's gonna judge answer completeness. This is the instruction. Does the output completely answer the user's question in the input? And it's gonna give me a yes, no based on, based on the first message in the message list. That's gonna be the user's question, and then the last message is gonna be the output that's by that's produced by, by the agent in this case. And so if I go and look at some of the traces here, I go and look at this, we can see that in this case, no, the answer was not or the the question was not answered. Actually, this is in Spanish. Unfortunately, I don't know Spanish. So I'm gonna have to go look to another one. Let's go and have a look here again. Okay. And, again, we're seeing that the ants we're seeing that it's judging this score. Oh, actually, so so now we're now on a scale of whether it answers the user's question or not. It's not binary, and so we can see that this answered the user's question with 70%, with 70%, success. And I can actually go and see the particular question that the user asked and, and or rather, I can go and see the question that user asked, which is all messages zero. That's this one here, and then this is the actual answer that was given. And so really helpful way again to do, to to kind of, like, get decision analysis at scale and begin to understand whether or not your agent is, in this case, answering your user's questions correctly. And so this is using Langchain's own decision model called Semif, that's also available to all of you today who are using Langchain or Langsmith on cloud SaaS. So I know that we kind of, we we kind of, like, breezed through that a little bit. There's definitely more more detail. And so, if anyone has any questions or if any part of that, needs further clarification, then I'm gonna stop sharing my screen now, and I'm gonna look at the q and a side of the, of of the session that we're currently in and and ideally answer any questions anyone might have. Stop that. Okay. Okay. So there's a few questions that I can see right now. Okay. So the first question is, is there a dashboard or centralized view where teams can see the issues that the evaluator services? So the answer to this is yes. Absolutely. In LangSmith dashboarding panel, of which, by the way, there's been a lot of great work done on recently, and and the experience for that has been, or rather, experience for that is, is is particularly nice at the moment. You can filter for feedback scores. In this case, the perceived error tuned is our feedback score. You can filter for feedback scores where perceived error tuned is one and and then get some some analysis, some insights into, you know, the percentage of total traces that there is a perceived error of, you know, where there's no error. You you can do a lot of great analysis around those traces, in the dashboarding experience when you filter on that feedback score, the perceived error tuned feedback score. Another one. Sorry. Very basic question. Okay. No basic questions, by the way. Taking a step back, how does a tuned evaluator differ from a traditional eval? Is it an is an evaluator auto managed by LangSmith? Yeah. Great question. Great question. So, yes, it is an evaluator auto managed by Langsmith. A traditional eval, you need to choose the model. You need to set the prompt. You need to be confident that the prompt and model are giving you highly accurate, you know, strong signal to then go and do your further analysis. You, again, yeah, you man you manage the the sort of you manage the inference, I suppose, by providing your API key. But, again, a lot of that is managed by you. We manage all of this end to end, in Langham Smith, and it's a great way to just flick it on and and kind of get confidently highly accurate strong signal, for for kind of, like, further analysis. No silly questions. And, hopefully, that was answered there. Okay. One more question, and, also, we're at the top of the time we're at the top of time here, so let me get this. Is the post trained evaluator model based, or does it use deterministic rules? It's model based. It's model based. And so we use a, we use a open model for the tuned evaluator, and we've post trained it. It is it is great for particular, perceived error workflows and is, yeah, is is is non deterministic. It's an order aggressive LLM much much the same way as, much the same way as a lot of the sort of reasoning models and and and kind of chat based models are today. So no determinism in it. There's, yeah, no determinism in it. Okay. So we're at the top of time. I see a question by Manoj. Manoj, I'd I'd love to take this offline as well. Your question is, you know, how how you use this feedback from the evaluator agents to tune them. That process is slightly more involved, and I'm happy to dive into I'm happy to dive into that more in an offline way. So, let's let's definitely sync. I'll I'll get your contact information after this, and and we can chat about that more. But that closes out the session. Thank you all for joining. I know that we kind of ran through that in a in a pretty action packed way. I hope all of you are really excited to to to kind of think about the ways you can attach signal to your traces. It's it's really like the beginning and the pillar of of doing really robust agent analysis at scale. And, hopefully, I've I've kind of I've I've given I've I've given some of you guys some ideas in terms of, where to where to start and where to jump in. Thank you very much. Jake from Langchain.