Transcript for "Evaluate Agent Behavior with Trajectories":
Hi, everyone. We're just gonna give, just maybe thirty more seconds for people to join us. So we'll get started in just a second. In the meantime, feel free to put in the chat where you are joining us from. I'm in San Francisco, so joining from Linkchain's headquarter office. Nice. I see Bangalore. Cool. More Bay Area, Maryland. Couple people from Canada. Awesome. Okay. Cool. It looks like, yeah, it looks like we've got a lot of people joining us, so I think we can get started. So thanks everyone for joining us today at our customer AMA. We're gonna be talking about evaluating agent behavior with trajectories. And really quickly, it's not a super packed agenda. So we've... We're gonna do intros really quick. We're gonna go over what trajectories are and then give a live demo and cover some use cases. And then we wanna leave at least half the time, at least ten minutes for live q and a. So I'm Bonnie Psevich. I'm in product marketing here at Langchain, and I cover observability and evals, and cover, that product suite. everyone. I'm Arjun Natarajan. I used to be a software engineer here at Langchain, and now I'm leading product for observability and evals. Yeah. Very excited to to talk about trajectories today. Okay. Really quickly how this works. As you have questions, feel free to put them in the q and a section. You don't have to wait till the presentation is over. Just ask your questions in there throughout. And we are recording the session, and we'll send it out after the call. Plus, at the end, we're gonna have a QR code so you can, share your feedback with us, about how the session went. Okay. So jumping right in, I'm gonna give some background, really quickly before we jump into trajectories. So building agents today requires a different type of development life cycle and a different set of tooling. So what we've seen is that the teams shipping the best agents have figured out how to ship early and they iterate systematically. These companies are following the agent development life cycle, which is parallel to the software development life cycle. And the life cycle has four parts, build, test, deploy, and monitor. Today, agents require more iteration. So So the difference between getting an agent to work once to building agents as a repeatable practice comes from having the right development life cycle in place. And since it's hard to predict how your users will interact with your agent and where it could fall short, you need a platform that allows you to feed those learnings back into the next build and test cycle. So next slide. And critical to this cycle are traces. So a trace captures an agent's full path from the inputs it received, the tools it called, to the final output it produced. And traces aren't just output for observability. They're the fuel to the whole agent development life cycle. So we've spent a lot of time as a company thinking about traces and how to build the best experiences around them. And to talk about one of these new experiences, we'll hand it over to Arjun. Thanks, Bonnie. Yeah. So we know that traces are at the center of the entire agent development life cycle. And what we've observed is that traces, in their current form are really, really useful and, make it... Makes it really easy to debug cases where you have short, agent workflows. Maybe they're single turns. But as you start to build longer running agents, these might occur over many turns, over many minutes or hours. Your agents might make multiple tool calls. They might have multiple retries, and they're probably gonna be invoking sub agents and handing off to them. And so as you have these longer running agents, we found that traces in their current form make it really hard to parse all of the nesting to find the actual path that the agent took. And doing the actual debugging and finding the information that matters to you just becomes a lot more time consuming when having to traverse individual traces. If we think about threads, where, again, threads are series of traces that are connected. So if you imagine some, like, agent conversation you're having, every time you make a request to the agent, that creates a trace. And so if you have multiple sort of requests to the agent within one conversation, those will... Those would create multiple traces within a thread. And if I wanted to do debugging in this case to understand what was the actual sequence of human and AI messages and and the high level steps that happened, parsing this today requires me to go through this tree across all of these traces, which is really quite cumbersome. And that's where trajectories really shine. With trajectories, we essentially parse these traces for you, and we extract out the actual sequence of human messages, AI messages, tool calls, and we structure them in a way that actually represents and is faithful to the actual sequence of conversations, that actually happened. If you think about, this case where you have sort of multiple turns and multiple LLM calls, often the LLM calls will encode the entire conversation history in each of their calls. And so if you're trying to understand, let's say, an LLM run five, or, like, the fifth, model call, what actually happened in that turn, there's a lot of duplicate information that you have to parse. And trajectories are really powerful because they find that unique, information boundary, and they collate that into a single list. So it makes it really easy to, like, deduplicate information and see what the actual sequence of user messages, AI messages, and tool calls actually were. I'm gonna hop into a bit of a demo now. So we are looking at, Chat Langchain, which is an agent that we have that you all can go to at chat.langchain.com, and you can use it to sort of ask questions about langchain. And you can see we have a bunch of different Chat Langchain threads here. If I click into any of these threads, you'll notice that there's a trajectory toggle. So, traditionally, you might be used to seeing this details toggle. If I select trajectory mode, you can see I... I'm actually able to see, like, a transcript of the actual conversation that happened. So these are the human messages that were, that were sent. These were the tool calls that were made. I can see for those tool calls what the request and response was, and I can also see the full AI responses. If we had sub agents in this in this thread, we would be able to see those as well. And importantly, I'm able to see this information across all of the turns in this thread. We think that trajectories are really good for getting a high level understanding of what actually happened in a conversation. But if there's ever a particular case where you need to debug in more detail, we still provide you the option to go deep into the details. You'll see this LOM call button on every message in a trajectory, and clicking this takes you into the details view at the exact point in the conversation that you selected so you can really understand, in much more granular detail than the trajectory view, that particular step. So we think that trajectories, again, are really good for getting a high level understanding of the sequence of events that happened in the trajectory, and then we still allow you to go super deep, into the details as you see fit. There's a lot of things that we allow you to do with trajectories, in LYNXsmith. So besides just viewing trajectories, you can also send them to other parts of the platform. So as an example, I could send my trajectory to an annotation queue. And when I do that, you'll see that in this annotation queue, I can actually see the trajectory and I can add feedback on top of it. We also allow you to add trajectories to datasets. So I could add this to a dataset here. And, again, I could use this for offline evals now. So the offline eval would run across the whole sequence of turns in the trajectory. And, finally, we support trajectories in many of the sort of online, operations we have. So automations, you can, run automations to send trajectories to annotation queues or datasets, and you can also run online evals on trajectories. So for example, if I wanted to create sort of a thread level evaluator, you'll see by default, we send the trajectory, to this online evaluator. So this is, again, the same rendering, the same trajectory. If any of you have created online emails before and you've done this, you know that a big step is actually sending the right information to the model. And at the run level, a lot of times, what you have to do is you have to find the particular slice of data, in the individual runs. And this is a super cumbersome process doing all this context management of what you send to the agent. And, again, because trajectories deduplicate a lot of that information, for thread level evals, we find it super easy to just send the trajectory directly to, to to the eval. So, yeah, we think that trajectories are a super ergonomic way. We have found that they make it a lot easier to do all these downstream operations. And in general, it just makes it a lot easier to do debugging for your high level, long running agents. And I think at this point, we are gonna stop sharing and turn it over to anyone who has questions. Hey, Arjun. So we have a couple questions. The first one, how does it handle multimodal content such as images, files, or audio in the messages? Yeah. So one great thing about trajectories is it's built on all of the same primitives that, that Lanksmith is built on, so runs, traces, and threads. So anything that's supported there, if you're sending multimodal content, should still be supported in the trajectory. One edge case here is, attachment support. So if you're... If you include images or, videos in the input and output, those should be faithfully represented. One thing that we don't support right now is, if you trace, attachments separately, that's something that, that is still being worked on. But for general case of of encoding images and videos and inputs or outputs, that should be supported. Okay. Great. Next question. So if an agent gets the final answer right, but it takes a bad path to get there, Can I catch that with trajectory evals? Yeah. Totally. So one of the things that we find, in in in old cases of evals is that it's it's really easy, and what a lot of people do is they only evaluate based on the final answer. What trajectories allow you to do is evaluate on the whole list of messages. And so you're not just looking at the final message, but you're actually looking at the entire sequence of messages. You're looking at, did the agent make the right tool calls? Do those tool calls have the right responses? So this is actually a very, very valid use case, and one of the things we think trajectories is most useful for, is understanding not just the final answer, but also the sequence of steps, to get to that answer. Okay. And does this work on historical threads already in my project or only on traces sent after launch? Yeah. So when we launched trajectories, we made sure that we supported everything out of the box. There should be very little instrumentation that you have to do, both in terms of, your client side, like tracing setup, as well as, your historical threads. Trajectory should work out of the box in most cases. I do believe in our docs section, we do have links to docs, and you can understand a little bit more detail in how trajectories, kind of work under the hood. There might be cases where your trajectories, are rendering in a way that you might not expect, And we allow you some levers, in your actual tracing setup to adjust how trajectories get rendered. So there are certain metadata tags that you can attach to runs, and that can affect, or change how the trajectories get rendered. So if you do see issues, again, when you're viewing a trajectory and it doesn't look as you expect, I would definitely go check out the docs and see the supported integrations, the metadata that we allow you to to attach to runs to kinda change or customize how the trajectory gets rendered. But, yeah, trajectory should work out of the box for most use cases. Okay. If I'm storing metadata at the thread level to organize or categorize, etcetera, is that info also carried over to trajectories? Yeah. So trajectories, are not inherently a new data structure. They're essentially, a projection of all of the data in a thread, and all of the data in the traces inside of the thread. So all of the information should get carried over and is accessible. In LYNXsmith, we technically don't support metadata at the thread level. We support metadata at the run level, and so all of those run level metadata should get carried over. We do, however, have thread level stats and feedback. And in the trajectory view, you'll see we support, showing all those feedbacks and those stats as well. So in general, all this high level information, feedback, metadata stats, should get carried up as well. Okay. How to implement the guardrail to isolate or how, yeah, how do I implement the guardrail to isolate if any PII data is being exchanged between tool calls or the agent communication? How to implement the guardrail to isolate? I'm not sure if this is quite trajectories related, but... Yeah. Yeah. I... So this sounds like, a case of trying to set up an evaluator. I would definitely encourage checking out. We do have templates around PII detection for evaluator cases. Yeah. I think this is probably worth, getting a little bit more detail on to fully understand the use case. But, again, trajectories should be quite useful for understanding, or sending in the right information to your LLM as judge evals to do something like this. You can sort of look at the actual responses of the tool calls and as well as, like, the agent, the AI messages. So you should be able to use sort of our templates that do PII, detection or create your own LLM as judge, evals if you have custom logic, in terms of your PII sort of, rule matching. One thing that we don't yet support that we are planning on is bringing trajectories to code evals. So if you're running evals on code, and you have, like, let's say, regexes for PII detection, that's something that we're working on to support with trajectories. Yeah. I I I would encourage checking out the docs here. I think they'll probably provide a little bit more detail than than we might be able to get into. But also, yeah, I'm happy to... If if there's a way that we can sync offline after this, I'm happy to to, provide more information if I get some more detail. Okay. And then are there any metrics or charts analysis available for trajectories out of the box? Yeah. This is a good question. Today, our metrics, in our monitoring page operates at the run level, and so we don't yet have support for trajectories, at the monitoring level. That's for dashboards and and alerts. You will notice that in the tracing page, the stats panel, which shows you, like, project level stats. If you go to the thread, tab when you're viewing your traces, you should see all of your thread stats. And, again, because trajectories are just a projection of the information in a thread, all those thread stats should be relevant in terms of the trajectories. So you should be able to see, like, high level stats at the project level for trajectories or threads, but we don't yet support. And it's one thing that we're working on, creating dashboards and alerts at the thread level. Okay. These two are kind of related, but, so how do I tell whether a bad session was caused by the agent itself, a tool call, or a sub agent? Yeah. I I I think this this is a really good question, and this is something that is is worth understanding what bad means. I think there's different cases of what bad can be, whether it's, you know, the agent going off and making, errant tool calls that it shouldn't be or maybe the agent, you know, isn't making any tool calls or maybe there's an issue, with the the sub agent. So if you... If there's clarity on what bad means, I think that helps you identify where to actually look, in the sequence of of events. In general, I think this part of evaluation still requires humans to have a good understanding of what they're looking for and where that might happen. The information is all sort of exposed to you. And if you're using an LLM as a judge, you can sort of prompt it in the directions of of places in the trajectory to look. But this is a case where going through your your traces and understanding what are common failure patterns can help you set up the evaluators in a way that, sort of guides it to to failure modes that you're seeing. Okay. And how does it handle sub agents' reasoning? So today, trajectories, provide a high level, preview of the sub agent. So when you are viewing a trajectory, you can see the sub agents that were invoked, and we show you a preview of, I believe, it's five of the the runs that the sub agent, or, like, five of the runs in the trace of the sub agent as well as the final sub agent response. And then if you wanted to go into more detail on the subagent, you can open up the full trace in the details. One thing that we're working on is true, nested support for subagents and so being able to view the entire subagent, including its reasoning, within the trajectory itself. So this is on the road map. And today, what you would what you would do is you would see the the sub agent preview and then go into the details for more information. And is there a performance ceiling? So what happens with very long sessions like hundreds of turns or thousands of messages? Yeah. Trajectories are a really awesome thing that we were able to build on top of our new database, SmithDB. So for those who who aren't aware, earlier this year, we we launched, our new purpose built database called SmithDB, which is perfect, or bespoke for agent agent traces. And that has unlocked for us a bunch of capabilities that allow us to do stuff really at the horizon or really at the cutting edge of sort of long running agents. We're seeing this happening more and more where agents are starting to run for hours and have tens of thousands of of runs in them. And so Trajectories is actually built entirely on top of SmithDB, and this allows us to very well support these exact cases. You will see when you load a trajectory that you should have no performance drop off, for these long cases. And, yeah, SmithDB has has done wonders for us in this regard, and we we think it really... Trajectories really shine in this exact case. Okay. We've got one more. Can we annotate trajectories with before agent events and after agent events? This is a great question. I think, if I understand the question correctly, it's asking if we can, add annotations to the trajectory, like, before the AI or before the, yeah, before the AI sort of responds with the message and then after. Is is someone... Is... Nikhil, are you able to confirm that that's the question? I wanna make sure I'm answering the right thing. Yeah. So this is something that, is definitely possible because, each message in the trajectory has an underlying LLM call. You should be able to add, annotations to each of the different runs, or each of the different messages in the trajectory. So before and after the agent is invoked, you can you can add feedback, at that level. Right now, when we render trajectories and the feedback, we we show those annotations kind of, in one flat list. So the annotation doesn't show up sort of at the message level in the rendering. But if you were using the annotation queues for this, this this should be possible. Okay. Cool. I think that was our last question. But before people hop off, we have resources to share with you. So, Arjun, if you wanna switch to that resources slide, these... On the slide, it's gonna show you a bunch of links, but they're also available under docs. So within this webinar platform, you should see, a doc section. So we have those linked directly so that you can click around and look at any of those docs. Next slide. And we have our big interrupt agent conference coming up. So if you happen to be in the area, it's in London next week, next Tuesday, and registration, it's still open. So if you wanna check it out, head to interrupt.langchain.com. And finally, on our last slide, thank you so much for joining. And if you wouldn't mind, yeah, just filling out this quick survey and letting us know how we did and also what else you want to see us share or answer questions about. So thanks, everyone, and thank you so much, Arjun, for joining us today. Thanks, everyone.