Video: The Future of Agents with Harrison Chase | Duration: 2752s | Summary: The Future of Agents with Harrison Chase | Chapters: Introducing LangSmith (0.52s), Agent Evolution Overview (117.47s), Persistent Agents (223.865s), Tasks to Goals (294.275s), Persistent Agent Features (457.66s), Building Agent Systems (557.71s), Agent Runtime Requirements (740.305s), LangSmith Deployments Overview (1102.02s), Testing and Improvement (1250.185s), Agent Governance (1449.865s), LangSmith Platform Features (1590.25s), Trace Data Control (1696.8s), LangSmith Coding Agents (1871.985s), Tool Gateway Security (1956.105s), Q&A Session (2057.26s), Agent Governance & Safety (2259.045s), Agent Memory Architecture (2470.93s), Closing Remarks (2636.38s)
Transcript for "The Future of Agents with Harrison Chase": Over the past three years, millions of developers have used lang chain frameworks to build agents. And today, we're putting everything we've learned into one platform, the new Lang Smith. Right now, everyone is building agents, but only a handful of them are transforming businesses. Those that do are proactive, specialized, and they get better over time. The only way to build these agents is to keep iterating on them, see what they did, measure it, fix it, and ship it again. Today, these steps are spread across multiple tools that weren't built for agents, and that's where LangSmith comes in. LangSmith is built for and powered by agents, supporting the entire agent development life cycle so you can build, run, improve, and govern every agent. Here's how it works. LangSmith lets anyone on your team build an agent in natural language or in code. You can use Langchain's open source harnesses or any harness. Deploy your agent in one click. LangSmith handles the infrastructure so it can run and scale reliably. Workday adopted LangSmith, and deploying a new agent went from five months to under a day. Every agent across your company is visible from one place, so you always know what's running. You can trace every step an agent takes and see exactly where it went wrong. Every time you make a change, Evals test the new version, so you know it's better before it ships. Lanesmith Engine monitors your agents twenty four seven to find failures. It writes the fixes and evals for you. All you have to do is review and merge. Teams using Lanesmith Engine improve their agents five times faster. Every run is governed by Lanesmith with model fallbacks, cost controls, and protection for sensitive data. Today, Lanesmith processes billions of traces a month with more than 7,000 customers relying on it every day. Our goal is to help every company own its intelligence, from the agents it builds to the systems that improve and govern them. That starts with helping teams build agents they can trust in production. So we're providing Lanesmith Engine for free to all new plus subscribers through the end of the year. Start building today at lanesmith.com. Hello. Thank you everyone for joining. Excited to be doing this webinar. We are gonna jump right into it, but there is a nice q and a section, on on the right. And if people add their questions there, we will get to them at the end, and we'll try to leave lots of time for that. So let me share my screen and, get right into it. Cool. So I I wanna talk a little bit about where we see agents going and some of the things that we're building to help with that. So I think it's actually a a really great timing to be doing this this presentation because I think there is a new, type of agent, a new step change in agent that... That's coming around. And so when I think of the past four years, really, and I think of the progression of AI and of agents and how we've built them, I think there's a few different kind of, like, you know, trends that we've seen. First, we started with with just simple LLMs, text in, text out. This was this was when LaneChain was started about a month before ChatGPT. And and and then after this came out, I'd say, like, the next era was was these messages and tool calling based, LLM, or APIs that that the providers exposed. And this is really where you could start creating conversational experiences and doing a bunch of sequential tool calling. And then reasoning models came out, and it just made the the the models and the agents way better. And and and and and after that, we saw the rise of coding agents. And and and and now, you know, it wasn't just single, model calls doing a lot of the work. It was it was these agents interacting with the environment. And and as I said right now, we're we're at the start of a new step change, and, I I I think of these these new things that that people are building as as persistent agents. And so what what exactly is persistence agents? And and and there's a few different kind of of terms that have been used over the past few months to describe, similar things. I I like persistent agents the best, but you might also have heard them called ambient agents. So that's something we use actually about almost a year ago at this point, two years ago at this point, actually. Always on agents is another thing that we've heard them called. So Anthropic wrote a blog on building always on agents. Proactive agents is is is another term. And then persistent agents comes from from Dylan Patel, or at least he's the first person I heard say it. And and I like that the best because I think there's a few different dimensions of these agents that that that really all of these, names are talking about. And I think persistent is kind of, like, the best name. And so when I think of persistent agents, I think of kind of, like, four different characteristics. They're they're goal oriented. They're always on. They accumulate context over time, and then they're multiplayer and multi surface. And so I I I wanna talk a little bit about, what these, what what these characteristics look like, and then I wanna talk about some of the things that we think are needed to build these. So so we... We've... As models and as agents have gotten better, there... There's been this kind of, like, shift from from tasks to goals. And so when I say task, I think of something like write this email. And, you know, the agent... You you you send that as a user to the agent. The agent will go off. It'll, you know, find whatever information it needs, call the kind of, like, write email tool. It's it's it's an action that the agent is doing. It's generally, like, pretty bounded. When I think of a goal, it might be something like maintain a blog or, you know, maintain inbox zero. And and that involves the agent running over an extended period of time, trying out different things. It's it's optimizing for an outcome, so like, you know, performance on the blog or or inbox zero. It's constantly ongoing, and it has to do a bunch of different kind of, like, task. It has to it has to break down these tasks as it goes along. And and so the shift from from tasks to goals, I think, is a a really important part of these of these persistent agents. And and one of the interesting things that I think emerges because of this is goal design. And I think I think we'll talk a lot more about goal engineering or goal design over the next, over the next however long. And and one of the key insights that one of our, users had, Actively AI, great company, they build sales agents, we wrote a blog with them. We're a blog on kind of, like, goal design and and how they basically aligned goals with with user feedback. The key insight being that the the hardest thing to get that is needed for these persistent agents is user feedback. And so if you align the goals with user feedback, then then your agent will be better off. And so, for example, they had... Previously, they had kind of, like, one agent per account, which is a very natural way to think about it. And and, you know, the the goal for each of those agents would be to, like, progress the account. But there's actually a bunch of different stages that are similar across all accounts, outbounding, early stage deal progression. And and when they reoriented their agents around those those goals and and then made it work across all kinds, that was actually better. And it was basically better because their customers needed to needed to basically teach the agent how to outbound, teach it how to kind of, like, progress early stage deals. And and by orienting the goals around what the customers found more natural to give feedback, like, it's easier to give feedback on outbounding across a bunch of different accounts and have those learnings transfer, that that proved to be really valuable. And so and so the key thing here is basically designing goals around feedback loops or just designing goals in general will become really important. Another part of these persistent agents is the fact that they're always on, so they react to a bunch of different things, not just user messages, but real world events or schedules that the developer of the agent might set up or schedules that the user of the agent might set up or schedules that the agent itself might might set up. And so all of these contribute to waking up the agent. And when I say waking up, really what that means is just sending another message to the agent, but it's not coming from a user looking at a at a chat box and typing something in. It's coming from from some of these event systems. And so this is a key part of of persistent agents. Persistent agents also learn over time. So you've now got them, you know, acting over longer time horizons on more ambiguous goals. And so and so as they react to things, they they they need to learn as well, and they need to keep that memory someplace so that it maintains consistency and makes progress towards towards its goal. And so memory is is a key part of of these persistent agents. And then these these these persistent agents are oftentimes multiplayer, so they work alongside humans, many at the same time. And so here you can see an example of Robin, which is an agent that we have internally and it's interacting, in Slack with a bunch of different users at the same time. And it's also multi surface. And so we also have a desktop app and a web app for Robin, and you can interact with your threads through through all three of those. And and and so that's another key part of these persistent agents. They're not just available in one place. They're they're available across a bunch of different, surfaces. So so so so those are the characteristics that we think make up persistent agents. So so how do you actually build them? And we... We've spent a lot of time thinking and talking with with builders about this. So so behavior comes from the whole system. So when we think of an agent, we generally think of kind of like three layers. So the the... You've got the model, you've got the harness, and you've got the context. The model are the LLMs, reasoning, tool calling, all all of those things make it better. The harness is really how that model interacts with the environment, with the outside world. So it's got tools, and it knows how to ex... It knows how to run-in a loop, and it knows how to interact with its sandbox or its file system. And and then and then context is the application specific stuff. And so these are instructions and facts and and feedback and memory, but also, like, application specific tools. And and so, generally, when we see people building these agents, most people will use a model off the shelf, whether it's a Frontier model or, or or an open weight model. They're they're usually not tuning their own to start. They'll usually start from a base harness or or use an off the shelf harness, and that's because there's actually a lot of things that go into these harnesses that make them quite complex, and I'll talk about that in a second. But then you bring all of your own context. So you bring your own tools, you bring your own skills, you bring your own memory, and and and that's that's the key part that that makes the agent really domain specific, vertical specific really work. So I said I'd talk about harnesses. So we have a harness of our own. It's called Langchain DeepAgents. It's open source. It's model agnostic, so it works with any of the models out there. And there's a bunch of stuff that we've added into it over the past year and a half at this point, that we think makes it really good, and we think makes it really good as a base harness. And I'll say more about that in a second. But we we we have, kind of, like, tools for interacting with file system and sandboxes and actually browsers now as well. We do a bunch of base undifferentiated context management for you. So so we support kind of, like, skills and loading skills. We have summarization for when threads get really long. When tool calls are big, we do a lot of context offloading. Prompt caching is really important to get right, and so we have stuff there. We support human in the loop. That's a key part of these of these long running agents as well because you're gonna need to ask for kind of, like, user feedback on things. And then we also support sub planning and sub agents and async sub agents, as as delegation, capabilities. And so all of these are general capabilities that you want your harness to have and you probably don't want to build from scratch. But you might also wanna modify your harness, and that's where where DeepAgents is, again, really great. DeepAgents is built to be a base harness. So what that means is that we want you to modify it. We want you to modify it through middleware, which is which is a a way that we have to to kind of, like, control the agent loop more. Or if you really want to, you can you can fork DeepAgents because it's open source and modify there. So so DeepAgents is not kind of, like, explicitly an end user, facing coding harness, although it's good at coding, but it's intended to be a base harness to to build upon. So so I wanna touch back on those kind of, like, four different properties of, of of of of of these persisting agents and then talk about what you need in your... What, like, what what parts are needed to make those a reality. So so so when I say agents are goal oriented, the main thing that makes them goal oriented will be some of the prompting that gets them to work towards that goal. And that prompting should also, you know, encourage them to be creative. When you're working towards a goal, it's not just like brute forcing something. You wanna consider a bunch of different paths, and and not just do this... If you're if you're waking up on a schedule and doing the same thing every thirty minutes, you're not gonna make a bunch of progress. You wanna know what you did before and and get creative and look at things. And so, it's it's a different type of prompting than just accomplishing a task, which is which is often, kind of, like, very brute force. And then this this is also where alignment comes into play, and... Both at the prompting level but also at the model level. And so what I mean by this is if you give a, agent a goal of kind of, like, removing all bugs from the code base, it might accomplish that goal by just deleting the whole code base. There... There's there's no there's no, there's no bugs left. It's accomplished the goal. But, of course, that's not what we want the agent to do. That's not what we want any, engineer or any any any system that we that we give that goal to to actually do. And this is where alignment of the models, really really comes into play. For the... For building this, like, always on aspect of the agents, you're probably gonna want a few things. You're gonna want crons. These are these are basically ways of running, runs in the background, and so you can set either kind of like a schedule for them, or or or as one off. You... You'll probably want a way for agents to schedule their own reminders because they will be doing work or interacting with the user, and they're... And and they'll want to wait at some point in time and and and do something. And so the... And and so having tools for the agent to interact with its own Kron schedules is really important. You'll want a bunch of event listeners as well. So so these agents should wake up, as they're listening to events, whether that's a new email coming in or a new Slack message being sent to it. And so these are, these are these are different event listeners you'll want to have. And then you'll also need to support kind of like queuing and steering. So these agents are always on. They're always working. And, and you might send a message to it as it receives another kind of, like, event from the background. And so how this main agent deals with all these messages coming in, whether it queues them up, or or whether it uses new messages to steer, you're you're gonna want kind of like, the the logic of your agent to handle that. You're gonna want this agent to learn over time what practically does that mean. Practically, that means you'll want some concept of user or org or agent or all of the above, memory. You'll want to determine whether the agent can write to memory directly as it's running, or does it write to notes? Does it write to a buffer that is then incorporated during dreaming? So dreaming is a process where the agent runs in the background, and looks at previous runs, previous, like, notes or buffer, other kind of, like, sources of info, and then updates its memory accordingly. And and and so, those are some of the considerations you'll wanna take into account, and that that can be different based on, how you structure memory and and and how complex it is. And then when you think about building, for multiplayer and multi surface, you're you're gonna want... You're gonna need to handle off for proper permissioning. If if me and my coworker are in a Slack channel together, you know, if if I if I start a thread and my co... Coworker, you know, adds on the next message, it shouldn't be able to access my Notion or my email even even though I started it. And so you'll want to make sure that that auth is really lined up with with channels where you have multiplayer, capabilities. And then you're gonna need to connect to it from multiple services as well. And so that means that any state you store will need to be, accessible, through through any of those. And any streaming or any any progress that you you make, you also want kind of, like, different UIs to kind of, like, support that. And so I've talked about the harness before, but that really only gets you about, like, half of the stuff that that we talked about in terms of the requirements. And so the harness affects how the agent behaves. And so it has the... That's where the prompts go. That's where the tools go. That's where some of the things like sub agents go. But a lot of what we talked about is also infrastructure around the harness. It's not actually in kind of like that core agent loop. It sits outside and and and and interacts with that. And so, this this is kind of what we call an agent run time. And and the way to think about it is you've got your app, where where users are interacting with the agent, and and you've got your harness here, but there's actually a ton of stuff that you need to build around it. So so you're gonna want some concept of kind of, like, execution that runs your agent and and and you'll... You're gonna wanna do this in in a durable manner. And what I mean by that is there's a concept of durable execution. This is where if the agent fails halfway through a run, it it doesn't just, like, start back from the beginning. It actually remembers, where where where it failed from and picks right back up from there. You're gonna want APIs to to take messages and then decide what to do with them. Does it does it reject new messages? Does it cue them? Does it steer them? You're gonna want a scheduler, that that has Kron's and sends reminders, and you're gonna want this agent to have tools to go there. You're gonna need to keep all the state of your agent. And so if it's running on your laptop, great. You you don't need to worry about that as much. You can just store its files there. But what about when it's running in the cloud? You're gonna wanna create kind of, like, the conversation and progress and history and and have that accessible, to to your app, but also to your agent. You're gonna wanna stream events in a in a reliable way, and and and and across kind of, like, devices as well. So if I start in Slack and then I move to my desktop app, I can resume that stream. And then and then you're gonna need your kind of, like, environment that the agent interacts with. And so we think of all this stuff as kind of like an agent run time, that solves a bunch of different things, and and that's what's in LangSmith deployments. So LangSmith deployments is part of LangSmith. It's it's, the the the name... What we really call it is like an agent server. So it takes, it takes the the the harness and it bundles it up. It adds in... We've got sandboxes there. It adds in a context hub for managing kind of, like, skills in a persistent manner. It's got cron jobs. It has a concept of threads for conversation. It does streaming, and it does it in a reliable way so you can rejoin to it. It it does off. It has a task queue for kind of, like, incoming messages and actually handles four different, ways of, of of multi texting, which is that steering and queuing. It's got checkpoints to kind of, like, maintain things. And so, again, this supports kind of, like, long running agents. You you define kind of, like, a bunch of different stuff in your agent, and then Lanesmith will run it in this agent server. And it and it makes building these always on agents that complete work easy. So so the the key part that I wanna emphasize is that you... You've got... When you think about building your your agent, you've got your... The the, like, context, the business logic of your agent, and that's the domain specific stuff, that that only you can provide. You've got the harness of the agent, and this is this is where, like, support for skills and support for sub agents and and and summarization and offloading is defined. And more and more, we see people using a base harness like DeepAgents for that. And then you've got this agent server. That is the infrastructure where where the agent lives, and that's what's in Lanesmith. Another thing that's in Lanesmith is manage deep agents. So I mentioned before that there's these three layers. There's this context, there's the harness, and then there's the infrastructure. And and so what manage deep agents does is it basically bundles the harness with with the infrastructure. So now you just define the logic of your agents. You define, you define kind of, like, the instructions and the skills and the memory and any kind of, like, schedules you wanna have and where you want people to interact with it from and any tools. And then and then, basically, we run it on managed deep agents, which is which is which is separate from the agent server. It's it's it's a it's a abstraction layer above the agent server. So, again, if you wanna run any harness you want, use the agent server. If you wanna run, if if if you want the easiest way that combines a harness with infrastructure, use managed deep agents. They're at different levels. And so this is also... This is available in Lanesmith and and and and can be easily deployed to from there. We've talked all about the runtime so far. And and I wanna emphasize that when we talk about the runtime, this is the intended behavior of the agent. This is what you want the agent to do. But as anyone who's built agents will tell you, you know, even if you tell an agent what to do, there's zero guarantee that it's actually gonna do that. And so and so there's two other missing pieces when you think about what it means to actually run and operate these agents reliably at scale. One is kind of like the the improvement loop and the observed behavior of the agent. So, again, you have this intended behavior in the run time, but what actually happens, you need to observe that and you need to use those observations to improve the agent's actual behavior over time. And so when we think about this this kind of, like, observation and and and testing loop, there's a few key parts of it. One is testing the agent. So you wanna test all aspects of the agent. You wanna, test that, you know, it... The... That it that it doesn't kind of, like, do too much for events. So what I mean by this is if you have an agent waking up every 30 minutes and and and it reaches out to the user every 30 minutes, that's too spammy. You want to kind of, like, test, you want... You wanna test that it does the right tool calls. You wanna test that it returns the right message. You wanna test that it, edits the state correctly. So so you wanna test the agent on a bunch of different dimensions. You wanna grade, like, the traces of the agent and not just the traces, but actually the end state of the whole agent against, against a rubric. So you so you want to, grade kind of like was the signal that came in relevant, was the research that it did... These are examples. But these are basically examples, that that show how you can grade a trace, the... So the full execution of an agent against a bunch of different dimensions. And so you'll... You're you're gonna wanna set up kind of like rubrics or evaluators, both online and offline for for some of these. And online, I mean, as agents are running, you grade them, so you don't have the ground truth, but you know it should do certain things. And then offline, you do have the ground truth. I mentioned this before, but but a key part or a new part of evaluating persistent agents is evaluating how it basically deals with with all the the signal... The the signals that are coming in or the the events that are coming in. And so you're gonna wanna test that it acts when it's worth acting on, and it's also silent when it's not worth acting on. And if if there's any noise or any missed opportunities, those are bad things. So this is this is just a new dimension of things that you're gonna wanna test with these persistent agents. And then the main thing that you're gonna wanna do is have this loop that basically takes, takes, traces, and and turns them into into kind of, like, test cases that you can run regression tests against. So, specifically, what I mean is you you might be running your agent in production, and, as it's running, you notice that it fails in some manners or you have a system that that that catches those failures, and then you turn those failures into evaluations that you can then basically use as unit tests for when you're making changes. And so, the the run time is the intended behavior of the agent. You've got this improvement loop that you're spinning due to the due to the, observed behavior of the agent. But then there's certain things that you just don't ever want your agent to be allowed to do, and this is where governance comes into play. And so governance defines what the agent can and cannot do. You you should not define this in a prompt. A prompt is suggestions. The the the agent can ignore it. It can get prompt injected. If there are specific things that you want the agent to to to, be able to do or not be able to do, you you want to enforce this at the governance, layer. And so you want to, enforce things like policy and identity and approval at basically, like, the choke points. And so that means when it calls tools, when it does network requests, including to the models, those are the places where you want to basically enforce these things. And a key part here is that those those permissions might depend on the context. So who is acting, what action it is, what surface it's in. And so, for example, this is a real example of of Slack bots we have. When you interact with with, one of our with one of our, company agents in Slack, if you're in DMs, you're allowed to kind of, like, access email, send emails from within the personal scope. When you're in a shared channel, you're not allowed to, and that's because there's other people kind of, like, in the channel, and there might be sensitive information that's in there. And and so this is a real example where permissions depend entirely on the context of where the agent is running. And so these are, like, the three aspects of agent behavior. We think they're... We think there's kind of, like, the intended behavior of the agent, the observed behavior of the agent, and then the governed behavior of the agent. And and and as agents become more powerful, you you need to own how they behave, and that's what we really mean when we say own your intelligence. And so when we talk about kind of, like, owning the... Your intelligence, we talk about owning the behavior of these agents across these three dimensions. You need to be able to specify what you want them to do, and you need to be able to observe them to make sure that they're doing, doing the right thing. And then you and then you need to be able to also set governance to enforce, what they do. I mentioned Lanesmith. I I wanna touch on a few places of of how Lanesmith is built for and powered by agents, actually. So in terms of the agent run time, the open source, this is this is where you define the behavior of your agent, and then Lanesmith is how you run it at scale. So I mentioned the agent server. I mentioned managed agents. And then you... I mentioned deep agents, which which are... Which is our open source, harness. And so these are really everything related to the agent runtime. On the agent improvement loop, you've got things like SmithDB, which is our our database for storing traces reliably, and then you've got observability and evals, which are are a key part. The other thing I wanna highlight is engine. So I mentioned there's this improvement loop that you want to spin. Engine is an agent that we actually built in Lanesmith to help spin it for you. So it sits on top of traces, and it monitors them for issues. And then based on those issues creates, creates, evals or creates fixes in your code base actually. If you haven't tried it out, I think this... Like, one of the things that we see is that you can send thousands or millions of traces to Lingsmith, and the value you get from them, you you can't look at them by hand anymore. And so then having an agent that runs on top is really, really valuable. And and so if you haven't tried out Lingsmith already, I would highly suggest it. And then agent governance. This is where you define what the agent can and cannot do. And so, there's... Again, I mentioned that you wanna set these at kind of like the choke points of the agent where where things go across the the wire, so to speak. And so the two main places that happens are when you call LLMs. So we have an LLM gateway. This is where you can control costs, control access to different models, set fallbacks, things like that, and then a tool gateway as well, and I'll and I'll talk about that in a second. And so I wanna talk about three new releases that we made in Lanesmith this this past week that all speak towards, this goal of of, you know, trying to make Lanesmith the best platform that... For these proactive voice on agents. I... After that, I'll I'll go into kind of, like, user questions. So if there are any that have popped up and you haven't already put them in the in the q and a box, please put them there. I'll I'll I'll get to them after that. And please upvote the ones that you like. Basically, what I'm gonna do is I'm gonna go down the order of ones that are most popular and answer them. So if you want ones answered, please please upvote them. So there's three new things that we launched in Lingsmith this week. First is basically trace data control. So so traces are super, important. They they they observe what your agent does, and they can be used to to to power the improvement loop. But they also contain a ton of sensitive information, and so you wanna make sure that they're not available to users, who who they shouldn't be available to. And so there's three things we added this week that are... That I think are really impactful for, safe kind of, like, usage and governance and access of traces. First, we added we added new controls over, the ability to see trace inputs and outputs. So what I mean by that is here you can see the skeleton of a trace. You can see what models was called, what the input tokens were, the cost was, the tools, but you can't see any of the inputs and outputs. And this is actually a really useful view to have, and you might want this to be the default view, for for for your developers because the inputs and outputs contain a ton of sensitive information, but there's... And and so... And your and your customers may have, basically, policies or contracts that don't allow you or your developers to see kind of, like, the inputs and outputs. And so by default, you can match them, but you can still get the sense of what's going on in the agent. The next thing you can do is you can basically manage these permissions, like who can see inputs and outputs, who can even access certain traces with, with with user groups. And so, previously, the way that we had it defined is that you could only set this at a user level. If you have different teams, that's obviously very cumbersome. And so one of the things we added was basically team by team permissions with user groups. And then lastly, the last thing we added was you can set trace level access policies using metadata. So what do I mean by this? So, basically, previously, you could... You would... If you if you wanted people to, see, like, the the... If, like... Let's say, by default, you had inputs and outputs masked, and then you're like, okay. I want I want someone to see the inputs and outputs for for particular traces. You would have to turn that on for all traces in the project. And you might have a bunch of different traces from different customers there or maybe the customer always wants... Only wants you to see a certain subset of traces. And so now you can you can set those policies using metadata. So based on the metadata that's available in the traces, you can say any any trace with, you know, metadata foo equals x y z, then, you know, user group a, b, c can see the inputs and outputs of. And this is a really useful kind of, like, combination of controls because by default, what we what we see is that people wanna give everyone in their org access to this kind of, like, you know, skeleton trace. But then when a user complains or when a user has an issue, they wanna turn on, they they wanna turn on the ability for a subset of users to see the traces for that particular user, which is often recorded in metadata. And so now you can do that with with this, and we think that's really powerful. Next, LangSmith for coding agents. So LangSmith is, the platform for, you know, improving and governing your your agent usage, whether it's agents that you build or agents that you use. And so, of course, a really popular type of agent that we're all using is coding agents. And so we added a dedicated view for coding agents in Lanesmith. So you can see here that, there's a few things that we've done. First, we made it so that you can connect GitHub so you can associate coding agent spend with outcomes. What PRs are being created and which traces lead to those outcomes? You... We add a bunch of different statistics that are coding agent specific. So you can see kind of, like, cost per merge PR, cost over time by model. You can group it by a bunch of different things. So you can group it by users. You can group it by model. You can group it by PR type. We we do things like, try to classify the different types of threads, so whether it's a bug fix or just exploration. And and so all of this is now available as a new, agent type in LangSmith. So you'll see here that there's this coding tag. And so if you go into LangSmith and you click kind of, like, create new agent and you select type coding, this is this is this is the view that you will get when you start tracing things here. We also have, integrations with all the major kind of, like, in popular coding harnesses. So things like Quadcode, Py, OpenCode, Codecs, all of them have integrations that you can trace to Linksysmith. And then lastly, the Linksysmith tool gateway. So I mentioned I talked about this. This is a this is a a way to basically secure and govern tool use for agents. So how it works, you've got all these different agents, not just, line graph agents, but you can use the tool gateway with coding agents or kind of like any any eight other other framework you might use. The tool gateway, what does it do? It it it keeps track of the identity of who is calling which tool. It gets a token and and and gets the token and stores it, for connecting to things. So we use OAuth. So it creates and attaches kind of like an OAuth token. It then looks at any policies that are available. So does the user or does the agent that's coming in have kind of, like, the ability to call this tool? Are there any guardrails you want on that? And then and then if all of that is met, it basically forwards the request and logs it to an audit log. So now you have this audit log, so you can see all the all the tool call traces and which agents are calling which tools and which human's behalf, and then it will forward it to the tools. And these can be built in tools that we host. These can be remote MCP servers or these can be kind of like remote tool servers that you host as well. And so this is a way to secure and govern kind of like the the tool use of agents. The very last thing I wanna mention before I go to some of the q and a is that we are running a state of agent engineering survey. We do this every year. It's very interesting to see how the the answers change year over year. We've purposefully kept a lot of the questions the same, so we have this good continuity between years. And so please, you know, go go take it. It should hopefully be pretty easy and pretty fun. And, if you do so, we'll we'll enter you to win limited edition lane chain swag as well. Alright. Let's go to some q and a. Awesome. Bunch of questions. What are what are the use cases that you see persistent agents being explored besides some type of AI assistant? Yeah. I mean, I I I I think, like, every every use case will become a persistent agent. Like, I I very strongly believe that. So I mentioned actively, in in the sales space, building persistent agents. Clay is another company that we just did a a a podcast with Max Agency on that's in the go to market space, and they are also building, towards persistent agents. I think, internally, we have, we have kind of, like, a persistent go to market agent as well. So go to market seems to be a common theme already. I actually think that's one category that will kind of, like, pop off the most. But we but we also have, a marketing agent. We call it content studio. It does things in the background, like maintain, links in blogs, and it actually, scheduled and sent a a tweet that I that I do every Friday, which is basically a roundup of what we launched. And then we also have a coding agent. And so there's stuff, like, reviewing, reviews is is one kind of, like, obvious one, but another one is, incidents. When incidents happen, you want a proactive kind of, like, persistent agent to respond to them and also learn from incidents over time. And so I I I can really see every domain being kind of, like, moving towards these persistent agents. In fact, one one one one thing that someone responded to me on Twitter on when I when I when I tweeted about persistent agents, persistent agents are just agents. Like, I I I think in a few years, we won't call them persistent agents. We'll just call them agents. The definition of agent keeps on changing and keeps on moving forward, and I really think every agent will will become a persistent agent. But it... But in a few years, we'll just call them agents. As agents, oh, we're going to this one. Cool. With the rise of always on agents, memory capacity and scalability could become a potential bottleneck. How is LinkedIn approaching this challenge? Are you exploring software level solutions, hardware partnerships, or other approaches to make agent memory more scalable and efficient? Yeah. It's it's a it's a good question. So so I think there's two different dimensions here. One is basically the fact that a lot of these always on or persistent agents need to basically run-in run-in a sandbox. And I think that that actually compute is is where we're seeing kind of, like, the most, kind of, like, squeeze for things. And so this this is why we built sandboxes into our product. I think it's a really easy way to manage and get this compute. We we use Google on... Under the hood, but we also have integrations with a bunch of other sandbox providers. With agent memory itself, so so my take on agent memory is right now, the actual, like, storage of memory is is not that hard. Like, that's not the hard part. And in fact, most people have, like, pretty small, like, memory things, all all things considered. The hard part is really deciding what to remember and and and then curating that over time and keeping that fresh and up to date. And so we are investigating much more on that... On on on those two things. How do you decide what to remember, and how do you how do you curate that that that that memory? Hopefully, we'll have some cool things to to launch there. As agents start taking real actions like payments, how do you think the rules for what an agent may do with the records and automatic contract should be handled? Yeah. Great question. So so, one, I think you absolutely need, audit log of all actions that the agent, has taken. And so I think I think that belongs, I I think you need something like a tool gateway, and that's why we're building it into Lanesmith. And so I think that that sort of thing belongs there. And and I think those I I I think that that audit log will be really important for things like auditors, and and getting sense there. In terms of, like, what rules an agent can do, I I think this is, I think this I think this comes down to two things. One, I think it comes down to the permissions that you set on the tools themselves, and this is again why having kind of, like, a tool policy layer. So so the two things are two... Or or tool gateway does a few things, but two of the things it does, one is this auto log. This other is this policy layer that you can set so that agents, can call a tool, can't call a tool. They can call a tool, but there's human in the loop, so a human has to approve it. And so having this policy layer... And and you can also imagine adding in semantic policy, things into that. Like, the agent can always call it with, like, these benign inputs or something like that. That will become really important. And then the other thing is alignment of these agents. Like, I really... Like, alignment has always been talked about as an as an as an issue. With with these persistent always on agents, I think it's way more of an issue than with these shorter task based things. Because now these things are running in the background. You don't see what they're doing. It's a huge issue. Can you go deeper on the continuous learning loop? I get it. High level. Collect traces, spot error patterns, make code changes. But how do you gain confidence that the changes are truly improving? Spotting error patterns, can we customize how this is done? What are best practices here? Yeah. Great question. I think there's, I... Okay. So I think there's three pieces to this. First, evals. Evals are kind of like how you, have confidence that the changes are truly improving, both online and offline. So offline evals can be used if you're making changes, run them before. Online evals can be used as a way to AB test things or just, like, make sure that when you make changes, things don't regress. Second important thing, traces. How do you spot error patterns? It all kind of, like, comes down, to traces. You need the system of record of what agent's done. Those are those are super valuable. And then the third thing that I'll say, and this gets more into, like, how do you how do you spot error patterns and how do you customize how this is done. So so, I think this whole process will be very bitter lesson. And so what I mean by that is, like, we we we have the engine. Engine does exactly what you're talking about here. It spots error patterns. Under the hood, it's just a coding agent. It's a coding agent with a particular set of skills and a particular set of tools and CLIs, but at the end of the day, it's it's just a coding agent. And and and when we build engine, we very purposefully try to lean into... Like, I... When when I think about memory as well, memory as well, I... And and when we talk about kind of, like, how do you curate memory, it's just gonna be a coding agent. I think a lot of these complex systems for doing memory are are gonna get better lessened, and it's really just gonna be a coding agent with a particular set of skills and a particular set of tools, running against, and running and updating something. And that's same for kind of, like, spotting error patterns. We're making it way easier to customize engine. We're gonna have some really cool launches first week of December that I think will make engine, even more goal oriented and even more persistent. And and and and I think will make it easier to customize as well. Can you comment more on how memory is being architected for these agents? For example, at LinkedIn internally, is it approach to build a Wiki or is there some... Yeah. Great question. Spending a lot of time, thinking about, thinking about memory these days. I wrote I wrote a... Like, I kind of touched on it. I I I I think memory will get bitter lesson pilled in in some ways. I, the way that we approach it at LaneChain... And, again, hopefully, we'll have more things to talk about here soon. But, so so I I I actually don't think the four... The storage format of memory matters that much. And I think there's different types of memory, and they might just need different storage formats. So for example, skills are totally a type of memory. Skills are procedural memory, how you do things. And so you're you're gonna wanna maintain skills, and skills have a specific format. Likewise, there's other kind of, like, just, like, knowledge and that might be best stored in OKF or in markdown files or Wiki or maybe even a graph database. Hex wrote a really good article and they... How you... They use graphs under the hood. And and so the storage format, I actually don't think is the super important thing. I think there's kind of, like, two key things to think about, and I touched on it briefly. But, one is basically as the agent runs, can it make updates to memory? Can it not make updates to memory? And and generally, where I think we're aligning internally is that it can... Some... Sometimes, it can... Or it it it can always it can always basically take notes because because as the agent is running, it might notice what's missing, it might notice what's out of date, and you don't wanna have to rediscover that later on. That's super valuable. And so you wanna give your agent a way to take notes as it's running. And then and then a separate question is, like, whether those notes are then immediately visible to other agents or other threads or whether you wait for kind of like a curator agent. And the curator agent is the dreaming agent. It's the agent that's running in the background. It runs on some schedule, and it will basically look at the notes that the agents have made. It will look at other sources as well, and and those sources, depend. So if you're if you're talking about skills, which are procedural things, how you do things, the best source to look at is agent traces. That's super valuable. If you're looking for just, like, arbitrary things of information, agent traces are valuable, but so are other things. Like, you might wanna look across Notion or across Slack and see other interactions and then start to incorporate that into a company knowledge base. One term that I that I used yesterday is, like, a knowledge factory. Like, I think you'll wanna build these knowledge factories, for for keeping knowledge up to date. And I think there's different types of knowledge, which makes it hard to have a generic thing. And so I'll probably I'll probably write a more detailed blog on that in in a in a bit. How do you think about these person agents running in one long session versus when you write to buffer long term memories and start a new session? Perhaps this is what the Dreaming concept is all about. Yeah. I think there's, I mean, so so so so one related thing is basically, like, for these always on agents, do you have one big thread or does does each time you message it in Slack and whenever, it starts a separate thread? And I think, like, I I I I don't know if there's a right answer here. Like, I think things like Muse and Instinct on the surface have one big thread. By the way, it's totally possible to have one big thread on the surface, but under the hood, actually use different threads based on kind of, like, the time and just have a really good memory system. I I don't know if I have a a a great answer here. I do think the... I do think regardless, you you want to give agents an ability to search other conversations, whether that's, like, one big conversation and you just give the agent the ability to search previous messages or it's a bunch of threads and you just give the agent the ability to search across threads. Both of those are important. All of that is important. And so, that's the important thing. Whether... And then whether it's one big thread or a bunch of, a bunch of separate ones, doesn't matter so much. We're at time. There's a bunch of good questions here. I I don't a 100%... I'm not gonna be able to answer all of them because I'm scrolling down. There's a lot. But, at at Linkchain, we're we're... I I I do really think we're at the start of these persistent agents. We're gonna put out a lot more content on these. I will I will look at a lot of these questions. And if there are repeated ones around memory or stuff like that, either I or our team will will will do some stuff on that or either a webinar or blog or things like that. So thank you all for coming. Thanks for all the questions. Let's go let's go build some agents.