Name: GTM Engineering Live: How We Built Our Paid Media Agent
Uploaded: 2026-09-23T19:30:50.091Z
Duration: 1 h 2 min 16 s
Description: GTM Engineering Live: How We Built Our Paid Media Agent

Transcript for "GTM Engineering Live: How We Built Our Paid Media Agent": Agents are a new kind of software. Endless inputs and non deterministic outputs. Build them the old way, and they break. Enter Lingsmith, the agent engineering It's model agnostic, cloud agnostic, and framework agnostic. Organization is shipping the best agents iterate with a system, and we call that system the agent development life cycle. Build, test, deploy, and monitor. You build with our open source frameworks. Deep agents, langchain, or langgraph. LangSmith Fleet enables anyone to build agents without code. Test agents using evals and experiments. Deploy changes in one click with LangSmith Deployment. Monitor every interaction from a single dashboard. Governance is built into every stage of the agent development cycle with Lanesmith LLM Gateway. Lanesmith Engine helps you improve agents autonomously and move through the development lifecycle more quickly. Blanksmith Engine is your agent for agent engineering. Welcome, everybody. Hope we all are having a good afternoon. Looks like a lot of evenings, some Europe, Chicago. Well, welcome in, everyone. So I'm Danny Lambert. I lead the GTM engineering team here at Langchain, and this is actually our first, installment of a new series that we're calling GTM Engineering Live. So once a month, for the foreseeable future, we'll be doing a virtual event just like this as well as an accompanying blog post, where we'll go deep into, internal agents that we've built. Essentially, what we built it for, how we built it, what we learned along the way, and then we'll leave twenty to thirty minutes at the back end, for q and a to answer any questions that anybody has about that specific agent, or Langchain more broadly. So our focus today is going to be on our paid media agent. This is an agent that we built for the marketing team. They were essentially looking to add and scale paid media as a way to build pipeline for our sales team across six different ad channels, and wanted a way to do that efficiently without having to hire additional headcount so that we could scale that up and make it contribute to our business, much faster than we normally would. So a little bit of information about what we're gonna cover before we hop in, is we'll do a little bit of housekeeping upfront to familiarize yourself with the virtual event space. If you haven't been on one of these events before, we'll quickly hit on our team as well as Linechain, and then I'll hand it over to Amal and Jan that will go into how we built the agent, give you a live demo, talk about, architecture and learnings, and then we'll close out, with, q and a at the back end. So a little bit of housekeeping before we jump in. I think the number one most asked question that we get on every, virtual event is, is this gonna be recorded? It will be recorded, and it will be available on demand, shortly after the event is over. Inside of the event experience, there's four places that you should be looking and engaging, if you choose to during the event. So probably the most important is all the way to the right, you'll see the q and a section. Currently, that is preloaded with all the questions that came in on the registration form. But if you guys want to add your questions in there, and I encourage you to upvote the questions that you find, the most interesting during the event. We'll try to get to as many of those as we can at the end, but we'll use the prioritization of the upvotes as the ones that we answer first. So definitely put your questions in there. Definitely upvote, what you find interesting. Next to that is docs. There's three links that we've put, in there to resources related to what we're covering today. So there's a link to the blog that goes in deep on some of the content that we're covering, as well as the open source repo for this agent, that you can clone and you can self host or host on managed deep agents. And then there is a link to the GTM engineering series. So if you like this content and you want to register for forward, events, you can do all of that there. And then the last one is the personalized demo button. So if you're building agents and you wanna learn more about Langchain or LangSmith, anything that we cover in this content, you click that button, and a member of our team will follow-up with you after the event is over. So before we jump into the actual content, let's talk a little bit about the team and the presenters. So as I mentioned, I'm Danny Lambert. I lead GTM Engineering. What GTM Engineering is at Langchain is we essentially build all of the, agents and automations to run our go to market team. And GTM is quite broad. So that's everything from, like, top of funnel functions like SDR marketing and partnerships, through to, like, post sales, technical support, account management, AMs and d DEs. So this team behind you builds and maintains all of the agents to make those teams more efficient and effective at their job. So it's myself, Jan, who is our software engineering manager, and Amal, who did the lion's share of the work on the paid media agent, and we'll be covering, you know, a lot of the content that we're walking through today. And just kind of to round it out for people who are maybe less familiar with Langchain, as you saw in that intro, video, we provide the plat platform for the entire agent development life cycle. So that build, test, deploy, monitor life cycle, helping organizations, move faster through that cycle and more reliably move through that cycle. We currently have half of the Fortune ten as customers, over 7,000 total active Langht Smith customers, and more than 200,000,000, monthly open source downloads, which is on par with the OpenAI SDK. And as you can see along, the bottom, we support a myriad of clients across a bunch of different industries and use cases, inclusive of Clay, Harvey, LinkedIn, Home Depot, and others. So pretty broad ranging experience of the different types of agents and helping organizations move those to production. So without further ado, I'm gonna pull up Amal and Jan, and we can get into, a demo and a little bit more on the paid media agent. Awesome. Hey, everybody. Great to great to well, not see you, but, have you all here. My name is Amal. Janice here as well. And, very excited to talk to you about our paid media agent and, share some learnings that we have, accumulated over time we built it. So let's start with just from the beginning, what is the paid media agent and what was, like, kind of the core functionality around the agent? Number one is primarily, you know, we have we're pushing, ads all on six platforms, ad platforms, Google, Reddit, LinkedIn, x, etcetera. And we need a way to, intelligently aggregate that data across all these all these platforms and provide reporting, analysis, recommendations, based on team's judgment. And so a big part of the agent is providing that weekly reporting, from several data sources, aggregating data from the warehouse, all the ad platforms, and providing judgment recommendations in the report. But also being able to proactively manage the campaigns and create new ones across new regions, new audiences, across platforms, as well as retarget, edit existing campaigns, run experiments. So the idea is just to to be able to enhance and automate a part of the workflow the team is managing on the paid media side. Some quick rundown of, like, the core value that we've so far have been observing. The agent, in addition to the paid media team, we the line change has increased the marketing pipeline about 20% in the last six months. We also have lowered our cost per per lead, per qualified lead, while also, increasing the monthly spends. There is a lot of demand for paid media, work at LimeChain as we scale to new markets. Well, also, we were able to save around 5 k per month on just, replacing the, reporting, by creating the agent and doing it all in house and automating that part. So without further ado, let me just share real quick how the agent works, in Slack directly and some of the, some of the core workflows that it has. So let me share my screen. Awesome. Okay. So we have our Slack paid media webinar channel. This is demo data, just so you know. So we have I'm gonna start the first question, which is gonna be around generating a weekly report. And while it's running, because it takes maybe, you know, a minute or so, I'm gonna run a couple more questions in parallel starting with, the following. So what I wanna do here is I want to compare our all five channels, Google, LinkedIn, Meta, and Reddit, on efficiency and the pipeline contribution, and kind of kind of ask the agent about what would you scale, which campaigns would you run more, what would you cut, would you rebudget anything. So, like, get that judgment from the agent based on the internal Wiki and documentation that it has from the team and from us, and can ask for recommendations what it's gonna do. So I'm gonna run that. And in Slack, the agent is operating with the team right now on some of the channels. And the reason for Slack is primarily, you know, you it's a great workspace where the work already happens. You have the teams working together, sharing reports, sharing analysis, asking questions, and the agent fits very organically into the entire workflow. And so now we have the agent running in the in the background. We can stream our tools. Right now, it's reviewing some documentation, internal wikis, calling tools, to LinkedIn x ads. It might be querying the BigQuery, to get some of the warehouse attribution, various, dates. Let's say right now it's doing, like, the last month or so. In the second one, I get some kind of results from agent, hopefully, with a great judgment about what to do next. And while that is running, let me check-in on our reporting. Also doing a bunch of work here, and I'll explain exactly how reporting flow works and how, the overall orchestration of the agent is is being done. So it's looking at meta, looking at Reddit. And just so you know, we have all these tools orchestrated, using the the some of the MCP servers that are available on the market as well as some custom tooling around the warehouse. So let me check-in back with the weekly one. And, actually, while this is running, I'll also run another one so that we can have some more examples. In this case, well, we have the weekly already. But in this case, I'm gonna also ask about a campaign, editing workflow with human in the loop attached to it. So I'm gonna say find all active Google search campaigns targeting The US market right now and prepare new versions of that targeting Germany and The UK with the same, audiences, similar keywords, but also optimized for, those regions with a bit of less of a budget. So 25% of the original US budget. And I'm gonna run that and see what it does. Now we have our weekly reporting being done, and this is something that's being done on a weekly basis on Mondays and provided to the team, in the mornings. It does report across all platforms in the last fourteen days. We can see here that it's giving me a per platform picture from Google, LinkedIn, x, and Meta, and Reddit, with some specific data around the particular campaigns, their CPL, cost per lead, which is this one looks like to be a total top spender, across all of the campaigns that we have. And, again, this is demo data. So just the just the over overview of the demo data. Now some recommendations here is let's see. So it's just saying, like, some of the things that we have are not set up correctly because it's demo data, but overall, we're gonna have some more recommendations in our reports here. And this is how we we get the reports. They're branded PDF reports and and language and branding that we get, for each platform. And in this case, each report is being done by the sub agent, delegated to that particular platform and data sources. We get our executive summary with the core metrics, and what is performing well, what doesn't, what needs to be updated or scaled. We have the summary across our spend, impressions, cost per leads, conversions from the warehouse, and, as well as any kind of, outcome measurement around what we track specifically internally, like content download conversions or webinar sign ups or product sign ups, as well as summary per campaign across the last fourteen days. And so that that will be done for each platform. And based on that, we can then make decisions of or of of what to do next, either by following the judgment by the agent or using that as as as a useful context for next steps. So that's reporting stuff. Let me go back to, this question right here. I asked about our five channels and what would it do, just overall, like, what would it recommend to do next? So it's saying here that across the last thirty days, it would increase the spend, I guess, or, oh, I guess. Yeah. So it's reading the the core metrics from those platforms, and it would lean on scaling. So it says, X and Meta carry the strongest pipeline contribution. And so they're cleanest candidates for increase in in budget for incremental budget. And then Google is the efficiency, its smallest spend, but it's high intent capture. So, you know, it understands the judgment and understands that would what, but would would need to scale X amount more because but to get some great results there, while Google is becoming pretty efficient at how we spend our budget. And then the last one, which is the most exciting, I guess, is it found the, U The US Google search campaigns in The US, and now we want to target or retarget them for the new audiences and new regions. So, it's gonna move our, this campaign to Germany at $35 a day, which is 25% of $140 per day. And so I'm gonna say okay, Ron. And what's gonna happen is it's going to create an approval workflow where before staging or creating new campaigns in those regions across platforms, is gonna first stage the edits for approval, and then the team members can, look into what is being updated, edit anything within the Slack block UI kits. And then on the proof, it's gonna call the API, update directly the platforms, and around the campaign or run the updated workflow. So just not to spend time, I guess, I can take a look at some of the ones that we had here. It's the same example. This is how this lock u the block UI looks like. Actually, it's done. There we go. So we have the first campaign for Europe, Germany. We have campaign type, all the same things that we had for The US campaign, the same settings, daily budget, delivery is paused for now, as well as the second one, which is for The UK, the same strategy, the same different market, and then the same one for, another Germany campaign and another UK campaign. And I can edit details, open the form, kind of update as I go, or approve or reject. So I can update any of these fields, have interactive forms, save changes, approve, stage, and go. So that's high level how the the thing works. Now let me share how it works from the architectural perspective, what's what's behind the whole thing. Now high level, as you saw, we have the Slack mention triggering the agent or a CronJob that's triggering the age run. In our case, it's weekly CronJob. We have the main paid media agent graph, who are using deep agents that harness, which is our lang chain harness, that runs on several profiles. We call run profiles. We'll talk about this later. We have our tools and the host that injects the credentials. The agent is operating in the sandbox by default, and so we need to inject the credentials at the tool calling time, instead of bringing it all into the sandbox. And so we have the host that executes the tools, injects credentials. We have MCT and warehouse access. And then we also have two modes. One is for just general q and a, and one is for, weekly reporting, which you saw where you have sub agent per platform orchestrating data and providing analysis back to the main parent agent, and that main parent main paid media agent is giving the the result back to, the user. And so a bit about the stack. We have our, interface, which is Slack right now. We can talk about Slack limitations as well, by the end of it, but Slack is fantastic for all of this, all the teamwork with an agent. Then Then we have the t page and harness that takes care of all of the context management, context engineering, planning, calling tools, abstracting away large tool calls into the files and working with the file system natively. We have our context, which we have a ton of Wiki, from LLM Wiki Biography, if you've heard about it. We have a lot of instructions and skills all copied into the sandbox where an agent is operating natively. Then we have tools, as I mentioned, add platform tools, MTP servers, and warehouse. Then in terms of the infra itself, we have, our lines by sandboxes that we operate from, which provides a secure environment for agents to execute code, run skill scripts, and, generate reports. And the platform itself, which is LangSmith, we have our deployment on LangSmith. We have tracing on LangSmith, a gateway for cost tracking on LangSmith, as well as the eval, evaluations as well. Now a bit about the harness, how deep agents is operating overall. We kind of take the abstraction that Langchain provides with the deep agent harness. You can think of it as the same way as as codex or cloud code harnesses where the the model, the LLM is operating in that run time, calling tools on loop. Right? And then there are several extra, tools around the the model, from context engineering tools. Like, again, if it's calling a tool with a large output, it's subtracting it away into the file system, so it can just read the file instead of getting all of this large tool output into the context window. That's obstructed by the deep agent's middleware. Then we have the orchestration of sub agents that's also obstructed by several, great functions. We have the file tools that the agent can call to read files in the sandbox, to read files outside of the sandbox. And then the approval workflows, which are primarily around if you wanna stop the graph execution and just have that Slack block UIKit rendering and you wanna approve something, the it's stopping the graph and, and and and showing that in in, and stopping the run of the agent before the approval is is completed. And then on top of this harness, we're adding extra layers, which is our instructions, our wiki, our skills, and some artifact templates, also more of a custom tooling for Slack, and the approval rules specifically for, like, new campaigns, for example. Now something interesting I wanna talk about here is which is my one of my my favorite parts of of the agent orchestration here is our agent profiles. So the idea behind agent profiles is that you want to create, a a custom profile and and inject, particular context, skills, tools, even maybe sandbox snapshots based on a user request. In our case right now, it's very simple. We have our Slack workflow, which is q and a. We have a Slack profile that, is triggered where we inject all of our core tools, the warehouse tools, and some core rights, approval, rules, right, for for campaign rights. And then we have the weekly schedule that has a lot smaller, profile with smaller tools and and and less tools overall and, like, specific skills that we inject into the sandbox only for that, run. And so what this allows you to do is, number one, be a lot more dynamic and and customize your your experience for the user. For example, we have some other agents internally, like the GCM agent that we talked about before that has this at scale for, 200 plus users with with custom profiles for each. And I I I would let Jan to to add a point here on the agent profiles. Yes. So as we have grown, the complexity and number of users and diversity of users of our system, we realized that it was better to create one single agent with access to a wide range of tools, a wide range of skills, and then use a factory pattern to select depending on the user, on their configuration, where was the agent being called from? Was it Slack, a UI, a background process? To configure the environment of that agent based on on our configuration? So what Amal is showing here is a small example where we truly have essentially two profiles. But as our systems are scaling, we're realizing that we have almost, like, 90 different versions of one one single agent. And, Amal, if you go to to the next slide, a lot of the environment, is actually configured by its computer, its environment. And we've made the decision not only in paid media, but in our broader GTM agent to effectively always use a sandbox. And one example of this is imagine you are hiring an analyst into your company, and you don't give that person a computer. Then it's gonna be really hard to perform analysis, to offload context, to organize information. So a lot of what we do, and even if it's not a coding task, because the models are trained to do coding, they perform really well on all of these different tasks if you shape the environment as a coding environment. And you'll see that by having sandboxes, the context will be managed a lot better. The models are, like, naturally trained to offload things into files, into particular markdown files or CSV files. And you will see that with the deep agent harness because we have tool eviction, meaning that when the tool output is over typically 20 k tokens, you would export that into the file system as as a file. The agent would just perform a lot better. And I think, Amal, you have some some metrics on the experimentation we did without a sandbox and with a sandbox. Yeah. Right. Yeah. With the sandbox, we have had quite a a lot faster and great better experience for the agent to manage its context overall. And to Jan's point, managing, evicting things on the file system, being trained on post ex semantics and reading reading files overall. So and and I think a part of it is also about how do you design the workspace for the agent. Right? If it's now performing in some computer, in a sandbox, there is an a bit of an art to creating the, the right context, that it can query, they can they can grab, across your, your agent use case. And so in our case, we have this workspace folder that we copy into the sandbox in every run that includes several things. It includes the core instructions about how to work within the workspace. And, you know, many of these things you might be familiar with if you work with agents locally in your projects, in your repos before, where you would have your agents. Md file, usually just providing some kind of a general instruction how to work with your project. In this case, providing, more of a, like, a context map into how to think about, working with this with this project, how to go through, how to find files, how to find skills, and what's, like, the general logic of paid media at LiveJain. Then something interesting is that we have the index.md, which is a a pattern that we got from LLM wikis for carpathy as well, where you just have, one file with all of the paths to everything else that is created, at deployment time, where you have where you where you create this provides paths into all the skill files, into, wiki files. And so the agent can just look at this index, query it, look at let's say, this is to find a skill like paid media skill, looks only at that gripped, line, and instead of reading everything all at once, going one by one. Right? And so we have skills, then we have some other paths around the the workspace for orchestration of tasks, and then we have our company wiki and a path for large tool results that is subtracted by deep agent's harness. Now, let's see here. Yeah. I think something interesting here about the context map is we have, the system prompt of the agent. It's not really too long. It's, like, pretty short because what it does, it just provides general rules and rule, but also just provides a a way to think about querying the right context in the in in the repo as I mentioned in the in the sandbox. And so it just says, you know, to find things, index read index.md, grab what you might need semantically, and then index is mapped into everything else into the workspace. Right? And so by the index by reading through the index, it understands that there's a path for this skill, there's a path for that Wiki, and then it reads that path and finds the skill very fast and more into two calls. Overall, I think it's it's a very interesting, paradigm of, like, orchestrating the context and letting the agents figure it out and following the better lesson overall so that it can scale, as we add a ton more skills and company, wiki, and and, templates and all of that context. I And, yeah, Jan, if you wanted to add a bit of a bit more here. Yeah. So if if you jump to the next slide, you'll see that in this age in particular, we have six skills and 19 Wiki files. But in our broader GTM system, you can get up to a 100 skills, many more Wiki files. And I think the main question here is, like, how do we come up with these skills? Because you can go online and and find, you know, from a lot of different companies, pretty standard, almost like vanilla skills. However, the performance of these agents is good because we spend a lot of time just being embedded in the business, talking to the experts from marketers to salespeople to deployed engineers, and we're constantly capturing the nuance of the business. So the the more time you spend almost, like, interviewing and compiling knowledge in markdown format, and then you bring that into the agent, and then you go back to those experts, you show and you let them play with the agent, and then you iterate over and over and over, that nuance, that last mile is really where I think the the true value in these internal agents, live. Totally. Yeah. And it's it's interesting because we we do spend a more majority of our time, like, really trying to understand how the team thinks about the problem and asking the questions that would be much, I guess, understandable by our local agents to implement very fast in the code base. And so it's more of like interviewing for our agents. So something interesting that I wanted to add based on the sandbox question that we had before, like, why sandbox by default? Jen has shared some great, comments around that. I wanted to mention that we also have the on top of just running files, running a code in the sandbox and executing scripts that can be tied to skills. In our case, since it's data analyst agent, there is gonna be a lot of warehouse tooling with large data, tool outputs with, and so the agent needs to run Python functions to analyze data, efficiently. But something interesting here is that just like the analogy that Janice provided around, providing an an analyst, a new hire with a computer, we also provide this computer already prepackaged with a ton of software, a ton of libraries from, obviously, Python to Pandas to DuckDV for to open by Excel, MacbotLib, and any kind of PDF libraries for, to be to to make it easier to render PDF and work with PDF files in the sandbox. And so it's almost like when you would provide the computer with bringing stuff software. We'll also start baking several things into the snapshot, like skills and and wiki. And the reason for that is and we have run some experiments. When we didn't have since we have so many skills and Wiki files that might be a little large, when you start the agent and you trigger the agent at runtime, when you copy skills and Wiki and all these files into the sandbox, it takes five to ten five to ten seconds or so. It's it's not a great experience in Slack overall, especially for faster q and a. And so what we found is it's kind of interesting if if you were able to, take those bigger files and inject them, into the into the, Docker image and then use that image as a template for your sandbox. And so, in addition to the software and the libraries, we're also adding some of the skills and bigger, context files into the snapshot. Right? Now something very cool about this agent is that we have 200 plus tools across, all the paid media channels that we have. And the reason for that is some of the providers that we have are providing a ton of the functionality, for covering the, kind of, covering the API, the raw API calls from the price from the app platforms. And so they cover all of them, packaged as as MCP server, and that MCP server provides 200 plus tools. Right? And when you if you were to inject all of these 200 plus tool descriptions into the context window, it's gonna equal around 120 tokens, maybe even more. What we had was around 120,000 tokens. And so what we found was it was not scalable because number one, it was expensive. Number two, the agent does not need to have all of these things in the system prompt injected in every run. And so what we did was we abstracted away by three tools. We took all these 200 plus tools, deferred them out of the sims out of the system prompt, and added a layer between them, which is a search ads tool, read ads tool, and read ads query. Right? So search ads tool would provide a, the agent can search they use that tool to search the tools by intent. That's something that Cloud Code is already doing. Codecos was doing pretty well as well, like, just the the paradigm of, like, searching across hundreds of tools. And so in our case, the agent can search as tools and the argument provides provide, like, a semantic, search. And then the middleware that we've built in deep agents is is querying with, by regex and then providing the, the the results back with the list of tools that and tool names and descriptions. Right? Now something interesting about isolation here is, in addition to all the orchestration we do for tools, we also decided to do some of the orchestration with sub agents. And the reason for that was primarily when we have weekly reports, we do we do need to have them, to work in parallel across all platforms as a lot of data aggregate. And therefore, if you have a sub agent per platform, you can parallelize everything much faster and and and have the the core judgment, the main model of the main agent that, delegates the parent agent, that delegates the task to sub agents to take care only of the judgment piece and and packaging instead of running all these queries all at once and going in a loop, for longer. And so what it does is just, isolates it into, each sub agent per platform. Each sub agent has their own tool to query that particular, Google platform, let's say, or Meta or the warehouse, create a PDF report, and provide it back to the main agent that then looks at that, reads, provides judgment based on the skills of Wiki that we already have. And something interesting here as well is when we do think about all of the all the system that we've built, how do we make sure that we keep track and gonna self improve as we go? Right? There's so many requests coming from the teams, and we wanna make sure that it's scaling efficiently. And so the way we have kind of organized the self improvement loop is right now, we have the the memory in our deep agents, which is this agents that indeed file where the agent on every run is appending learnings. And then on a weekly job, we look at that agents that indeed file, and then the agent that runs weekly would look at that, file and then extract all the learnings and update skills, or wiki files in parallel, and then that would be the updated version of them that then goes back into the into the new run. And then we clean up the agents that indeed filed to be shorter and to to allow for more addition, to this this file later. And the alternative to that is we have the protocol LangSmith engine that kind of takes that and attracts that away because it looks at all your traces. It looks at core findings. It looks at core errors that you might have with an agent. And, like, an agent for agent engineering would provide improvements, create a PR with with updates to, let's say, a tool call or, like, to a skill. So that kind of obstructs it away. So we're kind of moving from custom made self improvement to a locksmith engine improvement, which makes it a lot faster for us to review overall. And, one more thing I wanted to mention here is in addition to locksmith engine that abstracts it away for us, We also have the managed deep agents that we host as agent on where instead of all of us thinking about infrastructure, the sandboxing, the, the scheduling, the memory, and the tracing, and the Slack profile and Slack integrations, the managed agent's gonna abstract it all away. And just with one click, we're able to migrate and and and, have this agent deployed, with all the infra managed for us. That's something that's really useful that I found as I was engineer as working on this project. It took me a couple of weeks to self, self design the the infra, and we managed the page and I was able to do it quite fast. So without further ado, if, Jan, if you have any more comments you wanted to add here. No. I think we we could jump into, into q and a. Yeah. I know it was a lot of information. I'm sure you're gonna have a lot of questions, so we're happy to do so. I think maybe we'll just cover one quick slide before we hop into q and a for all the folks who had missed in the beginning. If you're looking for any next steps and resources on everything that we just talked about, they're included in the docs tab in between chat and q and a on the left. So this entire agent has been open sourced, and that open source repo is linked in the docs as well as a blog post that goes deep on some of the stuff that we talked about here today. And if you're interested in this type of session in the future, there's a link where you can register for our upcoming events where we talk about, building our custom agent UI. We're gonna talk about, our content studio agent that is essentially a multimedia, tool that allows us to produce any type of, image, video, audio, etcetera using an agent. So we're gonna have a a pretty wide variety of topics coming up. So if you found this interesting, feel free to register for the ones that we have upcoming. And with that, we can hop into, q and a. So I have queued up the top voted ones. Let's pull one up on screen here. How do you know when to use deep agents versus only using Langraph with five upvotes? Yeah. I can I can take this one? So deep agents is the higher level of abstraction of our frameworks, and you should think of that as a very sophisticated loop, agent loop. This is basically how codex or cloud code works, which is a loop with, lots of, like, in between steps, across the life cycle of the loop. That's what we call middleware. Land graph is lower level in that you can define deterministic workflows following nodes and edges. And so if you wanna build something that is much more deterministic, and now that we have cheaper classifiers and things like that, lang graph could make sense to increase reliability and reduce cost. But, typically, every complex agent will now be built on something like deep agents. Awesome. I'm gonna skip past a couple of the top questions to go to two that came up recently that were particularly relevant for, some of the more recent slides starting with this one from, Megan. What is the media agent's balance between having enough but not too much context and instructions in the system prompts versus delegating information to index.md and different wikis to preserve behavior, especially with 200 plus tools and skills? Yeah. That's a great question. Just to understand it correctly, I guess it would be around when is too much context. Right? When when do you have too much stuff going on into sandbox? Whenever you we whenever you add more stuff in into the agent. And I think, you know, we're at this point with even with the with GTM agent as well where we have hundred hundred plus, you know, tools and skills as well. And, the way that it scales right now is just what as I mentioned before, we we are on our on a on our own as engineers, very particular around what kind of skills are going into, the sandbox. If we have some kind of duplication, across skills that we already have based on request that we have, then, we're not adding this. We're just appending or updating it. Right? So it's about, like, looking at what you already have, cleaning it up if necessary, running those weekly jobs that would clean up the memory and update learnings. But also the even with the approach that we have right now where there is a context map going through index this from system prompt to index that MD to everything else in the workspace, the approach scales pretty well just because the if the if the model is smart enough and we're using frontier models primarily, then it can find real information very fast by using the file system tools. And I know, Jenna, if you wanna add anything else here. Yeah. And I've seen a bunch of questions that are kind of answered by the the idea that models are really good at agentic search, and models are really good at using POSIX. So, essentially, navigating through file systems and finding the information that they need. So I was looking at one of the questions that it was about, how is this different from putting all of these context into a knowledge base, a rack system, and all of that. So we're we're not doing rack at all. We're purely relying on files in a file system that the agent can, can search. DeepAgents comes with all of the file tools, that allows the models to find find the right files. And we're effectively using that for many things. First, finding the skills, offloading context, running computation or calculations in the case of the paid media agent. The broader concept is if you can push your skills and knowledge to be progressively disclosed and keep your system prompt for core behavior and orientation of that agent across the environment that you're creating, then you're getting the best of both worlds. In this case, this agent has 200 tools, but I'm sure that eventually these, systems will get a thousand tools, and they might get a thousand different skills, and you wanna be still context efficient. So, our system from you know, think of 20 k, 30 k, something like that ideally. And then everything else will be the agent finding finding the skills and the tools that it needs for that particular task. There are a couple different flavors of this question. So, I think this one was the best articulated, and we can kind of, collapse a couple of them into one. But how do you evaluate this agent, and do you have a continual self improvement loop in your agent Wiki, etcetera? Totally. Yep. So on the evaluation side, as I mentioned in the continuous improvement slide, we we do have the lines with engine that obstructs some of the logic away by running the looking at traces from the last several weeks, running the agent on their own harness that looks at what can be improved and optimized with its own agent engineering skills. So that, you know, that that part, we are looking and and using as as much as we can because it's pretty useful for us to afloat that part. At the same time, when it comes to how we judge the system, since it's so specific to our internal logic, if you have, like, LMS judge, for example, it would be a little hard to evaluate, the the output of judgment based on skills on Wiki that you have, that you collected from from users in your team. And since the agent does not run as much on at scale, we have, like, a smaller team that runs paid media, it's much easier to have a more custom or, I guess, a personalized approach, by working with the team, asking them if the output looks correct. Is there anything that they would like to update? They can also update the skills directly if they want to. And so it's more of, like, if if it was more of, deterministic flow, that would be easier to evaluate, but this is so much of the, semantic and judgment that even if smaller LMS judge would work here for evaluation, that'll be harder to understand, like, what's the exactly the benchmark you're looking for. So we're optimizing mostly for whether the output is, satisfiable by the team and the sound that they're looking for in the output. But I would also add that at the end of the day, evals are a proxy for user satisfaction. So you could have better scores in evals while having users less happy with the output of your agent. And in that case, what we always try to do is have, especially because it's an internal agent, a very, very quick feedback loop with our users. In fact, we sit next to our users, and so we very quickly understand what the ideal scenario is. And once once we can capture that, then we create the evals based on all of the feedback that we're getting. Because, otherwise, it's very easy to optimize for an LLM as a judge that is let's say in the GTM agent checking the quality of an email while your reps are complaining that the quality of the email is not good. Well, the problem was actually that the element of such ads is not quite representative of what what they're looking for. So when it's an internal agent, you can have the luxury of very quickly pushing changes, and we're pushing changes, like, every half an hour, every hour, very, very quickly. That way we get close to the ideal, reality of the agent as opposed to an evaluation. Awesome. Let's pull up this one from Peter. I forget which section, but he said, I'm not sure I follow. Are you proposing sandboxes as an architecture that saves you from having to make and maintain one agent for each task? Are you offloading some of the design decision making to the element runtime? For example, which tools and which data sources to use? I can I can take this one? Just to be clear, we are spinning up one sandbox. So, effectively one computer per thread. So every invocation of the agent will have its own computer with that persistent data layer. So it's hard drive and memory all at once. Then if you could pull up the question again so I I make sure I answer everything. What we do pay attention, we pay a lot of attention to making sure our skills are very, very concrete and nuanced. And so if you read our skills, it would be, like, extremely specific to the company, the process, and and we try to capture as much judgment as as possible. So, really, the the agent is not necessarily coming up with, like, completely random ways of of operating each time. The skills are very, very prescriptive. It just have a lot of freedom to use code and its environment to essentially answer edge cases that will come up. Because the the situation typically is that once you put these agents out and more people start to use them, they will start using the agent for things that you didn't design the agent for. And we've noticed with many of our systems, most of the use cases that we see in traces were use cases that we didn't build skills or even tools for because people just get really creative with combinations of tools and, like, their own particular workflows. So the way we try to solve that is, one, focus on capturing good judgment in skills. Second, make sure that the agent has degrees of freedom in its environment to call tools and use a sandbox. And three, build good governance at the edges. So before the agent actually takes some action in the external world, make sure you're gating that very clearly. And so that way, increase the freedom and power of the agent will make sure the gating is as strong as possible. That's how we we typically think about it. Okay. I, picked Harmon's example, but there's two or three others. The general sentiment in the others is, like, how do you determine which LLM to use? But his specific example was, do you use the LLM model for all skills or sub agents, or do you do LLM selection, which is a best fit for the specific task at hand? Yeah. Yeah. I can I can take the part on the paid media, and I know we have a lot of these, profiles for GTM agent that Jenny can take care of as well? So on the paid media side, we have, the way we think about it, for the agent is that more of these questions are data and and analytical questions and are pretty complex. So I do want to have a bigger model, a more front team model running. It the tasks right now are mostly the ones that need more of, like, running a little longer in the loop, querying data from different data sources, aggregating data in the sandbox, running Python functions, running custom code. And so for that, I don't want to have a small model, let's say, like an open source, GLM 5.3, for example. As I've seen this being running in the loop for much longer and getting confused by different data sources. Well, at the same time, right now, we're running, like, Opus for the main agent. And for the reporting, we can run LunaMax, for example. This LunaMax, the at least the last iteration of that is, will fit for it it's either it's it's smaller, but it's well fit for the data analytical tasks. And then we don't really differentiate between, like, running a model specifically for one skill. It's only running for a particular task or a workflow that we have predefined already. At the same time and I wanted I know that Jen had this this idea as well. In GTM agent, how we're thinking about running different smaller models or bigger models for different kind of tasks. Yeah. So I think there are two pieces. One is depending on the complexity of the query, you could select a different a different model. So that's that's something we're working on. But I think what's more interesting is to let the supervisor, pick the model that it's right depending on the kind of sub agent that it needs to run. I think we're moving away from codifying, prescribed agent architectures, like, literally writing that in code and instead moving that into skills. So the reason is when in the case of paid media, it's a very well defined set of tasks. So that's why AML has, like, you know, these sub agents, one per platform. Once the system becomes much more complex, you don't really know ahead of time the range of sub agents that would be the most optimal for your system. Instead, what you can do is capture the judgment of sub agent creation so that the supervisor agent can spin up sub agents with the right tools, model, and instructions at run time while the agent is actually processing that task. And that's also what we're seeing with, many other harnesses. I mean, if if if you look at, even Cloud Code or codex, you'll see that, they typically only have one general sub agent, and it's the task of the main agent to decide, what's the model and what is the task for for those sub agents. I think that goes again with the idea that we wanna give as much freedom as possible to the agent while capturing as much of the judgment as possible in the skills. Because if you do that, then the next generation of models is actually going to improve your system. If you have a lot of structure, then the next jump in intelligence is kinda going to fight against your structure. So you're gonna have to remove that structure to actually let that intelligence become, you know, something latent in your system. I think we've seen this with the model jump from 4.8 Opus to like 4.5, where before you had to write a lot of rules in your system prompts, now the agent needs more freedom and there's like a lot of general advice around x, for example, to have Office five have a bit more freedom and not define particular rules because it gets a lot more confused than you constrain the intelligence of the model. So we had to refactor quite a bit of of skills and and and and system prompt to make it, more of a guidance and overall way to think in mental models and principles rather than particular rules of if then then this. You know? This next one's a particularly interesting question because it's something that we've been spending a lot of, energy on lately. The question shortened is around how do you factor in total cost of ownership of the agent, categorizing different types of cost, and subsequently associating those costs with key performance indicators, to understand the the cost of the agent and the cost of the outputs of the agent and their subsequent value to the business. I think, Danny, I can take this one. We we've been doing a lot of this work in our broader GTM system, and you can think of this essentially as a BI, data warehouse problem where some of the costs will be tokens. A lot of your outcomes will not have to do with necessarily an agent run, but something that happens in your company such as sending an email, booking a meeting, closing a deal. So what you kinda start doing is creating your own data model for, number one, agent attribution. Is the agent actually touching any of these events? Is there any causality between these things? Is there any influence? Are these just, like, happening simultaneously, but was it actually because of human? I think that this one of the areas where we're spending a lot of the time, that we're doing in in in development. And, essentially, you you can need to start applying a business intelligence framework to to all of this. And it's it's really gonna depend on your business. I don't know if, Danny, you wanna comment on how how you were thinking about the GTM agent, in terms of outcomes and how we're measuring those KPIs. Yeah. We're definitely going to have a GTM engineering live at some point in the next two to three months specifically on this. As I alluded to upfront, like, our entire team's mandate is to make go to market functions more successful. So if we can't tie our agent costs back to business outcomes, like meetings, reply rates, time saved for certain types of work where there aren't more concrete outputs is, that's, like, how we illustrate the value of our work, you know, is is outcomes, not not outputs. So I would keep, checking your emails and checking the GTM Engineering live virtual series page because this is actually gonna be an entire event, that we talk about at some point in the future. We're still kind of formulating our strong opinions here. Like, it's in probably more of an exploratory mode, but, it's critically important to our team that we solve it, and, we're gonna be doing an event on that here shortly. Alright. I think we maybe have time for one or two more. Here we go. I didn't understand the point of storing large tool results and files instead of letting them into the model context directly? I think the large tool call results would eventually end up in the model context anyway. I'd love to know your thought process here. I can I can take this one? This is one my favorite question. So not necessarily. It doesn't necessarily have to end up in the context. The the reason is, the file tools that you the harness, in this case, Deep Agent provides to the model, they allow the the model to only say, I only want the first line or the first two lines, or I want, like, this section of the file that is, like, halfway through so you can offset. When it comes to actual quantitative data, it could use something like Pandas and say, actually, I don't wanna if you're an analyst, you don't wanna, you know, pick up a dataset and then read every single row in the dataset. What you're probably gonna do is create a pivot table, and you're gonna look at the structure. You're gonna look at the columns. And so in many cases, you will read less than 5% of the context, as compared to reading the entire dataset through the context of the model. And one way to think about it, I like to think about it as if your model is your control plane. This is about orchestration and intelligence. And then your file system is your data plane. That is your Excel. That is your pivot table. That is the thing that actually manipulates the data. And so you can look at your traces and see how if you're dealing with a quantitative thing like the paid media agent, how much of the actual numbers are going through the context versus how much of the actual synthesis or structure of those numbers are going through a context. And you're gonna say, well, my the more your agents doing control plane versus data plane, the more efficient it's gonna be because, by the way, not only it's gonna be cheaper, but it's gonna get much less confused. So this applies to quantitative, but also qualitative because it can grip specific words from, a markdown file, for example. So it's gonna be re eight models are really good at agentic search, and a lot of our architectural choices come from they're really good at searching and navigating, and gripping, through files. I think a part of that is also, since we have an integration with a warehouse, usually the warehouse would collect quite a bit of data. In the example that I've seen, the paid media agent querying the warehouse, the results that would come back from the tool would be extremely large. It would be the last two months of spend across every event, and you need to run a Python function or pandas to, like, analyze the just the one result and get that one output in in in your terminal. That's one piece of it. And I think, at the same time, when you work with Slack and or when you create user interfaces and the users are working with files, they upload bigger Excel models. They upload, like, large artifacts. And in relation to using a file system tools and the agent being so great at reading the files, it would just look at the artifact, read that, extract what necessary, and then, and use that for for the next job. Awesome. I think we have time for maybe one last one, and then we're gonna have to wrap up. This is a a timely one. What do you guys think about using nonautoregressive models like JEV for model routing in this case? I can I can I can start with that? I think Jeff has definitely revolutionized somehow the Twitter last week. It's been crazy. The one of the core things, and I I know that the team from Deep Agent side has created some great guidelines on how we use Jeff and the Deep Agent harness, for primarily like model routing for context summarization. Right? So it's sensitive decision model. To your previous question about, like, how would you select whether the model, is the right for for the context of the query. I think Jeff will be great here because you would just provide the question, right, as as the sample data and then would have Jeff decide based on the list of models and their kind of intelligence level, which one would fit the best. Same thing can go into harness engineering around, summarizing context, selecting the tools, and compare in relation to, like, our paid media agent with 200 plus tools. You can have Jeff instead of that abstraction that we built, Jeff selecting the tool for you, for, for the agent. So that would be, like, I think, a lot more semantic and, a lot faster overall and and and save tokens on turn calls trying to find a a tool using regex. The I think the only thing I would add is for the vast majority of our use cases, we just focus on using the best, most efficient frontier model, coding models effectively. What they do is effectively knowledge work, and, they allow us to do, like, very complex, heterogeneous tasks. And so we're always trying to find, like, what's the most efficient, frontier model possible, and and and that keeps pushing our system furthering the complexity of of the tasks that it can do. Awesome. Well, we're a little bit over time. Thank you all so much for attending. Really appreciate all the fantastic questions. And, again, if this was interesting to you, we'll have more of these in the future on a wide variety of different, agentic use cases, and all of you will also receive the recording, in your inboxes, shortly after the event is over. So really appreciate y'all being here. Hopefully, we'll, see you again on one of these soon. Have a great day. Thank you, everyone. Hello? Bye bye.