Name: GTM Engineering Live: How We Built Our Design Agent
Uploaded: 2026-10-28T19:31:18.404Z
Duration: 1 h 16 s
Description: GTM Engineering Live: How We Built Our Design Agent

Transcript for "GTM Engineering Live: How We Built Our Design Agent": Agents are a new kind of software. Endless inputs and non deterministic outputs. Build them the old way and they break. Enter Lingsmith, the agent engineering platform. It's model agnostic, agnostic, cloud agnostic, and framework agnostic. Organization is shipping the best agents iterate with a system, and we call that system the agent development life cycle. Build, test, deploy, and monitor. You build with our open source frameworks. Deep agents, langchain, or langgraph. LangSmith Fleet enables anyone to build agents without code. Test agents using evals and experiments. Deploy changes in one click with LangSmith Deployment. Monitor every interaction from a single dashboard. Governance is built into every stage of the agent development cycle with LangSmith LLM Gateway. LangSmith Engine helps you improve agents autonomously and move through the development lifecycle more quickly. LangSmith Engine is your agent for agent engineering. Welcome, everybody. I'm Danny Lambert. I lead the GTM engineering team here at Langchain, and I'm gonna be your host for today. Appreciate you all joining. Go ahead and share your name, where you're, coming in from in the chat. Let us know if you're hearing anything wrong on our side. If our audio is coming through, great. Or if you're having any issues, that's the place to let us know. But this is actually our second installment of the GTM engineering live series. So once a month, we will do a virtual event and an accompanying blog post about an internal agent that we built at Langchain for one of our teams. So last month, we did, the marketing agent that we built, the paid media marketing agent that we built. And this, month, we're talking about the design agent, otherwise known as Content Studio, that that we built in the last, couple of months here. So a little bit of, a heads up on what you can expect for this session. We're gonna be talking about the design agent and, essentially, where this came from was, our team has really been growing. We're putting out a lot of campaigns into the world across product launches, webinars, events, etcetera. And all of this was creating a tremendous amount of, demand and work for our design team, and kind of taking up all their times that they couldn't focus on more strategic initiatives. So we want a way to be able to, kind of offload some of their more tactical bandwidth to the agent and free up time for them. So that's essentially what we're gonna be talking about during this session is, Amal and Shreya from our side are gonna be demoing, what they built, talking about why we built it, how we built it, and then we're gonna open up at the back end, for about twenty, to thirty minutes of q and a. So before we hop into the real content, we're gonna do a little bit of housekeeping, intro to the team, a little intro on LangSmith, and then we'll get into, the real content in a couple of slides here. So, obviously, not all of you have been in this event platform before. So I just wanted to give you a quick orientation of where things are and how you can use them. So in your top, right hand corner, probably the most important is this q and a tab. So if you go in there now, there's a bunch of queued up questions that look like they came from me. That was actually the questions that y'all submitted on the registration form. So as we're going throughout the event, if you have questions, please put them into the q and a and also upvote other people's questions that you find the most interesting. That's how we'll prioritize that last twenty five or thirty minutes is look at the questions that have gotten the most upvotes and prioritize those. We'll try to fit them all in, but I know we already have, quite a few questions in there. Second is this docs tab. And so this is where you can get, ancillary resources from what we're showing you today. So this design agent has been open sourced, and that first one is the link to the repo. So you can access that at any time. The second is a link to our GTM engineering live series, what we're on right now. If you like this content and you want more of it, you can go there and subscribe for all of our, future events and keep track of, you know, what other agents we're building and when those events are coming up. And, last but not least is the corresponding blog post for this agent. So if you wanna read it in greater depth, that is already live and it's available to you there. And then the third piece is right above where I'm speaking right now, there's a personalized demo button. So if you're building agents and you're interested by what you see on the demo today, if you click that, a member of our team will reach out, and set up a demo for you and your organization and how you might be able to use Lionchain, in the ways that that we're demonstrating throughout this content. So before we hop in, a little bit about, the team and the presenters today. So as I mentioned, I'm Daniel Lambert, and I lead our GTM engineering team. What GTM engineering is at Langchain is we support all functions and go to market from top of funnel functions like, marketing and SDR and partnerships through sales, all the way to back office and post sale functions like deployed engineering, finance, legal, etcetera. And our mandate is to build agents that make those teams more efficient and more effective. So on that team is myself, Jan, who is our engineering manager, and Amal, who built the majority of the design agent and will be doing a lot of the presenting on today's session. And then we're really lucky to have Shreya here alongside of us. She is our brand designer. So anytime you see, Langchain out there in the world, she has touched that in some form or fashion and is just an incredibly important part of this agent being successful. Without her, we wouldn't be here, and I'm really lucky, to have her kind of co promoting with us today and talking about how we, taught the agent taste and made it actually valuable in the outputs that it's producing. And last but not least, before we get into the content, a little bit about Langchain for those of you who may be less familiar. So as you saw in that intro video, Langchain provides the platform, to help companies move along the agent development life cycle That build, test, deploy, and monitor loop, helping companies move along that faster and more reliably. We currently have half of the Fortune 10 as LangSmith customers, over 7,000 active LangSmith customers, and more than 200,000,000 monthly open source downloads. So that volume is on par with the OpenAI SDK. And you can see kind of along the bottom here a a myriad of the different companies that we have, the pleasure of working with, like Clay, LinkedIn, Home Depot. So we've seen a large variety of, use cases and industries and a lot of experience helping these organizations take agents from concept to production to scale. So if that's something that you're interested in, we certainly can help you along that same agent development life cycle and trajectory as we have these other businesses here. So with no further ado, I will bring up Amal and Shreya, and we can hop into how we built our, design agent. Alrighty. Hello, everybody. How are you doing? Great to see and have you all here. We're excited to talk about the content studio, our marketing and content agent, design agent. Just a bit of context. So far, content studio in the last two or three months has been really, really helpful for us and the marketing and design team, from being... Creating designs for many of our campaigns on socials and, over 3,000 assets so far, four and a half million views across socials, and median content, conversion has been increased as well. So really, really excited to to share our learnings as we go as we go through this presentation. There's been a lot of learnings, a lot of iterations for this agent. Many things didn't work in the beginning, and Shreya is the... We'll talk about that as well with all the anecdotes and evidence for that. But, yeah, excited to jump in and then share our learnings. So to start, Shreya here is our Batman and... At Langton. She is saving the Gotham from a lot of the, basically, a lot of requests are coming in from very, you know, small granular things and to change in Figma to, like, iterating on designs. And and there was a lot of design work that Danny mentioned in the beginning where the line chain moving so fast that there is product launches every week, there is many, many campaigns going every week. And so we had to find a scalable way to standardize and make the process more efficient for everybody in the company, but also enable everybody to to build with, and become more of, like, amplifying our content on socials and and build any kind of design they want in our language, in our taste, with our brand guidelines. And I know, Shrey, you you might have your anecdotes on this one, but, I found some comments that you have have have had on on Slack on this one as well. No. I think, Amal, this is, like, absolutely an accurate depiction of what my DMs look like. They're always started with requests, like you mentioned, from different teams. And it'll typically be, like, blog graphics, event flyers, campaigns, banners, just like anything you can think of. And, what's interesting about this is that, like, I want to be able to touch on all of this. I wanna be able to address all of these points. But at the same time, being one person, there's a tendency to... For you to get stretched thin. So, yeah, this is absolutely accurate, I will say. Totally. And I... What I what I love to what I love to see as I was building the agent for for Shrey and everybody was Shrey now is kind of operating in the higher layer of abstraction with all of these designs where she's now more of a creative director of of designs. And and ContentStudio is a little collaborator that helps execute a lot of these smaller tweaks and details that were spent on, most of the time were spent on. Now to say, honestly, it was not that... It was not the case for... From for the beginning. A lot of the first iterations, even, like, not as far as in August, were not as great. The agent was not really performing that well. We had to go through any iterations to teach it taste, teach it design understanding, to encode all the judgment into skills, into Wiki, into the context we're gonna talk about today. And, Shreya was a really, really, really big help for that because she was kind of educating and teaching the the the agent in Slack, with the rest of the team. And we basically saw the agent fail publicly many, many times in our in our Slack channel, going through iterations. But, that that made us learn a lot. And we're happy to say that right now, the agent is being used quite a lot by everybody in in our company from marketing, but also majority of tasks and and asks are coming outside of marketing, for product use, for product launch videos, for, you know, the the, any kind of amplification of social content. And people are now enabled to build their own designs and create their own designs and and and publish it online from their social. So it's, very excited that people are really liking that. I'm happy to see their happy faces in in the office, and, I know Shreya has has has liked the the agent as well so far, which I'm very excited. about. It's always great to, just... I'm at my desk and life feels so much simpler. And that moment of realization is that, like, wow moment, and you go through that wow moment, like, 10 times a day. And to also provide some color on what the historical process look for... Look like for us, I think earlier this year, we realized that, again, we wanna move faster. We wanna tackle these design requests in a much quicker, smoother way. And we started experiment with local coding agents. And, what we did was we had a brand skill, and that was basically, like, baked into the local coding agent. And then those briefs were what the, the agent worked on, and then the output that was produced was somewhat there. But then in this video, you're gonna see how ContentStudio tackled that and worked through the same brief. And, yeah, the model is the same. So, like, let's just, I think, watch the comparison. Right. Okay. Very exciting stuff. What you just saw was... It was a series of decisions that I would normally make in a design review anyway. What to remove, what to emphasize, how motion patterns should support the core messaging of what we're trying to communicate. And the main change was that we started building that judgment into the system around that model. This hiring chart that you see, it was made by a local agent with our brand skill loaded. But in my local setup, my reviews weren't automatically helping the next request that came in or the next person. And, again, the slime green is not part of our core branding. So with ContentStudio, the whole company works with a shared agent in Slack, and it has our actual assets, our standards, and memory for what we teach it. So it is essentially getting better with every request, every asset it creates. So feedback on one request can basically inform the next. And it also checks its own work, against those standards before it comes back to us, which is what I mean by a design brain. So more of that judgment I apply during review is now built into the process. I also wanna talk about this quote by Katie Dill that was particularly inspiring for both Amal and myself. You have to have a point of view because if you don't, AI will give it for you. It's... That point of view is basically what we mean by having taste or having that judgment. And it's like asking someone to pick an outfit for you. You can just tell someone, choose something nice, and you'll get their idea of nice. But unless they know you and your preferences, they'll give you probably a a wearable outfit that you might not really enjoy. So part of my job now is to make that judgment clear to the agent, what works for our brand, what works for us, what we're we're trying to communicate, and why we're trying to communicate that. And that's my point of view on how, on, like, how point of view actually shapes work beyond the pieces that we review. Now the bat signal reaches, the content studio, which is great. And I'm in content studio's DMs, and I'm spending more of my time, as Amal mentioned earlier, as a creative director. At Langchain, we we work on so many campaigns. There's probably, like, three massive, marketing campaigns that we worked on. This is aside from our product launches. And this way, I'm able to put more time into shaping those campaigns while these everyday requests are just running in autopilot. So here, the little, like, text thread that you see, that's a snippet from our, Okay. Jam ideation session for an upcoming campaign. I'm setting the direction, and it's proposing ideas, explaining its choices. And we're basically riffing off of our ideas, which is great because ContentStudio is now my co designer. And my role also includes shaping the system, setting standards, explaining my decisions. How do we scale? How do we forge this already strong identity? So, yeah, I would just like to say that for the developers here, build this with your designers. Give their feedback a home. Give it somewhere to live, and find them a way to inspect and steer their work. And for designers, honestly, just the reasoning behind your decisions, trust that trust that instinct and, use the time you get back and push your work even further. Yeah. Totally. As a developer of the agent, I would say that it would not exist without Shreya and her judgment. A lot of the things that that you see right now, the slides that Content Studio created are by Content Studio, the videos, everything has been due to the help of of Shreya, enabling the the agent to self improve and then encoding that judgment into it. And as a developer, I myself, when I was building this agent, learned so much about not only the agent architecture and how to even evaluate media assets and visuals to how do you run long long running tasks because a lot of these videos take some time and how do you cut costs, for example, and because they might be expensive. And so I wanna I wanna go into that and share some of my learnings here. Number one, I wanna start with the principle that I was following, which is design, in fact, is code. You probably already have worked with Cloud Code or with codecs, any kind of local agents, local harness, and all of them are writing code, and they're fantastic at it. You can build web apps, you can build any kind of applications, and all of these, when you see the UI and the web apps, they're writing code in JavaScript and HTML and CSS. With ContentStudio, everything that it does is fully in code. It's a coding agent in the end of the day that has some brand skills and and and context that allow it to write, to write it on brand. And so the reason why there can be... We didn't go fully with, like, let's say, transformation models or... I mean, transform models is that, with code, you have a granular ability to edit. The agent can go line by line into, like, a particular section of a... An HTML file or a CSS file and update it, and it makes it very scalable. So the agent can always iterate very fast. It already knows how to code very well, and so it makes it super scalable. For... In comparison, so let's say using an an agent to, to just offload it to a transform model, generate an image or generate pixels, and then has to work with pixels, because every iteration that would have to would have to be regenerating the asset. Now to be clear, we do have a hybrid of that sometimes when... But let's say we wanna have image generation or voiceovers or music or sound effects that the agent has tools to offload it to, the the the transform models, like, any kind of tools, like, eleven Labs or or something like that. So the... I guess, our first iterations, from the engineering side and according... On the agent side were, I guess, the intuition was if the models were not as great back then at at at design, maybe it made sense to to hard code a lot of these, component libraries or references and and rules around how the agent should create. Instead of let it design freely and writing code, we basically created a lot of component templates, from charts to layouts to thumbnails. Its main job was selecting those components and putting them on screen and rendering them instead of designing from scratch. And the reason why we did that was because, you know, we had to compensate for the intelligence of the model that was not really on par with what we need... Look... We're looking for. And so we provided a lot of that context, and we thought, you know, that would work well until, most of the designs that we're getting were in the same layout. They were all following the same the same kind of hierarchies, the same layouts, the same animations, the same motion graphics. And surprisingly so, I mean, because, everything that we gave it was just, you know, just select out of these 100 components and combine them instead of designing from scratch. Therefore, it would it would just combine them on on the screen. And so they were on brand. They were right, but they were not good. And there was not enough flexibility with the agent. And as we were growing, with the variety of of designs we wanted to follow, every time we'd have to be... Have a new design, we'd have to add it to the template that would not be really scalable, for the agent. And plus, new models were released. Opus five point five was incredible at design, as many of you might know. X was really booming on that. And so, we wanted to test it with this approach with Opus five five and was really, really limiting for the model. It was still... We saw a very different variety of of of things on on x. It was super cool to see different kinds of motion graphics, but it was not the case for us because the goal of the agent or the way it was instructed was to just choose from and assemble, layouts. And so we... The better lesson, is that we we had to change, and I actually removed a bunch of code, a bunch of rules as soon as Opus five five was released to try and see whether the the intelligence is there already to... For it to start designing from scratch and and just with some context and and and guidelines and principles of design instead of hard coded rules. And so the rule that I found was, like, rules written for a model's weakness expire with that model. You have to think kind of to build for the future. As as the new models come come by, you have to think, what is the scalable version of your environment of the harness that the new model can really be amplified, instead of guardrailing it into a a a deterministic box kind of. So an example of that, as soon as we removed a lot of these rules, and my... Most... Some of you might have seen it on our socials we posted, we had this version of the video that it did that was really exciting to watch. Every legend starts small. This one started as a single pixel. Its first poster? A disaster. But a legend never stays down. So it trained. Wrong blue? Fixed. Too big? Fixed. Off brand? Fixed. Every single day. It studied the wiki. It mastered its skills. It found its corner. Now it makes everything. Launch films, decks, diagrams, 3,000 pieces since August. And today, it's ready for the world. ContentStudio. Our design agent is open source. Go train your own. So is anybody a Rocky fan? Because I am, and that was amazing. And so the agent was able to create really, really cool stuff, as soon as we removed all these guardrails and rules, and now it's designing really, really amazing, scalable designs that we will not even be able to think through before, and it's a very fun experience to work with it internally. Now, what is really the agent and how it works? In our case, it's just three layers. It starts with the agent loop from DeepAgents, which is our general coding harness that is open source. Then we run this agent in production with managed DeepAgents that provides the runtime, the memory, the infra, that is managed, as well as on top, we built a lot of the tooling and and this custom media harness that we call it, the environment where the agent is operating in, with all these skills, wiki, contexts, and tools like hyper frames or, for example, like some other video engines and animation engines. And so DeepAgents provides us the agent loop with a lot of the context engineering, planning, file system tooling, management of skills of of sub agents. So if you use cloud code... Codecs, it's a very similar idea. It's a coding agent harness that provides a lot of the foundation for the agent to operate, reliably at scale. And the next layer would be the managed deep agents that takes that operating system loop, right, the deep agents, and puts it in production by adding infra, from tracing to deployment to sandbox management to a scheduling SDK that allows the agent to run-in the background, to a Slack integration that we use right now in Slack to to operate, with the team... With the teams in Slack, to management of secrets and all of that. So, really, really helpful for us to basically offload it all to Managed Deep agents for it to run the infra while we focus on how to make the design of the agent and the context better. And the the the the exciting piece is the media harness that we build on top that allows the agent to do all the things that you saw so far, from adding the custom sandbox image that we had, that we include the Chromium, the FFmpeg, the HyperFrames, and a lot of the brand, brand guidelines and tokens, and maybe references and logos and assets to the tooling, from, you know, all of the core tools like generating images and voiceovers and fetching music from different libraries and sound effects sources to paper, which is the visual canvas that allows the agent to collaborate with designers and with the team overall on the shared canvas and write designs there to, our Slack Wiki or, I guess, sorry, the skills and Wiki and a lot of references that we added with Shreya from her judgment around, and we'll talk about those skills later, as well as a little of the, orchestration that we added with async sub agents that allow the the lead agent to work more of, as a chief of staff, kind of giving the tasks to async sub agents, similarly to how Cloud Code also works with when you're coding with all the agents on the side. That allows it to run at scale on bigger projects and complex campaigns, and then the learning loop on top that allows it to self improve as we go. Now, the idea of a computer, that we gave to the designer, which is this agent is basically the the sandbox that we're talking about. We see the sandbox as really as a computer where you embed all of these tooling and and and references and skills into. And so the the agent looks more of like... The the workspace looks more like this, where you have one workspace folder with, the attachments that come in from Slack with a project folder that that is... And and the agent and agents collaborate on that one project folder together, with a brief markdown files to plan, and they all collaborate on this same file system, which is a coordination plane for every agent to work with. And they all review the designs together on that one project path. They collaborate on files, they design files, they iterate, and they they, then provide the the outputs into, into Slack. And it also includes the the knowledge, the skills, and the wiki embedded. In terms of tooling, as I mentioned, we had the the the tools for the, you know, basic rendering, like Chromium and Playwright, to writing HTML and and rendering that in the in the Chrome for the agent to then, you know, review its own work in Chromium and then send that to... And and export it as... In different formats to image generation, to charts and diagrams for all the more, more of the diagram... Architectural diagram heavy visuals, to motion engines like hyper frames. So it's really, really helpful. It's an open source framework for video editing and video engine for agents, to sound design and sound effects, as well as adding some extra layer on top of, like, general MCPs or Notion, HEX, GitHub, HEXA, so that the agent has more context of how the company operates and works overall. Then for the skills and the judgment layer, as I mentioned, we have those... The the core skills, that we encoded with with with Straya to the Wiki that primarily includes the... All the design assets, references, tokens, our logo assets and brand guidelines. We have specific gates as well that help the model understand or deterministically as tools whether, you know, the the... All the tokens aligned with what we're looking for, and whether all of the, you know, specific design principles aligned with what we're looking for, as well as a memory that is per user and per team global. The skills usually look pretty familiar as... That you might have seen before with many of these being, skill dot MD, a folder with a bunch of examples, references. Everything is in code, So we wanted to provide a lot of the references and examples so that the agent can follow them very... In a scalable way. So all these skills you would... You will see, people will be, like, with several subfolders to... With progressive disclosure so that they can really learn and understand what... How exactly the scope should work with real examples. And I wanted to say one more thing here about the architecture. A very big, I guess, scalable pattern for us was to give... To enable the agent to work as chief of staff, where it, assigns work to its sub agents, which each... Let's say we get a big motion graphic or, like, a big video launch, like that rocky launch. And in that see... In that sense, we have, like, four or five different acts or or stages of the video where each sub agent was kind of instructed by the lead to take over, work independently, look at the already core patterns created by the lead agent, a storyboard, a a diagram, a, kind of, core ideas, by this creative agent. Right? And then everybody else is looking at that and, and referencing that and and creating the assets, then then feedback into the main model and main agent. And this way, they all collaborate together, which allowed us to basically create a very long running tasks with some tasks taking, like, five, six hours sometimes. It's a really, really complex task. But this way, the agent kind of operates at a run time with nonstop, just looping itself, working with sub agents, working on the file system, and having a bunch of tools to basically report progress, check on the sub agents progress, and schedule itself to wake up if it... If needed to, like, to wait for sub agents as soon as as soon as they're done working. And one quick thing on the... How the agent works and self improves overall, There is a really cool pipeline where, you know, right now, if you tell the agent to, you know, update your your memory based on what I need, like, for example, if Shreya says, for this kind of graphic, I need this kind of principle and this kind of guideline, it's gonna go and update its own memory, either for only Shreya. But if Shreya asks, you know, make it for everybody, the agent will update the global memory. And we'll have a list of... In the background, we'll run background agents that will look at all the memory updates and look at what can be absorbed into skills or the wiki so that the memory layer can stay cleaner. And so this way, we constantly self improve the agent, and we, as engineers, we look at how the background agents improve the the system, and we approve, review, edit as the edits come into skills and the wiki. Now I know there's... The the the system is pretty complex. There's a lot of things, so I wanna leave a bunch of time for questions. The architecture is is very fun... Was was fun to work with and, really excited to jump into any kind of questions you might have on that front. And, yeah. Awesome. Well, Amal, thank you so much. I'll just leave you up here. because we're gonna do q and a here in a second and we'll... Drea and Jan on. But I I just wanna kinda outline next steps for people. If you found this interesting, we've tried our best to make this as accessible as possible, for all of you. So as I mentioned early, for the folks who may have missed it, we open sourced this project, this agent, and the repo is linked in the doc section in the top right hand corner, as well as there will be a recording of this as well as the full presentation sent out a couple hours after we're done via email. So if you wanna watch this back, if you wanna share it, if you wanna review the deck content, check your email in a couple hours when the recording is complete, and it will, be there for you. There's also the blog post. And in particular, we're gonna run this every month. So if you find this type of content interesting, we've committed to doing this at least once a month for the foreseeable future. So the future GTM Engineering Live are linked in the doc section. We have one in the calendar now, but we're gonna probably add another one, in the next couple of weeks here. So we're committed to doing this if y'all are committed to showing up. So please do, and and we'll be there. So now we're gonna pull everyone up on stage and do a little bit of live q and a. So let me grab Shreya and Jan and stop sharing. Cool. Alright. So I tried my best to prioritize these questions, while the presentation was going on. This one had the most upvotes. How does the agent manage memory, chat history, company business contacts, feedback, lesson learned, good practices, etcetera? Perfusser. Yeah. I can take that. So to to go back to that one slide that we talked about where, this... There is four layers of context. A bigger part of how foundationally the context is being managed is managed by our coding agent harness, which is deep agents, where the the the agent... The harness takes care of the, progressive disclosure, the middleware that allows the agent to progressively look at the right skill in the Wiki by gripping the file system with the file system tools. And then everything else will build on top with specific guidelines where... And and and the list of skills that I showed, and the Wiki with deterministic gates. And the memory is also abstracted away by managed deep agents, where they provide the abstraction for the agents who update its user memory scoped or the global memory. The memory layer can also be not only just one file, but with many of the sub folders, again, as a file system, memory as files, where it can add subpaths and update them, and then background agents would look at them and, iterate and clean up them as every day and move them into the skills if necessary. But that's kind of high level how how it works. And if, Gianni, if you wanna add any... Anything here, I know that you, you also were working a lot on on the overall coding harnesses and agent agent context. No. I think you you you covered everything, Emma. Cool. Cool. Next question with the most upvotes. How do you do evals for this harness? That's the most fun. Now I think that the the biggest question that we had was, how do you even evaluate visual outputs? In fact, I have tried several things. I have tried the running the Gemini flash models on every run to see whether... And provide a lot of context about our guidelines and and and and, deterministic visual, understanding. Basically, every time the main lead agent would look at... Would create some kind of an asset, it would go through a visual model, a multimodal LMM, basically, and and look at... And that Gemini flash provides feedback bay... Back into, the lead agent. And I thought that would cut cost, but also that would give, like, a more of a, deterministic, non deterministic gate for what good looks like to get... Add another layer of of of, of verification. But that turned out to be not as nice, because number one, Gemini, maybe it was the intelligence of the model, but, it would provide not a helpful feedback. It would explain, based on some of the weekend and skills that we had, but it it would confuse the main model, which is Opus in our case, by providing, you know, irrelevant feedback or something that we would... As... We as humans, we're not really, provide as the core iteration feedback, so we kind of took it out. And now the primary, way that we judge is really through Slack, and Slack is our eval set. You can think of, like, hundreds of iterations, hundreds of questions, hundreds of, assets being generated on the weekly basis, and we look at all of them as the reference as this dataset of of what good looks like and what the judgment looks like. And so we take that and we and we use our agents to look through it at scale and improve the system based on abstracting judgment, looking at comments, feedback, and putting it back into skills and evaluating before pushing to production locally. That's kind of the the main idea, for for evaluating that. But also, I would say, more more granularly, we have some eval sort on, let's say, if if there are deterministic gates on, again, like, all these brand token... Tokens that we have. For example, I have this slide that explains how even with our agent being very smart, since it's Opus five five, it follows all... Everything through very well. But what it allow... What the brand linting allows us to do, is to have a deterministic tool to look at whether designs that it's making are following our guidelines and give more specific checks on, let's say, whether this part of the brand token is is following the exact token that we need, that that Shrey has embedded into the system, or if something is off color or off brand. So that's another kind of verification that we have, and we do sometimes go through evaluating how agent would work with with the gates and and more of a deterministically rather than as as a judge. Yeah. I will add something else to how we think about evolves in the GTM engineering team because we deal with business outcomes constantly. So we don't we don't only care that, for example, a draft or a design is, you know, well written. We also care is it being used and is it having the actual business? Is it achieving the business objective? And so even in Langham, if you can use feedback, which is a a a tag that you can add to a run to incorporate things that, like Amal was saying, might come from human judgment directly. And so in all of our agents, we have thumbs up and thumbs down, that humans provide, and that would go directly into the feedback run. But we can also go one step further and say, well, did it actually have the effect that we wanted in the real world? So if we're sending an email as a draft, does it actually get back a reply? Does it actually book a meeting? And so it's almost like you're extending the definition of eval from going deterministic check. Does it follow the right trajectory? Then LLM as a judge that does it, you know, say what it's supposed to say? Does it follow our rules? Then next check does internally people want the agent to behave that way, so human feedback. And then the next step is, which might happen after a week or two in some cases, if it's a reply, did it actually have the effect that we wanted in the real world? So it's almost like we think of emails, we extend the definition of them. 100%. And I was also add that we added recently a... Based on that feedback to Lang Smith, the the tag. I added to Content Studio the, a way for it to verify whether content was used externally so that we can track ROI on that specific type of content. So since we do a lot of these designs, we need to understand... And they... And I also have a question on cost there as well. And some of them might be a little more expensive because it's running Opus five five. This is an expensive model by default, and all of these iterations, are... All of these designs are like long running tasks. And so the median can be like, let's say, like $5, 4 or $5. And they're... They're expensive. So we need to understand what's the ROI, whether this was used, whether this was used, let's say, in a social post or was used in an ad. And what's the engagement for that social post? What's the pipeline that generated from that social post? And so we can look at and track every asset that's linked to external publication and then look at the ROI of the task and and and basically show that it... That's... It's worth it or not. And then based on that, make more additions to the agent to maybe focus on more specific high ROI tasks that that makes sense for us. Awesome. Before I answer the... Or pull up the next question, we're gonna have a session either next month or the following month specifically on measuring ROI from the various agents that we've run. It's probably one of the more asked questions that we get and is now becoming kind of a core concept that we define prior to development at all. Like, we're trying to answer how we're gonna evaluate it before we actually start developing, and that's really changing how we prioritize and also changing about, like, how we orchestrate our work. But I guess I do wanna kinda pull up. There's a bunch of different questions here. You hit on it a little bit, Amal, about, costs. I mean, we we try to, like, package this up a little more cleanly and answer them them altogether is, like, kinda rough overall view of, like, how much this costs and the type of different costs per unique workflows, I think, would probably be helpful for people. And then how are we evaluating or thinking about if those costs are worth, you know, how much we're paying paying for it. I think maybe a summary there would kinda hit on the four or five different cost questions we have in the queue. Totally. Yeah. I think high level, content work is expensive due to the fact that the LLM has to review its own work, in our case, and look... And every... Let's say, when it reads the file that is an image, it's all, the... All all these tokens are very expensive because it's an image. It's base 64. Right? And so we, what I tried what I tried to do there is, you know, cut cost there, but turns out with that visual model example that I had, it was not as scalable and reliable. So I did make a decision to keep the agent to review its own work, because the output of that... Because the result of that is shorter tasks that are faster, that take less iterations and less loops because the the smart model reviews its own work and can make better decisions on the iterations and review. So I guess, you know, medium cost right now for those kind of content design tasks, it can vary from, let's say, the the complexity of the task. If the task is, design a thumbnail for YouTube, that can be a little easier because all it takes is one HTML file, maybe a couple of iteration... Iterations. It can it can cost less than, like, you know, 30¢. But then if we look at this Rocky film or, actually, that film that we showed in the beginning of the agent fixing its own work, that can cost $40.50 dollars more depending on how long you work on the task and how much you iterate. Right? Now that's why it's very important for us to understand what's the value of that work because we had an example where, you know, our head of marketing, Jesse, was looking at this really big slide slide deck. It was, like, 60 slides or so, and it was from very raw kind of bullet point notes. And one of... One might one might say, it will take two days of work or maybe, like, less or so... Less or more to really make this work in our design language for that big conference or, like, big event. Right? And so maybe... And and they will... You you can look at the amount of time it's it's gonna it's gonna cost and plus the amount of of of work by a by a third party, let's say, or a designer at my... In my cost as well. So, that job was maybe, like, you know, what was it, $50.60 dollars, or a 100, whatever that was. But, the the ROI was really, really great because it would take a lot more time, a lot more effort, a lot more money to make this happen. And so we're looking at those kind of out... Not even outliers, but, you know, looking at the high leverage tasks like that and trying to define how much they take, what's the effort, what's the alternative without content studio that would take us, and then try and compare that, and focus on those tasks more. I hope that answers the part of question, but feel free to jump in with any more. John, Jan, Shreya. Cool. I think this is a a particularly interesting question, maybe, yeah. for Freya around, speed. I'm curious about speed. I've noticed a lot of designers like to be hands on and iterate quickly and are used to working more synchronously on designs. How do you account for that? Did you work a lot on speed for the agent, or is it more about setting expectations of how you work with the agent? That is a great question. I do think, for designers who are producing a high volume of continent, we know that at Langstream, we're shipping across all teams really fast. I think the speed of iteration, is it's separate from the time it takes from... To finish production. So for small visual adjustments, for instance, I'm still pretty hands on, and that's how the shared canvas and paper that we have that matters. So we're still able to make edits pretty quickly, and, the agent can then use that, like, edited work in the next iteration. So every adjustment doesn't have to become, like, a chat exchange, which would potentially make take a lot more time. And I think for... Even for motion or for longer running tasks like, like a launch video, for instance, you know, we review storyboards and a timed version at each step before the final or the finished render, and that gives us something... Gives the whole team to react earlier in the process so that we can decide whether that direction is the route that we want to take. So that way, longer production jobs are still running in the background, but we're able to see your updates or progress updates as they come in. And I think that should answer the question. But, Amal, do you have anything that you want to add to that? Yeah. Yeah. I love... So I think, a really big part of the agent was that UX as you're talking about. Like, how do you enable the agent not only to give expectations of based on the complexity of a task, how long it might take? To Shreya's point, some designs might be pretty fast, but some of the more complex tasks can take hours and hours and hours. And and the issue that I faced first was we had an agent that was running not... Non synchronous sub agents, like, so synchronous sub agents. And it would delegate work, but they would stop the the main agent from, you know, from being able to to respond to the user because in deep agents, synchronous sub agents or many other harnesses, they stop the execution of the lead, and they wait until this agent is complete. And so I... To to the point about asynchronous sub agents, I moved that into async, had a lot of these things in the background, and then the lead agent would basically give an estimate, first of all, to the user. Let's say if a big complex task is gonna say, hey. I... This this might take me, like, two... Twenty, thirty minutes. I'm gonna give you a storyboard first, then I'm gonna give you, the, the animatic, which is, like, the... This smaller version of the animation. And then I'm gonna start a a production task to, like, as soon as you're approved to make it really, really scalable and and ready for you. Right? And so a big part of that coordination between the lead and async sub agents was that behavior we encoded in this harness where it has a bunch of tools to start the start the async sub agents, to give them tools, give them context, give them, define the model, define the the, the effort of that model. Right? So some agents might have OPUS, some agents might need GPT 6.1 low or whatever. So, it... It's it's it's coordinating all those long running tasks between the team of async sub agents, and, therefore, it can stay. Since it's async, the lead can be open to iterate with the... And start new async tasks with with the users or answer any questions as it's coming, give progress notes, maybe even schedule its own wake up for, like, let's say, for the next five minutes since it knows that there might be a task complete by then. And there's a lot of coordination between async sub agents to the lead with, like, kind of the the notification system where this async sub agent, as soon as they're done, there is a tool called back to the lead when it's done. So, like, similar to... If you use coding agents, it's gonna see a very similar behavior with CloudQuote, for example. And so this was, like, a really, really big unlock for a good UX in Slack that allows you to run long writing tasks and still be able to, like, give estimates, collaborate, and, provide expectations on how the job might look like. So here's another one with a good amount of upvotes. How do you define your design system? How do you review this at scale, and how do you use reviews to improve the next pass? Okay. I can I can maybe take this? To me, I feel like our design system is a living document and it's shared decisions that we make over time, across our team at Langchain. So I think it starts with the foundations. You have our font, our colors, our logos, the spacing rules, guidelines. Those are the principles that become the skeleton for any decision that we make, what someone should notice first, what feels most recognizably like LangChain, how our colors support the messaging, what motion is helping us communicate. And the agent basically needs examples and reasoning along those standards. So if I say that, you know, a certain layout works, I also want to explain why it works. And that could potentially... That's that's something... The reason I say the VIN document is because there's a series of decisions that might lead to us wanting to improve or, just, you know, like I said, we wanna continue to scale that identity over time. So, we're just... I wanna make sure that we have consistency and that there's room to explore as well. So a new campaign should accommodate room for a new idea. And, yeah, essentially, I would say that's like the the design system is a foundational layer, and then being able to build on top of that with newer concepts as we grow is the hope potentially for a brand. And and I would add that we had our design system in Figma. A lot of these really, kind of... Everything that Treya mentioned, which is around our... The way we have our logo assets to the spacing rules to color tokens to guidelines, that all would be in Figma. And when I was bringing it all into the agent, I basically had my coding agent look at the Figma and bring it all into, like, into into skills or or into Wiki that would provide references of what our assets look like. And that then would be referenced by the main agent, with either reading the skill and looking at what I have to design, similarly to what, you know, when you saw on the video, it was, like, making granular edits on what is good for us. It would look at that skill. It would provide that guideline on, like, what what this asset... Or how this asset should look like, how the logo should be placed, what size should it be, what fonts do we use. Right? And that would be both checked by... Through a skill, but also the linting is... If you remember that that slide, the the lint of the brand where it would run deterministic scripts that would check every rule that we have. And now as soon as we have a new campaign that we need to update it, we will update it in the coding in in the in the harness, and that would be pretty fast to iterate and update. So so, everything is based... There are several layers of verification, but Opus five five as the model is really, really smart that to to follow many of the skills by default as guidelines, and then sometimes it would not even need to use brand linting, and, it would provide the right output, from the beginning. Awesome. You've mentioned Opus five five and a couple other models throughout, but, what models, did we test with, Content Studio? Which worked best, and was it tested with small language models? Yeah. Great question. So in the beginning, well, I think to to to say the... I am a... I am I'm a supporter of of of bigger models and pushing the intelligence to the max. So I wanted to test with the bigger model first. As soon... And I started testing with... Back then it was Opus 4.8. It was a bit more expensive, so I wanted to to see whether it's really gonna... What's the, I guess, the the the ROI on my work as an engineer trying to make the... To compensate for the weakness of the smaller model and intelligence for a smaller model by adding all these components. As I mentioned, like, before, we had a ton of these, components added as, to the... As as references to smaller models so that they can basically assemble instead of designing. Because as soon as we'll start... When I ask them to design something from scratch, they would always look kind of not to the standard that we're looking for. And I thought with smaller model, if we were adding... If we were giving them the task of just assembling instead of designing, that would be really, really scalable. But in the end of the day, it was not the result we're looking for because, again, it would just assemble them all in one place, and copy paste the layouts and just duplicate that all the time, in comparison to a bigger model that would go crazy and do crazy things that are really exciting to look at and are engaging and are still on brand, for us. So, to answer your question, yeah, we tried smaller and the the bigger models, but, the results are much, much better with the bigger models. And I have not found success with a lot of the smaller models now. That to say, our async sub agents for smaller tasks, let's say, maybe designing, like, a very small asset, editing a small... Editing and doing a small tweak to an asset that is already existent, that can be used because the main model can define its, its sub agents. It can it can define the the model effort, the model type, and it would use a small model if necessary. But that's up to the judgment of the main model, which is Office. It's another interesting question. If it's been trained on the use cases Langchain had internally, what would be the step for us as users to optimize slash make it better for a larger set of applications specific to their business, I assume? Yeah. I can take it and straight for free to jump in as well. I know that it's something that we did overall. The open source repo that we have in the docs section would have a lot of the kind of more generic skills that we abstracted away from our specific branding. And so that would include, kind of guidelines even for coding agents locally. If you provide them your website, if you provide provide them your brand guidelines or brand books, it will look at them, the... Can decompose them and put them into skills. So the onboarding with the repo with your coding agent should be, hopefully, seamless when it will analyze your branding, guidelines and then put them into skills and hopefully have a a good enough, output in the first try, by encoding that as an... Into... Based on the judgment that we have about, you know, creating this harness. So it would add this to skills. It would add those to Wiki. It would design the environment, similar to what we have based on your brand tokens and guidelines. But I think... And, Shreya, feel free to jump in. Might be more... Might... There might need some more iterations as you are working through this in Slack or whatever environment you choose. It took us many iterations even with our branding skills, as Shreya mentioned. But, Shreya, curious to hear your take. No. I think, you've basically covered it all. Yeah. It did... We... It took us many tries to get to the point where we are today. I largely just agree with what you're saying. Yeah. Yeah. I think I think, we try to make it easier for for you guys to have on a post source project. We can just start with a good foundation and then for your coding agents to update the relevant context and the harness. But it might take a couple more tries to to to make it right. And if you have any questions, please, feel free to reach any... To any of us, and we can we can help out. We are coming up on time, so I think this is likely to be our our last question. But I think this is an interesting, conceptual question. I think it's a little different than what we solved, but probably one that's interesting to answer. Can ContentStudio be tweaked to allow for multiple design documents for multiple clients? We're pretty single player in that we're solving it for Langchain. I assume maybe they're an agency and want to try to leverage this for a bunch of different clients, and kind of have a similar output. How would you think about that? Totally. And, that's a great question. I think in the end of the day, context... The way you design the environment and the context can be really scalable. So let's say for us, we encode our design, but let's say if we wanted to have it pretty flexible, we can have, that context being, let's say, stored in a shared space. Let's say, even Google Drive. Right? If you have, like, Google Drive of each of your clients as as as subfolders and folders and have the tools for the agents to look at them before designing something, you can add a bunch of skills to, like, to to explain exactly what to look for, how to find them, and how to use them. And then the agent will just bring them into the sandbox, let's say, and then use them for that particular design. So you can make the environment pretty scalable, and and flexible if you if you host that context somewhere somewhere else, or bring it into the sandbox as you as you run the the agent. But I think how you ask them to design... Either you design the the the folders yourself in Google Drive, let's say, to... You ask your eight... Your clients to add that or, you do it yourself, that's really up up to you, I guess. Awesome. Well, I think we are at time, with what we can cover for q and a. We're all super, super appreciative of y'all hanging out with us. Really hope you'll join us again for our future events. And all of us are available on LinkedIn. So if you wanna ask us any questions, you can reach us there, and as well as... This will all be shared, as I mentioned before, in an email following the event. So very appreciative of you all being here, and have a great rest of your day. Bye bye.