Name: [Customer AMA] Scaling Agents Across Your Organization with LangSmith Fleet
Uploaded: 2026-05-06T21:16:10.954Z
Duration: 41 min 44 s
Description: [Customer AMA] Scaling Agents Across Your Organization with LangSmith Fleet

Transcript for "[Customer AMA] Scaling Agents Across Your Organization with LangSmith Fleet": Alright. Well, welcome, everyone. Thanks so much for joining today's AMA. We have a very fun topic queued up for you today. Lanesmith Fleet. Here, let's let's dive into it. Ariel, would you like to start the start Yes. Let me share my screen here. the presentation? Awesome. There we go. Next slide. So let's get the boring stuff out of the way. Who are we? Introductions. So I'm Carlos. I am responsible for account management here on the West and API version. If I have a name or face that you aren't familiar with, you may have interactions with my counterpart, Lauren Quilty, out on the East Coast. I will be playing the part of cohost in today's AMA session to the host, Ariel. Ariel, would you like to introduce yourself? Yeah. Hey, everyone. I'm Ariel. I'm part of the customer enablement team as a customer engineer. I work on delivering workshops to help you get more acclimated with the blank chain ecosystem. Awesome. Awesome. So today, what's the agenda like? We'll have a mix of informative literature as well as being hands on with Fleet in the actual product. By the way, Fleet is new nomenclature. If Fleet is new familiar with agent, which is our to, building agents. So fleet, new name moving forward. Agenda. So what we'll talk actually is, does it fit into the whole ecosystem and the details of the. How Carlos, you're a bit choppy, so I can continue on with the agenda here. Update. So, And we'll have, q and a. gonna be, Yeah. I'll take it from here. Thanks, Carlos. So how to participate? Feel free to drop questions in the chat. We're gonna be going through about twenty minutes of an overview of Fleet, and then we'll be answering questions right afterwards. The session is being recorded and will be distributed shortly after. And after this recording, please feel free to leave us feedback as we're always looking to improve. Alright. So first, before going into Fleet, I wanna get us situated with common questions that we get. Where does Fleet sit inside the ecosystem and when to use Fleet? So Fleet is one of our build tools. It's a no code way to build agents, and it's typically used for automating everyday tasks. So ones that are lower stakes, like reading through your inbox, providing a summary, doing research on the back end as well. Then when it comes to using, agents or developing agents that are external facing or revenue critical is where we see more customers landing towards our code solutions, which include LangGraph, Langchain, and DeepAgents. Great. So let's go into what we're going to build today. We're going to walk through a use case of building a market intelligence agent that provides us a report that summarizes for a given week where competitors land, what have they released, and help our team be able to pivot off of that. So let's go right into building an agent. Here I'm on the homepage of Fleet. So Fleet lives inside LangSmith. If you're navigating into fleet, you can come directly into Langht Smith. In the top left hand corner, you could select fleet. So fleet also has the ability to observe traces, which we'll do, evaluators, and is also deployed on Linksmith. So before we go into creating our use case, one of the quick ways of also developing an agent on fleet is via templates. Here are a couple that we've already created. One of them being similar to the use case we're gonna walk through, which is a competitor intelligence Slack bot. So within this template, we have already pie compiled agent instructions, integrations, which we'll dive into, as well as skills. So these are quick ways to start creating an agent and then tailor this based on your specific use case. For integrations, what makes Fleet very powerful as a no code builder is that it integrates with many common tools that you use today. For example, for productivity tools like g Suite, communication tools like Slack, and also research tools, which we're going to use one of them, Tavoly specifically, to do research on competitors. And another resource that our agent has access to are skills, which are reusable instructions. So a common skill that we use in our org is our brand skill. So just to quickly show you, a skill requires a name description, and then it's just plain text instructions on our guidelines here. So here we have styling, instructions set, what our colors are, our font sizes, and how this could be used across the org is I could start creating an agent that maybe is going to develop a report and make sure that it's consistent with our brand guidelines. So this is a great way to have consistency and also be able to share skills across agents. So as your organization starts to develop more agents, you may want to share this across teams, across the org. And this is this is where workspace agents comes into play. So Workspace agents, similar to templates, already has instructions, tools, as well has settings on whether they can be ran, edited, as well as cloned. Agents can also be shared with a subset of individuals, and you could also have private agents just to yourself. Great. So I've kind of touched on here that you can create agents in different ways. One being a template. Second, natural language is what we're going to do next. And you could also set up an agent manually by specifying the agent's memory and files, organically on your own. So here, we're going to start developing this market intelligence agent. But before actually having it run on a repeated basis, we're just going to ask it to develop a single report. So I'm gonna copy over a prompt. Fairly simple. I'm just asking Fleet provide me a report on what Lynx Smith Fleet's top competitors released in the last week. And here, I can also add skills. So I can add that skill that I mentioned previously, the lang chain brand skill, and say something like use this for formatting the report. Under the hood, Langsmith Langsmith Fleet is using deep agents, and we can also specify here what model we wanted to use. So Lynx Smith Fleet is model agnostic. You could select different models as well from different providers. And you also have the ability to upload files. So maybe you've had a previous report that you're now looking to automate. You can upload that report so that your agent tries to create something in a similar format, And you could also give it computer access if you want, maybe, additional files, that you have saved down for it to have access to. So here, we now send this prompt off to our agent. And because this is underlying a deep agent, it's under the hood working, through long horizon tasks. So this is going to take a bit of time, which is why I have very very similar prompt, already complete so we can actually walk through it and see what's happening under the hood. So this is the same prompt, also pointing towards two different skills to generate the report. And initially, our agent started with just looking to take our initial prompt into a more refined task. So in this small snippet, we could see that took our initial prompt and then added some instructions on top of how it was going to go about reviewing competitors and ultimately delivered a result. We'll go through a little bit later of what this looks like in detail in a trace. But as of right now, let's see what the output was. So this agent, completed its task, series of tasks, and wrote to a file, which is this one here, a report of top competitors, something that we could share within our teams and possibly, help us action or prioritize different features. What's really useful is you can try to amend your file, manually here by selecting the elements and specifying maybe you want your text to be smaller, larger, or, have other styling elements, you can do so. And you can also amend using natural language. So here I had specified that I wanted to make the heading larger, which it did, and it also applied this across headers. So at this stage, we've created an agent using Oh, sorry. We've used, one time, occurrence of creating a report, which was this report. But now what happens if we want to actually run this on a cadence? So maybe every single week, we wanna get a report back, that does research on our competitors, and then we can observe, maybe that report, say, in Slack. So if you're going through this flow on your own, you'd see a pop up here that says turn this into a reusable agent, which is what I had clicked in this circumstance. And once you look to turn this to be reusable, the agent will ask you a few questions. So there are two questions here that it had asked. It responded, I have everything I need. The agent's going to run on a weekly basis. How do you want the report to be delivered? So there are several different ways it could be delivered. I selected Slack channel and then also what ID I want the post to be reported to. Once you, answer questions based that your agent, asked to be configured, you will then see the agent is generated in this panel, and you'll be able to actually view your agent in more detail. So that's what we're going to do next. We're actually gonna observe inside the agent how do we actually improve upon it, modify it, and also observe its behavior. Great. So right now, I'm inside the editor of this particular agent and fleet. So it's the same one here. I just clicked view agent, edit, and then now I can see, two different panels here. On the left hand side, there's a chat window where I can interact with the agent and have feedback on improvements. Then on the right hand side is the agent editor. And one of the resources here that we had touched upon earlier was the toolbox. So we can see here that our agent has used different tools to create that report without me having to specify those tools. This included, Tableau for web search and also Slack to send the message. But what if I actually want to have our agent look at maybe x as an example? So look at social media posts. Perhaps we noticed that competitors are putting a lot of releases there. So I'm gonna put here in the agent chat window, I'd like to use x as a research tool for competitor releases. While that starts, going, I'm gonna show you other ways to add tools so you can add tools manually as well here. There's a list of them. If you wanted to bring in your own tool, you can add an MCP server here. So maybe it's a connection to your own database or your own, resources. And if you have an MCP endpoint, you would enter in the URL here and then authenticate and save the server. You could also add custom MCPs in your settings as well. So now if we look at the response from our agent, we see here that it identified there are several tools that we could use for x in order to search posts, user posts, and get posts from lists. And, in this let me actually refresh this here. Great. It's asking us if we want to add all relevant tools or to keep it minimal. So I'm gonna say add all tools. And our expectation is that the tool should show up on the right hand side and also update our memory. So let's actually dive into the memory of our agent and the instructions that are set. So in this file explorer, our agent automatically created a set of files that it has access to. So without us having to create this detailed prompt, it's done it on its own. It started with a role, so you're a weekly competitive intelligence agent and also a specific series of tasks that needs to complete, which are specified down below. So different steps for research, triage, and then ultimately, its role to post to Slack and how to format that report. Within our file system, we also see what skills our agent has access to. Skills also are really useful because they load based on when the agent deems them necessary, so they're not loading up or taking up context, in between. Then our agent also decided that for this given task, it'd be best to use sub agents. And our sub agents have their own memory here of what their tasks are and how it should go about research. So it's really useful about sub agents. One of the reasons is that it's faster. So instead of having to, have one single agent sequentially just wait to research each and every competitor, we can have multiple, agents review competitors at the same time as well, because they're running in isolation. If there was a failure, it'd be easier to recover. In this file, we also have our configuration, which includes the triggering of our agent and tools that it has access to. So if we go back to our agent editor, we could see in our toolbox that those tools related to x are now added automatically. So without having to actually code it up or manually input these tools, we've been able to do, we've been able to have natural language here in order to instruct our agent to add tools for us. And if we actually go back to our file system here and if I search x, We don't see it in this instance. It's in, my other instance that it chose to add x here. It might have asked us down below if we wanna add it to memory, but that's something that we could also do. We can specify maybe when to use x, when to use web search by just entering our cursor here and adding, some text and saving down our agent. Great. So as of right now, we've been able to modify our agent by using natural language here. But in reality, maybe this is not where we have all of our communication. So this is where channels comes into play. You may want to interact with our agent outside of the space, for example, in Slack, and be able to quickly reference it to maybe create a competitor report on the fly or possibly update maybe for a future run. So here, we can connect to Slack in this case, and all we would have to do is, reference our agent with the at symbol, and then we can interact with our agent in a different environment outside of this. Another thing to mention is, well, how does our agent actually have the credentials to run these different tools to access different channels? And this is where agent identity comes into play. So there's two options here, clause and assistance. So clause are essentially, fixed credentials that are more doesn't really matter who's interacting with this or more widely used. Whereas assistance let's imagine that instead we have an agent that's going through our inbox and giving us a summary per day of what we should be focusing on. That's our own credentials. So it would be based on each user, and in that case, we would be selecting assistance here. Okay. So that was a bit of inside the agent, and I'd like to shift our focus over to governance, observability, and deployment. So when you're running your agent in the real world, you may not feel comfortable with it starting to send reports right away, maybe in your team Slack channel. Perhaps you want to actually review its output each time until you get more confident in how it's, delivering each report so that it's non deterministic. So what you can do here is set human in the loop instances where your agent will need to wait for your approval in order to continue. Here in your inbox so just click out here. In your inbox is where you'd see in this example, I'm going to, wait for approval for my agent to be able to post to Slack. So in my inbox, I could see my Market Intelligence Weekly agent needs my approval in order to continue to make that post. And the agent inbox generally is very useful because you may have not just one agent, but several agents, running different tasks. And this is your centralized place to see what may need your attention, but also see what's completed as well. So in this case, we have our weekly agent that looks for competitors. So it was fired to run, and it spun up different sub agents in order to review different competitors. Then once it was ready to send that Slack message, it asked for our approval. So I could come in here and either accept or I can maybe specify what I'd like the agent to change. Maybe there's a competitor that's missing or maybe the formatting is not how I'd like it. I can leave that feedback here, reject it, and then have my agent make the modification on its own. And if we wanted to actually have our agent make modifications and update its memory without us having to do so, we can have it actually set to update its own memory. So within the agent editor, under settings, there's the option to have the agent auto improve. So maybe your agent lives inside Slack, and we have different teammates who are asking for ad hoc reports to compare against other competitors. Maybe there's a single competitor that keeps coming up every time, but isn't isn't in our daily run or a weekly run. Fleet will review conversations daily and determine if there's reason to update its memory and can do so automatically on its own for auto improve, or it can still recognize that there may be opportunity to update its memory and require approval, which will also be sent to the agent inbox, which you could review and then approve to update the memory. Great. So that was a little bit about human in the loop of how you can have more control over your agent. Maybe when you build it and you're looking to have it run several times but haven't trusted the output yet, it could be a great option to set. Next, observability. So what's really great about Fleet is that it lives within Lanesmith, so automatically traces are captured. So if I navigate back to this agent, within your settings, you can view agent traces, which will bring you over to LYNX Myth. I have one already opened here. This is an example coming from the same competitive intelligence agent where the input is that we had a cron job that was fired, so that weekly occurrence for this agent, this deep agent to run. And we could see on the right hand side down below the agent's output and all of the intermediate steps in between, which could be a lot to take in. So we could use Poly to maybe summarize this trace, perhaps give us some oversight of maybe how many tool calls we're using, so we can get some insight as to how wide the search is. As well, if we're finding that this is starting to take longer, maybe not just five minutes, but it's starting to take twenty minutes for this deep agent to continue or finish its task, we can come over to the waterfall view. And here we could see the total time of our deep agent, which in this case is a little less than five minutes. And we can also observe its steps when it's reasoning, when it's calling different tools. And, also, we can observe, the sub agents here and their steps. So just as a recap, links with traces are automatically being logged for links with fleet. So you don't have to set this up. And that's one of the benefits is that it's just easy to set up your agent and then automatically see traces and be able to observe how you may want to improve upon them. And then lastly here, running your agent in the real world, you may want to call it like an API so you can access it again outside of just this environment. So if we go back to our agent editor here, under settings, we could find code snippets. So if I select this here, our agent is by default deployed on Linksmith via Linksmith Cloud. We have our agent ID and our API URL. If we wanted to port this agent and maybe give it to an engineer so that they can continue working on this deep agent or have it called within a different place, we can provide this set of code. And we can also download the agent config as a zip file. So these files that I walked you through, the agent's memory, the skills it's accessing, some instructions for sub agents, config and tools, that's all portable, something that you can download and then use for another purpose, maybe to extend upon your agent. Alright. I know we have probably just a few minutes before I wanna get to questions. So there's two other, areas that I wanted to show you quickly on Fleet. And if I open up the side panel here, a common question that we see is how do I actually control the usage of the agents? Don't want cost to skyrocket and also wanna make sure that we have caps maybe across users. So here in the usage, you can quickly view how agents are being applied within, your workspace, what the cost is, how many runs, how many agents, and you could set limits as well. So dollar value limit per agent as well as per user. This may give you some insight to maybe change change certain agents from being daily to weekly if you're looking to save on costs. As well, in settings is where you can manage at the Oregon workspace level RBAC, so permissioning for users, and you can also add custom MCP servers down below as well. So I know that was a little bit fast, but, hopefully, we've been able to or you've been able to kind of see that Fleet is a very easy way to get started with creating agents. That initial prompt that I had was fairly simple. And from that, Fleet was able to create a memory file, that was more expansive and more detailed without me having to do so, and then also select what tools to use to complete task. And one of the benefits here of Fleet is that there is no infrastructure required, so automatically connected to traces as well, automatically deployed. Fleet is useful because it can be both used by technical and nontechnical folks within the org since it's using natural language in order to develop and amend your fleet agent. And you can also take fleet outside of this environment that I've walked you through and bring it into maybe a more native spot for your org like Slack. So with that, happy to answer. any questions. I'm jumping in here. Just Carlos has some connection issues for the day, but hi, everyone. I'm Angeline on the marketing side. We are gonna jump directly to questions right now, but I see that, Isaac, you asked the questions about, for integrations, do you have, Asana? And if not, are you planning on adding it? So, Andrew, thank you for answering that. Asana isn't a built in integration, but, Ariel, maybe you can kinda share in terms of how, do folks can configure custom tools and skills and integrations to, Fleet. Yeah. So if you don't see an integration here, you can go back to settings, and you can add the MCP server. So, for example, in this case, for Asana, and you'd be able to connect to that and view it as your integration as well in your edit edit agent area. If I go back to edit, here you can also connect to say, like, the Asana MCP here and then save the server. Alright. Probably move on to the next question. How are Fleet Agents different from Linkedin agents or DeepAgents? Yeah. So Fleet is our no code agent builder. So it's a UI that sits on top of deep agents. And typically, if I go back to that previous slide, just to give a visual here. Typically, we see customers use Fleet for lower risk. So like automating everyday tasks, keeping things internally within the org, and then using pro code solutions like lang chain, lang graph, and deep agents when they want more control, and it may be an agent that's external facing or revenue critical. Great. And then for the next question, how do I know which agents would be best built with Fleet versus, normal DeepAgents? Yeah. So I think it's kind of similar, where you probably start with the fleet for the everyday tasks. But, also, I guess, depend, you can also take it from the user perspective. So within your org, if you want to give the ability for nontechnical users to be able to develop agents, that's when, Fleet becomes very useful since they just need to use natural language to develop and also modify. Then for, the more technical folks and more external facing agents, that's when using deep agents framework would be really useful. 100% agree with that. Me and marketing, I will definitely be the person who's using Fleet. So agree. Alright. The next question is, how does the agent improve over time? Yes. That's a great question. So let's jump back in here and go into our inbox or actually I'll go into the edit view again. So in our settings, we could see here that we have an option to auto improve. So our agent reviews our conversations on a regular basis. So these threads that we're having, these conversations that we're having with it. And if, for example, this competitive intelligence agent, if it sees that we want the report structured in a certain way every time, maybe we're providing that feedback that we want to also add, not just recent releases, but maybe funds that were, raised between our competitors. They'll observe that that's a frequent thing that we're asking for and then they possibly auto improve by adding that into the memory, so in that agents MD file so that going forward, it'll add that context as well. So that's the auto way of doing it. And then if you just wanted to improve the agent, yourself, you could amend this file by just, going right into it and then saving any modifications you have or using natural language to tell the agent, give us certain instructions and have it update that way as well. Alright. Thanks. And then the next question here is, do I have to use the Fleet UI, or can my agent live in Slack? Might have mentioned that. Yeah. That's one of the benefits is that you don't have to use the Fleet UI. You can have it connect to, a more native tool that you use on a regular basis. So the example that, I've used this one for is within Slack. So this agent lives within a Slack channel, and I can just reference the agent and ask it to maybe create a competitive analysis on the fly or ask it to make a modification for a future report. And because it's connected to Slack, it'll see my prompt, my input, and then can move forward with that task or maybe updating the memory from there. Great. Can the next question is, can I deploy fleet agents in our own environment? So right now, we have, the ability to deploy on Linksmith Cloud, for self hosted that's currently in beta, but something that, Carlos can connect further on if needed. Alright. And the next question is, can my engineers extend a fleet agent in code? Yeah. It's a great question. So within the same space, the agent editor, if we go over to settings here within view code snippets, you can provide the snippet to an agent to sorry, to your engineer to call this agent, within a different maybe workspace or area. But if you want your engineers to maybe create start creating a more detailed deep agent, based off of this, you can go to settings and export, the configurations of this agent, which will provide your engineer these files that we have here in the explorer. Alright. And this is going to be the last question, unless if anyone else have questions, feel free to drop it in the chat. But, how do we make sure that agent, the agent doesn't take any high stakes action without us? Yeah. This is very critical, especially when you're you just created your first agent and you want to make sure that, that you can trust it before it's maybe sending off reports or emails or or whatever output on its own. That's where the agent inbox becomes very important. So it's right underneath chat. And this is a centralized place of across all your agents where you'll get notifications on successful completions, but also those that need your attention. So to answer that question specifically, you'll see that your agent, needs approval, which you could set here. So in this case, this agent that we just walked through, I have it selected or enabled that this tool requires human approval before executing. So before sending out, I need to review it. And then what would happen is that in the agent inbox, I would see that needs approval notification. And then after review, I can accept it, and it'll send out. Great. Well, thank you so much, Ariel, for the presentation today and thank you all so much for attending today's session. We will be sending a recorded version of this, after, we wrap up. And, again, if there are any questions or anything that you have, for our account management team, feel free to go ahead and message us. Or there's also some, docs and also some more details on the doc side on the right hand side there. And, yeah, feel free to, also send us, kind of, like, what your thoughts are about the session. We'll be sending a survey, after the session. And we also have Interrupt, which is our annual conference. It will be based in San Francisco. I believe you can still get tickets right now, but there is a live stream that's happening. So if you can't make it, just go on interrupt, our interrupt website, and you can actually register there as well. Well, I think that's all. Thank you all so much. Thanks, everyone. Thank you.