Name: Building Agents for Life Sciences: Deep Life Sci Workshop | October 8
Uploaded: 2026-10-08T18:31:31.359Z
Duration: 59 min 56 s
Description: Building Agents for Life Sciences: Deep Life Sci Workshop | October 8

Transcript for "Building Agents for Life Sciences: Deep Life Sci Workshop | October 8": Agents are a new kind of software. Endless inputs and non deterministic outputs. Build them the old way, and they break. Enter Lingsmith, the agent engineering It's model agnostic, cloud agnostic, and framework agnostic. Organization is shipping the best agents iterate with a system, and we call that system the agent development life cycle. Build, test, deploy, and monitor. You build with our open source frameworks. Deep agents, langchain, or langgraph. LangSmith Fleet enables anyone to build agents without code. Test agents using evals and experiments. Deploy changes in one click with LangSmith Deployment. Monitor every interaction from a single dashboard. Governance is built into every stage of the agent development cycle with Lanesmith LLM Gateway. Lanesmith Engine helps you improve agents autonomously and move through the development lifecycle more quickly. Blanksmith Engine is your agent for agent engineering. Hello, everyone. It's great to be here. Let's, we'll give about thirty seconds for people to finish, trickling in, and then we'll get started. Alright. Thank you all for being here for this workshop on, agent engineering for the life sciences. I am Michael Cunningham. I'm a deployed engineer, here at Langchain. And, you know, generally, what I'm doing is is kind of, like, helping companies, get get started with Langchain's tech, bring it into their operations, really get the value, out of everything that we offer for, agent development. But I actually have a background in, life science. I, did lab work for about seven years if you count my undergrad, and, spent some time in in pharma doing drug development. So this is still kind of a very active area of interest of mine, and I've kind of been developing some tools here at Langchain to to, help companies, you know, life size companies in particular, really kinda, again, get the get them the most advantage out of, Agen to k I. So a little bit about Langchain. We were actually, the the original LLM and agent, orchestration framework. So, Langchain came out around, the end of twenty twenty two right when ChatGPT did, and people were still kind of figuring out, you know, how to, how to how to, use LLMs. Right? This was very early days. And Langchain was really kind of the first, framework that actually allowed people to, easily build, you know, the the, you know, sort of the very first agents and and to LLM automations. Since then, it's really evolved, quite a bit into a couple of different, open source, you know, libraries and and pieces of software that we offer, to allow people to, to build agents and build other, LLM automations. And then we have, let... LangSmith, which is our commercial platform, which is, essentially intended, you know, once you've built your agents, with the open source software or with anybody's open source software because we're very widely, compatible to actually then, take those agents into deployment, make sure they're reliable, make sure that they can, scale and so on and so forth. And one kind of recurring theme that you'll you'll get a lot here is that, building agents is actually the easy part. But then getting them into production, making sure that they are are working properly, and, you know, and and, up up to the quality that you expect for actual production apps, that's the hard part. And that's kind of where, Lang Smith comes in. Now the life sciences are really just, poised for a transformation with AgenTic dot ai. We have, you know, many of you, if you're in pharma, you may have heard of, Arum's law. Right? This is the, Moore's law backwards. So, you know, with Moore's law, we've seen this exponential increase in, computation power and this incredible, you know, drop in cost. But the, the opposite thing has actually been happening in, in pharma. Right? The the cost of bringing a new drug to market is actually, increasing exponentially over time. And, you know, this obviously is is just a really, tragic situation given that, you know, the the the purpose of of pharma and of life science is to is to bring life saving cures to people. So, this really is something that, you know, if if, AgenTic AI can reverse this, this is really what, this could really do a lot of of just incredible good. So, you know, there's a lot of contributors to, this problem. You know, one of them is, that the amount of literature... Scientific literature is actually growing exponentially, which sounds on its surface like a good thing. But, you know, when you are... And, you know, I'm speaking from experience here. You're a scientist trying to, keep up to date with with the the state of the art. It's it's just, you know, there's a deluge of papers. It's impossible to keep up with all of it. But luckily now with LLMs and and with agents, you know, agents can read, hundreds of papers at at at once. Right? You know, in in in in parallel. And and sort of pull these sort of needle, needle insights out of the... Just the haystack of literature. There's also this issue of, evidence being, sort of siloed. So, you know, you have, papers, you know, scattered across journals and so on and so forth. You have clinical trials. You have, you know, maybe your company's internal data. You have online databases of, of, of, you know, bioinformatic data. Right? Like, all of this is just... Is scattered to the four winds, and it's very hard to, sort of bring it all together, to get these insights in a in a central place. But luckily with agents, if you have, if you have your agent appropriately hooked up to all of those data sources, then it can it can, you know, bring that all together and synthesize it. And then, once you actually have, that that data in one place, then you have to do an... You have to actually analyze it. Right? And and you have to to, again, kinda synthesize that information into actual, useful insights. And, you know, one one really powerful paradigm that we've been seeing with agents is now the ability to write and execute code. And, you know, this is not necessarily a a trivial thing to to implement. You know, if any of you have been, kinda, you know, monitoring the news lately, you'll you'll know that, companies, you know, some of the big labs have gotten themselves in... Selves in trouble with, not properly sandboxing their agents. Right? So this is this is, something that that is a a technical challenge that requires, you know, software that is is suitable to keep these agents, properly, you know, contained when they... When they're executing arbitrary code. So in the interest of of kind of helping to bring that that AgenTek AI revolution to, to the life sciences and just, you know, sort of really illustrate what's possible with our technology. We've been building something called DeepLifeSci. Basically, what this is is this is an agent that is intended as sort of a general purpose, assistant for, life scientists who are doing this kind of work. It's hooked up to a couple of different, important scientific data sources. So, it has connectors for, PubMed. So it has, access to basically the full corpus of, of scientific abstracts that are out there. Also, PubMed Commons, full text. Right? So, there are millions of of articles out there that are, you know, where the full text is available through, through PubMed and has access to those. Clinicaltrials.gov. Right? So the the, government run repository of all clinical trials run-in The United States. And then, general web search. Right? There are a lot of questions that are, you know, not necessarily answerable just through, these sort of scientific data sources, and you need a more general, interface with the web that has access to that. And then as well as the ability to, to upload files. Right? So you can actually take your own data, your own, your own files, and, and and upload them to the agent. Now DeepLifeSci is a, what we call a multi agent system. This is a very, common pattern for for agents these days. So it has, what we call an orchestrator agent, which basically, you know, that is sort of the the core central agent that does most of the work and interacts with the user. But then that agent actually has the ability to, delegate, work to what we call sub agents. And the reason for this is, you know, let's say you have, hundreds of different papers that might be relevant to, to a a a scientific question the agent is working on, but, you don't really have the ability to, to to filter them. You you know, easily just, you don't wanna do give all of those abstracts or all of those papers to the orchestrator agent and and have it review them one by one. That would just take too long and it would get overloaded. And so what it can actually do is basically say, okay. I'm going to spawn a 100 sub agents and I'm gonna give each one of them a paper and say, you know, review this and give the, you know, tell me if it's relevant or give the, you know, give the most important, insights from it. And then the the outputs here are, you know, it it can generate, you know, scientific insights. Right? So so just, you know, reviews of literature reviews with, cited answers. Right? Going to, whatever paper or, clinical trial the inform... The the information was, was gained from. Right? That, that's the ability to generate citations to do, data analysis in a a safely contained sandbox. So this has access to, what we call a Lang Smith sandbox, which I'll I'll talk about, in a second. But that that actually, you know, is keeping the agent again safely, or the agent's code safely contained so that if there is any kind of, any kind of mistake that it makes, it's not actually affecting, your your, your system or your computer in any way. And then also has the ability to generate, files and figures. Right? So, you know, this can be, you know, scientific figures. This can be, you know, plots, charts, all that kind of thing. And then also, you know, if you're doing arbitrary data analysis, you can, the agent can then output data files. Right? Like a like a CSV or an Excel. It has the ability to generate that. Pretty much anything that, can be generated with code, the agent has the ability to, to generate that and then push that to the user. So without further ado, let's take a look at, what this actually looks like, you know, in person. So this is DeepLifeSight. It has kind of a, standard sort of chat GPT like, interface, you know, very simple and easy to use. And we can give it a question, like, let's say, we're interested in Moderna's, phase three mRNA vaccine trial for melanoma. We can say, you know, give the basic details on this. You know, many of you, if you are in, the pharma industry, maybe, have, you know, heard about this. It it made headlines lately when, you know, maybe the first first really, encouraging sign of effectiveness in a late stage trial for a a cancer vaccine. And so, the agent is able to find that. It actually says, okay, you know, here's the trial. Here's the citation. So we can actually try... Trace that right back to, the clinical trials registry. And, and then, you know, it it says, okay. You know, here's the enrollment arms, main question, so on and so forth. But it's not finding any results. So this agent should have web search, but it doesn't seem like it's finding that, the the web... The the results, which are posted online but not on clinicaltrials.gov. So something's wrong with the web search, and we'll actually get to that in a little bit. We have an issue with our agent. We're gonna have to fix it. So, you know, that's an example of, how it can, you know, just pull a single, a single clinical trial from the registry. But then, let's say now, you know, we want to expand our search and we wanna say, okay. You know, find all the other phase three, cancer vaccine trials for melanoma, save them to a CSV, give it... Give us the trial ID, title, status, sponsor, so on and so forth. Right? So we're actually doing sort of a full, full search across, across the clinical trial registry trying to get a, you know, a comprehensive view here. So if we look under the hood at what the agent is actually doing, so what it... It has a, a tool for searching clinicaltrials.gov. This this, this all looks a little bit messy because the agent is actually, searching in code. Right? So it it has the ability to compose these these tools into, these sort of, sophisticated code based, or code based, searches. So it's it's doing this sort of Boolean search. Looking through the, the clinical trials registry, it says, okay. Initial results, we have 221 trials here. You know, the... But then it's not... That is just looking at melanoma. Phase three hasn't added cancer yet. Okay. Now it adds, or excuse me, hasn't added, mRNA yet. Right? So now it it adds, or, you know, it adds vaccine, and it's going through this process of, you know, narrowing it down to the, the the relevant trials. And then, you know, now it's it's saying, okay. I've I've identified the main, ones that are labeled as vaccine, but there are some that might not have, the word vaccine in them. And so we need to look for those. Now one thing about about agents that distinguishes them from, traditional software, and and this is really important to remember when we're we're working with agents here, is that agents are nondeterministic. Right? Traditional software, works the same way every time. Agents, because they're using an LLM, there's some there's some element of randomness here. And so that is, you know, that's what necessitates a platform like LangSmith, which I'll get to to in a minute, that actually, enables you to, ensure that even though, you know, again, the agent is going through this this search process a different way each time, that at the end of the day, it is, you know, the the the results that it's it's getting to are, are the right ones. Right? So even if it's taking a slightly different search path, in each case, ideally, again, we we we should see it, you know, finding the correct answer each time. So we... If we can see, the agent, you know, finished its search, it's identified, all of these trials, And then it's generated this CSV file that we can then, download. Right? And for further analysis or, you know, if we need to pass that off to some other other stakeholder, and it has the... You know, it it cites this back to the, the actual, clinical trials registry. Right? So this is an example of how you can use this to, you know, again, kind of review review the, the corpus of clinical trials out there and and, you know, identify ones of interest. Now this also has an integration with PubMed. So if we're looking... We wanna look at the scientific literature, you know, let's say we're interested in, this particular protein's relationship to chronic pain. You know, maybe let's say we're looking into this as a possible, a possible, drug target. And so, you know, now the agent will go ahead and, look through, look through PubMed. So it has these... The set of, of search tools. And it's, you know, it's it's doing, like, a Boolean PubMed search here, narrowing it down to just the, the articles of interest. And then we can see here it's, again, very messy because this is all, you know, this is sort of in, agent language. It's working in code. But you can kinda see, and it's a little cut off, but it... It's actually, giving a prompt to its its sub agents. So it's actually spawning some sub agents here to review, review the... These these papers, or these abstracts, and saying, okay. You know, for this abstract, you know, find these these pieces of information and then return them to me. And it's doing this with, you know, it looks like about 45, papers at once. And so it gets the gets the result from these, or it gets the results from the agents that it's it's fanned out. It hits an error here, but, you know, because this is, you know, because these LLMs have gotten very smart, and it has what's, it has this very intelligently built harness. We can get to what a harness is in a second. But as this very intelligently built harness around it, it's actually able to correct on the fly and say, okay. You know, I I made an error here. I can make up for that. And, you know, it it it, revises how it's it's, it's calling the tool. So it's still going through this process of searching through all these abstracts. It's it's intended to be very thorough, so it can sometimes, you know, take a minute to do, to, you know, to go through this this search process, especially when you're when you're dealing with this kind of scale. And, again, it it... You know, each time it kind of... It takes a different, different different path, so this can be a little bit variable, but still doing its search. In the meantime, somebody asks, how does this product differ from CloudScience? Well, first of all, this is, completely free and open source. Right? So CloudScience is a, is a, a paid subscription. This is completely free. You know, you can, take this... You know, I... I'll I'll link you to the GitHub repo at the end, and, you know, set it up yourself, modify it in any way you want. And that really is, I I think, a a a a major plus of of our open source approach, you know, particularly for the the harness that we've used to build this, is that you're not locked into one vendor. You're not locked into, you know, you're not locked into a a specific company's paid product. You can, you you know, you can pick something like this up and just modify it in any way you want. Right? So that's that's really, I think, one of the biggest, value propositions here. And, again, this is, you know, we're we're we're not... This this is not a, a a paid product of ours, right, in any way. This is not not something we're trying to sell. So, so it's, you know, this is just kind of a demonstration, of the the capabilities that come with, Langchain's technology. Although we... I definitely do encourage you, to try this out. And if it is something that that you find valuable to, absolutely, you know, set it up for yourself and and, and use it. We have another question. Does it enable, title slash abstract search, or also full publication search? So, it has all three. It, generally, it does, generally does abstract search, because the... It has access to the full corpus of of, you know, I think it's 29,000,000 abstracts on PubMed. There are only about, 8,000,000, I think, open access papers. So that's through through PMC Commons. So, generally, what it'll do is if it's trying to, like, summarize the literature, it'll mainly look at abstracts. But then if a, if the papers that it's looking through are, are publicly available, it will use the full the full text of the paper. So, you know, let's say now we're interested... You know, this is just one of a set of targets that we're actually interested in. And we can say, for each of these targets, you know, get a count of their abstract mentions for chronic pain over the last five years. So we're saying, you know, we're interested in in targeting chronic pain. We have this... Oh, I'm sorry. I completely forgot to attach the actual, file here. So, we have this list of chronic pain targets we're interested in. This... You know, so we can upload this, the CSV file, and, you know, maybe apologize to the agent. Oh, I'm sorry. I forgot to, forgot to upload that. But... Okay. So it it, it, you know, it takes in the CSV file. It's able to, to read it. It sees what these proteins are, and then it's gonna go and do a a PubMed search to actually count the the number of, of citation... Or excuse me, of, of, you know, mentions of this, this these... Each of these proteins in, in PubMed abstracts over the past five years. And so, you know, this is kind of an example of, like... Let's say, you know, we are a, drug company that's that's, you know, screening these for... As as potential targets. We can say, okay. You know, where where is the, where has existing work been been done before? Right? And we can see, okay. It says, here's how many, times each of these was was co mentioned with, with chronic pain. So if we go back to, you know, go back here, the... This is architected in something called, deep agents. Right? So so I I talked a little bit earlier about harness. You know, I mentioned that a couple of times. So basically, what a harness is is, this is everything that's built around the agent in order to... Or around the LLM in order to turn it into an agent. Right? So, right off the... You know, out of the box, an LLM is just a a text generator. It it can't, you know, it it can't execute code. It can't, even, you know, sort of go back and forth sequentially with the user. All of that has to be has to be added on and that's what we call a harness. Now, Cloud Code is an example of a harness. Codex is a harness. You know, the... There... A lot of the major companies have their own, harnesses. But ours, which is DeepAgents, is actually completely open source. Right? So, you can... You know, just like DeepLifeSci, you can modify it in any way you want, and you can use it to build build whatever you want completely, completely free. So, and this is a a really modern, agent harness. You know, it... It's... The the best practices here are just changing all the time. And so, you know, we we constantly keep this up to date and look for, you know, the latest best practices and the latest features that, you know, that modern harnesses have. And, you know, so some of these include, like, a, a code interpreter. So, you know, when we were looking at the the tool calls the agent was making, you know, you'll... I'm sure you saw that, again, it was very, very sort of, like, dense and and sometimes hard to read. That's because the agent, isn't, you know, it... It's not just communicating in natural language. It's actually communicating in in code. So, the code interpreter gives it the ability to do that. It has a, a sandbox. So, you know, again, the the... This is, giving it the ability to execute code in this safely contained environment where, you know, there's no risk of it, you know, causing causing damage if it makes some kind of mistake. Right? Now that sandbox comes with, a couple of features like, the ability to, have skill files. For those who aren't familiar, basically, this is, these are are sort of like instructions for how to carry out a certain task. And so the agent can actually read that in from the file system, when it needs it. And, you know, this this means that it doesn't have to, have the the full set of information about its its, you know, how do... How it approaches its tasks, in its its system prompt. You can actually just stash this in a skill file, and the agent can retrieve it whenever it wants. Things like, summarization. Right? So you reach, you know, you reach the end of the LLM's context window. What happens? That, DeepAgents automatically, compacts that. Right? So it preserves only the most important information and lets the, lets the agent, continue. The sandbox also comes with a file system. So, you know, this is basically just like the file system that you would have, on your computer, but, again, within this this, contained sandbox. So the agent is actually, you know, can write files, read files, you know, execute code that that generates files, like that that CSV that we saw. You know, that's that's generated by the agent, in code and then and then pushed to the user. It also has the ability to delegate to sub agents. So this is... This, comes right out of the box with with deep agents. So, again, you know, that... This is just a very common pattern now to... Instead of just having a single agent that's expected to do everything, actually give it the ability to offload, you know, certain parts of its, certain parts of its work to, to to sub agents. And then there's human in the loop. And, you know, this is a really important part of agent engineering as well because, you know, agents... Again, they can fail. They can make mistakes. And, you know, some actions are, are, you know, perfectly fine, you know, like an agent just just reading some pieces of information. You don't have to worry about that. But then when you actually start getting to, actions that are irreversible, right, like deleting data or, you know, making modifications to, to your system. Right? That's something that you don't want the agent to just be able to do, you know, without any kind of permission. And so, DeepAgents allows for what we call human in the loop interrupts, under certain conditions where if the... You know, you can say, if you want... If the agent wants to do a certain action, you can require it to, to get permission from the human user, before. So this is kind of a... You know, this really enhances the, the the the safety here. Now there's a lot of advantages to using something like DeepAgents. You know, kinda getting back to what I... You know, where the earlier discussion about Claude Science, like, these... The other providers' harnesses, are often, you know, closed source. Right? And and they are intended specifically for you to use that provider's models. And as basically anybody who's been working in AI, or using it, you know, lately knows, like, companies' AI bills are just through the roof. Right? It's it's it's a it's it's a real problem. And these these models, especially for the major providers like Anthropic and OpenAI, are just... You know, they're very expensive. Right? And there's this really growing interest in, open models. Right? So the ability to, to, you know, take models from providers that are not closed source or to actually, like, hop back and forth between the, you know, these different providers models because, like like, the frontier is changing every day. Right? Like, you know, I'm sure you've seen, you know, one day, Anthropic will be in the lead and the other day, OpenAI will be in the lead and then it goes back and forth. And and we... You know, it's it's really critical to have the ability to, to adapt, adapt quickly and not get locked into one vendor. So provider agnosticism is is kinda really important here. Also just highly customizable. Right? So all of those those, features that I was showing, on the last slide, those are all, you know, those those can all be enabled or or or disabled, at will. Right? This is very modular. So, you know, let... Let's say you don't need a sandbox. You don't have to to give it one. You know, you can take human in the loop or leave it. Sub agents, you can turn them off. Right? So it really allows you to just easily compose whatever kind of agent, you're trying to build. And probably, you know, maybe the most important one is production readiness. And this is because, you know, kinda going back to what I mentioned at at the beginning, actually just building an agent is is not that hard. It's, you know, you you can you can throw together an agent, pretty quickly. The really hard part is once you have created that agent, does it actually work? Right? And, you know, it... That is... It is... It... I've seen many, many cases of, oh, we built this agent. Did you test it? Oh, no. Okay. Let's test it. It doesn't work. Right? That's that's that's very common. And even when you, you know, when you've tested your agent and you wanna put it into production, you know, some of the... Some other agent harnesses are just not really well suited to that. You know, they're they're not really designed to be production ready. DeepAgents is designed in such a way that it's very easy to take, an agent that you've built with DeepAgents and put it directly, into production. So this, you know, this kinda brings us to something called the agent development life cycle. Right? And and this is really, again, kind of inspired by that insight that building, just building an agent is the tip of the iceberg. Right? And so, this... You know, building is the first step of this process. But then once you build the agent, you have to test it. Right? You have to make sure, that it's actually doing what it's, what it's intended to do, that it's not making, you know, egregious errors, that's not costing too much money, that's not taking forever to run. All of that has to be tested. There's deployment. Right? So, you know, you have the agent, you have the agent ready. You feel like it's... Okay. It's... You know, we we can put this in front of users. Now we have to deploy it. That's non trivial. Right? You have to deal with, infrastructure considerations, so on and so forth. And then even, you know, once it's deployed, now you're getting it in front of, you know, 10 times, a 100 times as many people as we're seeing it in testing. And inevitably, you're going to start to, to see, you know, bugs and and issues that you hadn't seen before. Right? And so monitoring the agent, making sure that, you know, those... That you're identifying those issues and fixing them, that's another critical part of this. And this all sort of feeds into this this life cycle. Right? And and I'm sure that, you know, many of you are probably familiar with the, software development life cycle. But, you know, the again, agents being a very different type of technology, very different type of software, they need their, their own development life cycle. And, really, what, what Langchain is is here to do, what we specialize in is enabling all stages of that the agent development life cycle, for for customers. So, you know, this starts with our open source software like DeepAgents for actually building these agents in the first place, then testing them. So we have datasets, evaluations, experiments. I'll show all that in a minute. You know, we we offer, you know, a suite of software for that. Then there's deployment. Right? Actually getting these agents into production, you know, running in a in a scalable and, in in reliable way, and then monitoring them. Right? So, you know, seeing how well they're performing, making sure that, you know, as issues are are cropping up, they're being addressed. You know, we... That is... That's part of our platform, as well. So our one stop shop for this is called LangSmith, and we'll take a look at it here. So this is, you know, again, kind of intended to be the, you know, the single platform for all stages of the agent development life cycle. So bring this all together, so that, you know, you can take your agents from from building, testing, deploying, monitoring, all... You know, that's all right here. Right? The fundamental building block of, of Langhamt is traces. So if we go here to DeepLifeSci. Right? These are essentially records of what the agent, what the agent was doing. Right? So if we say here, you know, this was, the... We were actually looking at the, at the record from this... You know, the the conversation that I had with it earlier where I was saying, okay. You know, summarize it... The relationship between this protein and, and chronic pain. Right? Here's here's the input I gave it, and then we can see, you know, that triggered this call to chat open AI. So so we can see what model was called. We can see the, you know, time it took to the first token. We can see, you know, input, number of input tokens, input token cost, cash, you know, how many cash, cash cash tokens were created, output, number of output tokens, how many of those were reasoning. All of this is is visible right here. So we have this very detailed, rich breakdown. And we can see, you know, here's the system prompt. Right? So we can see everything going into the agent. We have the system prompt. We have what the user gave it. And then if we go to output, we can see this is what the LLM actually, you know, gave back. Right? So we have the the text it it wrote. We have, the call it made to, the tool call it made, to list this directory. And if we actually go to that directory or that, tool call, we can see, okay. You know, here was the, the input, the input path that it was given. Here was the output. So listed this file. And so, basically, what these traces are are just, again, a a a comprehensive sort of under the hood record of, everything that that is actually going on in the the agent harness when the agent is running. And so, you know, if we hit issues or we're trying to understand, like, why the agent did what it did, we can take a look through the traces and it's all visible, right here. Now once we have an agent built and we want to, to start testing it and making sure that it's working properly, then we have what are called datasets and experiments. And we can kinda think of these as like, sort of like standardized tests that you give to, to your agent. So, you know, that... That's that's a a a data set. Right? A data set is like a set of these questions that you're asking the agent. And then an experiment is an actual run of that test. Right? So if we see, you know, this was an experiment that we ran on the agent. You know, it's named after the model that was used as the orchestrator, with, you know, its thinking setting and then what model was used as the, the sub agent. Right? And we can see... Okay. You know, 12 questions were run. All of them produce citations. All of them produce the expected artifacts, and then it got 92% of the questions right. Right? So 11 out of the 12. And see when it was run. We can see the cost. We can see, how long the median question took so we can... And as well as ninety ninth percentile. So we can kinda get a sense of, you know, how how quickly is the agent, running. We have all that information right here. Could dive further into that and actually take a look at the individual questions. So, again, this is sort of like... You know, if you think of EVALS as sort of like the SAT for agents, you know, each of these is like one of the SAT questions. You can say, okay. You know, here are the questions that we gave, like, what, nuclear localization signal did, this author identify in this protein? And then we can... We have the rubric. Right? So this is actually the expected answer, kinda like the answer key. And then the actual output that the agent gave. And so, you know, that answer is gonna be checked against that rubric. And if we click on this again, we can actually, again, see the trace. So, you know, if the agent is, didn't produce the the result that we expect, we can actually look in there and see, you know, why did that happen. Right? If we go in here, you know, we can see, okay, this produced the correct citations. It, it wasn't expected to produce an artifact, so we don't have any feedback there. It got the rubric right. This is how long it took. This is how many tokens it used. Input, output, total tokens, cost. Right? All of this is visible, for each of these, each of these, runs of this this this agent right here. Now I think what really the the most powerful thing that this then enables is, actually comparing them. Right? So, in this case, we ran the agent with, Luna six. Right? GPT six... Or excuse me, g... GPT six Luna. In this case, we ran GPT 5.6 Luna. Right? So the older version. And we can actually click both of those and then compare them side by side. And so, you know, the view that we get here is basically, you know, all of the questions, again, same kind of thing. But then we can see the outputs for... In this case, this was, six, and then this was, 5.6. And we can see, you know, again, did they produce the right citations? Did they get the rubric right? How long did they take to run? Token usage cost, again, all visible right here. And what's interesting here is we can actually see... And this is this was a, you know, a real experiment, that, Luna's... Or GBT 6 Luna actually, as the sub agent, actually took a little bit longer than 5.6, and cost more, for the same accuracy. And so this kind of enables you, you know, when... Whenever these Frontier Labs release a new model, you can see, you know, oh, this this one blew the previous one out of the water on the benchmarks, and they have all these benchmark scores. But, you know, those are easy to game. And so this is... You know, the the the real value for you is is are these new models actually improving your agents. Right? That's that's the important thing. And so this enables you to actually do these kind of experiments and say, okay. Let's fact check this and see, you know, if we... If we're swapping out the the old model for the new one, is that actually improving performance? This allows you to actually test that. And so any kind of sort of AB testing that you wanna do as you improve your agent, this is this is the way to do it. So once you have the agent ready to go, and if you've done that kind of testing, you're confident, like, okay. This agent, you know, we're... We think this is ready to be put in front of in front of, you know, our users or customers, then it comes time to deploy it. And so this is... This brings us to, LangSmith deployments. And this is, basically, you know, just a, an easy way to get these agents up and running in the cloud. Our our open source, like, DeepAgents is intended to work, sort of hand in glove with these kind of deployments. So, you know, that's just really, really seamless and easy to do. We can see this is the, current deployment for DeepLifeSci. We can see this is the, this is the amount of compute we've allocated for it, and as well as for the database. We have the URL version. Right? All of this is visible here. And then we can see the traces. So, again, that same trace view. And, you know, so if we're trying to, we're trying to see, you know, what the agent is actually doing when when real customers are talking to it, right, real users, then we we have an easy, you know, easy view of that. Right? Then we can go to, monitoring and actually see... You know, get kind of a a health check of the agent. So if we look back at, like, the last, last couple of days, you know, this is when it... When, we saw the agent actually being run. We have, you know, what the latency looked like for that. And then we could get, you know, a health check on the deployment. So memory usage, CPU usage, you know, so on and so forth. All of this is is visible right here. So we have this just easy dashboard for, you know, seeing what's actually going on, with the agent. So, really, you know, this is the... This is just kinda scratching the surface of the features in LanghSmith. I mean, there's there's a lot of stuff here that that enables, you know, all stages of this this agent development process. But this kinda gives you, you know... I I think looking at these just gives you an overview of some of the most important parts of of that, that process. But there is one other thing that I really wanna show because I think it's it's maybe, out of all the things we have, it it may be the one that I just think is is the the coolest. And this is LangSmith Engine. So, you know, this sort of sits on top of the the agent development life cycle and acts as sort of like an agent for improving your agents. So, you know, all of the the steps of this process to, you know, to, feed into this this, you know, this this challenge of, okay. You know, once we have an agent built and it's running, we're monitoring it. Right? Then users are gonna be using it. They're gonna be hitting issues. And then how do we improve the the agent? Right? So so the improvement is sort of this this cycle. Right? Do we do that manually? Well, to some extent, you know, that's that's something, you know, something you can do and some some improvements do need to be made manually. But, the great thing about engine is engine is actually, sort of a big step forward to actually automating that improvement process. So what this is, again, is an agent that is actually reading the Langham traces, right, that I that I I showed. Right? So the records of what the agent is doing. It's also, hooked up to the GitHub repo that that it is running the agent. This is actually optional. You don't have to hook it up to the GitHub repo, but I definitely recommend it because I I think that's, you know, that's really how you get the most value out of engine. So it's able to see the the code that that the agent is running on. And they can see the prompts, the skills, you know, the tools given to the agent. All of that is is directly visible to to engine. And so it's taking that in and, you know, identifying issues that are coming up as as users use it and then suggesting prompt fixes, skill fixes, code fixes, you know, all of the, the the different solutions that might, you know, prevent the agent from making the errors that you're seeing, in production. It's... It proposes those and then, is able to create draft PRs against your... Against that GitHub repo. So saying, hey. Here's what's wrong. Here's what I think the fix should be, and and basically showing that to your your engineers. So if we take a look at what, this actually looks like back in, in Lang Smith. So we can go to engine and we'll create a new project. We'll say, okay. We want to, improve the, DeepLifeSci. We'll take a look at GitHub and we'll say deep, Deep Lifesci. So we're pointing that at the GitHub repo. If we want, we can point it like a specific folder, you know, if you're if you're working off a, a mono repo here or a specific branch if if, you know, you're deploying from a specific branch. And then we can decide whether or not we want to, open a draft PR for every fix. You know, some teams like to do that. Some teams don't. You know, some of them don't like, having an agent pushing directly to their GitHub. They wanna give the human choice, you know, others don't. So that's an option. And then you can also, select, if you want, what things you're you're mainly trying to, trying to, address. Right? So let's say, you know, the agent is running slowly or it's it's unreliable. Let's say it's cost, you know, cost a lot. You can select that. And then we have different analysis analysis levels that are available. Right? So, how deeply you want, engine to look, to to to look, how many traces you want it to pull in. Because if you're doing this at at large scale, it's... Engine is only going to select a a subset of the traces, generally, you know, in order to, you know, stop costs from exploding just because there's, you know, there there there may be thousands or, you know, for some of our customers, like millions of traces that, for it to analyze, it only wants to take a a a sample of that. Right? So we can decide how much we wanna sample. And then we could start analyzing if we want. I'm actually going to, instead show the, one we already have running because it takes a little bit of time. You know, once you once you you set a engine up, it takes about twenty minutes for the first, the first, you know, sort of issues to pop up because, obviously, it needs a little bit of time to, to, analyze those traces. So, if you'll recall earlier when we were playing around with the agent, this was actually a deliberately broken version of the agent, that has a, a broken web search tool. And so, basically, what's happening is is the agent is passing, or excuse me, the code is passing, anthropic specific parameters to the OpenAI web search, spec. And so, you know, that that obviously breaks the web search. And engine is able... Been able to identify this and is then saying, okay. You know, here's the issue. Here's where it's occurring over time. Right? We're seeing it on on these days and these traces. And it's actually citing evidence. So it's saying, okay. You know, here are the, here are, eight traces where we found that. We can, you know, see all of those. And then we can even click into them. So, you know, again, we get that trace view. We can see, you know, where that issue is occurring, actually visualize it. And then, and then what engine does, it says, okay. You know, I found this issue. I've I've identified why it's happening, and I'm gonna propose a fix. And so we have the, the the actual diffs that it's proposing. So it's adding, this code. This is this is the test, you know, unit test and then it's actually, you know, proposing this change to the, the the code that defines the the web search model, in order to to address this. Now, once you have that, you can actually, raise it as a PR. Right? So, like, let's say we think this looks good. We can go open PR and then that will open a PR directly against the, the Git repo. Right? And so we can see it says what, why, how, and then the, the the, you know, the files that are changed. Right? All the, you know, the standard, git PR view. And then we can have our engineers review that, you know, see if we like the the changing... Changes that, engine is proposing. And then if we want, we can, we can merge that in. Now, lastly, you know, once you once you've actually, you know, raised that PR, merged it in, then, you wanna see is that... Did that actually fix the problem. Right? I mean, we don't wanna just take engine at its word that this this solution fixes the problem. In a lot of cases, you know, agents are very nondeterministic, and so, you know, a a prompt change might not actually address the behavior. So that's why engine actually proposes, examples that we can then test the agent against. So if we look at this, we can see, you know, these are all questions, that the agent failed on. And, you know, here was the input. This was the wrong output that was given. And then engine actually writes, a new spec to be given to the grader, like, this is what, this is what we should see. Right? And so that we can actually add those to a dataset, and then that becomes a new test that we can give to the the agent. Right? So, like, you know, earlier, I showed, that again kind of like standardized tests. For the agents, we can actually, create a new one just based on these examples and then run that against the new version of the agent to ensure that, that that fix that engine has proposed actually, actually address the issue. So, anyway, that's that's LangSmith. That's engine. Really appreciate you, you know, learning about this. Like I said, I think, I think this is all just amazingly cool. I think engine is just... Is is amazing. And, if you are interested in checking out, you know, some of the stuff that I've shown here, definitely encourage you to do so. So, DeepLifesize available right here. Again, it's, it's free and open source, so you can take it, you can pick it up, play with it. You know, if you actually, you know, if you see it as something that you, could wanna put in put in production at your company, absolutely, you know, talk to me. Again, complete... It's completely free, and I'm happy to help, happy to help you with that, modify it in any way you want. Deep agents, you know, if you wanna build an agent for something slightly different, that's that's available right here. Again, completely free and open source. You can do whatever you want with it. And then I have a little surprise. We're giving out $50 in free LangSmith credits to everyone, everyone here. So if you scan that, you can actually, submit, submit a form to to get, free credits. Definitely encourage you, to try that out. You know, take advantage of all those features in LangSmith that I showed. So, you know, really to kind of... You know, once you once you build your agents with deep agents, take them to the next level. And, you know, lastly, I, as you may be able to tell, I'm a huge nerd about this stuff. I I, I love Agenca AI, love building agents, also love pharma and bio, love to geek out about it. So if you're... If you wanna chat about it, you know, whether that's, again, about, AI and pharma or if you're interested in, in Langchain and LangSmith, and, you know, you want some help getting started with it, always happy to talk. So, you know, connect with me on LinkedIn and and, you know, let's let's let's chat. So, yeah, thanks thanks, everyone. And with that, I will take, some questions here. Let's look at the q and a. Let's see. Deep dive on context management. Oh, okay. Here. Current limitations on the capabilities of deep... DeepLifeSci. Yeah. So so guardrails, you can... Hitting guardrails can definitely be an issue with certain models. I, the the the reason that this is is set up to use GPT 5.6 terra instead of, instead of some of the the top line, you know, Sol or, or Astro models, is that, you know, those often hit guardrails. So, this this is kind of a a thorny issue that I think we're all kind of, you know, working our way through as as biologists and life scientists. You know, we want... We wanna have the latest powerful models, but, obviously, there's there are safety considerations. People... You know, there's risk of people misusing them. So, yes, absolutely. This is, you know, you can use any model you want with this, but if you'd use some of the the more, top line ones out of OpenAI or Anthropic, there is the risk of of hitting, hitting issues with guardrails. Let's see. Next question. How do you evaluate traces for, this complex workflow? Yeah. So, you know, the... There's a couple of different ways you can approach this. If you're for... When you're testing the agent, right, then in that case, you can you can... You know, you're you're you're writing the the test yourself in many cases, and, and and writing that rubric. Right? So, like, all those those, questions that I showed and then the, the rubric that they're being compared against, you know, I I wrote those, manually with a little bit of, research assistance. Right? So I'm I'm using, you know, other other models to kind of search the the the literature, and, and and find ground truth. Right? You you you... When we write these kind of evals, we need to make sure that the that the the the truth that the agent is being compared against actually is the truth. Right? This is actually... In some ways, I'd say this is the hardest part of writing evals and and it remains somewhat an unsolved problem, for, like, how to generate true ground truth. But, you know, generally, as of right now, that's a a mostly, manual process. And then how do you evaluate the traces? You know, generally, if I see an agent like this, give... Giving the wrong answer, first of all, I'll just take a manual look at the traces, and oftentimes something will pop up like there will be a bug or, you know, some obvious issue with, with its reasoning that you can then fix with a a prompt change. But then once you get it to scale, that's where something like engine comes in because, you know, you can do that with two or three, you know, or four, traces where it's it's made an error. You can't do that with hundreds, right, for an agent in production. That's why engine is so great because, it's able to actually, you know, comb through those traces at scale and, you know, and identify issues. See. I think we have some time for a couple more questions. Are there tool calls to, CDISC, biomedical concepts? Refresh my memory. Let me... See the ISC. Oh, okay. Yeah. The, clinical clinical data. Have not... Haven't integrated this with, other clinical data platforms yet. You know, the, the the data integrations are something I'm still actively working on. So there's a lot of a lot of those to still add. Definitely, I will make a note of that and and, and look into to adding that. I think, you know, some of the top priorities right now are like, archive and bio archive. It can technically reach those using, using web search, but it it, it's always better to have a a, a purpose built integration like for PubMed or or, or clinical trials. So that's that's something that I'm I'm working on. How do you optimize for, cost and complexity? Do you run with the expensive model and let the smaller model train on or, is the harness tuned for the expensive run? Yeah. Great question. So, you know, we don't necessarily train models in house. I mean, you know, that's generally something that's left to the the big model providers. We do, actually do some fine tuning. So we we have something called SmithTune. I definitely... You know, if you're looking to fine tune models yourself, I definitely encourage you to look into that. But, you know, there's there's a couple of different schools of thought on this. I think, generally, the generally, the approach that you wanna take, when you're building an agent is to start with a more expensive or capable model just as a proof of concept. Right? So you can see, okay, this is what the agent is, like, maximally, you know, at at using the best models, this is what agents are capable of. Right? Like, if this... If the smartest model can't do it, then maybe it's it's beyond the capabilities of current agents. That's... That is a very shrinking... That's a shrinking, number of things these days. I mean, the models have gotten wicked good, but, but, you know, there's still some cases that are are beyond the capabilities of current LLMs. So start with the smart model, see if it can do the task. If it can, then, you know, try switching to the cheaper one and see, you know, is is it still holding up just as well? If if the smart model can do it and the the the less intelligent model can't, then that's a case for, you know, adding more, you know, adding changes to the prompt, skills, things like that, because that... You know, that way, you know an LLM can do it. It's just the the cheap LLM can't or can't do it without some assistance since we have to kinda give it that assistance. See, I think we have time for a couple more questions here. Does this agent have connection to, open FDA for review letters and drug programs? Also, very good question. Again, as of right now, it doesn't have a dedicated connector. It can, find that, I believe, through a web search. I I... I'd have to go back and and confirm that. But, so in theory, it can. But I will... I'll I'll definitely look into adding that as as one of the, dedicated connectors because, again, those are a lot faster. Is there a way to enable some users to do a simplified Lang Smith testing in the customer facing chat itself, if end users, for example, want to evaluate differences in models without having access to, to the full LangSmith. So so you can actually get a, sign up for a LangSmith account free. It's, you you you get a, honestly, a surprising amount of usage. But before I I even worked at Langchain, I actually, was, you know, I was building some agents on the side and, and, you know, set set up a personal LangSmith account. I was pretty impressed with the amount of usage I was able to get, get out of it free. So if you want to, you know, to test, LangSmith, I definitely encourage you to, again, just sign up for the free account, try it out. It actually, I believe it it does come with sandbox usage as well. So you can even run something like deep, you know, DeepLifeSci with a sandbox, and and get, you know, a decent amount of usage out of that free. Alright. Maybe, one more question. Are there similar frameworks for other science domains? Also great question. I I think, I haven't looked as extensively into, things like physics, chemistry. I actually do, you know, would like this to be something that that can also do, do a decent amount of work in chemistry because chemistry and and and biology obviously have a lot of, overlap and kind of blend together. Yeah. More more specifically, things like, you know, physics, even math, things like that. I am not as familiar with the current state of things. I'm sure there are. Yeah. I'd I'd... I definitely look for them. I'm sure they're out there. Cool. I think that, that just... Oh, actually, here. We'll we'll we'll take we'll take one more. With the underlying data changing, month to month, is there a recommended method for creating evaluation datasets, limiting range, find deterministic answers? Yeah. I mean, that that is... That's definitely an issue that a lot of agent builders hit. You know, ground truth is a... An ephemeral thing if if the the... If your data is always shifting. Right? Okay. You know, we say, how many papers are there on, you know, this specific drug this year? Well, that's, you know, that's changing over time. Right? There's a couple of different ways to do that. You know, one is, getting the ground truth dynamically. So you can actually, have... Instead of a fixed answer, you can actually set up a, a program, right, to to generate that that ground truth at, at test time. Right? So that is actually, being calculated and then... Or or, you know, found dynamically and then fed to the the agent that's that's giving the grade. So, you know, that's that's one way of doing it. Another way, you know, is is just to try to avoid, you know, avoid questions that are, prone to, you know, prone to having the the objective answer, you know, shift quickly. I I think, you know, that... That's a good place to start, but you definitely are going to want to, especially any kind of agent that's, like, working with your your business data or something like that where that's just something that's shifting all the time. Then you wanna get into dynamically actually generating that ground truth instead of just having it, like, predefined, in one place in a file. Alright. I think that's it. You know, thank thank you everyone again for coming. Really appreciate it. Like I said, absolutely reach out to me, if you're if you're interested in chatting, about AI, about pharma, about LangSmith. You know, I love to connect with people, love to talk. So, definitely hope you'll check out DeepLifeSci, check out LangSmith, and and, build some cool stuff. So thanks.