Name: Build More with LangSmith: AMA
Uploaded: 2026-08-26T15:00:44.609Z
Duration: 37 min 16 s
Description: Build More with LangSmith: AMA

Transcript for "Build More with LangSmith: AMA": Agents are a new kind of software. Endless inputs and non deterministic outputs. Build them the old way, and they break. Enter Lingsmith, the agent engineering It's model agnostic, cloud agnostic, and framework agnostic. Organization is shipping the best agents iterate with a system, and we call that system the agent development life cycle. Build, test, deploy, and monitor. You build with our open source frameworks. Deep agents, langchain, or langgraph. LangSmith Fleet enables anyone to build agents without code. Test agents using evals and experiments. Deploy changes in one click with LangSmith Deployment. Monitor every interaction from a single dashboard. Governance is built into every stage of the agent development cycle with Lanesmith LLM Gateway. Lanesmith Engine helps you improve agents autonomously and move through the development lifecycle more quickly. Blanksmith Engine is your agent for agent engineering. Hi, everyone. Welcome. Happy Wednesday. We are here to do another ask me anything session. And today, we have Jake joining us. We're gonna talk about tuned evaluators, which we're really excited about. So we'll start with some quick intros. We only have thirty minutes, so we're gonna get to the good stuff. If we haven't met yet, I'm Lauren. I am an account manager here at Langchain working with customers using LangSmith to get the most out of their subscription with us, and joined by Jake today. You wanna intro yourself? Of course. Hi, guys. Nice to meet everybody. My name's Jake. I'm on the applied research team here at Langchain, also known as Langchain Labs, where we're focusing a lot on the applications of, post training and and, some some more sort of applied research based, methodologies and techniques that we are looking to bring into the way that we build agents at Langchain. And so really excited today to talk to you about, the first manifestation of that. And, yeah, really excited to dive in. Great to meet everybody. Great. So here's how this is gonna go. We'll do quick intros. Jake's gonna talk through tuned evaluators, what they are, why we built them. We'll do a demo. And then at the end, we will leave plenty of time for a q and a. So here's the how it works. As questions come up, drop them in the q and a section, which is over on the right side of your screen. We will be recording this session, and we'll distribute it after the call. We also want your feedback. What are topics you'd like to see on future sessions? Where can we go deeper? Where can we best help you all? So, yeah, without further ado, let's let's dig in. Awesome. Thank you so much, Lauren. So really excited to talk about, LangSmith tuned evaluators this morning. This is something that the labs team at Langshane has been working on now, well, experimentally, I suppose, for the past, for the past two months, but something that as of last Tuesday is now available in the LangSmith platform. And the, the the reason or or or, I I suppose, why, we decided to build tuned evaluators is that a big part of the, the mission of Langchain Labs and and the research that we're doing is understanding how we can, mine production traces for really useful signal for your agents. And so sort of what I mean by that is, in reality, and I think it's no surprise to anybody, the the amount of agents being built has absolutely skyrocketed in the past, particularly in the past six months, but definitely in the past couple of years. And the result of that is that the traces or the exhaust or that, you know, the trajectories that these agents are emitting have also, you know, exponentially increased in number. And if you're an agent engineer and you're building good agents, then it's your job to, read between the lines to understand and and extract useful signal from all of the traces that your agents are producing. It's the first step that you would take in improving your agents is is sort of understanding where your agent's going wrong, and then hopefully kind of putting in a variety of of steps and mechanisms to fix it. The problem with that or rather actually and and the way that you might do that today is by, is by evaluating your agent as it produces its traces. At LangSmith or rather at Langchain, we call this online evaluation, and, that's just a fancy way of saying applying different, evaluators that might be things like an LLM as judge on traces that your agents are producing, or it might be more deterministic like code based evaluators. But the most common we often see is is LMS judge. And so you might have, you know, you might have judges that look for your agent's hallucination, or you might have judges that look for how concise your agent was. You can imagine these are running now on all of the traces that your agents are producing. The problem is if you have an agent that is, particularly high volume, has a lot of users interacting with it, then that becomes very expensive, and and often very efficient using frontier models, at at at scale. You know, if you use an Opus or if you use a Sol, for example, to actually go and judge a lot of these traces that your agents are producing, you can imagine that's gonna get expensive very quickly. And so what it's often meant is that teams have had to choose between sampling the traces that their agents are producing to pass to these judges, which means obviously only taking a percentage of the traces that your agents are producing, or using a more lightweight model, where where you effectively sacrifice accuracy to be able to afford, and and and run an NLM as judge on all of those traces. So we noticed this we noticed this kind of coverage problem, we call it, at at particularly within the labs team, and we recognize that, actually, this is a a potentially perfect application for post training a specific evaluation or a specific evaluator for, different evaluation tasks that we can run on traces in this online evaluation method. And so the idea is that we wanted to design a turnkey evaluator for specific objectives that you can apply to your agent traces. Also as well, I I just might I I kind of wanna wanna come back and briefly mention as well. When teams are tasked with building online evaluators, they have to define things like the model that they're using, the prompts that they wanna use, and, a variety of other kind of configuration steps to go and set up that online evaluator. Now, you know, that's that's great because it offers you flexibility, but also as well, it's it can be quite cumbersome and a bit of a burden to manage and, and sort of, again, validate that that that's been done correctly because, obviously, it's gonna be something that your team's investing in. And, yeah, it it can be something that is that is quite expensive. And so what we noticed was that, for some teams, that that would that was also slightly a blocker as well beyond the fact that they have to, you know, sample for cost and also as well kind of make that accuracy trade off. The actual the actual sort of, management steps of setting up that evaluator as well just compounds and and adds to already what is, a a nontrivial task. And so we wanted to provide a turnkey evaluator for a specific objective. What this will do or rather, and and we'll kind of dive into the first one of these that we've built today. But the idea of these is that it's a fully, fully finished, fully managed, online evaluator that has a post train model in it that is, for a specific evaluation task. And so we manage it end to end. We've tested it. We've benchmarked it, and it's a great way to begin to extract signal or useful information from those production traces. And the way that that will effectively work, for those of you who are familiar with, with with applying online evaluators to tracing projects or, yeah, to to to tracing projects is that you'll go into a tracing project. That's where all of your agent traces are being generated. You'll identify, you'll identify the perceived or you'll identify the tuned evaluator that you're looking for and and sort of, like, have an idea of of where you wanna apply it. What this will then do is you'll in the same way that you add existing evaluators, you can go and turn it on. It will go and verify that the, tracing project has the correct structure for it because we manage this end to end, and, this will be right now, this is in a trial period or rather is in a is in a, a trial period for for some customers, but, for others, it's it's a paid product and part of, part of LangSmith's offering. We wanna make sure that it will actually, you know, validate and run on the traces that it's designed to run on. And so there's a validation mechanism and a validation step that makes sure that it's running on, or or rather it's yeah. It it has the correct structure to run. It'll attach to your tracing project, and then it'll begin to assign feedback to your traces that then that you can then use in your agent improvement loop. Okay. So, and finally, at a high level before we dive into a specific, or rather our first tuned evaluator, really, what's quite important is that all of this is managed end to end. We've we've gone through the, the rigor of benchmarking and testing different prompts of versioning and controlling the evaluator configuration. So that's like the way that it, the way, you know, the way that it, attaches feedback and the way that it, the way that the actual judge model functions and works, and we'll dive into some some sort of benchmarks that we've done there in a moment, and the infrastructure as well, to be able to offer this to you so that you can run it on all of your traces. So we use Fireworks for this. Fireworks are a, for those of you who who haven't used them, a really fantastic inference provider, that provide a variety of, kind of open models off the shelf, but also the ability to go and post train a variety of different open models, which is what we've done in this case, to offer you our first tuned evaluator, which is perceived error. So perceived error is a probably one of the most useful signals to understand whether or not your conversational agents, giving users a good experience. What it looks for is evidence in conversations that the agent has made an error. And by conversations, I mean threads or I mean, like, multi turn interactions. This is specifically designed for conversational agents where you have, you know, some assistant or some agent, and it's speaking with some user. Obviously, if you're the designer or you're the builder of an agent like that, then it's very important to you that that users are having a positive experience in those kinds of conversation conversational interactions. But, again, if if you've got, an agent that's running at really high volume, then in order to have strong coverage across those conversations, you need to be able to run a highly accurate, you know, economically affordable evaluator, and that's what perceived error is. So it looks for things like explicit evidence, you know, the use of correcting an agent or, them rejecting an action or the agent, you know, the agent might be looping or the interaction might drift. All of these subtle signs in a conversation that indicates whether or not there is, you know, a perceived error or there is a problem in that conversation. Did the agent make a mistake? And the results of our of of our post training are fantastic. We offer the perceived error evaluator per evaluator in this case, this is this is a a benchmark that we, that we developed, and and it's showing the cost per 100 evaluations, running over some running over our benchmark of, conversational threads or conversational interactions. And this is where our perceived error model and evaluator because, you know, to have an evaluator, you need both the prompt and then also as well the model. This is where it scores. So it's more accurate than all Frontier open and closed source models, and it's cheaper than all of them as well. So, really, what this kind of unlocks is the ability to have, above OPUS level, above sole level accuracy on judging whether or not there is perceived error in your conversations, for, again, a fraction of the cost of some of those larger, reasoning models. The reason why we think this is extremely important and if we just sort of, like, take a step back a second because I know that we've, you know, we've we've kind of, like, you know, gone into gone into some detail here. The reason why we think this is really important is that, again, as I mentioned at the beginning, if you're an agent engineer, your task is to first understand and and you're tasked with improving your agent. Then your task is to first understand where your agent is making mistakes. The next part is, okay, once I have understood where those mistakes are happening, I need to mark where those mistakes are happening, and I then need to do something useful with those traces or with that signal to then make sure that my agents my my agents' behavior improves over time. What's really what's really great here is that tuned evaluators, and in this case, perceived error, gives teams give teams the flexibility to be able to go and work into whatever their agent improvement loop looks like, based on the fact that they can just consume those traces or they can consume those threads that have those feedback scores on them or rather have been flagged for where there is a perceived error. And we'll do a demo in a moment, or rather I'll show you what that looks like on actual traces in a moment, and and, hopefully, it'll begin to sort of crystallize how you can then take this to to either pass maybe to a coding agent to sort of, like, do further analysis. Maybe you wanna create datasets and actually, you know, do some do some further benchmarking based on traces that have had perceived error. And then you can just kind of, like, fold this into your your broader agent improvement loop. And now you've now you've, you know, can confidently kind of improve where where your agent is making mistakes and also know that you've got great coverage over all of your conversations because you're running this this, you know, tuned purpose specific evaluator, to to look for perceived error. Awesome. So let me actually show you what that looks like in practice. I am on a tracing project in Lang Smith at the moment, and this is a conversational agent. It's called chat lang chain. For those of you who actually for those of you who haven't used it, I really encourage you to go to chat.langchain.com. It's a great place where, users can ask questions about, you know, lang chain, the the the frameworks that we have, maybe questions about Lang Smith. You can see that in my tracing project, right, if I wanna go and add an evaluator as I would here, I have the option to add this Langchain tuned evaluator up in the top corner. It's the first thing you see. And clicking into it, you'll see that I only need to set of of or, actually, the the idea is that a lot of this is out of the box, and so I just need to effectively attach it to a tracing project and click save. I don't need to design the prompt. I ideally don't need to change the sampling rate because, again, this is, this is extremely cost effective. And I ideally also as well don't need to set any filters. You'll see that we run this on threads that have more than two turns. And the idea here is that if I'm judging a conversation, then a simple input and an agent response, that doesn't constitute a conversation, it's quite hard to infer whether or not the user had a a a good experience or or they had some perceived error from that. So we we control that it runs on more than two threads. We've had this running, by the way, on conversations, and so you can see, by the way, for my tracing project for chat lang chain, we have threads, and so there are conversations. Now what I can do is I can go and filter for, my feedback key. It leaves a feedback key called perceived error tuned. And in this case, let's actually go and look for all the times where it has found a perceived error. So in this case, one. And let's go and look at let's just pull up this conversation, for example. We'll go to the last run-in the thread, and we'll see this feedback key attached. So we'll see one. One is yes. There is a perceived error. Zero is no. Kind of bullying yes, no, and then it'll give a reason as well. And so in this case, we can see that an error has been flagged, and a user has asked some question about sub agents. So the user's follow-up to sub agents clarifies that they wanted information specifically about sub agents, which the first answer didn't adequately address. And so in this case, the user's having to kind of, like, follow-up and clarify, based on based on the the response of the agent, and and therefore, you know, that clarification has obviously constituted a perceived error. And so this is useful because now I know that users are kind of having to kind of, like, ask twice. Perhaps the prompting or perhaps the way that my agent is built is is not kind of like answering users' questions at first at first ask. If I, like, if I come and inspect another one, this one has four turns now, so there's, but there's, you know, some this conversation has continued for a little bit longer. And in this case, the user's reporting an error when the agent has suggested some kind of code or formatting. And so in this case, the error occurs because the template contains JSON curly braces that need to be escaped, which the assistant did not mention in its prior answer. And so in this case, our user is asking our agent for, some kind of code. I you know, I don't we don't have the full context here, but it's quite clear that the agent provided an answer that actually led to the user, you know, receiving an error. And so this is ideally something that as the designers of chat lang chain, as the designers of this kind of system that produces a a a variety of different guidance on how to use our open source and also how to how to kind of, like, interface with our different products, We we we wanna make sure that it's not providing users with with code that will error, and so this is another great way of seeing where there was, something that we might need to address. So that's how it works. It's actually really not that complicated. I just it it's extremely it's extremely straightforward. I just go and add a new evaluator. I effectively attach it to my tracing project that has threads, and I save. And then you'll see that I'll have I'll begin to get all of this really rich feedback, really rich signal that's being attached to my traces, and I can then use this in further analysis. You know, I I can set up a variety of automations. I can, again, point I can use the lengths with MCP or CLI to pass to a coding agent and then sort of, like, ask it to understand what's happening. I can I can consume it throughout a variety of other different parts of the LangSmith platform? So that is perceived error. Low cost, highly accurate, signal being attached to your traces that you can then use to go and enrich and make sure that your agent is giving users the best experience. That concludes the demo. We'd love to we'd love to answer any questions if there are any. I haven't been able to see the chat, unfortunately, but I know that we also have, I I know that we also have, I think we have another kind of ten minutes or so. So happy to kind of answer any questions or, or, yeah, kind of take this take this wherever. Great. I just had a customer maybe last week asking me for evaluators. Can you guys just choose the model for me? Like, I don't wanna have to worry about that. So excited to go back to them with with tuned evaluators. Nice. We we have a lot of different customers with a lot of different types of agents, customer support, research, coding. How well does this, transfer across domains? Like, will a. support agent and a coding agent get the same quality of judgment? Yeah. Great question. So I think, I think, the the the short answer is it maps really well across domains. And the reason why is the signals that we trained our model on, the perceived error model on, are, are effectively domain agnostic in that they aren't verify like, a perceived error is not, is not objectively judging whether or not a domain specific answer is true. So in this case, you know, in in the case of sort of, like, maybe coding or, or, something maybe, like, more specific. Maybe there's some sort of, like, niche financial service workflow. The the perceived error model isn't judging the agent's response. It's it's picking up on signals from the user's response or from the user's interaction with the agent. And so that that might be things like the user being frustrated in the language that they use. Right? So that's obviously gonna map very well across a variety of different domains, or, the user having to clarify what they meant because the agent didn't understand it the first time, or the agent looping. You can you can hopefully, you can hopefully sort of believe me when I say that these we noticed, when when sort of, like, developing the model that the, the flagging of CPIR maps really well across unseen domains because those those different sort of, you know, aspects of the conversation that I just mentioned are, are sort of, like, ubiquitous to all different types of domains. And so that was our intention. When when offering a tuned evaluator again, ideally, we sort of, like, take as much of worrying about domain spec domain, specificity out of out of sort of, you know, the the kind of, like, problem thinking sphere of of of the customer and and just give them something that allows them to to actually look for, like, user based signals that map across across domains. And so, yeah, the totally great. at at a variety of different domains, and hopefully should be really useful, regardless of of where you're building in. Great. Had a question come in, which I think is a really good one. With companies that have, like, internal specific data that LLMs wouldn't know, can we tune, specifically to app data, or how would it know when it's a true hallucination in that case? Yeah. So so great question. So in this case, we aren't looking for hallucinations. And and if we were looking for hallucinations, then we absolutely would need to have some way to ground that in what is the source of truth for each customer, or rather for each user. And that that's actually why, by the way, if you've built LangSmith evaluators before, and online evaluators, you'll see that there are, well, there's this LangSmith tuned one now, but then there's also the ability to add custom evaluators. And so without because basically what we've done is the the prompt, heavily influences the model that we use. And because we're looking at perceived error in this case, the problem that we've designed, we think, is is what is is sort of best for mapping across different domains. If I wanted to sort of tune out and make it specific to my, to sort of, like, my particular use case, then we recommend that you actually go and build one that is custom that that might be able to or rather that might reference sort of domain specific language, or or sort of be more grounded when looking for things like hallucination. But in in this case as well, as as I just mentioned, because we believe that this maps really well across domains, particularly in kinda, like, the user agent interaction, you know, set of signals, then, then then, yeah, there there there would be no way to there would be no way to change that. We are as well kind of, like, looking at, at at adding more tuned evaluators to Langhtsmith and sort of, part of this is understanding is understanding what would be really important to to to customers, and if we think it makes sense to sort of, like, map that across, or or rather to offer that in a way that that could be useful for a variety of different people as we have with perceived error. Great question. Yeah. Another part of the question, was how has the fine tuning process been? How long does that usually take? I'm not sure if that's asking, like, how long does it take to set up and for the eval to run. Yeah. Yeah. Yeah. Does. that Yeah. Yeah. Yeah. No. Great question. So I'm gonna assume, that that it's it's around sort of, like, the data gathering process, around everything that everything that needs to be everything that needs to happen to go from, to go from sort of, traces, I suppose, to to a fine tune model. What's really great, and I hope that sort of this this is sort of starting to kick the gears into place forever on this call, is that your traces in LangSmith are a fantastic way to build up the datasets that you need to go and tune a model like we've done for perceived error. And so the the short answer to the question is it probably took us, around around sort of from the beginning of doing research to to having something released, it probably took us, like, a month a month and a a month and a week. But, really, it's actually it can take us it can take as short as an afternoon if you already have the data that you know you wanna use for tuning this evaluator. The technique that we use to post train this model was SFT, so supervised fine tuning. And, basically, that says, hey. Here's a dataset. Here's an example of all of the here's an example of of of what good looks like. And in this case, this was like, we had a dataset of conversations, human agent conversations, and we did a and then we we went through this labeling process of of labeling conversations. Yes. This has a perceived error. No. This doesn't. And you effectively need that, and you need, an inference provider's training APIs. In this case, we use Fireworks, and it's just an iterative process. So you pick a model, and and you have a dataset you feed to it. You feed that to the to the APIs of of the inference provider, and and you sort of work through and continue to to kind of, like, iterate on that until you get to a point where you're happy with its performance. So, again, it took we were very thorough with it, so it probably took us, you know, a a month or so. But, again, this is something that is becoming, a a lot easier to do as the tooling for post training becomes, you know, a a lot more easy to access. Great. Maybe the last question. For perceived error, is there a good, like, a target rate that people should shoot for? Like, is there a benchmark for a normal amount of perceived error? Yeah. Great question. So we noticed, so this is gonna be very, you know, very heavily dependent on, on your agent building practices. And and so, to give to give the audience sort of a a benchmark idea, our chat lang chain agent, we're really happy with it. It it definitely could could be, you know, it definitely could be tweaked and improved, but but it's something that we've worked on over time. Our perceived error rate on our chat line chain evaluate error, I think, was around, I I I think was kinda, like, roughly around, like, eighteen eighteen to 19%. Now, obviously, that directly correlates with things like the user saying, hey. You know, the the code that you gave me produces an error, or, no. You know, I asked this, but you answered this. And and so I think anecdotally, if we all sort of, like, think about our experience with with kind of conversational agents today, they definitely still do have errors. I I can't think back to a time where, like, something has been so perfect in in my interaction with it. And so, like, maybe a good benchmark is is sort of, like, 19%. That's where we are. If if you're super proficient agent builders, maybe that number's a lot a lot lower. But I I think that sort of you can use this and sort of iteratively work against it, and and that's, again, one of the nice things about about this improvement loop. Great. There's more questions that I'm happy to, you know, sort of stay over by a minute or two. Does that work for you, Jake? Yeah. If Of course. can stay. There's some good questions here. Is there a dashboard that will show the issues identified during the evaluation process? Great great question. So there's a variety of different, there's a variety of different things that we can do to once we have these, once we have these perceived error, feedback keys. In LangSmith, there's something called insights. And so insights is a way to run a let's see. Actually, so I I I haven't run this recently. And so, I need to run this, but, basically, what I can then do is go and categorize all of the different I can go and categorize all of actually, here we go. So this this is from the previous run. So right now, an insights job will, take in a, optionally, a filtered set of traces and understand the reasonings behind why, you know, or or, like, effectively cluster those traces into different groups. And so what I've done is I've run an insights job, and I filtered it for I filtered it for when there is perceived error. And so you can see that 49 well, 40% of my perceived errors are coming from the agent misinterpreting what the user asked. And so in this case, I I I sort of did a breakdown here. And, actually, at a at a high level, your agent misclassified basic math requests, failure stem from intent routing, lack of fallback handling. And so this is just one way of dashboarding or showing, those perceived errors in something that we offer through LangSmith. You can see as well response quality was was another sort of another part of this where, over long docs or tool dumps, in this case, implementation misuse, it gives incorrect APIs or graph controls. And so the short answer is, out of the box, LangSmith has dashboarding. I can then go and dashboard on perceived errors. I can run an insights job. Or there's something called LangSmith engine, for those of you who haven't looked at it, that sort of offers more of this end to end, which I encourage you to go and look at in the docs. I won't dive into now just for the sake of time. So hopefully that answers that question. Someone asks, is the post trained evaluator model based or deterministic, and what types of of evaluations does it support? Model based or deterministic? I'm not sure I understand the question. So so the model will always run per evaluation, and the result of the evaluation that it produces, as I showed, is a Boolean one zero on whether or not it perceives the error and then a and then a string reasoning field. And so, we use a lot we we, yeah, we we used a a reasoning model that, that outputs that we've kind of instructed to output in that format. Hopefully, that answers the question. I I don't know if it did. Okay. One question. This this may require some follow-up offline, but, someone asked following up on the question. If you wanna fine tune on top of 500 gigs of company data and let a pool of agents run the tasks overnight, over days, Is there a connector to handle GPU compute directly and flipping the compute based on how heavy the agent task would be? Yeah. That's a that's a great question. So the Fireworks infrastructure that we used fortunately manages all of this for you. And so the the GPUs that are required in the training jobs that you run are, they horizontally scale with the amount of data that you're processing. Actually, as well, much in the same way that once you have its a tuned model, you can you can horizontally scale the GPUs that you use to support, you know, different workloads. And so and so all of this is sort of abstracted away from you when you use the fine tuning APIs of, Fireworks, for example. I've been asked the question in a in a yeah. I know that I know that was a relatively involved question, but Fireworks handles, all of that for you. Great. Awesome. Okay. Last I last question. I keep saying that, but we really will make this the last one. Everyone has. such questions. For built in perceived error analysis, is there a way to input more fields? For example, being able to input the user could build a profile on that specific user having more perceived error based on things like x user has been having a negative experience. So the way that it's built today, it it only looks at the conversation that's happened. Would be really interested to sort of understand maybe, we can chat about we can chat about this more offline. Would be really interested to sort of understand what you would like to see be provided or or be made available to to the evaluator as well. And and and if we haven't already, I'll follow-up with a link to the, to the to the announcement of of tuned evaluators, and there's a field at the bottom of it that asks, if if something that you know, if if there is something that is really interesting to you that you'd like to build or that you'd like us to come back looking to building out or you have questions around, then then please fill out this form and and, actually, it it'll kind of feed directly to me. But in short, the an in short, the perceived error evaluated today only looks at the user conversations from the thread that it's evaluating, so no external sort of stimuli. Great. With that, we'll wrap it up. I know we're over. As a reminder, everyone, we're bringing interrupts to London and NYC. I think Jake is MC for both. I'm gonna see both of these. Yeah. So. I'll be. If you wanna meet Jake in person, come to New York and London. New York on the September 24. London, October 13, we'd love to meet you in person. As Jake mentioned, tune evaluators are live in your LangSmith account if you have it to test it out. If you wanna talk about it more, reach out to your account manager or account executive. Send us a note on our website. Sales will get in touch with you. We're happy to talk about it more. We want your feedback, as Jake mentioned. We'll send out a recording of this. And if we didn't get to your question, we will follow-up as well. So thanks for for coming and for the engagement, Awesome. and thank you, Jake. This is exciting. If it was a pleasure to do it, have a great day, everybody. Bye. Thanks, everyone.