Name: Building a Harness with Jev
Uploaded: 2026-09-22T19:15:55.440Z
Duration: 48 min 4 s
Description: Building a Harness with Jev

Transcript for "Building a Harness with Jev": Agents are a new kind of software. Endless inputs and non deterministic outputs. Build them the old way, and they break. Enter Lingsmith, the agent engineering It's model agnostic, cloud agnostic, and framework agnostic. Organization is shipping the best agents iterate with a system, and we call that system the agent development life cycle. Build, test, deploy, and monitor. You build with our open source frameworks. Deep agents, langchain, or langgraph. LangSmith Fleet enables anyone to build agents without code. Test agents using evals and experiments. Deploy changes in one click with LangSmith Deployment. Monitor every interaction from a single dashboard. Governance is built into every stage of the agent development cycle with Lanesmith LLM Gateway. Lanesmith Engine helps you improve agents autonomously and move through the development lifecycle more quickly. Blanksmith Engine is your agent for agent engineering. Folks, welcome. We are super excited to get started with this, building a harness with JEV, lane chain and type safe webinar. Yeah. So, first, I'm gonna introduce, our two guests today, Ali from Type Safe and Hunter from Langchain. Do you guys wanna introduce yourself? Sure. Hi. I am Ali. I am, I am developer relations at TypeSafe. You've seen my name all around our Discord. If you've been in there, you'll be seeing a lot more of me and more demo videos and generally just out here teaching people how to, how to use TypeSafe, how to think in TypeSafe, how to think about System One models in general. And yeah, super excited to be here. Yeah. And then I'm Hunter. I'm one of the technical leads on the open source team here at Langchain. Super excited about Jeff. I've been pretty much nonstop experimenting with it over the weekend. Yeah. Super exciting stuff, and I'm super glad to be here and talk to everybody. And, I mean, Sydney, the, Product Manager for our open source team at Lynching, and close partner with Hunter on all of our recent JEV work. Yeah, I think everybody's lost a little bit of sleep in the past, week experimenting and, yeah, learning all there is to learn about JEV, so we're super excited to chat more about it today. Ali, maybe you could start. Could you explain in simple terms what is JEV? Why is everyone so excited about it? That's gonna be the hardest question to answer this entire time is, like, how to explain Jev in simple terms. Like, so Jev is an entirely like different class of model from when we're talking about, you know, most of the time today when we're talking about models we're talking about LLMs. And Jev is different from that. Like fundamentally, Jev does not generate text. It is, it is not a, like, generative AI in that sense where it is to, like, create text or create images. Jev is Jev is about making, fast decisions from, like, specifically well scoped questions with, with type safe schemas, for each of them. So Jev fits into the overall picture of AI where there's we've seen over the past couple of years, a huge increase in using these very heavy LLMs to do what is ultimately like a tiny decision. Like, oh, which department should this support ticket be routed to? Or is this email a promotion or is it spam or is it whatever? Or is this tool call like a safe tool call? Like, does it or does it, do do perform any write operations? Stuff like that. And so Jev is a a new class of model that we call system one models and it fits into that kind of picture. And then the thing I always like to say, although this is a lang chain audience so probably don't need to say, but it does confuse some people like Jev is not an end user product out of the gate. It is an API product. It does not do anything until a programmer, whether a meet space programmer or an AI programmer, does it until it puts that into software. That's like the first question that comes up is like, oh, how do I just point open code at Jev? And it's like, well, it's a little more complicated than that. Like, you use it inside your software to make what I like to call smart software. Yeah. That that I think that was a great, simple explanation of, Jeb. You mentioned it's called a system one model. What does what does that mean? So it comes from and I always forget the author's name, but there's a a book called, Thinking Fast and Slow. And, and the author of that came up with this way of labeling, like, kind of two different ways of thinking. And system one thinking is the like it's the fast, intuition side of thinking. It is it is quick determinations, where system two is like, multi step reasoning thinking. Right? You have to think about this and put it together with that and connect the dots and then you have output. And so system one model like Jeff is a fast intuition based model. It is about I like to I like to describe it sometimes to be like the questions you should ask Jeff, you should be able to present it to a panel of like super smart humans and they could answer in like five seconds. Right? It's like it it's not a you have to dig through and make multi step reasoning like arguments to understand the output. It is like, it is very very smart but the system one thinking type of questions is a different type, like it is a it is a subset of all of the problems you might have in the world. Yeah. Yeah. That's a good explanation. Would you say that, like, traditional LLMs kind of fall into that, like, system two thinking? a it's a good question. I think I think before reasoning models, I think LLMs were much more, like, system one thinking in general. Right? They would just kinda, like, answer quickly and but it it was very intuitive and oftentimes, like, not trustworthy answer back in those days. Now with reasoning, I would call that, like, very much oftentimes that is system two thinking. Right? When you send off a request to, like, a big Frontier LLM, right, it's going and it's like looking in especially if it's in an agentic loop. It's like looking things up but is making connections in order to, like, to to to find that. But what I what I also like to think of is, like, the system two oftentimes that is integrated with our system one is your code. Like code, a lot of code is the system two thinking. It is the putting pieces together, connecting them together, composing a bunch of system one decisions into a sort of reason to final output. Yeah. That that makes sense. You mentioned you've mentioned a couple times, like, Jev is really helpful when you, like, embed it in code. Can you say a little bit more about, like, what kinds of decisions, Jev is good at making in code and kind of why it's this, like, code first model? Yeah. So I'll start with the the end of that where you talked about, like, why is it a code first model? And a a lot of that just comes from us seeing what is needed in the world. Right? We see the future and I think this is probably not even, like, I I bet it's not controversial to say that, like, the vast majority of automation in the future will be machine to machine automation. Right? Is machine to machine communication already a huge amount of the world's infrastructure is machine to machine communication. Like API's talking all over the place, a whole bunch of things working together. Right? So that's how like most of our digital world works is code talking to other code. And so, Jev, like the system one model concept and Jev is about fitting into that picture. It is like we already need our computer to talk to another computer, So when we wanna bring the incredible, like, power of AI, the the intelligence that we want to, like, access from these models, why isn't why why isn't it, Why shouldn't it be a machine native format to begin with? And that is like the kind of foundational thinking behind, like, why JEV is shaped the way it is. Like and to expand on it a little bit, like, I've, for a while, I've liked I like developing with LLMs. I like putting LLMs into into these, like, decision points. Before I started working at Type Safe and, like, saw this, I was, like, using it for a lot of those. But it always there was always like an itch in the back of my head that like, I thought, isn't this absurd? Like, isn't this completely zany that we are we we we have this big huge heavy machine, this LLM machine that is doing a huge amount of computation to output human language, to output, like, text. And then we use that to, like, do tool calls or invoke CLIs, for example, which themselves a CLI is a human interface. Right? It was designed to allow a human to interact with a computer system in in like a way that's like semantic and that we can like memorize. Right? But ultimately it turns it back into just like raw machine code and goes off and does its thing. So like, we've got this weird bridge in the middle where like to get a computer to use a computer, we're having it like enter like human text generation communication space, which is just so so so so inefficient. Like why? Why are we like it just feels like standing up in a hammock award is a phrase that I don't know if anyone else uses in the whole world, but my, my spouse's professor used that one time and I really like that, which is like, it's the most complicated way of trying to solve a solution. Like, why are we doing it this way? And so Jeff just feels like coming back down into like sanity, coming back into reality of like, if it's machine to machine, there should be no there should be no pros generating model, like, in the middle of that, unless you need pros generated for your output. You know, that's fine. That's still a huge place for LLMs to sit in all of this, but there's a lot of places where it ought not to be. Yeah. Yeah. I feel like, I think Hunter shared this analogy recently. It kinda feels like we, you know, LLMs became popularized, and we were like, oh my gosh. Like, this is a hammer tool. Like, everything looks like a nail. Right? Like, let's just use this for everything. It can do a good enough job, but often they don't, like, fit super well in, in code. And so we kind of made these, like, tack on bridges, like tool calling and structured outputs to be able to support LLMs in code driven applications, specifically the structured outputs. I think we're pretty transformative in the early LLM days. Like, okay, it produced text and then also something adhering to a JSON schema. But, again, way, overinvesting in the, like, computation. And and that shows both in the, like, cost and latency that we see from OLLM responses for a lot of, I think, pretty simple decisions. So one other question that I had. Jev has kind of three types of, like, decisions that it can make or questions. Maybe you could talk through those. I think I have a slide here. Yeah. Could you talk through kind of these? Yeah. So this is like this is the actual manifestation of, like, that machine to machine communication. Right? The the the the type safe, question. So when you, when you do a call to Jeff, you are giving it like a state as input. That's what you got up there, the high I've been trying to connect my Stripe account for three days. Right? So the state is like the thing that you are evaluating. And then you give it an array of questions. You can give it one, you can give it a 100 of them, but each of the questions is one of three types. So we've got choice here which is to pick one option from a list. So in this example it's like, oh, which team should handle this? Billing, technical, or sales. Right? So it's like it's it works best when there's, like, going to be one dominant answer and the choices don't have a huge amount of overlap. But when I say works best, there's always, like, a gazillion different, like, nuances and ways to talk about this. And we, like, even our docs only scratch the surface and we're like adding to it as we learn from the community. But so a choice is to pick one option from a list. It's like it's the closest analog to like what people would call a classifier. Right? It's just like which of these is this thing? The and I would say choice is like probably the the the question type I use the most, like maybe second to Newell. Score is about rating the state along an axis. So you're you're gonna get back a value from like zero to n. You define how many levels there are. Let's just pretend we were doing zero to four. Okay? So or actually here we go. We got we got we got zero to two. Calm, frustrated, very angry in this in this example here. So you're saying how frustrated the customer appears. So score is all about rating the state on a single axis of, of determination. That's like the most important thing to know about it when using it and this actually goes for kind of all of our questions is you really need to break down the questions to the point where they can't be broken down anymore. Make sure you're not asking a compound question. Don't put multiple different like axes of criteria in a single score. You should do two scores if there are actually like two axes of information. So, the huge power from score comes from the way that you define the levels. So we don't see it here in this like quick overview but the calm, frustrated and very angry, each of those is defined by semantic, description of what those levels are. In this case, it might just say calm, frustrated, very angry but it'll always work best if you like give it you should be able to like clearly define what a one means in your scale. And again, it's like giving it to a panel of humans. Right? It should be defined well that a panel of humans could be like, oh, I one is very well defined, two is very well defined, And what that means is when you use that same question with the same criteria to evaluate a whole bunch of different states, maybe they're a bunch of documents, maybe they're, live chat messages, there's a whole bunch of different things. It could be log lines, output in an observability system. You can trust that if you got a score of like one or 1.5 or two across multiple different things that they mean the same thing. That it like because they meet the same well defined criteria. It makes it so much more powerful than just like a blind scale from like if you just did if you just did like level zero is like, you know, calm and level 10 is very angry with no descriptions in the middle. You'd be like, well, what it actually is a six? What is a seven? What is a four? Like, can you really be sure that that has a firm description? So that's score. And then the final one is Newell. Newell is is kind of the simplest. It really is just asking if the answer to a question is yes or if the statement is true. So you just ask like a simple straightforward thing like, the message conveys, urgency or time sensitivity. There's always a like a little bit of a yellow flag if you're putting an or in a newel because that's like, oh, is that actually asking about two different things? In this case, urgency and time sensitivity, I would argue, can probably like be the same thing. But so you're just asking a true, false, a yes, no, and you get the probability that the answer is yes or that the statement is true. Since everyone asks, Newell is popularly explained by, by everybody else at the company to be short for Bernoulli probability. But my personal head canon is that it means a non binary Boolean. It is because ultimately, it is like a Boolean answer, but it's not it's not a binary state. It is continuous from zero to one. Nice. I will yeah. I I just saw the Bernoulli thing before this, but I like your I like your take better. So at LaneChain, we think a lot about, you know, how can we help people build agents. And one of kind of our biggest foundational pillars that LaneChain is like, we wanna help people build agents with any model. Right? The model for that's right for one task is not gonna be right for another. And then also, you know, there are new models every week, and these new models are good at different things. So we were very excited to to see Jev come out, and help support people building agents with Jev. Maybe, Hunter, you could talk a little bit more. This is a a bit of a two pronged question, but how do we see Jev fitting into agents and kind of how can you build it into your agent furnace? I can let you screen share as well as if you want. Yeah. I got a little bit of a demo, cooked up, which I think I can I can use to demonstrate better in words? But, yeah, I I, you know, at Langchain, we've been talking a lot about, you know, what makes a good agent harness, how do we, you know, get, like, the most useful output out of a model. Usually, the way that we interface with that just because that's how, other people interface with models is, some kind of opinionated harness, the agent loop, that sort of thing. Like, we we develop a ton of opinions about how we want that best to look. When JEM first came out, we, like, initially thought of it as you know, there's a couple of super interesting problems that we can solve directly inside of, the the harness just like straight to do. There's the things that there's questions that we wanted answered, and not that great of a way to do it because if we threw LLMs to make decisions for us, at every step, then, you know, gets very expensive, very lengthy. There's just, like, a ton of ton of complications that come with it. So let me and pray for me. I know that live demos are a curse, but, we're gonna try it anyways. Let's see. So, Just oh, go ahead. yeah, dude, something I wanna say I can I was gonna start another demo? Yeah. But was just gonna say to add, like, a little bit of color, we think about the core kind of agent loop as, like, send a request to a model, an LLM, to be specific. And then the LLM can kind of reason, call tools, and do this, like, tool calling loop until it decides it's done with a given task. And as Ali mentioned, like, you can't just plug in Jev for, like, an LLM as a direct substitute. That's not the way that JEV is designed, both in terms of its functionality and API. Hunter is going to cover how we think about plugging it into the harness because it is still very useful in agentic applications. Yeah. For the agent harness specifically, there's, a couple of points. If you or anyone that's used, like, codex or cloud code or something of that variety, you might notice that, when it takes certain actions, it usually classifies it as risky. I think everybody's heard the, like, pattern case where, you know, an agent goes and deletes a production database, and then all of a sudden everything's on fire, and we have to blame the LLM for it. Sort of what Claude and Codex adopted, pretty early was to have some kind of classification step. This was, like, abstracted away behind their harness and and how they implemented it, to classify if a decision was risky or not. Now with JEV, just because it's so cheap and inexpensive, we can, you know, leverage just leverage a very similar paradigm just, in in the agent harness. So this, like, basic example I have here, and this is very link chain forward code if you guys are familiar. You can go look at our docs. I I think there's a lot of concepts here, you know, we could we can chat about separately too, but, yeah, just wanna so, like, to to go front to back, we have these two tools right here. You can see that we just are updating something in a CRM, or we're deleting a customer account. We can classify this as, like, a pretty destructive action. We usually want, like, some explicit intent before before this happens. We don't want an agent to just do this automatically. The way that we can add this into the agent, lang chain agents have this notion of middleware, which is basically just at different points in the loop, we can choose what to do and organize context, in in different ways. So with lang chain type safe, this is something that, if you go and install lang chain type safe, you can take this straight to boot. I specify auto mode middleware. This is saying, in cloud and codecs, this sort of behavior is called auto mode. We specify some tools, the two tools that we have up here. Specify some instructions saying, if this action is going to be dangerous or not. We set some of those criteria that that Ali mentioned, kinda one is the true and false of when we want these tools to be rejected or not. And then let me run this real quick. And then if I run this here, this is a pretty safe result. Customer yeah. C123, add the internal note. You know, pretty safe actions. We can go ahead and run it. Yep. Okay. Serum note added. This was fine. The status was success. If we take something like a destructive call, if we say, I want to delete this customer. I don't want confirmation. If I run this, then at this point, Jeff, right before the tool call gets made, is going to decide if it's risky or not according to the criteria that we have up here, up here. And then status error. Yep. It was classified as risky, so we didn't wanna execute the tool. We lost your screen share, by the way. oh, sorry about that. Just a moment ago. Cool. Yeah. Anyways, you could see that it errored out. It was classified as risky, so we didn't execute it. Another case that we like so, yeah, this is this is pretty exciting because before the way that we had to do this was make another LLM call, which was pretty expensive. It was, like, very latent, just because model had a reason over the tool call and yeah. It it yeah. The implementation was pretty nice, but whichever makes it a lot simpler. A second kinda case that we pretty immediately latched onto was this idea of model routing. So depending on the task, I know models are gonna be good for certain things. I don't wanna slam Astra on on every single task that's that's possible. So we could specify a fast model or a, more in-depth model. So very similar before, this is another middleware that you can export from. lost we lost your screen sharing again. Oh. Oh, interesting. I'm just being the the the the chat. No. puppet. I appreciate it. I appreciate it. Let's try this. This might be a little bit better. Does that work? Yeah. Yeah. We got it. Okay. Cool. Yeah. Specify the router in the same way. Fast and yeah. It specify the instructions for when we should use different models. And we're gonna try running this one more time. So I can run this, rewrite, like, super based construction. We're gonna use a a cheap frontier model for it. Yep. It selected the fast route, so you can see that it opted to use Luna for this specific case. Jeff made that decision, and it was super fast because of it. We can also do this one. This is probably gonna take a while because Sol's gonna go and do Sol things, but, I mean, it's like, yeah, we we can into it that, like, it's gonna take a while to execute. That means that, you know, Sol picked this path. And the other interesting thing, to come out of this is with this, we can, because we're using this middleware, we get tracing into the system out of the box. So we can see exactly what's happening. We can see, all the decisions it's making. I think this is one of the yeah. So this was the model router's middleware that we saw. So by looking, I can see that before the agent runs, this what is that's what this before agent thing is doing. We're running a a classifier call against Jeff and determining what the output is. So we pass in some details about the different, models you can that Jeff can choose from, pass passing some of the state with the initial query and things like that. And then in the output, we can see that it shows fast, which means that we opted to use, Luna for this case. And one interesting thing that Hunter is showcasing here is, like, you know, with OLMs, I think it's pretty common knowledge that you want kind of observability into what they're doing. Right? Like, OLMs are sort of a black box. You wanna be able to tell, you know, what were my inputs? What were my outputs? Can I score how you know, evaluate how my agentic applications are doing and, you know, change my configuration for my agent to make it perform better? And observability and evals are still, like, very important for, even the, like, decision model class that JEV is. And so LYNXsmith is is one great way to kind of visualize your state in, state out, and score, how that's doing for your your. use cases. I also I I really I really like you showing that observability, because one of the one of the, like, top tips that I give to people when they start building with Jev is to like no matter what system you're in, build a way to get auditability, of the actual type safe calls. Because the the most likely, especially if you're like vibe coding to start with a lot of people that's like how they're doing it. The, the the models out there are pretty good at like understanding how to think about, like, system one questions and kind of, like, compose them together. We have found that they are not typically very good at actually writing the instructions and the criteria. We think that'll get better with time as they become more trained on more projects that use like Jev and system one models. But, I I like to give two tips which is like make sure that you have a way that you can like inspect all of the calls that happen answers are like as you're debugging and figuring out like what's working. And then the second is something that I forgot. Oh, especially if you're vibe coding, tell your agent to like put all of the copy, all of the like instructions, criteria, anything that's being fed into those into like one centralized place or some place that you can easily go in and tweak it. Because that's gonna be if nothing else gets a human touch in your code base and you don't open up a single other single other, code file, you are going to want to look at that one. Because you're going to be able to get huge performance, like an accuracy increases by just taking some questions that sometimes are written very poorly and just making them more sensible. Yeah. Thank you. Thank you, Hunter, for the demo and, Ali, for the tips. I think, you know, this is just a very new type of model, and so everyone's gonna need to slowly learn how we can, like, best develop with it. One interesting resource that I stumbled upon was, this resource called, Jevify. It's just basically a skill. I'll paste the link in the chat. And this is, like, a prompt that you can give a coding agent, to help it look over and scan your agent code and figure out like, where am I using an LLM for decisions that actually would be much better with the decision model and how can we plug that in? How can we architect that you know, to save on latency and cost? And then also importantly, like, where is an LOM the right tool here? And, like, we need to leave that. So excited about that. I think it would be great if we could saw that specific project and and and love it. Like, we we wanted to if we had had like more time leading up to launch. We talked a lot about creating, like, a brainstorming skill, which basically is. exactly what this person has created. Like, because there's oftentimes, like, low hanging fruit, especially in, like, an LLM integrated piece of software. There's usually low hanging fruit for JEV to be, like, a swap in, and then sometimes less obvious things. And so the Jevify, the Jevify skill there is is pretty darn cool. Is. is there one more one one more thing I can show, because I made some promises, and this landed literally five minutes ago, inside of Linksmith. I just want to sorry to sorry to hijack real quick, but, this is the thing that I love about working about, working at start ups is that, you know, we could ship so quick. We have nice rendering now, so you can go in and actually interpret some of the things that are inside of your links with trace. So you saw before that it was, like, pretty ugly JSON output, but, now we get, nicer rendering for all the different, choice types in inside of links with. This, is awesome. love that. Yeah. Anyways, click of. working at Linksmith oh, sorry. Yeah. At LinkedIn on Linksmith. Awesome. Well, we're gonna save some time at the end for, community questions. But, Ally, we have a couple of questions for you first, both that. we've seen on on Twitter and just internally from langchain dev power users. So, one thing that we're curious about, I believe the context length, for dev is 32 ks tokens right now. Is that correct? It's basically it's like a it's slightly complicated. It's 32 k, it's 32 k for the state plus your largest question. So, like, if you imagine the state and the question together is 32 k. And then there's around a 64 k cap for the state plus all of the questions together. That's just when you would need to, like, separate it into into multiple calls. There. So our official answer to, multimodal, do we support images is not yet, with no further, expanding upon that, vague promise. And I would also give the same answer to things like increased context length. We also wanna go faster, cheaper, like this is this is the beginning. And if you read like the launch blog post on our website, like, written by Diogo, it really does explain like where we're at in this. We see this as like the beginning of this revolution that we that we want to help, you know, drive forward. And so, yes, improving in every angle is, is what we expect, but now I'm not not putting any specifics or promises on timing or any of that, but follow us because we're we're we're moving fast. Yeah. Yeah. That's great. I saw also Diogo put out some cool content, I think, yesterday about, like, how to think about JEV for coding agents, and, like, how it can plug into some of those pieces. It seems like interesting use cases that were highlighted, were along the dimensions of, like, progressive disclosure and tool selection, and then also, like, RLMs and code mode seemed like an interesting case. Are there any other, really compelling use cases that you've seen or, like, JEV in production, that's interesting, man. for folks to know about? It's like the it's so it's so vast, like, what people are doing and they're moving so fast. I wanna be like, let me just open my Twitter feed and like look at look at the top 10 posts. I bet all of them will blow me away at this point. Right? And that's the the most exciting thing about where we're at right now is, like, we knew we knew the community would take a lot of these things and run with it. And, and we are seeing, we are seeing that. I like, you already talked about, like, some of the the I think what I'd call low hanging fruit like, using it for like a safety classification of tool calls, model routing especially like upfront where you're like, oh, is this overall question like does it require this that the other thing? I I my personal, like, biggest excitement is the projects that I see that are in the real time, real time use cases. And real time was like a it was a big, like, moment for us internally because we we always called, like, we knew that we were very fast and we knew that we were very cheap. And then all of a sudden, it was like I had an epiphany when I was like, wait, when you're fast and cheap, that unlocks real time. Like, if you're too expensive, trying to use something in real time, it'll just become cost prohibitive. And if you're too slow, you just can't keep up with real time. But real time means that, like, real time chat filtering to me. I made a demo for this early on, but didn't we didn't end up using it for our launch, demos and now people have already made better versions of like like live filtering a massive streaming Twitch stream of, you know, tons and tons and tons of chat when you get these channels that have got, you know, like a 100,000 viewers. Right? You can filter those live and at only around a hundred, hundred and fifty millisecond delay in making it appear, it feels like real time. Like it doesn't even matter. There's like more latency than that just from like Twitch's like displaying it. Right? So you can you can you can because you can score them along an axis. You can even make it where the user can just like move like a scrubber of like how filtered do you want it. Do you want only the most like productive, like interesting questions being asked or do you want all the emoji spam? Right? And you can just like literally just like turn a dial. And like that's just like so cool to be able to do that. And in that parallel call, you can also classify them as questions or statements or hype messages like all at the same time, all in that hundred milliseconds for every single message, and it won't break the bank. So, yeah, I just there's so much. That's just, like, the first one that came to my head. I think one of the I I am glad you mentioned the real time element of it because, I think in, like, one of the launch posts, I saw the demo you guys did, where you got Jeff to play Doom real time. I that. was, like, one of those moments that was, like, this is this is incredible. Things are so cheap and fast now. It's gonna open up like a whole new paradigm. Yep. That. was my baby. The doom, demo has. been my favorite thing to to work on the last, the last couple months here. Nice. Well, yeah, that was it it blew me away. I think it blew a lot of people away. So, props to you. Props to the whole team, for for building it and yeah. Thank you. Yeah. Yeah. Agreed. I think, you know, Twitter has really done something else with Jev over the past six days or so. I I feel like I learn about a new use case, like, every five minutes. Are there any other interesting before we pivot to the, the community questions, any other, like, road map items that you wanna share? I know, like, you know, fast and cheap faster and cheaper is, like, obviously encompasses a lot of work, on your own. Not I would say not not specifically, you know. It's like it's like it's like a watch this space. You never know what next week will be, with, like, how much incredible demand there has been. Right, like, in this immediate moment, we are, like, working on increasing capacity to get more people in. At least at the start of this, our sign ups were still paused, as we, as we manage demand to make sure that we have, like, high quality service for everyone that's currently accessing JEV. That could have changed in the last forty minutes. I have no idea. I'm not watching Slack. But, but, yeah. So that's, like, the immediate focus, but but more more stuff more stuff is coming. One last question. Have you guys gotten any sleep in the past couple of days? Not a lot. Not a lot. It's been it has been it has been wild and crazy. And everyone is working like incredibly hard. And but we're also it's like it's also so many smiles. It's so much smiles here. Like, people are excited. The energy is just electric in the office. Amazing. Well, with that, let's let's pivot to a couple of the questions from folks on the webinar. The most uploaded one right now is, is JEV trained on the same data as a typical LOM? So the so it is it it it is Sorry. I'm like reading my, reading my notes on this one because I don't do I'm like not an ML person and I'm not involved in like the actual training and stuff. We do use a mixture of pre trained models to train ours. Our, our data that we use to train our model is all synthetic data created in house, and that's part of our, you know, part of our secret sauce. But, but it goes it goes far beyond, the process goes far beyond post training. And there's a lot of hand waviness in there because we do have we we we do have breakthroughs, which is part of why we're able to, like, create what we've created and for it to be this amazing. And so we're gonna hold on to some of that secret sauce for now. Nice. I'm kind of aggregating a theme that I'm seeing across some of these questions, which is like, how do you kind of tailor the either, like, confidence or, probabilities that you set in your code as, thresholds for, like, okay. If the confidence is above x, then we'll do this, or if the probability is above x. Is that, yeah, it seems pretty domain specific. How do you think about that? I would say that's exactly right is is that it is domain specific. My I've I think this is written, like, multiple places in the, like, tips in our documentation where it's, like, you should test your domain, like test your actual like data and use case to make sure that you under like first of all, make sure that it's like in distribution. Make sure that like Jeb is good for your use case. Don't blindly shove it into production without testing. But, but also like that's where you should be looking at like what the thresholds are, you know. If you're finding yourself setting a threshold like super low, like especially like below 50% for renewal because that almost certainly means that either there's a problem and like this is a, like an aspect of our model jaggedness where maybe we just like aren't good at that particular use case yet. Or it means that maybe your questions need some significant tweaking. But, so it is about it is about testing it and then I will also say that a big thing about using JEV effectively is to like confidence thresholds is only one way to use the probabilities. What I think is really important to know is that the confidence that's returned by our API, that is a computation, is like a statistics computation on the probability map that you get back. It is it is a deterministic computation that you could do yourself. That is not a silver bullet. It does not it represents in, in I think primary use cases, it's a good analog for confidence. And but that is the cases where you are expecting, like, a single dominant answer. It most the confidence is mostly a measure of how much is there, like, disagreement? How much is there a distributed spread in the probability map? And you could have use cases where you actually expect two or three of the possible options to be competing for top probability, and in which case, your confidence is gonna be, like, super, super, super low because it's that's the confidence in there being a single dominant answer. But in some of these use cases, that's actually normal and okay. And so I also find that coding agents, they really like to see that confidence and be like, yeah, we should always do a different thing if confidence is low. That's not always the case. Like I built a, like a a shopping cart. Actually, the doom demo is a great example. The doom demo says, which monster, you know, which where should we be looking each frame. Right? Like this monster, that monster, should we look at that item, etcetera, etcetera. None of those is like a, oh, yeah. That one's gonna be like a 100% and everything else is gonna be 1%. No. No. No. There's like actually a lot of like solid options. And so we just take whatever one is the top choice. You could even do some fancy things like adding a bit of like decay where you make the the most recently picked choice like stickier and so it requires more of a swing to change it. Like that's an interesting pattern I think to play with. But if we were to like naively and what some of the first like vibed up versions of the of the doom demo brains, used confidence gating. It was like the a the agent was like, oh, confidence is low on this, so I should just fall back to, like, looking forward or something. It's like, no. No. No. No. No. No. No. That's like a bad use of confidence because it just doesn't apply in the situation because we expect there to be competing answers. We just wanna pick the best one. So that's like just one of many potential examples of where you don't want to just blindly trust confidence to work in your particular use case. It's a good shortcut for a lot of simple use cases, but, like, always test, always check, and be creative. There could be other probability math. Sometimes you just want, like, the top probability raw percentage, some that you'd want a threshold on. Sometimes it might be the top probability minus the second highest or divided by the second highest probability, gives you a different thing. Like, all of these are different, like, statistics math approaches to how to deal with it. Yeah. That's super interesting. Anecdotally, I was working on, like, a pretty naive application of Jev, just using it to help me class, like, add labels to new GitHub issues so that we could kind of automate more of our triage use case. efficiently. And I was kind of perplexed by some of the, like, probabilities and confidences that I was getting back in some of the results and was like, oh, I I need to dig more into, like, how to properly understand this for this use case, and just, like, calibrate myself as the developer. Yeah. I think so maybe time for one more question. I think this one is interesting. It says, as far as I understand, JEV's, status input token limit is, yeah, around the 32 k that we talked about. Are there any decision degradation, consequences on large inputs? How should we think about that for JEV? That's a really good question. We have a, a doc page in our docs, called model jaggedness, and I highly recommend checking it out. It is our this is a part of our, like, one of our, like, cornerstones of our company culture and, like, our brand the our brand is, honesty. Like, we don't we're not we don't wanna say Jeff is better at anything that it actually is. Right? We we are not in overpromise, culture here. And so the Jaggedness article is, like, our list of, like, known ways that we know that the model doesn't do great in. And I believe because I've read so much things recently, so I could be wrong. But I almost I'm almost positive that one of the things in there is talking about long contexts, and that it can degrade, accuracy if you put in a huge amount of state. I would say the guideline for that is, is make sure you're only providing in the state what is actually necessary for making the determination. You don't wanna just like throw the whole world at it and then ask it to make a determination about a part that is like just a tiny slice of it. Now sometimes you don't have any other choice. Right? Sometimes you truly are evaluating a large unstructured like wall of text and you don't you can't trim that down in code. That's why you're trying to use an AI model to like to evaluate. In which case, you know, iterate on questions, figure out like what works best, test it. But in a lot of cases, we're we're constructing the state from multiple bits of data that we have in code. And so that comes down to the system designer, to you as the programmer, to like consider, is this part actually needed? Do you need to add that to the state? Do you need to add this to the state? It's also better, like more cost effective to like not like throw more tokens at something you don't need. Java is so cheap that most of the time you can just like throw extra at it and not care. But, you should you should be mindful about what you're including and definitely test that. That that's like a big thing and also in the doom demo, like the state, I played a lot with like how much detail to provide in the state and at one point I had like a recent actions array. This is like everything that happened recently but it was so granular. It was like every like bullet of damage and stuff that it would it ate up a huge amount of the state and it did cause the like the overall intelligence to drop a little. So I was like okay we should like clean that up. That's like kind of a pointless amount of detail. So there is there is, like, good tweaking to work with that. That being said, every single thing on the model jaggedness page, you know, is things that we think could get even better in the future. Yeah. Yeah. I I really appreciate that transparency, and I think it's great. Like, you know, thinking back to the Jabify stuff, it's like, where should we use Jab? Where should we not? It's not, an answer for all use cases, but can be really great for some. And I think feels very full circle to end this conversation talking about basically context engineering for JEV. Right? Like, we have come up with many names for that in the past. In fact, we've run webinars on harness engineering, on loop engineering, on fill in the you know, now we're talking about JEV engineering maybe. But, yeah, I think it's still important to think about, like, this is this is still a model. Like, how can we optimize what goes in in order to optimize what comes out? And with that, I just wanted to say, thank you so much, Ali and Hunter, for joining for this, almost emergency webinar on, building a harness with JEV. We are so excited to see the enthusiasm around the JEV launch. We're very, bullish on on using it and other decision models internally. And, yeah. Thank you so much for joining us, and thanks folks for joining and your good questions. Thank you. And I will also take an opportunity to plug our Discord. We wanna see what you're building. We wanna see your cool projects. If Discord's not your thing, you can also, like, tweet it to us on x. That's also cool, but, like, a lot of people are showing a lot of really cool stuff on our Discord and a lot of our developers are, engaging with the community in there too. So we'd love to see you. I can plus one the Discord. It's a pretty cool place. I've been I've been lurking in there for a little bit. Yeah.