Name: The Agentic Operating Model - Aligning People, Process, and Technology for Production AI at Scale
Uploaded: 2026-10-29T11:34:01.704Z
Duration: 39 min 44 s
Description: The Agentic Operating Model - Aligning People, Process, and Technology for Production AI at Scale

Transcript for "The Agentic Operating Model - Aligning People, Process, and Technology for Production AI at Scale": Good morning, everyone. Where is everybody joining in from? Me and my colleague, Dred. Oh, Paris, Germany. That's fantastic. Me and my colleague, Dred, are in London. I'm complaining about the weather every single time, so I'm gonna do it from here. But, Salona, that's amazing. I'm jealous about the weather. The UK. Okay. So we're on the same boat. We have Frankfurt. The US. Whereabouts from US? It seems like it seems like it must be very late for you guys. Cape Town. Hi. Singapore. India. Fantastic. Amsterdam. I feel like... I think about any country you guys are mentioning or every city you guys are mentioning, and I'm like, we... Like, you probably have better better weather. Grass is always greener. India, Spain, Edinburgh. Yeah. We do have very different time zones here, it seems. Are people chiming... Our river is unbeatable today. Pakistan. Are people chiming in from work, or are you calling in from, you know, after hours? Work. Okay. Fantastic. Sweden, but not having great... Greater River. Okay. Fantastic. K. I think we are ready to start. The recording of this session will be available as well as the slides from the session. So, if you want to rewatch something, this will be available to you. So... Okay. Fantastic. Welcome, and thanks for spending an hour with us. My name is Marta Stangirzka, and I'm a deployed engineer here at LandChain. And my team's role is really advising customers on how to build production agents. This session is drawn from our ebook, the agentic operating model, and it describes how the enterprise, how enterprises can align people, process, and technology so agents make it into production and keep improving once they are live. You can drop your questions in the q and a panel as we go. And huge shout out to my colleague colleague, Dred, who is moderating the workshop and will be answering your questions as well. But first, let's start with what is Langchain and LangSmith. K. So LangSmith is our platform for agent engineering. It is used by half of the Fortune 10, and we have over 7,000 active customers. We also have 300,000,000 monthly OSS downloads, which makes us very widely adopted across the engineering space. This actually puts us at a similar scale as the OpenAI SDK. And we truly work with incredible companies. You can see the logos over here. So we have high growth startups like Harvey and Vantaa to large enterprises like Nvidia and ServiceNow. We speak to the customers on a daily basis and really understand what it takes from an organizational point of view to deploy your agent into production successfully. Our agentic operating model follows as you need more than just really good technology to be successful in your journey to wide AI adoption. This is what I want to share with you today. So traditional software is deterministic. We have unit tests that validate logic, CIDI pipelines, and monitoring alerts on exceptions that should not happen. Agents break every one of assumptions of traditional software, and you probably see it in your daily work if you work with AI. They are probabilistic and prompt driven. So you measure quality with evaluation suites rather than pass fail unit tests. And when something goes wrong and multiple things can go wrong, you inspect the reasoning in a trace, not a stack trace. And the failure modes are different. We can have an answer that sounds plausible but is plain wrong, a goal that drifts, a tool used in a wrong way, or simply a cost overrun. And here's the important thing that most enterprises miss. The builder population is different. It's no longer only software engineers. Anyone can write a prompt, define an evaluation, or write a tool, and then thus shape an agent. AI fluency becomes the new bottleneck, which is why it's not a tooling problem alone. So here's the pattern behind most stalled programs, and you likely have seen this in some organizations you work for. So you can have a well staffed team, but without an improvement process, it ships agents that break silently. You can have a rigorous process, but if you don't have the right tooling, the manual overhead will stop you from scaling. And you can have powerful technology, but in poorly structured teams, it produces fragmented, ungoverned systems. You can drop examples in the chat of what you have seen, and I bet you a lot of people will relate. So each one of these examples gets one pillar right and misses the others. Getting agents into production at scale means moving all three together, which makes this an organizational transformation rather than simply a technology update. So what does it look like when all of those three line up? This is the whole model on one slide. We have three vertical pillars, people, including those who build and those who you build for, so those who adopt, process, so the agent development life cycle, and technology. Running all of those, across those three pillars are those five cross cutting concerns. So you have governance, security, fill ups, compliance, and interoperability. So what does that mean? Well, first, and I'm gonna state the obvious here, agent engineering is a new discipline. Agent behavior lives in traces, not in code. That means that prompt engineering, evaluation, and observability are engineering priorities from day one, and if they are not, they should be. Treating them as an afterthought after the agent ships is the most expensive mistake we see enterprises make. Second, you need to establish enabling constraints. The goal is tricky because you need to let teams move fast without breaking things with enterprise consistency for infrastructure and decentralized teams and non engineers builders on top. Third, you need continuous improvement and a process of moving from a production failure to a measurable quality improvement. Today, we'll talk about how you can do it with LangSmith, our enterprise software, but really, this is a process that you just need to start thinking about. Here is the structure of the presentation. So we have five parts. People first, because this is where most programs actually stall. Then the process, the agent development life cycle. Technology, so the stack behind every phase. Then the cross cutting concerns, so you have risk, cost, standards, and independence. And then finally, the journey, how organizations mature, what it looks like in practice, and the factors that decide success. So let's start with people. Okay. So people, and I want to emphasize this, not only who builds, but also who adopts and how the teams can grow. So let's start with the most common failure. We see some enterprise AI programs go into an extreme. So what are the extremes? So you can be too centralized. So one team will build every agent, and you'll make the delivery become a bottleneck. To decentralize. So each team picks its own stack and you get sprawl, duplicated infrastructure and ungoverned risk. This can also happen... We see a lot of teams, building their own skills, but, like, this can lead to a lot of duplicated work if the same... If the different people from different teams are creating skills that are doing the same thing. And you probably have seen this in your own organization. And you can be too engineer only. So you exclude the operator, analysts, and subject matter experts that are ultimately closest to the work, and they never participate. And the institutional knowledge and agent needs and what makes the organization special stays trapped outside of the loop. All three are equally common. You will probably recognize at least one of them in your organization. The operating model resolves all three with one structure. So we have the three concentric rings. Ring one is the platform engineers, a horizontal platform, applied AI, or platform governance team. They own the model gateway, tool gateway, evaluation infrastructure, observability backplane, audit logging, policy engine, and reference patterns. They build a substrate and set the governance part. Then you have the agent engineers in rank two. So domain engineering teams and product operation customer and finance. In the core... Correctly structured organization, they self serve the platform to build domain agents on paved path patterns, and they own risk classification at design time and evaluation thresholds for what they ship. Co building with the platform team is the optional accelerator here, not a prerequisite. And finally, we come to rank three, the biggest one. This is the non technical builders. The SMEs, the operators, analysts, support, marketing, sales, finance, agile, anyone who holds the domain knowledge and agent needs. They own prompts, evaluation criteria, ground truth, and feedback, and increasingly, they own agents. In fact, in LangSmith, we actually have our own tool for no code users in order to enable the ring free. The key point here is that governance, security, and compliance are responsibilities baked into all those three rings. Rings one... Ring one provides the shared infrastructure. Ring two share... Ring two and three share the accountability. But what I want to emphasize here is that you don't need all of those three rings formed at once. Early in the journey, most organizations do not need a hard line between platform and agent engineering. You can collapse rings one and two into a single team that builds the infrastructure, the pattern, and the first domain agent, while ring three stays distinct. As adoption grows across teams, the distinction between rings one and two emerges organically. Whatever the size run it as an agent engineering hub, not a gatekeeping committee. Ring one produces patterns, primitive, and reference implementations and spreads outward through rings two and three. Gatekeeping is top down and slow, while a HAB is collaborative and multiplies enablement. Think of it as train the trainer. Extend it one ring further to include the people who actually do the work that the agents are automating. So ring three, as you can guess, is actually where most strategies fall short. So So most enterprise AI strategies have a serious ring one plan and a workable ring two plan. Ring three is where they fall short. And this is also the largest fraction of the work... Of where the workforce sits. In the classical ML, you don't really have to wonder of, like, how to answer the question of who builds the model. This is a small team of specialists. With agents, however, the people who know what good looks like in the domain can shape the agent directly. The same goes for evaluation. A support lead who has triaged 10,000 tickets can define the eval that matters more, that will evaluate more accurately than if you give this job to a data scientist who has two steps removed. This is why we build LUNIXMEF fleet as well to make that practical. A non engineer describes the agent's behavior in natural language, and the agent runs on the same runtime, observability, and governance backbone as the code build one. So this leads us to the so what question. Who do you actually hire? Well, building an agent capable organization does not require replacing existing talent. Agent engineering is really three existing disciplines, each expanded. You have product thinking defining what the agent should do. So writing the prompts that drive its behavior and the evaluations that say whether it's working. It lives in all three rings, and ring three is the most underrated source. You have new skills here, though. So you have prompt design, eval criteria, failure cluster analysis, and iteration loops over specs. Then you have the engineering that builds the tools the agent calls runtime, human and the loop pulses and platform primitives. The new skills here are tool calling patterns, state and memory, durable execution, sandbox isolation, and prompts as artifacts. And then data science. They build the systems that measure performance, evaluation harness, AB testing, monitoring and error analysis on production traces. And again, you have new skills here. LLM as a judge design, building datasets, regression suites, and cost per per correct answer thinking. So your staffing pattern should really follow from that. So typical anchor we recommend is three hires per discipline. So you have one senior agent PM with eval and prompt fluency, one agent platform engineer, one applied AI engineer, and one evaluation engineer essentially with red team experience. At the org level, you often add one AI security engineer and one conversational or trust designer. Everyone else comes from rescuing the people you already have, and adoption is the multiplier. Think about an enterprise that hires brilliant engineers and never trains the rest of the workforce. That's basically like building a formula one car and forgetting how to teach everyone else to drive. The programs that you can implement within your organization and that work share a few features. Implement a mandatory AI literacy training with role specific tracks. Have communities of practice where ring free builders share working agents. Make sure you recognize the building of the agents and not only using them. And have a no code first policy for new automations under a defined risk threshold with a paved path into rank two if the agent grows. I see that in my work a lot. I see people developing brilliant agents, but then the organization's not knowing how to push for getting those agents in front of many other users internally or externally. You need a process that defines that. And in our engagement, really, we emphasize the rescuing part. If you rescue your organization, this will out... Outlast you. The habits will stay with the team, and that's ultimately the goal. K. So now that we know who builds, let's look at how they build. So it's the process, the agent development life cycle. Here's the look. Okay. So you have... You probably heard of the traditional software development life cycle and that follows a predictable arc. You define requirements, build, test, and ship. The ADLC, so agent development life cycle, is shorter, continuous, and driven by evaluation rather than specs. And you have four phases that run-in sequence for every single agent change. You build, you compose model calls, tool prompts, and retrieval and choose the right abstraction. Tests, so you have datasets, evaluators, experiments, and passfail gates. Then you deploy. You need durable run time, human in the loop checkpoints, and gradual borrow out. And then you monitor. So here, the things like traces, signals, feedbacks, and dashboards come into play. But you need those two underlining practices that run continuously while this loop is spinning... Wheel... Sorry, wheel is spinning. So you iterate. You find the failure, identify root cause, fix, and reevaluate in days rather than quarters. And then you govern. So you need to govern costs, tool access, and discoverability under, and you need to have it under control as the portfolio grows. Of the four phases, test is the one that decides whether your compound install. Evaluation is the most universally weak area in enterprise AI programs. Treating it as a continuous practice rather than the deployment checkpoint is the single biggest differentiator between organizations that compound on their AI investment on... And those install. Have a look at those datasets and think about Autelgrit which ones of these you see in your organization or which ones of these you have seen. So datasets are your ground truth. And if you think about how you create those datasets you can use automations and real in production traces for a lot of them. So you've got golden sets that validate against real world truth. They can be created by your SMEs from production traces. You have synthetic sets. And like, I think about it like, why would I need synthetic sets if I have real in production data? Well, synthetic sets cover edge cases. Trajectory sets verify the execution path for the graph. So were the right tools being called? Regression sets come from previously resolved failures to make sure... So let's say you improve your agent a few times a week or a dozen times a week. What happens, like, after you implement the change, how do you test whether, the previous failures are not gonna reoccur? This is where you need regression test. And finally, adversarial tests. They test the injection and jailbreak resistance. Ring free SMEs are usually the most valuable contributors here, while data science owns the framework. And then you need evaluators. And evaluators score runs against those datasets, each with a trade off. Think about deterministic checks: they're cheap, so you can use them for every verifiable assertion. LLLMs are judged, scores criteria like grind... Grounding, policy adherence, and helpfulness, but its trade off is high. So use it selectively and use align evals to keep the judge calibrated to human judgment. You can do that in LangSmith. You can select, your online evaluator and align it to your examples. Think about it as few shot examples for data science. I think it's really important because this is a very easy way to show the LLM as a judge of what actually good looks like. Human annotation queues, they're expensive, they take time to produce but they do generate ground truth. Testing tells you what fails. The next step is turning that into a fix. So the value of traces, evals, and dashboards is only realized when teams can move from a signal to diagnosis to fix and do it repeatedly and depletes production demands. In practice, this rule rarely breaks because of lack of data. It breaks because interpreting that data is manual, slow, and competes with the work of building. The teams shipping the best agent make this cycle a first class discipline. They treat production failures as structured inputs, cluster them by patterns, trace them to root cause, propose a targeted fix, and verify the fix holds before it reaches users again. The fixes are specific. The prompt that ignores instructions gets examples and clearer constraints. A capability limit gets a model upgrade for that node only. A knowledge gap gets better retrieval. Tool misuse gets tighter tool schemas. As trace volume grows, running this by hand becomes really difficult, goes without saying. And this is where the automation comes in. Make sure that you define certain thresholds and send certain, and send certain traces into your datasets for an easier evaluation. We'll come back to the... To that in the technology section. Before that, deploy and monitor. Deploy is about reliable execution environments, not just shipping code. Here, we have the durable run time with checkpointed state, human and the look checkpoints for set sensitive operations, sandboxes for code writing and file system access, context app for the behavior files that shape the end agent, versioned and managed independently from agent code. And you can also have gradual rollout. You can have canary from five to 10%, then 25, 50, and so on, each stage monitored before advancing. Monitoring needs four kinds of visibility. It needs the traces, so full trajectory of a run, signals, automated quality scoring such as L and M as a judge and policy compliance. Feedback, explicit ratings and implicit signals like escalation, and this is where ring free usually is the biggest source. And then dashboards. So be creative with your dashboard. Make sure you monitor usage, latency, cost, evaluator scores, and failure patterns. Standard metrics like uptime that we are used to in software development will miss the agent specific features. Sorry, agent specific failures. So what happens if my agent uses the wrong tool, the wrong context, or skips an approval step? Let's move to part three, the technology that supports each phase of the life cycle. Here's our stack. So unlike point solutions that solve one problem in isolation, these components are designed so each strengthen the other. At the build layer, we have langchain, langgraph, and deep agents, our MIT licensed open source frameworks that are very popular with builders. We also have a massive, langchain community with events in London, New York, and San Francisco. So make sure you check that and get engaged in order to see how other people are building. We do also have LangSmith Fleet, the no code builder and the rank free entry point. LangSmith is where the inner loop lives. Observability, evaluation, prompt management, and datasets, plus insights and engine for actionable observability. We also have LangSmith deployments, which is the runtime for stateful agents, and sandboxes that give isolated execution for the riskier part of agent work. SmithDB is the data layer underneath, built for the shape of agent telemetry. And the Smith... LanghSmith LLM gateway is the single enforcement point for credentials, spend, personal identifiable information, leakage control, and audit. What we see is that most enterprises arrive at Lang Smith for observability. I have multiple calls a day about observability and evals, but observability is just the start of agent improvement. Here's how these product offerings fit into the agent development loop. Okay. So we have the build, test, deploy, and monitor with each element of our stack fitting into these, and all of those circle around traces and Smith dp, which is our database for traces. On the outer, on on the outer, part of the loop, you have the LLM gateway. So you govern how those traces and insights are being used. But in order to iterate very quickly, you can use tools like LangSmith Engine, which is your agent for agent engineering. Okay. So actionable observability is how you can spin this improvement loop that we just saw quicker. So you can use LUNIXmith engine that runs the iteration cycle on your behalf. It clusters production issues by severity, diagnoses root cause within hours rather than just waiting for a customer complaint, proposes a targeted fix such as a prompt edit, a context hub update, or a code PR and generates the evaluators and dataset examples that keep that failure from returning. This is that regression dataset that we enable to... We enable you to create quickly. Insights does automated pattern discovery. It clusters agent behavior, shows where the agent performs well versus poorly, and surfaces drift over time. It's important to notice that insights actually are your entry point for your no code users. It offers plain English read on how an agent is actually being used. And then you have chat. Chat is an interactive copilot for trace analysis and debugging. What we find is that Teams' operating engine ship improvements weekly, private and monthly, and catch regressions before users do. We also use it internally. One of the technology highlights for me is LLM Gateway. This is particularly important for our enterprise customers. With gateway, you can control costs, so you set, monitor, and enforce spend limits at different levels. So you can control limits at organization, workspace, user, and API key levels across different LLM providers. You ensure reliability. You set and enforce rate limits and configure model fallbacks to keep agents available when model providers aren't reachable. Model fallbacks can be configured on the agent level, but what I want you to understand is that it also can be configured on an organization level. You can also... With gateway, you can also protect sensitive data. So you detect and redact personal identifiable information and secrets from model calls. Sometimes it's as easy as to answer the security question of what happens if somebody inputs a password into my cloud code? And indeed, you can use tools like gateways to protect your organization from that. Part four: Cross cutting concerns. We have governance, security, FIML ops, compliance, and interoperability. And these are not a separate work stream. They are how the people and process commitments, how the people and process commitments become enforceable for the technology. So we have four lenses, risk, cost, standards, and independence. You need to start with risk. I really want you to have a look at this slide and think about it. So do you, within your current organization, have a way to classify how risky your specific agents are? As agents move from read only retrieval to taken actions, governance becomes as critical as agent logic. The agentic operating model classifies every agent into one of those three tiers with controls scaled to the tier. This classification should be reevaluated whenever tool or model authorization changes. We have the tier one, so the bonded agents. These are your agents that perform read only and informational operation, like retrieval, summarization and knowledge based Q and A. Controls are input and output filtering, audit logging, and evaluation gadgets and deploy. Tier two: activating. The agents take consequential actions like writing to systems of record or modifying customer facing data. The controls are everything in tier one as well as human in the loop approval, scoped tool tokens, rate limits, and real time policy enforcement. Tier three are those autonomous or higher stake agents, long horizon regulated or financial work. Here are the controls that are everything in tier two plus pre deployment red team simulation, dedicated runtime guardrails, behavioral anomaly monitoring, and name accountable owner, regulatory mapping, and sandbox code execution. Each tier inherits the controls below it, and they are enforced in one place, the LLM gateway, not scattered across agents. The same gateway anchors cost. So it goes without saying agent systems consume tokens continuously, call multiple models per interaction, and run-in long lived background processes. Without an explicit operating model for cost, agentic workloads silently destroy unit economics. There are three levers all enforced at the LLM gateway level. So we have routing, outer route to cheaper models when quality permits and reserve frontier models for the high stakes nodes. You have caching, so you can eliminate redundant model calls. You also have budgets, so set hard limits per organization, workspace key, or user, token budgets per agent, spent alerts and, per agent, spent alerts and token and wall clock caps on long running autonomous agents. The foundation, I think, is cost attribution per agent, so product teams on their own unit economics, per tenant per... For chargeback, per feature, per node, and per interaction, which links cost to business outcomes. I think it's important to track cost per correct answer alongside accuracy. So as you see on the screen, the BRITE model is the cheapest model that meets the quality bar, not the most capable one available. In fact, one of our new product offerings is SmithTune, which is supervised fine tuned model. So we enable you to fine tune the fine tune the models and get better answers from a potentially cheaper model. Cost is one question your leadership will ask. Your regulators will ask other questions. The agentic model is designed to be compatible with the major external guidelines and regulations an enterprise already has to answer to. The risk taxonomy maps to the risk classification each one requires and the LLM gateway, SmithDB and audit logging produce the edit... Produce the evidence. Okay. So there are a couple of different standards listed on the screen. So the first one is NIST AI RMF, and its generative AI profile asks you to govern, map, measure, and manage across the lifestyle. Across the life cycle, I'm sorry. The EU AI act asks for risk classification, transparency and human oversight for high risk systems. The ISO regulation asks for an AI management system which is the operating model itself plus the continuous ADLC. And the last one maps the gateway observability in SwiftB. You can find all of these listed and how we answer them in the, agentic operating model ebook that I will link towards the very end. Now let's move on to talk about journey. Okay. So how do you scale agent maturity? You have four stages. And I think we... You should now see a live poll on the event, on the screen. Okay. Let's see what the results are. Is it coming off key? No. It isn't. Natasha? Okay. Oh, I see. Okay. Let's wait a little bit longer. I can see Druid answering the questions. Okay. Natasha, could you pull it up on the screen, please? Okay. And we seem to have some technical difficulties here. Okay. Fantastic. Why does everyone just say it in chat? And I think I like... Mhmm. Yeah. If you guys... I I think we have some technical difficulties here. So if you can comment in the chat so we we see where people are in their journey, that would be very useful. Do you have a polls tab? Right? We do. Okay. So let me chat about those stages. Okay. Someone has currently built and shipped AI agents into production. Fantastic. Thank you so much. How was that experience building and shipping for you? Scaling. Fantastic. Exploring and building. Okay. Fantastic. So what... How we think about it is that there are four stages. We have another person building and scaling, explore and build. Yeah. Sometimes people are on the, on the intersection of two stages. Okay. So most enterprises entering agent development today are in stage one or two. So stage one is actually exploring. You prototype with one team, tracing is new or inconsistent. We have manual testing, whatever each team picked. Stage two is building. So a team is in production with basic basic tracing and offline evals for one or two applications. We have a platform team forming, and first, SME reviewers appear. Then we have operating. So multiple apps in production with online evals, human review and quality is actively managed, all three disciplines in place, and FinOps is treated as a quality dimension. Where the organizations aspire to be is scaling. So you want agent engineering to be an organizational capability with universal AI fluency, automated improvement loops, and proactive governance. The path to stage four does not happen by accident, and I'm sure a lot of people on the on the call here have seen it, how difficult it is to progress between stages. It's also, arguably, does not happen as a pure technology upgrade. Every transition is a people, process, and technology moving together. And most organizations are uneven. You can have strong observability with weak evals or strong evals with little human alignment. The gap between dimensions is the opportunity. It's also an opportunity to get there before... To get to scaling before everybody else. Here's what it looks like for the two very different organizations that we have worked with. So we have worked with two organizations at a very different scale, just to give you an example of scale and how many agents and productions our customers are, developing. At first, a global automotive matter... Manufacturer with a clear mandate. Get AI into production across supply chain, manufacturing, R and D and customer facing properties without a centralized bottleneck. The deployment time when deploying with us decreased from three months to one week. They have about 50 agents in production, 300,000 traces a week, and internal multi agent platform serving 2,000, 20,000 daily active users. We also work with a global telecommunications provider. In fact, telco is a big, is a big region area for us. They prioritize governance at scale. Their platform is serving over a 100 enterprise customers. They use self hosted Langham across multiple entities, and they have PII filtering and access isolation built in, not retrofitted. And that's exactly the agentic operating model. Okay. So to sum up, the shift to agents is an organizational transformation rather than a technology upgrade. The technology is here today. The open source build layer, language map observability evaluation, closed loop improvement, the run time and sandboxes, and the gateway as a single enforcement point. What remains is the organizational will to adopt a new way of building software where evaluation precedes deployment, traces precede debugging, production data precedes the next iteration. And every employee, not only engineers participates in agent creation and improvement. The agentic operating model is how an enterprise puts that into practice across people, process and technology. So if you want to go deeper, the full ebook gives you a much deeper overview on every section we covered today including the detailed tables behind each slide. As we already said, the recording of this webinar will be available to you And if you would like to talk about where your organization sits on your journey, speak with the Langchain team today. Fantastic. Dhrit, how are we doing with questions? All good. Oh, good. Fantastic. How do they receive the recording of today's session? So you will receive an email after the session. It should be later today with the recording of the session. Cool. Thank you so much, everyone. Thank you.