Measuring AI's Impact on the L&D Team

Welcome & Series Roadmap

Hello, and welcome to the Designing with Love Podcast. I am your host, Jackie Pellegrin, where my goal is to bring you information, tips, and tricks as an instructional designer. Hello, instructional designers and educators. Welcome to episode 127 of the Designing with Love Podcast. As we continue through the 2026 lineup, we're also moving through the AI Ready Designer Series. Last time, we talked about collaboration with IT, security, and procurement, so AI adoption doesn't stall. Today, we'll track lightweight workflow metrics so you can prove impact with credibility. So grab your notebook, a cup of coffee, and settle in as we explore this topic together. Before we jump in, a quick note. This is a 12-episode arc and each episode builds on the last. In this 12-episode AI ready designer series, we'll move through five AI ready checkpoints each time. So you always leave with something practical you can apply right away. Alright, let's jump into checkpoint one.

Why AI Needs Real Evidence

Here's the shift. AI is no longer just something L and D teams are experimenting with on the side. It is starting to affect real workflows, how we draft content, how we create scenarios, how we review materials, how we summarize feedback, how we build job aids, how we support SMEs, and how we move projects forward. That means leaders may be asking, is this actually helping? Are we saving time? Is the quality better? Are we reducing rework? Is the team using AI responsibly? Is this worth the investment? And those are fair questions because excitement is not evidence. Anecdotes can help tell the story, but they are not enough on their own. If we want AI to become a sustainable part of L ⁇ D work, we need to show its impact in ways that are simple, honest, and credible. The goal is not to prove that AI is magical. The goal is to understand where it is helping, where it is not helping, and where we need better guardrails or training. AI adoption gets stronger when evidence is clear. So that's the big shift. AI use needs evidence. Now let's ground this in what still matters most when we measure anything in learning and development. What doesn't change is this. Measurement is not about collecting numbers just to have numbers. Measurement should help us make better decisions. For LD teams, that means asking what should we keep doing? What should we improve? What should we stop doing? Where is AI saving time? Where's AI creating risk? What does the team pause take to? What doesn't change is this. Measurement is not about collecting numbers just to have numbers. Measurement should help us make better decisions. For LD teams, that means asking, what should we keep doing? What should we improve? What should we stop doing? Where is AI saving time? Where is AI creating risk? Where does the team need more support? This is important because AI measurement can easily become performative. We might be tempted to report big, exciting numbers like, we generated 50 drafts, we saved 100 hours, we used AI on 30 projects. But numbers need context. 50 drafts do not matter if they created more we work. 100 saved hours do not matter if quality dropped. 30 AI assisted projects do not matter if the outputs were inconsistent or risky. So the constant is this. Measure what helps the team make smarter decisions. Not everything needs to be measured, and not everything that is easy to count is meaningful. And when we forget that, AI's impact measurement can go sideways pretty quickly.

Three Ways Metrics Go Wrong

So let's talk about the risks. There are three common measurement risks I want to name in this episode. Risk number one, vanity metrics. A vanity metric looks impressive, but it does not tell you much. For example, number of AI prompts used, number of jobs generated, number of tools tested, and number of assets created. Those numbers may be interesting, but they do not automatically prove value. A team could generate more content and still not improve quality, speed, or learner experience. So instead of only asking how much did we produce, ask what improved because of this? Risk number two, overclaiming. This happens when we give AI too much credit without enough evidence. For example, saying AI saved us 40 hours. Maybe it did. But how do we know? Compared to what? Was the project similar to previous ones? Did review time increase? Did the SME spend more time fixing errors? Did the team spend extra time learning the tool? A more credible statement might be on three similar projects, AI assisted drafting reduced first draft development time by an average of 25%, while SME review time stayed about the same. That's less flashy but more believable. Risk number three, invisible costs. AI can save time in one part of the workflow and add work somewhere else. For example, faster drafting, but more fact checking, faster scenario generation, but more brand editing. Faster outlines, but more stakeholder alignment needed. Faster content production, but more security review required. So when measuring AI, we need to look at the full workflow, not just the exciting part where the tool looks fast. So if those are the risks, the upgrade is to use a small set of practical metrics that show the real story without overwhelming the team. Here's the

The AI Impact Snapshot Metrics

upgrade. Instead of building a massive measurement system, start with a simple AI impact snapshot. This is a lightweight way to track AI's impact on your LND team across five areas time, quality, rework, consistency, and confidence. Let's walk through each one. Time. Where is AI reducing time in the workflow? Examples include first draft development time, outline creation time, scenario drafting time, job aid creation time, feedback summary time, revision turnaround time. You do not need perfect time tracking. Even a simple before and after estimate can be useful if you are consistent. For example, before AI, scenario drafts usually took two hours. With AI assisted drafting and human review, they now take about 45 minutes. That tells a useful story. Quality. Are the outputs better, the same, or worse? Quality can be measured through SME review comments, QA checklist results, fewer factual errors, stronger alignment to objectives, better scenario realism, and improved accessibility checks. This matters because speed without quality is not a win. Rework, are we reducing revision cycles? This is one of the most practical metrics for LND teams. Here you can track number of review rounds, types of issues found, pause, take two. Rework, are we reducing revision cycles? This is one of the most practical metrics for L and D teams. Here you can track number of review rounds, types of issues found, time spent revising, repeat feedback themes, and whether the same issues kept coming back. If AI helps create cleaner first drafts, rework should decrease over time. But if AI creates polished drafts with hidden problems, rework may increase. That's important to know. Consistency. Are we getting more consistent outputs across the team? This connects back to your knowledge vault. Consistency might include common templates, shared terminology, stronger brand alignment, repeated use of approved prompts, consistent QA steps, similar structure across modules or assets. AI can create inconsistency if everyone uses it differently, but with shared patterns and standards, AI can actually help teams become more consistent. Confidence. Does the team feel more capable using AI responsibly? This can be tracked through quick pulse checks. Do team members know when AI is appropriate? Do they know what data not to paste? Do they feel confident reviewing AI outputs? Do they know when to involve IT, security, or procurement? Do they have reusable prompts and standards? Confidence matters because adoption is not just about access, it is about people knowing how to use the tools well.

A Simple 30-Day Measurement Loop

Now let's turn that snapshot into a simple workflow you can use monthly without turning measurement into another full-time job. Here's your next move. Use a simple 30-day AI measurement loop. It has four steps. Choose one workflow, pick two metrics, track lightly, review and adjust. Step number one, choose one workflow. Do not try to measure everything at once. Pick one workflow where AI is already being used or could be useful. Examples include drafting scenarios, summarizing learner feedback, creating job aids, building first draft outlines, rewriting for clarity, generating quiz questions, and preparing for SME interview questions. Keep the scope small. Step number two, pick two metrics. Choose one speed metric and one quality metric. For example, time to draft plus SME revision notes, number of review rounds plus QA checklist score, scenario drafting time plus scenario quality rating, feedback summary time plus usefulness rating. This keeps your measurement balanced. You're not just asking was it faster, you're also asking was it useful? Step number three, track lightly. Use a simple note, spreadsheet, trello card, or project tracker. You might track project name, AI use case, tool used, time estimate, review notes, quality issues found, and lessons learned. The key is to make it easy enough that people will actually do it. Step number four, review and adjust. At the end of 30 days, ask the following Where did AI help? Where did it create extra work? What should we standardize? What guardrails need to be clearer? What should we stop using AI for? What should we test next? This turns AI adoption into a learning process for the team. Not hype, not panic, just continuous improvement.

Job Aid Example That Reveals Truth

Let me give you a quick field note so you can hear what this looks like in a real LD workflow. Imagine your L and D team starts using AI to draft job aids. At first, the team is excited because first drafts come together quickly, but instead of just saying AI is saving time, you track two simple metrics in 30 days, time to first draft and number of SME revision comments. After a month, you notice something interesting. The time to draft first dropped pause, take two. After a month, you notice something interesting. The time to first draft dropped from about 90 minutes to 30 minutes. That's real improvement. But the SME comments stayed high because the drafts were missing organization specific terminology. So the team does not abandon AI, they improve the workflow. They add the terminology list from the knowledge vault to the prompt. The next month, SME comments drop because the drafts are faster and more aligned. That is the kind of measurement that actually helps. It does not just prove impact, it proves the process.

Weekly Challenge & AI Impact Compass

Alright, let's make this practical with this week's checkpoint challenge. Here's your checkpoint challenge for the week. Pick one AI assisted workflow you already use or want to test. Then choose two metrics, one speed metric and one quality metric. For example, time to first draft, number of review comments, number of revision rounds, QA checklist score, and usefulness rating from a SME. Track those for 30 days. That's enough to start telling a more credible story about AI's impact. And to make that easier, I created a quick companion resource for this episode. Before you go, I made an interactive companion called AI Impact Compass. It's a click-through guide you can use to choose lightweight metrics and track AI's impact on your LD workflow without overcomplicating it. If this episode helped you, please follow or subscribe and share it with a designer or L and D leader who wants to prove AI's impact with credibility, not hype. AI can help L and D teams move faster, but speed alone is not the story. The real question is, are we improving quality, reducing rework, creating consistency, building confidence, and making smarter decisions as a team? This is what credible measurement helps us see.

Closing Quote & Final Sendoff

Before I conclude this episode, here's an inspiring quote by William Bruce Cameron, a sociologist and author. Not everything that can be counted counts, and not everything that counts can be counted. Thanks for spending time with me today. Until next time, keep it practical, keep it human, and keep designing with love. Thank you for taking some time to listen to this podcast episode today. Your support means the world to me. If you'd like to help keep the podcast going, you can share it with a friend or colleague, leave a heartfelt review, or offer a monetary contribution. Every act of support, big or small, makes a difference, and I'm truly thankful for you.