Assess What Matters In An AI World with Hamza Sami

Purpose of Assessment With AI

Jackie Pelegrin 0:01

Hello, and welcome to the Designing with Love podcast. I am your host, Jackie Pelegrin, where my goal is to bring you information, tips, and tricks as an instructional designer. Hello, instructional designers and educators. Welcome to episode 120 of the Designing with Love podcast. You're listening to part three, the finale of my series with Hamza Sami, where we focus on assessment that shows real learning when AI is in the mix. Welcome back to the show, Hamza.

Hamza Sami 0:34

Thank you, Jackie. Thank you for having me back.

Jackie Pelegrin 0:37

Yes, I'm so glad to have you back. And at the time of this recording, we're in the middle of January. So uh so it's really exciting. And you were um on earlier, episode 100. Now this is episode 120. So it's amazing. I love it. Great. So to anchor our thinking and first principles, what are some core purposes of assessment that must stay front and center when AI tools are available?

Hamza Sami 1:05

Okay, so when we go back to the first principles, the most important thing to remember is uh why do we assist? Assessment is a means, not an end. The end is learning, uh, real understanding, comprehension, the ability to apply knowledge meaningfully. So uh even when AI tools are available, uh the core purpose of assessment must be uh the center and the focus. Uh assessment exists to support learning, not just to measure it. Uh we have assessment for learning. Uh assessment for learning, what also we call it formative assessment, and we have uh assessment of learning, which is uh what we call summative assessment at the end of the course. So um to focus on the formative assessment, uh it should help students see uh see what they understand and and where they are struggling and what to do next. Uh AI can assess here uh by offering feedback, for example, uh, but the purpose remains uh helping learner uh learners build clarity and confidence, not just produce correct uh outputs. Uh now assessment uh should should make uh us see how students uh think, should make thinking visible. Uh so we design assessments uh that reveal how learner how learners reason, how how they make decisions, how they connect ideas. Uh also here AI can help to generate content, but it cannot replace uh the learner uh to explain why they made uh some

Formative Checks & Trust Building

Hamza Sami 2:54

choices in their submissions, for example. Uh also uh uh assessment should guide instruction and feedback. Uh it's not only for students, it's also for educators. It tells us uh as assessors or educators what is working, what needs to be revisiting, for example, um where are the misconceptions relies, or um where are the students are struggling, so it gives us uh a heads up uh before it's too late. Uh also we can uh talk about how the assessment uh builds the learner responsibility and self-evaluation. Uh as we said, it's not uh it's not only for the uh learners or the for the instructors, but it can help to evaluate uh the students' uh uh uh uh personal like uh comprehension and understanding of the material. Uh if it's designed well, uh the assessment can help students to become better judges uh of their work. So they can self-reflect. Um this can be done uh by self-assessment or peer feedback or reflection. Um which uh for example, in AI-rich environment, uh this becomes more important uh because students need to evaluate not just their work but also the outputs of uh of the AI uh content. So the key point here is the purpose of assessment doesn't change uh eventually, but uh the understanding, the growth, the meaningful learning stay the goal. Uh what changes is the uh are the tools uh we use to get there. So instead of asking uh how do we stop uh students from using AI, because we cannot stop them from using AI, it's there. And it's it's a valuable uh tool if we think about it also. But we should ask how do we design assessment that helps them uh to understand and explain uh and apply what they are learning with or without AI.

Jackie Pelegrin 5:09

Right. Wow, that's amazing, Hamza. I love that. Um you know, you talked a little bit about the formative, right, and the summative. So um because we want those formative or learning and progress checks to build trust, uh what do you think are some low-stakes checkpoints that can help students practice with AI openly and surface some of those misconceptions that may surface a little bit earlier in the process instead of later on?

Hamza Sami 5:36

Okay, so first we need to guide the students on how to use uh AI in their study. Uh we cannot just um like assess them uh without giving them instructions and guidance on how of the responsible use of AI. Uh in my uh in my master's uh dissertation, actually, I uh I was uh as I told you, I was uh researching the use of ChGBT in uh in higher education. So uh after I finished uh it helped me to build uh a student guide to responsible use of AI. Uh so we built this uh guide and we shared it with uh with our students, uh where we explained the purpose of uh like the the guide, of course, and uh how to use uh AI in their studies. We gave them examples, uh the responsibilities, the limitations and the risks of using AI, how how how can they misuse AI and it it uh negatively affect them, and we gave them examples of the misuse uh so they can avoid it. Uh we also gave them the uh like uh how to cite uh or acknowledge their use. It's very important uh so they can uh openly uh um share with with us that they are using AI. It's not something to hide.

Jackie Pelegrin 7:05

Right.

Hamza Sami 7:05

Also, we recommended in the guide we recommended some AI tools uh for different purposes. So um trust is everything. If learners feel they have to hide uh their use their use of AI, we will lose the visibility into their own thinking.

Jackie Pelegrin 7:23

Right, exactly.

Hamza Sami 7:24

Uh yeah, so the most effective low stakes to answer your questions are the ones that invite students to use AI openly, intentionally, and critically. Uh one simple and powerful approach we can talk about, and actually we do in our center, uh it's in the scheme of work for the teachers, we advise them to use it in every session. It's in class AI supported exercise or discussion. So, for example, uh the teacher would ask the students to use AI tool to generate uh an initial response to a prompt, for example, about a topic in the in that subject. Uh then they draft a paragraph or explanation, then they compare and uh with another source like a book or an article or um any any reliable resource. So um the key is that AI output is not the final product, uh, but it's a starting point for the learning. So once the AI generated, the AI response is generated. Uh here we we we we we began the assessment, the formative assessment by uh guided discussions. So we ask students uh what did the the AI do well here? Uh what are uh like the are uh for example, is the content uh accurate in terms of the information? Did they double check the the content, the accuracy? It is reliable, uh it is j generalized, not specific to their prompt. So these steps uh help students to see uh that w like the uh how AI can sound confident while still being wrong or incomplete or biased sometimes.

Jackie Pelegrin 9:20

Right.

Hamza Sami 9:21

Yeah, so you can also build trust through reflection-based checkpoints uh after using AI. Uh students can write their own reflection, uh like how uh how did AI help them, for example, where did it fall short, or did uh they have uh to uh to correct or rethink themselves, you

Documentation, Citations, & AI Transparency

Hamza Sami 9:42

know? So these are some of the uh like uh uh checks that or low stakes checkpoints, yeah.

Jackie Pelegrin 9:49

I like that. That's great. And do you usually in the guide, do you ask students or do the the faculty um ask students to provide screenshots of the conversations that they're having with AI so the faculty can kind of see what the process looks like and then they can kind of look at the reflection and see, oh, I see you know where where they their thought process went. And um, because of the for the assignments that I grade that have students use AI explicitly, um, we asked for screenshots. So I'm not sure. Did they do that as well at the center?

Hamza Sami 10:23

Of course. This is actually is uh in the policy of uh the citing and uh the citing policy, how to cite references. Uh we added this uh in details on how to uh acknowledge the use of AI in their assignments. So, as you said, yeah, one of the ways is to take screenshots and to write how they used AI. Uh uh, we also uh guided them to share the link of the conversation so we can uh so the assessor can see all the like the questions and the answers. And how did they like amend the that uh answer or what what's their contribution to that uh uh to that response of the AI generated content?

Jackie Pelegrin 11:13

Right.

Hamza Sami 11:13

So of course, yeah, uh using a screenshot or uh they can add it in the appendix of the of the submission of the if it's a report or even if it's a presentation, they can add in the reference list, they can add the link to the and site uh Chat GDT. So uh we have um uh like an overview of how the students uh used uh AI in their submission.

Jackie Pelegrin 11:41

Wow, that's great. I love that. Um so you talked a little bit about reflection, and that's one of my favorite ways to have students kind of think about, you know, like you said, what AI did well at, where it fell short, and maybe what they do differently next time. I've had students do that too, like what they would do differently. I like that because then they're always thinking critically, right, about the use of AI and how it's being used in their work, not just academically, but then they they tend to think about the future of work and how AI is going to be their partner along the way, right? Their creative partner. So uh we know evidence should reflect the learner's capability. And this is something you and I have talked about before, um, but I think it's becoming more and more um uh uh what am I trying to say? Like it's becoming at the forefront, I think, especially in higher education and workplace training, is authentic artifacts and authentic assessment. So when we look at things such as drafts, um process notes, and maybe even oral defenses, right? A presentation or live demos, uh what are some things uh, you know, like those? Do you think there's some that work best in in that environment, especially with AI, or um are there some that are just kind of low-hanging fruit that you think we can we can really focus on? Um yeah, what are what are your thoughts on that?

Hamza Sami 13:05

I I agree with you, Jeff and processes uh and process notes uh work well because they make their thinking visible. So when students submit early versions of their work and explain what they uh how they got there, sorry. We can quickly spot met understandings uh or gaps in their reasoning or over reliance on tools uh like AI. So uh more importantly, we can address these issues before it's too late, before they come embedded in the final submission. That's why uh when we say evidence uh should reflect uh the learner's capability, the key idea here is we need to see learning while it's happening, not just as a final product, uh which what also what we call uh assessment as learning. So we assess the the learners as they are learning. Uh it's it's similar to the formative or the assessment uh of learning or for learning, sorry. Uh so this can also happen in the class discussion, uh, which is one of the most effective authentic artifacts we have. Uh in our center, we we we as I told you, like in the AI uh based exercise, also when students talk through their ideas, uh why they chose the this certain approach, for example, or how they interpreted uh a concept, uh we we get a direct evidence of their understanding, how they are thinking. Uh so

Authentic Evidence & Making Thinking Visible

Hamza Sami 14:40

these conversations act uh real time on for uh informative assessment and uh build a shared understanding of of quality and expectations from the uh teachers also. Uh also you mentioned oral defenses and live uh demos, which are uh especially like uh valuable because they are uh uh they require uh learners to uh to connect knowledge to action.

Jackie Pelegrin 15:05

Right.

Hamza Sami 15:06

For for example, even a short explanation like walk me through your thinking. How did you reach to this point? Uh it can reveal far more than a polished writing from uh on a report or an essay, you know. Uh so if a student truly understands the content, they can explain it. Uh they can uh respond to the question. Uh but if they don't, um they it's not their work. You know, you will know it's not their work, it's just a copy-paste from uh an AI response. Uh so uh the best authentic evidence is not about catching mistakes. We're not looking for mistakes, we uh it's uh creative uh creating uh visible moments of understanding through discussion, reflection, uh guided uh practices. Yeah.

Jackie Pelegrin 15:58

Right. I love that. You know, in the instructional design program that where I teach at Grand Canyon University here in Arizona, they they built the program in such a way that it's project-based. Uh and so for example, I'm right now, I've I've got a class of 25 students, Hansa, that I'm teaching right now. So it's a big class, and it's the introduction course to instructional design. So it's their first, very first ID course that they take in the program. And so I have students that are most of them are K through 12 educators, and I have some that work in the military and uh one that he's worked with the US Navy, and so he's he's got some of that experience. But majority of them are K through 12 educators. So they they're having that imposter syndrome right now, and they're like, uh, this is a lot, you know, it's a lot to learn, and I agree. But what they have to do through this whole entire course is they they have a glossary um and of instructional design terms, so they're learning about that, but each week they're working on some and they're adding to that document. But then throughout that, they're also working on a project, an instructional solution. So they have to look where's there a gap, whether it's in my workplace or it's in the community. Uh, maybe it could be at their church or at a community center or something like that. Wherever it is, you know, there's a gap somewhere. And um, so they have to do a needs assessment and then they have to do a high-level design document. And their assignment that was due last week was actually where they had to create their instructional materials and then they had to do a reflection on that and talk about why, you know, they chose the specific delivery method. Were there any changes that they made to their materials from the high-level design document? And some of my students actually talked about that and they talked about how they realized as they were building their materials that they needed to make some changes based on uh what they discovered and working with the learners. So it was really exciting to see that um and see them work through that. So I love those types of authentic project-based assessments where they get to work through that. And then this week they have to actually deliver to an uh their target audience, whether it's one person or it's several people, and then they write about that. So it's really neat that they are actually going through the ADI model, right? And I get to I get to see them work through it. And yeah, you're right, it doesn't have to look polished right away. What matters most is that they're working through that and they're seeing, okay, this is my first instance of doing this, but I'm able to learn as I do it and as I go through it. And I have one student in particular that kept working through the this one pro the her project at the beginning, and she's like, I just keep wanting, I keep wanting to work on it. And I'm like, you're not gonna get it perfect. It's just not gonna, it's not gonna be that way. And that's not how it is in the real world either, because you know, an employer is not gonna say, yeah, have 12, you know, have 16 weeks to work on this. You may not get that. They may want it in two weeks. So um, so I I I keep having a reminder. I'm like, you know, you're not gonna get things perfect. You just have to get it to the point where you feel like it's to, you know, it's gonna benefit the learners the most, and then you have to release it because you're gonna be working on multiple projects at a time, not just one. They're gonna expect you to balance multiple projects. So you can't put all your effort in one project because everything else is gonna fall by the wayside. So uh she so she's learning that really early. And so it's a good learning experience for them. So I'm glad you brought that up with uh, you know, those opportunities where it's not always the finished product product that we're looking to grade. We want to we wanna see their progress along the way. I think that's just as important. Yeah.

Hamza Sami 19:44

I agree, especially in their case as students, they are learning. So we are don't expect from them uh a final result that uh just like when they work for a company, they want to it's a different case, you know. Right, they are learning, so they can uh make errors, they can um ch change their way of thinking or their approach in the next project. But if they don't have a project they just learn the theory or they they if they don't apply it, they if they don't like get their hands dirty, they they will not learn from their own mistakes on their own uh like from the feedback. They will not get feedback of course.

Jackie Pelegrin 20:24

Right, exactly. I like that you mentioned that the the the dirtiness of it. Yeah it's okay to have things that aren't aren't polished and perfect. So um during prototyping I I call it dirty design and I heard that somewhere and I like that I like the dirty design where it's it's not refined, it's not perfect. And that's what models like SAM and rapid development are all about. It's all about giving your stakeholders giving your subject matter experts something that is not refined and is not perfect because you don't want to spend too much time making something that's perfect for them. And then you have to go back and make a bunch of changes. I learned that the hard way in my first project as an ID where I gave them feedback I I asked them please look at this please look at the storyboard right before I go and build it in captivate. And she did she looked at all the feedback and it was wonderful and I'm like okay now I know what to do because this included a screen it included a demo. So as you you probably if you've used captivate before and anybody that's listening has used captivate before when you go to do a demo in captivate each click, each uh drop down anything that you do is uh it's like a slide in captivate. So if you go it's not like PowerPoint where you can just take out something and then it's okay. You have to go and re-record that whole simulation again from start from scratch. And so yeah and so I learned the hard way and I went to go and then I created that in Captivate and sent it to her before I I um had it tested with the a few learners because I had selected a few learners that

Project-Based Learning & Real Constraints

Jackie Pelegrin 22:00

could test it out for me before I implemented it and rolled it out to the whole entire department in enrollment. And so I had her test it out again and she's like wait no that's not right there and that's not correct there. Can you just take that out and I'm like no I have to redo the whole simulation again. And she's like oh I'm sorry I didn't know that. But what I should have done as a structural designer is I should have been more explicit about what I was looking for and what I needed in my first initial feedback. So I learned about that and I was like oh that was a hard lesson. And I'm like okay now I know in the future to be more explicit about what kind of feedback I need and that I really need her to look at certain things and not just keep it surface level because she just didn't she didn't realize that she didn't know and she felt bad and I was like it's okay it's not your fault. It's uh you know it was just miscommunication between us and I just wasn't clear enough. So I I learned early on with that. So yeah that's so true. Yeah. So let's let's transition into rubrics handlack because I think rubrics I think sometimes they at least in higher ed they kind of get a bad wrap sometimes and I think but I but rubrics are such a good way to to see where students are at and where they need to improve right because that's that's what we and accreditation higher education is all about accreditation. We we utilize that rubrics are utilized as that type of uh quantitative right as uh assessment that we can utilize to show accrediting bodies the accrediting bodies that hey students are either they're learning it or they're not when it comes to competencies and things like that. So so we want rubrics to they we we want this to separate the polish from the understanding. So what's some criteria that you think can help distinguish that AI assisted refinement from genuine knowledge and skill um where do you think that that can be kind of separated a little bit or yeah especially with educators right?

Hamza Sami 24:04

Yeah I think uh the key shift here is uh they should uh prioritize the evidence of thinking over surface quality uh and to distinguish the AI assessed refinement from genuine uh knowledge and skills uh criteria needs to focus on the higher levels of bloom's taxonomy I I know you uh Jack you're a fan of bloom's taxonomy yes uh so uh these higher levels uh for example the critical thinking the analysis the evaluation uh we can focus more on these uh the justification the demonstration so they can apply it uh not just uh like no or explain uh because these can be done like the first levels they can be done easily in AI and we wouldn't know uh who did what you know uh unless the student uh acknowledge it so first uh the strong rubrics uh should assess reasoning and decision making so instead of asking what the answer is we can ask the learner to explain why they chose uh this particular approach for example uh criteria might include the quality the quality of the justification or the uh the logic behind the conclusion uh or how well students connect evidence to to their claims uh AI can help refine the language of course but it cannot replace uh the learner's reasoning uh process uh uh reasoning processes you know also uh yeah rubrics should uh also value the application uh in the in context uh the context uh is very important especially if uh if it's the uh if the assignment provides the the context uh so when students apply the concept uh to uh a new or realistic or uh for example scenario uh it becomes much much more harder uh to uh to rely solely on on AI so we need to see the students input in in that in that sense uh criteria here can focus on how uh well uh learners adapt knowledge or solve problems that don't have a single answer to them you know uh all the higher order skills uh another powerful uh criterion is uh the process transparency uh we mentioned how they can they often we discuss the use uh of AI uh rubrics can reward students for documenting how they arrive to the final uh like for example drafts uh you mentioned screenshots uh of the use of the AI um decision logs reflective notes uh or even uh short oral explanations so this will separate the genuine learning from the surface level uh polish uh one we talked about uh by making thinking visible i love that yeah that's great so we need to take our rubrics and and all of like bring them to the next level and not just look at the lower level thinking skills but the higher level thinking skills yeah I love that that's great and uh instead of like you said describe or explain you know take it to that higher level where they have to dem we're we're actually grading them on their demonstration and creation things like that yeah I love that yeah that's great wonderful yeah because the because the when you evaluate something you need to define it first you need to explain it first then evaluate so it's already there you know um right it's already in the in the answer of the of the student's work yeah or if they explain something you know they will if if like you will understand if they evaluate if you evaluate something um you already know what it is so the first levels are already covered by just going to the higher levels yeah exactly right I love that so given the need for the summative or end of module checks that we we like to have as well which are just as valuable which formats do you think are most most valuable in this uh such as performance tasks capstones or maybe scenario based exams which ones do you think hold up best with AI in play? Okay so when AI is in play summative assessment uh still matters uh but

Rapid Prototyping & Feedback Lessons

Hamza Sami 28:40

uh we count as evidence uh uh needs to evolve uh what we count is uh the evidence uh we we want the evidence to evolve you know so the format uh assesses uh the formats assess uh not just a final product but the process uh and the judgment and the reasoning behind it uh that's why uh you mentioned before space uh formats uh tend to be the most resilient in that case uh for example presentations combined with discussion uh qa afterwards uh are very powerful and effective in the uh in this case when learners present their work and they engage in questioning they have to explain their decision uh they respond to challenges they don't expect for example sometimes the uh the questions from their assessors uh they need to explain their own thinking uh in real time so here we will understand uh if the students uh uh like even if uh AI supported parts of the preparation discussion revealed the depth of their understanding uh the critical thinking the the ownership of their work also uh reflective blogs uh it's very important uh uh it grants the experience uh are another strong uh like summative component you know when learners uh can uh connect theory to their own context uh whether academic or workplace uh based uh they demonstrate understanding that can't be easily understood uh reflections on for example what worked what didn't work uh and why uh this show the evaluation and learning uh transfer not just uh to connect reproduction you know uh another thing we use uh in our evidence uh is portfolio of evidence uh in the assignment we also uh add that uh to some parts of the assignment so depends on the uh the subject of course uh it it holds up uh well in ai uh environment a good portfolio in uh uh it can include uh multiple art artifacts like triad revision revisions um feedback responses and final output which allow uh the uh the assessors to see the growth over time and distinguish the refinement from real learning um capstone's projects uh require learning to solve authentic and context contextual problems uh it's very effective uh with the process demonstration also or the or the document the documentation yeah so uh another one you mentioned which is very uh also I I like this one because uh it you mentioned about uh like in your s with your students you you ask them to do a project with a real client so the scenario based exams can uh play a huge role in that uh but they work best when they are open-ended and applied uh asking learners justify their decision rather than recall information um so the format that hold up best uh like uh in my opinion uh uh uh with AI are those that require explanation and justification or emphasize process over polish uh and involve interaction and reflection uh and real world uh application so um basically when submittive assessment asks uh asks learn to show how they think not just uh what they submit uh ai becomes uh a support tool not a shortcut right I love that that's great I like those types of projects too those are great and one thing that's uh that they revised the instructional design program at the college recently and what they did was they incorporated more group work into that so that that way they get the opportunity to collaborate together and work on say for example they uh there was one group project that my students have to work on in a class that uh that I taught recently um a couple months ago and they had to do a group project where they had to work on this uh job aid together for uh a company and it was they were active the it was the scenario was basically they had to work on a customer service job aid for a company and then talk about just different aspects of that.

Jackie Pelegrin 33:19

So they

Rubrics for Reasoning Over Polish

Jackie Pelegrin 33:20

each had to work on that and then they had to do a presentation about the job aid itself and why they made the decisions they did, right, as a team. So it was great because not only did it allow them to be able to work on this project together, but it also helped them to learn about communication, collaboration, all those things that we do as instructional designers, we're we never work alone in a project. We always work with a team. So it was great. I love that part um and it's and it's authentic right because it's a real world application uh like you said earlier. And so I think the more that we can make those types of um things more authentic I think it you know and again like we said earlier that we're not expecting them to not use AI. We actually want them to use it but we want them to you know not replace the work they do but have it come alongside them. So those types of projects um are great too yeah I really like those yeah wonderful so for our bonus question Hamza um to offer some quick wins what are uh three assessment moves that any instructor could implement during the next semester or term that you'd like to offer okay first uh we can uh talk about adding a short explain your your thinking checkpoint uh this can be as simple as asking the student to include uh a brief paragraph for example or audio note or any class explanation answering questions like what why did you choose this approach?

Hamza Sami 34:53

Uh what was the most challenging part? What would you do differently next time uh this small uh move shifts from uh shifts uh assessment from evaluating polish uh to evaluating understand uh and it works whether students use ai or not uh another way uh another way way also uh build in a short discussion or uh peer feedback moment uh this could be a five minute in the class discussion uh think pair share you know uh or a uh brief peer review using uh guided prompts when students explain the the ideas to others or respond to questions they clarify their own understanding and develop uh evaluating uh evaluative judgment uh it's also one of the easiest ways to assess learning authentically without adding uh you know the gradient load on the assessor uh this is uh yeah uh we use that a lot in our uh in our center with our students and it helped us uh a lot actually with some students who would uh like you know miss some uh some criteria in their assignment you know especially the the passing criteria so we know they uh understand that concept for example uh we know they uh they covered it in class discussion so we would just uh refer to that in class discussion and they would cover the criteria you know so uh I think that's uh also effective uh third one uh we uh explain uh we can ask uh the students to explain how they used AI uh with citing uh the process we we talked about um uh how to cite it you know the screenshots or a link to uh the chat GBT conversation uh and the prompt and explain how the uh they reached to that output or or how their output was modified you know to reflect their own contributions so they how they okay so this is the output from chat GBT but as a student how did I uh change it to to match the context I'm looking for right and to reflect on it also yeah I think these the uh that uh that are effective uh I advise uh the instructors to use uh in their term or next semester yeah I love it great wonderful so as we wrap up this episode Hamza here's the the final question I want to to put out there um because we've been talking a lot about assessment in education but also in the workplace as well so if you could change one assessment habit across a department maybe if we're thinking about corporate right what would it be and and why uh I think uh I would uh change the practice of collecting final reports or essays without ever discussing them with the learner uh before AI uh we could use that you know like just some at the end of the assignment submit a report uh or uh an analysis report or you know but now uh it's not affected at all. Uh it's not that it's not important, of course it's important. Uh but right now in many courses students submit a polished product uh and receive a grade and move on uh with little or no opportunity to explain their thinking or ask questions or reflect on the on the feedback. So when that happens the assessment becomes a one-way transaction you know uh instead of a learning uh experience the purpose is remember we talked uh in the beginning of the episode about the purpose of assessment is learning and uh understanding it's difficult for the assessor to judge also uh we I remember a lot of time I was put in a very uh tricky spot where I question is this the I know the learner is is hard working for example or even if they were not it's the same because I can see the the language of the of the AI in their submission uh and if in the report especially in the written format uh submission uh especially if I don't uh see their progress during the semester so I will also I will directly go to to questioning or suspecting that they used AI. So and sometimes it it happens that we have to do an extra sorry assessment uh uh format for example like a discussion or presentation we ask them to present or discuss their uh understanding um so the issue is not with the essay or report itself Uh these are very valuable formats, of course, and they will need them in the the world of general. But it's without dialogue we are only assessing the output, uh not the student, not the understanding behind it. And in an AI enabled world, the gap becomes uh even wider. So a simple change would be to prepare written submission with a brief discussion touchdown. This is what we do in our center. Uh like for example, uh we are an assignment-based, we use assignment-based assessment. So in every assignment there's a presentation, there must be a presentation as a part, and another part, for example, a written format. For the written formats, uh like reports, essays, even reflective journals and so on. Uh we ask the students to like in a to give a five-minute uh any class discussion uh in front of the assessor and another uh reviewer, for example. Uh and uh we asked them some questions to make sure like we asked them first to walk us through their uh the report, what they did as a content, and how they reached to their conclusion. Then we asked them questions about uh the report, like in

Resilient Summative Formats With AI

Hamza Sami 41:20

in their submission to make sure they understand what they wrote. So once that is done, uh we we know that the students understand that's the purpose of it, remember? It's not that even if they even if they relied over overly on AI, uh they can expand uh what they wrote. So from the learner's perspective, uh this idea will reinforce the goal is understanding, not just completion. Uh builds uh metagognition uh and self-assessment skills for this for the learners and makes feedback feel meaningful rather than final. So uh and from the assessor's perspective, it improves the assessment validity without adding uh excessive uh grading mode. Uh we we talked about that also.

Jackie Pelegrin 42:13

Right.

Hamza Sami 42:13

So you're not changing yeah.

Jackie Pelegrin 42:17

I like that. Yeah, that's great.

Hamza Sami 42:21

So you're not changing the the assignment, you are changing how learning is evident, you know. Uh is uh how how to to how what how the evidence is uh or the evidence is is evaluated. So stop letting the assessment uh end at submission, but start uh letting it continue through conversation, through dialogue.

Jackie Pelegrin 42:43

Right. I like that. And then the students don't feel like they're doing busy work, right? They don't they feel like what they're doing is meaningful, and the instructors will also feel like I'm not just grading a bunch of assignments and doing that, but they feel like they're actually grading the students' progress in in each of the assessments. And it's more of a journey, not just okay, this is the destination, then we're gonna go on to the next, right? It's a it's a whole journey for them that we're we're actually seeing along the way as they do that. I like that.

Hamza Sami 43:15

Yeah, I agree. It's uh also rewarding when you see the the students uh uh input at the end of the semester and their final submission. You feel the the value uh of uh of the learning uh uh experience throughout the semester and you and they actually understood what the the what the subject is.

Jackie Pelegrin 43:38

Right, exactly. They walk away and then when they go to their next course, they take some of what they learned and they apply it to the next course, and then you just uh and there's instances where I've had students uh multiple times in courses where it's like, oh, I have this this student again. And that I actually like that because then I get to see their progress through the program. And I'm not just in one part of their learning experience, but I'm able to actually uh give them feedback along the way. And so I like when I'm able to have students multiple times throughout different courses in the program. So it's exciting. Yeah, I'm sure the and then the students enjoy that as well because if they know my grading style and they know how I give feedback and it helps them along the way, then they're like, oh, I get Professor Pellegrin again. And so uh so they like they like that because they know how how I go about assessing and and grading their work. So um, because if if an instructor is passive in what they're doing and they're just saying, great job, great job, great job, and that's it, and they're not giving them that that good uh you know type of feedback, then all it is is like you said, it's just a a grade, right? It's just like here you go, here's your grade, and then there's no meaning behind it. So um, so I think one thing that instructors need to do too is is really give solid feedback and make it make them meaningful and don't just say good job and that's it. It's like, uh, you need, you know, they need a little bit more than that. And uh look for look for those opportunities that they can improve because there's always room for improvement. Um, you know, you and I have been in this this field for a long time and we still learn um how to improve our work. So, you know, if we're to, you know, I've been in higher ed almost 20 years and I'm still I'm still finding ways to make improvements um because now AI is around. So it's like, oh, okay, how can I learn to work with it and uh and improve the work that I do without losing the human touch to it. So yeah, so it's just yeah, it's so important. Absolutely. I love it. This was a great conversation, Yamsa. Thank you so much for closing out this series with a clear assessment toolkit that helps us to prioritize evidence of learning over AI fluency alone. I think that's so important. Um, so if anyone missed earlier parts, catch part one on AI norms and misuse, and part two on autonomy with SDL and CPD. And then make sure to share the series with a colleague who is redesigning a course. So thanks again, Hamza. I look forward to maybe doing another series with you. So feel free to hop back on and uh the door is always open for you.

Hamza Sami 46:24

Thank you, Jackie, for having me and I look forward to it. Thank you so much.

Jackie Pelegrin 46:32

Thank you for taking some time to listen to this podcast episode today. Your support means the world to me. If you'd like to help keep the podcast going, you can share it with a friend or colleague, leave a heartfelt review, or offer a monetary contribution. Every act of support, big or small, makes a difference, and I'm truly thankful for you.