It’s common practice for educators to create their own assessments including selecting and/or creating the individual questions. This may be fine in many cases, but there are huge issues you need to be aware of when making this choice.
To illustrate this potential problem, consider what happens when office buildings need to perform maintenance on their fire extinguishers. Every year or two, fire extinguishers need to be inspected and calibrated by an expert. The problem is that most people know so little about fire extinguishers that they wouldn’t be able to verify whether the expert did her job or lied and did nothing.
So, a common recommendation is to hire one company to check the fire extinguishers and another company to do the maintenance. That way you’ve got a system of checks and balances to help ensure that you’re getting the outcome you’d like.
The process of educating and assessing students is very similar to what happens with fire extinguishers. When educators create their own assessments, they often select questions that assess what they taught their students, not what the standards measure.
If that happens, then you’ve got a big problem. You’ve got no system of checks and balances. You might think your students are proficient based on your assessment scores, but those scores could be a false positive measure because they are only assessing what was taught and not the actual standard.
To mitigate this potential problem, consider incorporating questions created by a third party instead of questions created by you or from your curriculum. As you and your colleagues look at each of the questions, some will seem great and some will seem crazy. The crazy questions are the most important to inspect in detail. Specifically, there are likely two possibilities:
- the question seems crazy because it is poorly constructed and unrealistic
- the question seems crazy because it really measures the standard and is significantly different from what you have been teaching
Without this system of checks and balances, it’s very similar to only having one company check your fire extinguishers and maintain them. Perhaps they’ll work fine, but you don’t want to find out the hard way that your fire extinguishers or assessments are not working properly.
What do you think? Where am I right and where am I off? Please let me know in the comments.
For me, this post is qua light bulb moment! As a learning coach in an elementary school, I often have teams of teachers eliminating specific questions on chapter or benchmark assessments because they say, “We haven’t even taught anything like this! The students won’t even understand this question.” Your point that it is not about what was taught, but rather about what the standards measure, is right on! I find that when students struggle and don’t know how to answer a question, we as teachers want to avoid those types of questions because WE feel that WE have failed as facilitators. We think it is the question’s fault, rather than the fact that the student doesn’t yet have a grasp of the standard.
Agreed Deanna. I don’t claim to know the answers, but I think we need to have meaningful conversations about what students need to know. Perhaps making the assessment before teaching is necessary. Deciding which questions best assess student understanding is an important part of it.
You lost me in the first part of this post, since I’m not sure what it means to assess the “actual standard,” as you put it, or “what the standards measure” (since I don’t think they measure anything?)
In most states, we use the Common Core standards which are just that–standards, not assessments, They are fairly broad and vague as to what they actually mean. So we can look to places like Illustrative Mathematics, our individual state’s common core assessment, our textbook’s so-called “common core aligned” tests… but as far as I know there is no definitive way to “assess the actual standard” other than cobbling together the resources available to us that fit best with our sequencing and understanding of what the standard is saying.
I agree that we should collaborate with colleagues and make sure we are giving exams that make sense.
Thanks for your pushback. I was probably sloppy with my phrasing, but let me better explain what I mean. Consider this standard:
CCSS.MATH.CONTENT.5.NF.A.1
Add and subtract fractions with unlike denominators (including mixed numbers) by replacing given fractions with equivalent fractions in such a way as to produce an equivalent sum or difference of fractions with like denominators.
I think that educators should have conversations with their peers about how students should demonstrate understanding for this standard. As you mentioned, having “no definitive way to ‘assess the actual standard'” does not make this easier.
That being said, when a teacher makes the assessment and the lesson, there is the potential that the collaboration we both want may not happen.
Our district creates its own common assessments and our teachers sometimes question certain assessment items. We question whether an item is within our outside the standard. We question if an assessment item is truly assessing what we want to measure. We question the item type and if we are giving student’s exposure to some item types that are on the PARCC.
We also question how we assess in terms of inter-rater reliability and making inferences on the student’s level of proficiency. Just today we had an email exchange on assessing multiplying and dividing decimals. What inference can be made when a student multiplies or divides and the decimal isn’t accurate? Perhaps the assessment item wasn’t designed well. No questions on reasonableness or estimation were on the assessment and, looking back, instruction in that area could have been reinforced.
Another set of eyes, an educator well versed in grade level standards and experienced in assessment design is needed. Standards experts likely exist in middle and high school. Assessment design experts might be another story.
Exactly Mary. We need to have these conversations. We need to question many more things than we are doing. The quest for figuring out the answer to those questions is likely to be eye-opening for all. It may show how far off the assessment questions are OR it may show that we were farther off than we realized in our understanding of the standard.
Asking which standards are being addressed in an assessment and to what degree is important. Depth of knowledge is important to ensure that students are being asked to strategically think in some of the problems. I would much rather assign problems with multiple solutions that show how students are reasoning rather than using only recall and skills and concepts problems. The Webb’s DOK worksheets from last week’s Empowered problem Solvers workshop were thought provoking.
Thanks Jami. It’s all about these conversations. I think the synergy of educators talking together about the validity of these questions is part of the growing process.
I completely agree with you. We must teach for transfer, which means we need to assess a little differently than students were taught to see if they are able to transfer their learning. This is how you know a student has mastered a concept in my opinion. If we test the exact same way we guided them through the process, how do we know if they have just temporarily memorized a procedure or if they understand and can apply their learning to new and different situations? Exposing students to a wide variety of problems within a standard is key to student achievement. In my opinion, I welcome new and different approaches to standards because I want to see what kids truly know and are able to do when faced with challenges in their learning journey.
Thanks Amy. I believe I understand what you mean by “teach for transfer.” What I agree with you about is that we need to be really thoughtful about the questions we use to assess understanding. How do we really measure whether students have a robotic understanding or have a deep and authenatic understanding?
Here is a response I’m considering posting. If you have a minute, can you give me your opinion?
I wonder if the question of standards and assessments reveals a deeper flaw in our education system. It seems that we continue to operate in an industrial-age mindset. That is, we act as if raw materials (kindergartners) can be placed in one end of the system and manipulated in certain ways (textbooks, lessons, standards, assessments, etc.) so that we will get a desired product at the other end.
Children don’t learn what we want them to. Instead they construct their own understanding of their world based on their lived experiences. In my opinion, until we change our view of children, and change our education system, children will continue to pretend to learn what their teachers want them to and pretend to care about showing it on the prescribed assessments.
Who fills and checks the fire extinguishers? Well, what if we’re dealing with a flood and not a fire?
That’s a very thoughtful and reasonable perspective, Scott. Learning is certainly complex, and the way you described it reminds me of the research behind Cognitively Guided Instruction (CGI), which uses the understandings students already have as a foundation for building new learning.
If you come up with any more insights, please share them as I have more questions than answers to what you wrote.
The use of success criteria helps with defining what mastery of the standard looks like. Developing an assessment after determining success criteria certainly helps with making sure the standards, instruction, and assessments are better aligned. There is no magic bullet to this and it certainly takes time!
Yes, definitely takes time and success criteria sounds like it would help. Nice seeing you in SD.