It’s challenging to have a conversation about Depth of Knowledge (DOK) when everyone has different ideas of what it is. So, I’ve decided to refine how I distinguish between DOK levels and turn it into a tool that can be used to facilitate a conversation. The two-page tool covers topics from 1st grade through high school as well as topics ranging from number sense, fractions, geometry, probability, and functions. The pictures below give you a preview of the tool and you can download it by clicking the “Download PDF” button at the bottom of the page.
Page One
Page Two
Some things to consider:
- Some people may make the case that the depth of knowledge level could be raised by asking student to explain themselves. I don’t necessarily disagree with that, but it is noteworthy that all of these problems focus on the thinking that has to be done to solve them. None of them ask for explanations.
- All of the DOK 2 and DOK 3 problems come from openmiddle.com. Open Middle is a collaborative website where all problems are free and new submissions are welcome. The more problems people submit, the more problems we all have access to. It was co-founded by Nanette Johnson and I with help from Bryan Anderson.
What do you disagree with? How can this tool be refined to make it even stronger? What other topics would you add?
I jumped on the DoK bandwagon a long time ago and totally agree with you on how important it is for teachers to understand and reflect upon the cognitive demand of the tasks they ask their students to do. Depth of Knowledge is a very complex idea in and of itself, and probably not as black and white as we math teachers would like it to be.
I recently had the opportunity to attend a workshop on Depth of Knowledge facilitated by Dr. Webb himself. One of the things that Dr. Webb emphasized throughout the workshop was the difference between difficulty and complexity. I think many teachers believe they are offering a rigorous mathematics curriculum because they assign students lots of difficult problems, most of which require little more than the application of an algorithm or procedure that was modeled in class by the teacher. His other major point was that we should rely on the definitions of each level to help determine the DoK of a task, and that such determinations should really be done collaboratively rather than by a single person.
I think the examples you have collected are great illustrations of the different DoK levels, but there were a lot of powerful conversations that day built upon understanding the definitions, the essential differences between the levels and what level to assign a problem based on the definition. He also made it clear that he does not endorse graphics commonly associated with Depth of Knowledge, such as the “Wheel ‘o Verbs”, and even Hess’s Cognitive Rigor Matrix (which I really like & was the source of my a-ha moment regarding the importance of DoK).
One other topic that came up was the rigor of the CCSSM (DoK was developed to judge the alignment of assessment questions to the standards they were claiming to assess). Analysis of the CCSSM show that most of the standards fall at DoK 2, therefore most of the tasks we ask students to do should be at DoK 2.
P.S. Thanks for sharing the OpenMiddle sight. Love it.
Jennifer, I agree with everything that you are saying. Please share your resources with me. I have a 13 year-old daughter, who is having some challenges with applying the skills learned however, I know it’s an easy fix because they need to have their instruction level at a higher level as well as all worksheets and not only assessments and quizzes.
Thank you so much for this. The part that resonates the most with me is “His other major point was that we should rely on the definitions of each level to help determine the DoK of a task, and that such determinations should really be done collaboratively rather than by a single person.” I think that is a great articulation of what has been bouncing around in my head.
The way I’ve been describing it is that a group’s discussion about what DOK level a task is and why they believe it is more valuable than simply having a single person define the DOK level. That is actually something I hope happens as a result of this tool. How wonderful would it be to have an argument between teachers regarding whether one of the problems I’ve listed is DOK 2 or DOK 3? To actually construct their argument or critique another’s reasoning, there would be such great thoughts being shared!
Much appreciated.
This tools is very interesting for math. How did you determine the levels? How could it be generalized to use in other subject areas? For example, in physics, similar calculations are being performed. How could I make a depth of knowledge rubric similar to this one?
Thanks!
Hi Ani. I so wish I had deeper understandings of other subjects such that I could better answer this. I have seen good examples but nothing ever so encompassing as the tool I’ve made. If you come up with something, please share it with me so I can forward it to others. Thanks!
Ani, I think Level 1 would be use “this formula” to answer a question.
Level 2 might be, “how would you change this one parameter to achieve this one goal?”
Level 3: “Here are several parameters to change, how would you change them to find the max or min”.
Level 4: “Here is an authentic, unique application. Optimize a solution.”
I was specifically thinking about free fall and trajectory problems when I wrote this…
I agree with what’s been said in the post and comments so far: distinguishing between DOK levels is valuable, as are the discussions as those determinations are being made. I think it’s still helpful, though, to discuss the boundary between DOK 2 and 3 more specifically. While acknowledging that there’s no easy distinction, is this a correct generalization of the examples you’ve presented?
A problem that requires conceptual understanding is DOK 2; a problem that requires conceptual understanding AND strategic thinking crosses the threshold to DOK 3.
Zack, that is pretty succinctly said. I don’t know if that definition is always the case, but it seems to be from my experiences.
I am posing these same ideas to my PLC and was trying to come up with definitions of my own. I like Zack’s general idea, but I was wondering what principals guided you, Robert, when making this rubric.
Hi Ben. I was primarily guided by the categories defined by Norman Webb combined with the experiences of having done the problem. For example, if you look at the DOK 3 for first grade, you could solve it via random guess and check. Otherwise you need to strategically think about where you’re going to put a 9, and then an 8, and so on. So, there was definitely a lot of experimenting involved to find examples that fit well.
I have been very confused recently with the idea of DOK levels and have started to do a lot of reading and research on the topic. Thank you for all of your articles, explanations and work with this. You have helped me grasp the idea much more firmly. However, I still have a lot of questions. For instance, on the Iowa Core site that I frequent, the DOK levels have been established for the standard.
*Here you can see what I am referring to under the words For Educators:
https://iowacore.gov/iowa-core/grade/6/mathematics/the-number-system/apply-and-extend-previous-understandings-of-multiplication-and-division-to-divide-fractions-by-fractions./6.ns.a.1
This site is obviously aligning the standard with a DOK level. Does this mean that this standard is capped at a DOK level 2 and that problems and assessment questions can’t reach a DOK 3 or 4 within this standard?
Hi Megan,
Thank you for bringing this to my attention. I am the State Supervisor for Mathematics at the Iowa Department of Education. We definitely want to work to make sure the document is as easy to understand as possible. This is a common question across our state and we have been trying to address it. Let me explain my understanding and you can let me know what you think.
The Iowa Core Standards contain the content standards and standards for mathematical practices. The Iowa Core “content standards” contain the DOK for content standards only. These DOKs were set using work from Ed West and summarized in the “Determining the Cognitive Complexity of the Iowa Core in Literacy and Mathematics,” document in the link here: https://drive.google.com/file/d/0By-KPiWr7RBBdVN3bGg2cG13V19Uc3VENnBnbDhtVlQ0MG84/view?usp=sharing
Since, the DOKs are at an individual standard level, then they only apply to the concepts and the procedures of that individual standard. The DOKs do not apply to task that incorporate the clusters level of the standards or the standards for mathematical practices. Rigorous mathematical tasks, that include many content standards and the standards for mathematical practices, would tend to be a DOK level of 2 – 4.
On the Iowa Core Standards pages, we have included assessment targets links for support on what this might look like. Here is one as an example: Supporting Information for Target B [m]: 6.NS.A in blue at the top of the page located here: https://iowacore.gov/iowa-core/subject/mathematics/6/the-number-system/apply-and-extend-previous-understandings-of-multiplication-and-division-to-divide-fractions-by-fractions.
Additionally resources that might help and support your work are located below:
Iowa Core Mathematics Resource pages:
https://iowacore.gov/content/mathematics-resources
Assessment Targets Defined for Grades 3-8, HS
Standards to Assessment Target Alignment Guidebook (help understand the information in the above Assessment Targets Defined)
https://docs.google.com/document/d/1U6IQtYZdxQkIQLFA5gzH6OMNkgodyrZlkDFHX9pbiE8/edit?usp=sharing
Additionally, there are numerous resources on the Iowa Core Mathematics Resource site for tasks.
Performance Tasks
https://iowacore.gov/content/formative-assessment-mathematics
Teacher Resources
https://iowacore.gov/content/teacher-resources-mathematics
Plus, any task from Achieve the Core or 3 Act Task from Dan Meyer, Robert Kaplinsky, or Graham Fletcher is sure to hit the mark.
Please reach out to me if you have anymore questions at [email protected] I would love to have you join one of our Virtual Networks for Teachers, Instructional Coaches, or administrators. We offer asynchronous and synchronous learning across the state and have over 1,000 instructional coaches that participate.
Hey Robert,
What do you think about adding Kindergarten? I’m thinking K.NBT.1 would be a good standard to include (since this AWESOME task is already on openmiddle.com: http://www.openmiddle.com/teen-number-with-10-frames/
Funny you should ask Brian. My blog post next week is a new DOK Matrix that’s just for K-5 and includes Kindergarten. The following week will be 6 through Calculus. Check it out on the 24th and let me know what you think.
Thank you for sharing these ideas! I love your resources to help with the discussion of what DOK is. The comments have been very enlightening about how we view DOK. I am still struggling with the difference of DOK 2 and DOK 3 in some of the examples and am wondering if there would be a better example for DOK 3. For instance, under the idea of adding whole numbers in DOK 3, there is the focus of making the largest sum. To me, there is not a very distinct difference between DOK 2 and DOK 3. What is DOK 3 stated to come up with a variety of way to make 67 by filling in the boxes below? Would that get at DOK 3 more?
Hi Samantha. Have you seen this blog post? http://robertkaplinsky.com/3steps/
It might give you more of the details you are looking for.
The decimal problem you sent to get as close to one using 1-9 only once definitely took 5 tries using logical inferences to make decisions about which data to manipulate (in spite of having done these problems before. I like the fact that there is no specific algorithm, just the time to figure out how to move values to achieve the goal. Of course, if helps if you know the totals for different combinations before trying sets of numbers (e.g. 7+8+9=24), so you can think about which numbers may or may not share the same place value. Fun and engaging!
Thanks Carol. I also like how the problem helps bring out conceptual issues like deeper understanding of place value. Glad you found it useful.
Your examples remind me of the 4 categories for mathematical tasks – memorization, procedures without connections, procedures with connections, and doing mathematics. Wondering how much overlap we might find in the criteria for both…
Interesting. Not sure how those categories are defined. The first two don’t look very deep.
This is true. The lower level categories are not as cognitively demanding as the other two levels. Lots out there about the 4 categories for characterizing the cognitive demands of a task, but this is a simple place to start. http://maccss.ncdpi.wikispaces.net/file/view/BRIEF_Pedagogy_Tasks.pdf/609099971/BRIEF_Pedagogy_Tasks.pdf
The DOK 1 example for 2nd grade could also be labeled a”procedures without connections” task and the DOK 2 & 3 examples could be “procedures with connections” tasks. Much harder to find good “doing mathematics” tasks, especially ones that fit in the table, just as the same might be true for DOK 4 problems.
Hi Robert,
We have just switched our grading system to a 4-point scale, and we have been told that we should be using rubrics to grade our assessments. There has been lots of confusion. We were given a sample rubric which uses DOK level language, so some teachers are grading assignments based on DOK levels. Our school-wide sample rubric uses the following:
Competent with distinction
Competent
Basic Competency
Developing Competence
The wording is confusing them because for example competent with distinction is described as:
The student has consistently and independently demonstrated the ability to analyze and synthesize content-specific knowledge and skills in a new task, across content areas, or in an authentic experience. This seems to describe DOK 4 but it is being used as a sample rubric for grading assessments. I am just not sure how to explain this to my administration, do you have any insights that would be helpful?
Gosh, that does not sound like a good situation. It sounds like a Vegas situation to me (https://robertkaplinsky.com/can-going-las-vegas-teach-us-education/) where everyone agrees that rubrics would be useful yet everyone has very different ideas of what exactly that should look like.
I’ll go on the record and say that aligning your grading system to the four levels of DOK is not a good idea.
This blog post MIGHT be useful though, but it’s not really a great fit: https://robertkaplinsky.com/4-ways-assess-3-act-tasks/
I’m trying to download the file but am unable to. When I filled in my personal email, I’ve received an email saying that I’m signed up to receive weekly emails but no link for the PDF. How do I get this PDF?
Sorry it’s taken me so long to reply to you. Did you ever get this sorted out? If you got the other email, then that means you entered your email address correctly. If you didn’t get the email with the PDF, it likely means it went to your SPAM folder. If you still want this and can’t get it to work again, email me at [email protected].