It’s challenging to have a conversation about Depth of Knowledge (DOK) when everyone has different ideas of what it is.  So, I’ve decided to refine how I distinguish between DOK levels and turn it into a tool that can be used to facilitate a conversation.  The two-page tool covers topics from 1st grade through high school as well as topics ranging from number sense, fractions, geometry, probability, and functions.  The pictures below give you a preview of the tool and you can download it by clicking the “Download PDF” button at the bottom of the page.

Page One
Page Two

 

Some things to consider:

  • Some people may make the case that the depth of knowledge level could be raised by asking student to explain themselves.  I don’t necessarily disagree with that, but it is noteworthy that all of these problems focus on the thinking that has to be done to solve them.  None of them ask for explanations.
  • All of the DOK 2 and DOK 3 problems come from openmiddle.com.  Open Middle is a collaborative website where all problems are free and new submissions are welcome.  The more problems people submit, the more problems we all have access to.  It was co-founded by Nanette Johnson and I with help from Bryan Anderson.

 

What do you disagree with?  How can this tool be refined to make it even stronger?  What other topics would you add?

27 Comments

  1. I jumped on the DoK bandwagon a long time ago and totally agree with you on how important it is for teachers to understand and reflect upon the cognitive demand of the tasks they ask their students to do. Depth of Knowledge is a very complex idea in and of itself, and probably not as black and white as we math teachers would like it to be.

    I recently had the opportunity to attend a workshop on Depth of Knowledge facilitated by Dr. Webb himself. One of the things that Dr. Webb emphasized throughout the workshop was the difference between difficulty and complexity. I think many teachers believe they are offering a rigorous mathematics curriculum because they assign students lots of difficult problems, most of which require little more than the application of an algorithm or procedure that was modeled in class by the teacher. His other major point was that we should rely on the definitions of each level to help determine the DoK of a task, and that such determinations should really be done collaboratively rather than by a single person.

    I think the examples you have collected are great illustrations of the different DoK levels, but there were a lot of powerful conversations that day built upon understanding the definitions, the essential differences between the levels and what level to assign a problem based on the definition. He also made it clear that he does not endorse graphics commonly associated with Depth of Knowledge, such as the “Wheel ‘o Verbs”, and even Hess’s Cognitive Rigor Matrix (which I really like & was the source of my a-ha moment regarding the importance of DoK).

    One other topic that came up was the rigor of the CCSSM (DoK was developed to judge the alignment of assessment questions to the standards they were claiming to assess). Analysis of the CCSSM show that most of the standards fall at DoK 2, therefore most of the tasks we ask students to do should be at DoK 2.

    P.S. Thanks for sharing the OpenMiddle sight. Love it.

    • Jennifer, I agree with everything that you are saying. Please share your resources with me. I have a 13 year-old daughter, who is having some challenges with applying the skills learned however, I know it’s an easy fix because they need to have their instruction level at a higher level as well as all worksheets and not only assessments and quizzes.

  2. Thank you so much for this. The part that resonates the most with me is “His other major point was that we should rely on the definitions of each level to help determine the DoK of a task, and that such determinations should really be done collaboratively rather than by a single person.” I think that is a great articulation of what has been bouncing around in my head.

    The way I’ve been describing it is that a group’s discussion about what DOK level a task is and why they believe it is more valuable than simply having a single person define the DOK level. That is actually something I hope happens as a result of this tool. How wonderful would it be to have an argument between teachers regarding whether one of the problems I’ve listed is DOK 2 or DOK 3? To actually construct their argument or critique another’s reasoning, there would be such great thoughts being shared!

    Much appreciated.

  3. This tools is very interesting for math. How did you determine the levels? How could it be generalized to use in other subject areas? For example, in physics, similar calculations are being performed. How could I make a depth of knowledge rubric similar to this one?

    Thanks!

    • Hi Ani. I so wish I had deeper understandings of other subjects such that I could better answer this. I have seen good examples but nothing ever so encompassing as the tool I’ve made. If you come up with something, please share it with me so I can forward it to others. Thanks!

    • Ani, I think Level 1 would be use “this formula” to answer a question.

      Level 2 might be, “how would you change this one parameter to achieve this one goal?”

      Level 3: “Here are several parameters to change, how would you change them to find the max or min”.

      Level 4: “Here is an authentic, unique application. Optimize a solution.”

      I was specifically thinking about free fall and trajectory problems when I wrote this…

  4. I agree with what’s been said in the post and comments so far: distinguishing between DOK levels is valuable, as are the discussions as those determinations are being made. I think it’s still helpful, though, to discuss the boundary between DOK 2 and 3 more specifically. While acknowledging that there’s no easy distinction, is this a correct generalization of the examples you’ve presented?

    A problem that requires conceptual understanding is DOK 2; a problem that requires conceptual understanding AND strategic thinking crosses the threshold to DOK 3.

    • Zack, that is pretty succinctly said. I don’t know if that definition is always the case, but it seems to be from my experiences.

    • I am posing these same ideas to my PLC and was trying to come up with definitions of my own. I like Zack’s general idea, but I was wondering what principals guided you, Robert, when making this rubric.

      • Hi Ben. I was primarily guided by the categories defined by Norman Webb combined with the experiences of having done the problem. For example, if you look at the DOK 3 for first grade, you could solve it via random guess and check. Otherwise you need to strategically think about where you’re going to put a 9, and then an 8, and so on. So, there was definitely a lot of experimenting involved to find examples that fit well.

  5. I have been very confused recently with the idea of DOK levels and have started to do a lot of reading and research on the topic. Thank you for all of your articles, explanations and work with this. You have helped me grasp the idea much more firmly. However, I still have a lot of questions. For instance, on the Iowa Core site that I frequent, the DOK levels have been established for the standard.

    *Here you can see what I am referring to under the words For Educators:
    https://iowacore.gov/iowa-core/grade/6/mathematics/the-number-system/apply-and-extend-previous-understandings-of-multiplication-and-division-to-divide-fractions-by-fractions./6.ns.a.1

    This site is obviously aligning the standard with a DOK level. Does this mean that this standard is capped at a DOK level 2 and that problems and assessment questions can’t reach a DOK 3 or 4 within this standard?

    • Funny you should ask Brian. My blog post next week is a new DOK Matrix that’s just for K-5 and includes Kindergarten. The following week will be 6 through Calculus. Check it out on the 24th and let me know what you think.

  6. Thank you for sharing these ideas! I love your resources to help with the discussion of what DOK is. The comments have been very enlightening about how we view DOK. I am still struggling with the difference of DOK 2 and DOK 3 in some of the examples and am wondering if there would be a better example for DOK 3. For instance, under the idea of adding whole numbers in DOK 3, there is the focus of making the largest sum. To me, there is not a very distinct difference between DOK 2 and DOK 3. What is DOK 3 stated to come up with a variety of way to make 67 by filling in the boxes below? Would that get at DOK 3 more?

  7. The decimal problem you sent to get as close to one using 1-9 only once definitely took 5 tries using logical inferences to make decisions about which data to manipulate (in spite of having done these problems before. I like the fact that there is no specific algorithm, just the time to figure out how to move values to achieve the goal. Of course, if helps if you know the totals for different combinations before trying sets of numbers (e.g. 7+8+9=24), so you can think about which numbers may or may not share the same place value. Fun and engaging!

    • Thanks Carol. I also like how the problem helps bring out conceptual issues like deeper understanding of place value. Glad you found it useful.

  8. Your examples remind me of the 4 categories for mathematical tasks – memorization, procedures without connections, procedures with connections, and doing mathematics. Wondering how much overlap we might find in the criteria for both…

      • This is true. The lower level categories are not as cognitively demanding as the other two levels. Lots out there about the 4 categories for characterizing the cognitive demands of a task, but this is a simple place to start. http://maccss.ncdpi.wikispaces.net/file/view/BRIEF_Pedagogy_Tasks.pdf/609099971/BRIEF_Pedagogy_Tasks.pdf
        The DOK 1 example for 2nd grade could also be labeled a”procedures without connections” task and the DOK 2 & 3 examples could be “procedures with connections” tasks. Much harder to find good “doing mathematics” tasks, especially ones that fit in the table, just as the same might be true for DOK 4 problems.

  9. Hi Robert,
    We have just switched our grading system to a 4-point scale, and we have been told that we should be using rubrics to grade our assessments. There has been lots of confusion. We were given a sample rubric which uses DOK level language, so some teachers are grading assignments based on DOK levels. Our school-wide sample rubric uses the following:
    Competent with distinction
    Competent
    Basic Competency
    Developing Competence
    The wording is confusing them because for example competent with distinction is described as:
    The student has consistently and independently demonstrated the ability to analyze and synthesize content-specific knowledge and skills in a new task, across content areas, or in an authentic experience. This seems to describe DOK 4 but it is being used as a sample rubric for grading assessments. I am just not sure how to explain this to my administration, do you have any insights that would be helpful?

  10. I’m trying to download the file but am unable to. When I filled in my personal email, I’ve received an email saying that I’m signed up to receive weekly emails but no link for the PDF. How do I get this PDF?

    • Sorry it’s taken me so long to reply to you. Did you ever get this sorted out? If you got the other email, then that means you entered your email address correctly. If you didn’t get the email with the PDF, it likely means it went to your SPAM folder. If you still want this and can’t get it to work again, email me at [email protected].

Leave a Reply

Your email address will not be published. Required fields are marked *

Post comment