Only a few months ago, I used the words “complex” and “complicated” interchangeably as synonyms for “difficult.” Recently I learned about the differences between their meanings and I now see their significance everywhere, especially in math education. Specifically, I am concerned about Depth of Knowledge (DOK) being viewed as a complicated issue when it is really complex.
Dr. Sholom Glouberman and Dr. Brenda Zimmerman use a great metaphor for describing the difference between complicated and complex. Consider how sending a rocket to the moon and raising a child are both considered difficult, but for different reasons. When you send a rocket to the moon you have to follow a long list of detailed procedures. If you follow them carefully, you can reliably repeat that procedure with a high probability of success. There is no such thing for parenting! Even if you expertly raised one child and wrote down everything you did, following that same set of procedures carefully for the next child does not ensure a high probability of success. There are so many interwoven factors with raising a child that no set of instructions can guarantee success. Accordingly, sending a rocket to the moon is a complicated process, whereas raising a child is complex.
Complicated situations are so much easier to deal with: with enough experience you can create a set of procedures to handle almost every possibility. What winds up happening though is that people want to apply techniques to complex situations that are more appropriate for complicated situations. Below is an example of what I mean.
Have you ever seen something called Pinterest Fails? People will post recipes for creating a food or dessert on Pinterest and others will try to replicate them. It seems so simple: just follow the steps and you will have the same results. As you can see below, it isn’t always so easy. The reality is that there are a number of factors that are hard to encapsulate: temperature of the cupcake and frosting, device to deliver the frosting, type of cookie used, etc.
I see the same sorts of things happening with Depth of Knowledge. Consider the DOK Matrix I made to distinguishing between Depth of Knowledge levels pictured below that features many problems from Open Middle. Take a couple of minutes and work your way down a column or two of your choice. You will feel how much more cognitively demanding each level gets.
How do you articulate this to others? How do you determine the depth of knowledge level of a different problem? I get that it would be so useful to be able to treat DOK like it was complicated: simply follow these steps and you can be doing higher order thinking with your students too. The reality though is that depth of knowledge is far more complex than that and we have to be very careful about how we articulate it.
I want to share a few approaches to articulating DOK that are commonly shared on the Internet and my thoughts on them. I must emphasize that everyone who has created or shared one of these tools did so out of the best intentions. So, while I want to take an honest look at each tool, it is not my intention to belittle anyone.
Looking more closely, issues start to appear. For example, the words “construct” and “compare” are both in DOK 2 and DOK 3. Many verbs, like “revise”, “create”, or “use” could be a variety of DOK levels, depending on how they are used. So, while this tool may be popular, it misses much of DOK’s nuances.
Source: Unknown
When I think about it now, I realize that posters like these may do more harm than good. Do those prompts help students use the same level of cognitive demand as what you felt using the DOK tool or is it oversimplifying the process? Like with the DOK verb wheel, will everyone who sees these realize that or will they think that any time students “create” something it will be DOK 4? Again, I see this as a complicated approach to a complex problem.
Source: Penny Lund
Source: Tracy Watanabe
I asked her about the flowchart on Twitter and she pointed me to the blog post it originated from as she felt it was being “used out of context.” She later revised it and shared this updated flowchart (below). I admire Tracy’s goal of helping people by freely sharing her work but I still have major concerns about the flowchart.
A flowchart is the quintessential response to dealing with a complicated problem: if this happens, do this, then if that happens, do that. I appreciate her attempt at refining the flowchart, but I don’t think that any flowchart could ever summarize Depth of Knowledge’s complexity. Instead, teachers may look at this as a way to skip the messy process of exploring DOK’s complexity and instead use the flowchart to make decisions.
Source: Tracy Watanabe
My tool for distinguishing between depth of knowledge levels takes a complex approach. It is not as easy to use quickly as other tools designed for complicated situations, but I believe that it helps develop a deeper appreciation for the nuanced differences between levels.
I’m sure I will continue to learn more and revise my thinking about DOK over the coming years. Please let me know what you think. What am I missing? What am I right about?
Very nicely said. You have done very well disentangling difficulty from complexity along with informing on complicated work.
Thank you Norman. This means a lot coming from the man responsible for creating Depth of Knowledge! I really appreciate you letting me know that I am on the right track.
Thanks for this post Robert and the many others you’ve made to help clarify my understanding of D.O.K levels. It seems so much easier to slap an adjective on a question and call it a level but that’s how we’ve missed the mark.
Lots of misconceptions out there, so thanks for clearing the water.
Thanks Graham. If only that worked!
I agree with Fletchy that this is more than just examining verbs and nouns in a standard. I think you are on the right track Robert to look at standards in context – that is the levels of application in a problem solving situation that are aligned to the standard. This is how I see your tool being of use. Great start!
Linda @learnmoremath
Thanks Linda. Yeah, definitely merits looking at each standard in more depth (no pun intended).
Thank you for this, Robert. I am so tired of conferences where intelligent people waste an hour discussing the “correct” DOK level of a single problem because of one word IN the problem. DOK shouldn’t be “dumbed down”!
Thanks Kim. I agree. In general I am not very concerned with determining the precise level of each task. I am more concerned with exposing students to an appropriate level of rigor and knowing enough about the problems to figure out which ones are more challenging.
The level of a task is not important to me but having my teachers realize that students have to think at a deeper level in some tasks is very important. I challenge my teachers to be intentional in the tasks they pose. Thank you for sharing your work. Your DOK ‘problems’ have been very helpful as we continue to learn.
Thanks Michele! We’re all in this together trying to make sense of it all. Glad they have been helpful.
Yes!- and if you as a teacher work at good tasks regularly you will discover students’ approaches could very well change the DoK level you perceived. I really appreciate the charts and posters of questions I have never used the verb chart because of the confusion better to work with a problem set and sort for all of us!
I’m thinking the focus is put on identifying DOK levels as a way to incorporate higher levels of rigor to instruction and assessment. How can we address that?
Maybe I am not certain what your concern is. To some degree you want to know how rigorous a problem is but that shouldn’t be the sole focus.
I think it helps us understand the difference between rigorous and “hard.” Finding and manipulating fractions with a common denominator of 63 or 77 is hard but most likely not rigorous. Making sure students can do rigorous work without falling into the trap of hard is important (and challenging)!
I’m concerned to see teachers being led to hide behind the DOK but not really “postholing” concepts.
Ok, so then assuming “postholing” means “authentically understanding” (I had to Google that term), yes, hopefully everyone will have deeper understandings as we continue to have more conversations on the topic.
Thank you. This is a great way to think about DOK. Indeed, your analysis could be extended to describe education as a whole. Just as raising children is complex, so is educating them.
Thanks Pedro. Yes, I see applications of this topic in so many other areas as well, for example problem-based learning approaches.
Robert-
This is well written and your analogy at the beginning of the article clearly shows the difference between complicated and complex! Thank you for sharing that example.
I lead a lot of rigor work in our district, and the main point that I try to emphasize is that rigor is no longer a list of characteristics…we must get to the meat of the task, standard, or end result to determine whether or not what we’re designing for students is rigorous. One other point we must begin discussing is the relevance of the task at hand. Students must be exposed to real-world problems to wrestle with in order to see how the school content connects (and I’m not talking about real-world word problems). So often we spend many hours designing a “rigorous” task for students to not attempt due to it’s artificial parameters.
Love this article and it’s already been shared out!
Melissa
Thanks for kind words Melissa. I agree with you and I emphasize that teachers should not be spending their time creating these tasks initially. My first problems are awful. You need to use quite a few before you start to understand the complexities of creating problems. I look forward to reading your thoughts again in the future.
Thought provoking and interesting. I appreciate the work you did clarifying complex versus complicated. Thanks for sharing.
No problem Michelle. I’m glad it resonated with you. It was fun finding just the right images to make the point.
Thanks Robert, I think this a great tool. How I’ve been planning my lessons is by linking learning intentions to the achievement standards to ensure they’re valid and then choosing an authentic task or tasks to achieve lesson objectives. Yourtool will further help me to gauge what skills I can target or develop with each task or activity.
Thank you very much.
Awesome. If it can save you time, then my goal has been accomplished. Thanks for the kind words.
DOK designates the context in which students are expected to transfer and use what they are learning. Bloom’s defines the level of thinking to demonstrate. The DOK Wheel is an inaccurate representation of DOK. Both Norman Webb and Karin Hess have refuted it. It”s also not a credible source. I invite you to check out my own graphic is designed and included in my book Now That’s a Good Question! from ASCD that depicts the DOK levels as context ceilings.
Hi Erik. Thanks for chiming in. Did you have a chance to thoroughly read what I wrote about the DOK wheel? I think that you’d see that I agree with you. I am not suggesting it as a good option. Also, it might be interesting to note who left the first comment on this blog post. Thanks.
Yes, I did – and I enjoyed your blog post. You explained DOK very clearly and well. I wish Norman Webb would speak up and share what exactly is depth of knowledge.
Thank you Robert! Great food for thought. I appreciated this blog post and the approach to thinking about depth of knowledge. It reminds me of my philosophy on teaching, which is to teach kids HOW to think. DOK is not something we can formulate or find a “quick fix” answer to. Most of all, it should be a part of our thought process when planning for rigorous and coherent lessons.
Great point Angela. Helping kids learn how to think is always going to be important.
Love the article. Do you mind if I reference it in professional development I present on the topic? It would be a great addition and help teachers to “think past” the verbs and phrases they so badly want to use.
No problem at all Lindsay. Hope your training goes well.
I love this blog post. Thank you, Robert! It is hard to talk about DOK without talking about a specific subject, and I love your DOK chart that shows the difference between DOK 1, 2, and 3. I also really love that you evaluate the current tools that are available. I don’t think there is ever a perfect system for evaluating DOK, but I haven’t loved anything that has been put out there thus far.
I think that your blog is a great way to start a conversation at a school level. A great PD opportunity would be to have a math team read your blog and then have a conversation about the examples you give about the different levels of DOK and the DOK tools. I think that a lot of great ideas could get flowing using this blog post. So thank you.
Thank you for the kind words, Karen. I have two DOK matrices for elementary (http://robertkaplinsky.com/depth-knowledge-matrix-elementary-math/) and secondary (http://robertkaplinsky.com/depth-knowledge-matrix-secondary-math/) that you may want to check out too.
Using dok formats i let my students pick questions from the several levels . Students are encouraged to make multiple attempts – this year i am using a more portfolio approach to these with an intended audience of a gallery walk at the end for parents to come in and share a math evening of their progress – with all their trials and thinking. After attempting several such questions students can pick one from each level in their presentation.,
I found myself with the unique opportunity of sitting with Dr. Webb at a dinner conference. The idea of the “wheel” with verbs came up and Dr. Webb wholeheartedly disagreed for the exact reasons you indicate here. It was interesting to me his take on that. He made it clear that it was never meant to be a simple answer for educators to use a verb that would be automatically associated with a DOK level. Thank you for your blog… cause to think and discuss!!
Thanks for validating that Dottie. The first comment on this blog post was actually from Dr. Webb and he seemed to agree as well.
Thank you! I really like the DOK flowchart. Would 3-Act Math tasks be DOK 4 level questions?
Thanks Kelly. I’m glad you like it. Yes, some problem-based lessons (or 3-act math tasks) would be DOK 4. I’m not sure I can say that all are.
This is so helpful as I try to add this to my curriculum.
Thank yoU!
I loved (and related to!) your analogies! This is a really interesting approach to understanding DOK and how it can best be used in the classroom. They last flowchart breaks it down in such a way that helps further clarify my understanding. As an teacher candidate, I concur this is great addition to add to my wheelhouse! Thank you for sharing!
Hi Mindy. Just to be clear, the last flowchart was not something I felt was necessarily a good fit for DOK. It approaches it in more of a complicated way (rather than complex). Something to consider.
I understand! At least I think I do…DOK is not a “follow the steps” strategy”. There are many influences that will affect learning and student understanding.
Thanks for the post Robert. The DOK wheel appproach is somewhat similar to Bloom’s Taxonomy. I’ll be using the wheel to deliver a better instruction. For teachers it can become a greeat gadget for class instruction and assessment especially if you perhaps have the cut-out wheel and actually spin it! By doing so you can force yourself to inquire deeper. The DOK wheel is my new “retro” gadget. I am also planning on students having their DOK wheel and for future assignments as wrap-up they will spin and respond to the resulting DOK category. Endless strategies tahnks to a great “spin”!
Hi Roberto. It wasn’t my intention to make it seem like I was endorsing the DOK wheel. I’m actually not a fan of it. It might be worth giving this post another read.
I remember when I started teaching about 22 years ago, that we used a lot of the DOK 3 framework with students. Esp. the problem about the coins. Today, I don’t feel that we ask enough of our students and there are certainly gaps with students of poverty. Today, I was asking my 6th graders to do Prime factorization. Because they do not know their multiplication tables fluidly, they totally shut down even with a multiplication chart and Sieve of Eronstothenses present. It’s mind boggling that they don’t want to at least try. It’s sad too.
That’s certainly a complex situation. Between treating kids like robots and the socio-emotional components, it’s really hard to make progress sometimes.
Nice to know that we are all experiencing the same and being frustrated about it. I teach Algebra 2 and have found that my students have no knowledge about the concepts done in their algebra 1 class.
I am also having the biggest problem with calculation using the multiplication tables. Really, at grade 10 and you cannot divide or multiply .
They count with their fingers. I am also thinking of using the multiplication chart. Hope this helps to eliminate some of their difficulties. Very sad indeed.
I asked this question about DOK in our Assessment PD day last week with not much luck in terms of answers. It bothers me that it doesn’t have a direct answer… hence being a math teacher :-). I see curriculum standards and I use the basketball analogy of I can ask a layup question (easy), Freethrow question( decent), 3 point question (kinda hard), and a half-court shot (really hard) for a particular concept/standard, but how do I know what the appropriate level of DOK is? It is an ongoing process but your article makes me feel better about how complex I feel it is as well.
Thanks
Tyson
In terms of your question about DOK level, I think I can answer that one. Look at the verbs in your standards. While I don’t have a perfect correlation, if it says “Know…” it is usually a lower DOK level than if it said “Use…” or “Apply…” Be careful with this one though as this is how we got the verb wheel. It doesn’t always work backwards though… like just saying “Use” now makes it DOK3.
It would be great to have the DOK posters but more detailed for math! Anyone have those???
Have you seen these Stephanie? https://robertkaplinsky.com/tag/dok-matrix/
Yes, I’ve already printed those out! I’m looking for dok posters like those above that are pink, purple, blue and green. I don’t think the DOK 4 poster has enough math relations on it…
Ok. Just keep in mind the concerns I express about them. Those posters look cute but have issues.
Hi Robert,
This is a great article. I’ve always struggled with the DOK charts that are around and reading your article with the included charts has now given me a different look. Thank you.
My pleasure, Yvonne. Thanks for stopping by.
Do you have other examples of DOK in other content areas such as physical and health at the middle and high school level?
Unfortunately I do not, Joseph.
Have you ever looked in to SOLO Taxonomy? It too can feel like a verb list, but I think the tasks behind it could be used here too.
No, I’m not familiar with it. I’ll keep an eye open for it.
Interesting flow chart.. the real question is when a missing skill is needed from 2 years prior, when is this material covered.?
Thank you for this blog. A lot to consider here, and to discuss!