I have a whole new level of respect for anyone who has ever worked on putting together a conference, especially if he or she had to review the speaker proposals. I just finished reviewing over 240 proposals for the 2015 NCTM Regional Conference in Nashville and I’ve learned quite a bit that I want to share with you. Prior to working on this program committee, I had basically no understanding of what happened between applying to speak and the speakers being chosen. My hope is that reflecting on what I have learned will help us all have a better idea of where the process is currently at and start thinking about what can be done to improve it.

The process begins with aspiring presenters submitting their proposals to speak when the application window opens. All presenters, except for the keynote speaker, are required to submit their proposals during this window. This includes the invited and encouraged speakers (explained here). During this period of time, the program committee received training from NCTM on how to use the online evaluation system to review the proposals. This was run very efficiently by the NCTM staff member who walked us through the process of using the database.  As I mentioned in the previous post, NCTM uses a rubric that measures how well the proposals answer the questions listed on page 4 and 5 of this document under “Proposal Rating.”

In the first month after the application window closes, NCTM staff members do an initial proposal review which includes making sure that speakers who opted to apply to all three regional conferences have been properly sorted.  About one month after the application window closes, the program committee is given a two-week window to evaluate all the proposals in their queue.  Each program committee member is required to review proposals for two grade bands.  The available grade bands are PreK-2, 3-5, 6-8, 9-12, Higher Education, Pre-service & In-service, and Research.  This double checking is intended to better ensure each proposal receives a fair evaluation.

To submit their evaluation, each reviewer submits a numerical score associated with how well each of the rubric questions were answered.  Once that is done, there is a textbox where the reviewer enters their comments.  Finally, each reviewer chooses “Accept,” “Maybe,” or “Reject” for the proposal before clicking the submit button.  Once that is completed, the total points earned is summed up and the next proposal is shown.

The NCTM staff member further explained what happens at this point, “For the Regional Conferences, after the proposal is reviewed three times (once by NCTM staff and twice by program committee members), the program committee chair goes through and makes the ‘final decisions.’ At this point, the chair sees a numeric rating based on committee reviews, as well as the written comments from the comment box.  If the chair makes the decision to reject a proposal, they must choose an option from listing. The accept/waitlist/reject emails are sent out through a database merge. Rejected proposals include the reason for rejection that the chair had selected.”  At that point potential speakers receive their emails letting them know their proposal’s status and the cycle is complete.

Additionally, I was fortunate to have access to data on proposal submissions for the 2010-2014 NCTM regional conferences.  While I am not able to share the raw data, I did create two charts that I thought provided useful information for people to consider.  Click on the charts for larger sizes.

Prior to 2014, the NCTM regional conferences had only two types of presentations: sessions and gallery workshops.  In 2014 a third type of session, burst, was introduced.  So, the chart below breaks down the percentage of proposals submitted for the 2014 NCTM regional conferences by submission type.  It is noteworthy that the Higher Education and Research strands only offer session and burst options.

 

The chart below shows data for 2010-2014 NCTM regional conferences in terms of the percentage of proposal submissions by grade band.  To be clear, this is not the percentage of sessions that are ultimately accepted, just the percentage initially submitted.

 

With that information presented in a more objective manner, I am now going to share my own thoughts.  I have broken them down into three sections:

  • Experiences and reflections
  • Proposal writing tips
  • Thoughts on improving the NCTM speaker proposal review process

 

Experiences and Reflections
  • I had wondered whether proposal reviewers got to see who was submitting the proposal and use that as part of the decision making process.  Program committee members do get to see the presenter’s name, organization, city, and state.
  • It is challenging being familiar with all aspects of PreK-16 and beyond.  I found myself having to look up quite a few names and acronyms that proposals contained.  For example, I had never heard of a Rekenrek.
  • I have looked at the programs for many conferences and often wondered why some people have multiple proposals accepted and while other proposals were rejected. What I learned was that the reviewer’s primary goal is to evaluate the proposal.  What are you supposed to do when someone submits two or three really good proposals? I guess you could reject some of them, but when they are much better than others, it is hard to deny them.
  • It’s not fair but name recognition helps make a decision on a borderline proposal.  That being said, name recognition is relative to the reviewer as some well known names are unknown to others.

 

Proposal Tips
  • General
    • Have a math educator who is unfamiliar with the content of your proposal read your submission and tell you what your presentation is about. That is essentially what program committee members will be doing. If the math educator can’t clearly explain your presentation back to you, then you need to revise your submission. I read a number of proposals where I was left unsure as to what would be covered.
    • Think about whether you would want to go to your own session just based on the title and/or description. I got the feeling that some presenters were more concerned about lecturing on their research rather than thinking about whether it would be presented in a manner that anyone would want to hear.
    • Submit your proposal early if possible. The sooner you submit, the higher up you go in the queue. If you are later in the queue the reviewer may have already read several proposals like yours and will see yours as redundant. Based on the data I have access to, it appears that about 85% of the total proposals submitted were received in the last two weeks of the application period.
    • I reviewed over ten proposals on both interactive notebooks and on flipped classrooms. Even though they may all be well written, obviously they can’t all be accepted and it is hard figuring out which ones are the best when they are so similar.
    • Limit your use of generic filler text that could be about almost any presentation.  Stick with specifics about your presentation.  Here are some examples of filler text:
      • This workshop will present hands-on activities and projects that inspire the challenged learner as well as challenge the inspired learner.
      • Teachers can motivate their students with fun and relevant content!
      • Participants will be actively involved in an activity that can easily be incorporated into all grade levels.
      • Come ready to experience an activity that gets learners actively thinking about these concepts!
    • I realize that proposals have character limits but don’t assume that everyone knows the acronyms you are using. For example, CRA got used a lot and I did not know what it meant.  People deciding on whether they want to approve your proposal or come to your session may not either.  It makes much more sense as Concrete to Representational to Abstract.
    • Similarly, if you reference a person, don’t expect everyone to know who he or she is. For example, I love Peg Smith’s Five Practices but the reality is that if I mention her in a proposal, many reviewers evaluating my proposal and teachers picking my session from the program may be unfamiliar with her.
  • Proofreading
    • Triple check your proposal’s content, grammar, and spelling before you submit. There were numerous spelling and grammar errors that detracted from the proposal’s overall quality. For example, if you’re doing a presentation on online learning and are spelling “Khan Academy” as “Kahn Academy,” it doesn’t reassure the reviewer that you are familiar with the content.
  • Grade band
    • Consider picking a single specific grade band. If you pick a broader grade band like grades 3-8, your proposal will either be evaluated by the 3-5 person or 6-8 person. So picking a specific band gives you more control.
    • About 5% of the proposals were submitted for the wrong grade band. For example there was a Geogebra based analytical geometry presentation in the PreK-2 band. While that may not necessarily eliminate your proposal, it certainly isn’t going to help it.
  • Speaking experience
    • For speaking experience, make it short and sweet. Reviewers are trying to quickly determine if you have relevant presentation experience. Putting lots of text makes that much harder. Don’t list papers you wrote or workshops you gave for your district. Just make it clear that this is not the first time you have ever presented at a large state, regional, or national conference.
    • If it IS your first time presenting, seriously consider doing local or state level conferences before you apply for NCTM. It will give you valuable experience and make your proposal look much stronger.
    • Having no speaking experience does not rule out a really good proposal but does make a borderline proposal seem less likely to be accepted.
  • Commercial organizations (and avoiding sounding like one if you’re not)
    • Businesses doing infomercial style presentations about their products have to pay to present at an NCTM conference and are not allowed to use the traditional proposal submission process. If your proposals sound like it may be a commercial presentation, even if it is not, it may get rejected.
    • If you mention using free resources like games or apps, make sure you state that they are free resources or it might look like you are doing a commercial presentation.
    • If your organization is listed as a business and not a school district, it begins to look like it will be a commercial presentation. This is especially true when your presentation mentions that you will be using educational products and your company just happens to sell that product.  If that isn’t the case, make it obvious.

 

Thoughts on Improving this Process
  • (Lack of) using data
    • The elephant in the room is that reviewers are not given any data from applicant’s previous NCTM presentations.  It pains me that in an age where assessing students to make decisions is so important, we are not doing the same when it comes to assessing presenters.  How do we reconcile the reality that the best presenters should be presenting at NCTM conferences with the reality that there are still bad sessions at NCTM conferences?
    • Clearly not every applicant has presented for NCTM (and therefore has evaluation data available) but some have and without that data, there is no way to tell the difference between a presenter who has done mediocre NCTM presentations and someone who has done an excellent job.
    • Dan Meyer recently blogged about how the 2014 California Math Council (CMC) South conference digitized their presentation evaluation system.  He stated that, “conference committee chair Brian Shay told me via email, ‘Historically, we use the data and comments to guide our decision-making process. If we see speakers with low reviews, we don’t always accept their proposal for next year.’”  If this is already happening at the state level, why not at the national level?
  • Evaluation rubric
    • The rubric used to evaluate proposals needs updating.  This is more a realization that the things we value are changing than a critique that the current rubric is bad.  Here are three specific concerns:
      • The “Focus on mathematics” category requires a reviewer to evaluate “What tangible mathematics will this presentation showcase?”  I totally understand that question, but what about proposals on increasing student discourse, parent/community involvement, interactive student notebooks, or writing about math?  Those are all valuable, even if there is not “tangible mathematics” and it doesn’t feel right scoring them down.
      • Speakers who submit proposals for gallery workshop presentations are asked about the manipulatives they will be using.  This seems strange as you don’t always have to use manipulatives to be actively engaged in hands-on mathematics.
      • One decision we have to make is whether a proposal is grade level appropriate. For example, what do you do about a grade 6-8 proposal on fraction operations?  California used to have fraction operations as a middle school standard but now it is primarily part of grades 3-5 under CCSS (except for dividing a fraction by a fraction in 6th grade). So, if someone submits a proposal on fraction operations for the 6-8 grade band, it isn’t clear whether that is the wrong grade band as it may be the appropriate grade level in non-CCSS states and/or Canada.
  • Program committee
    • The people selected for the program committee greatly influence the presenters ultimately chosen to speak at the conference.  As a hypothetical example, if the majority of the program committee work in the same grade band, live in the same geographic area, and collaborate in the same manner, then the scope of who and what they are familiar with is limited.  We can’t realistically aspire to have a more inclusive organization when we don’t do the same with our program committee.
    • Accordingly, it is critical that program committees are diverse in terms of:
      • Grade bands (at least one person from each of PreK-2, 3-5, 6-8, 9-12, Higher Education, Pre-service & In-service, and Research)
      • Geographic region (at least one person from Canada, the northern, southern, eastern, and western regions of the United States)
      • Gender
      • Ethnicity
      • Collaboration style (brick and mortar as well as online communities)
      • Experience level (newer teachers through veterans)
      • Etc.

 

My hope is that this blog post is a conversation starter about working together towards positive changes.  It is clear to me that many people work very hard on doing right by math educators.  As always, I believe that if we collectively brainstorm improvements, the final result will be much stronger.

What other thoughts and concerns do you have?  What was I wrong about?  What have I missed?  What other questions do you want answers to?

23 Comments

  1. Spot on, Robert. Reading through this I found myself nodding in agreement. It was difficult to make calls on some presentations. Some were obviously good or obviously bad, but a lot were in the middle. I was surprised to find that some sessions I thought did not sound good at all, but then when I got down to the person’s speaking experience, I saw they had spoken on the same topic at other NCTM conferences. That concerns me. I think having evaluations of presenters at NCTM is a necessary piece if we are going to ensure we have EXCELLENT presenters, but I think it also comes down to your point about needing reviewers from every grade band, experience, etc. If the person reviewing PreK-2 does not have a background in PreK-2, they can easily accept a session that is well-written but is not incorporating new focuses in early math ed. I definitely felt that way when having to review the 6-8 proposals. It’s been a long time since I was immersed in that grade band and I found myself not knowing for sure if the topic was appropriate for the new focuses in the middle school.

    One thing I would add is that there were multiple times I wished there was a sort/filter feature that I could find all the sessions that mention “number sense” in PreK-2. Then I could pick the best ones, score them high, and score the rest in relation to those ‘best ones.’

    My last thought….after the review process, one question I am left with is; How many of the proposals were from actual classroom teachers? Or district math coaches? It seemed like the majority of presenters were from universities or companies (some product, some consulting), but I don’t know if the data is there to back up my general sense.

    It was a very enlightening process, overwhelming at times, but it was a great opportunity to have some insight. Plus it has been a great way to connect and learn from other great math minds…like you.

    –Christina

    Reply

    • Those are two very good points. Your first suggestion would be very useful in helping move from “the FIRST proposals on a topic get accepted” to “the BEST proposals on a topic get accepted”.

      Your second suggestion is also strong because I often wonder why so many presentations are from people who are no longer working with children. We definitely want a balanced set of speakers.

      Thank you!

      Reply

  2. Thank you Robert for taking the time to share this valuable information. As you know, I do not have any experience presenting outside of my district. However, I am getting ready to apply for GLAMC in February. I know this information will be helpful when I am writing my proposal.

    Reply

  3. As we’ve discussed, I’m on the 2016 NCTM Annual Meeting Program Committee and our experiences and thoughts on this are similar. Thanks for writing these three posts. The first was helpful and timely as I prepared to meet the committee in September.

    Obviously, I haven’t taken part in this process yet, so I’m reluctant to give proposal tips to others at this time. Nevertheless, I have submitted proposals in recent years and this peak behind the scenes was fascinating and, at times, surprising. As a potential speaker, I’d write a description that was intended to pique the curiosity of potential attendees, not spell out what attendees would be doing. This is what I want when I read the program. What’s the question that drives your workshop/session? If I, too, am asking myself this question, I’m at your session. But tell me the answer to this question in the description and I’ll find another session. Having said that, it is important that the educator evaluating your proposal has a very clear picture of what attendees will be doing. I just question whether the description that appears in the program is the place for that; there are other parts in the online application where this information would come out. All of this is to say if I want to increase the likelihood that my future proposals are accepted, I may have to change how I view my description. I wouldn’t have known this without having this experience.

    You’ll be happy to know that many of your thoughts on improving this process were talked about at that initial weekend of meetings that I attended. There will be changes made to the rubric as a result of these discussions. The “tangible mathematics” criteria concerned us all. A session on rational expressions likely scores higher here than a session addressing access and equity, or mathematical discourse, or elements of task design, etc. and this didn’t sit right with us.

    The gallery workshop vs. session thing was something I was prepared to bring up. Nothing against manipulatives. Big fan. But if I want attendees to participate by solving problems and discussing pedagogy, if I want to circulate and then include participants’ ideas — as I would in the classroom — then I shouldn’t have to say that I’ll be using pattern blocks to be able to do so. The good news is that most rooms at the convention centre in SF will have tables. That should be reflected in the online application.

    I’m glad that you bring up the grade bands. As a Canadian, no states, nevermind Common Core State Standards. So I’ve learned to largely ignore them. You’ve identified the biggest difference in terms of concepts and content: fraction operations show up later (Grades 7 & 8) in our standards than in CCSSM. (Integer operations show up in 7 & 8 here as well, which I believe is later than in CCSSM.) My other concern with grade bands has to do with narrow-mindedness. One of the most well-received activities that I’ve done with secondary math teachers in the last year was to have them explore each of the four operations across the grades in terms of fundamental meanings, concrete/pictorial models, symbolic representations, algorithms, etc. Not one teacher complained “Why do we have to learn about two digit multiplication?” or “What’s subtracting whole numbers have to do with subtracting radicals?” They found this glimpse at how these concepts were taught in K-7 fascinating and made connections to these concepts as they appear in 8-12.

    More later…

    Reply

    • I am very curious to see how your experiences in evaluating proposals ultimately compare to my own. My default personality trait is being optimistic that things will always work out so I really hope we can improve the process. Hopefully you will already be seeing some of those changes.

      Reply

  5. Thanks for writing about this very important issue. As background, I was on the 2004 Program Committee for Philadelphia, and worked as the Director of Research at NCTM from 2010-2012. Disclaimers out there, I agree with many of the issues and some are still things that were brought up in 2004! My concerns in 2004 started with just the volume of proposals to review in that 2 weeks. Bias does start to creep in as you are rushing to get them done. As a staff member who did the reviews, it was just as bad, because we did them on top of our regular work.

    But I’d say overall the annual meeting is too big. There are too many sessions to wade through and too many of them are not of high quality. One of the difficulties in using reviews of past presentations is that so few attendees actually submit reviews. It is hard to not invite someone back when they only have 2 reviews, both negative, from their past 3 presentations.

    As for the number of teacher presenters, those numbers have been going down, but that is a function of the ability for teachers to take off to go to the conference. They have to commit almost a year before the presentations. They may or may not know if they are going to be in the same school or have the same principal who approve the time away, often right in the middle of the state testing season, which also affects supervisors, who may have attended NCSM at the beginning of the week and cannot be away that long. That is a tall ask and one of the reasons for moving the dates from the spring. When you add availability of travel funds, it can be impossible for many teachers to commit by May 1 for a possible presentation in April. On the other hand, that is much easier for faculty, and they have many more incentives to participate, as these kinds of presentations can count for promotion and tenure, depending on their university. I actually don’t think this is a major issue as there is no evidence that practicing classroom teachers do a better job at these presentations than others. That would require further looks at the data.

    I am glad to hear that things are changing, slowly but surely. But part of the problem is the need for NCTM to better embrace technology in a range of things that impact the ability to provide the data and ensure there is better data collection. And some is the community being less conservative about change. One anecdote to illustrate. In my position, I ran the NCTM Research Conference. In our meetings with the contractors for AV, I refused to give the option for overhead projectors. We are the only meeting this major contractor has that still uses them. They only keep them just for NCTM. I offerred document cameras, but no more overhead projectors. The folks in charge of regional conferences and annual meeting were too scared because when they tried to eliminate them, there was an outcry. Many of the things that are not working stay because there is a vocal community that wants them. If others want change, they have to be just as vocal, and within the organization.

    Reply

    • Thank you Karen. You bring up numerous interesting points:

      – It was certainly a lot of proposals to cover in a short period of time. Not having more experience than I do, I didn’t know why there was such a hurry to finish the process in two weeks when the conference was about a year away.

      – Though I have much less experience than you, I don’t necessarily believe that the conference itself is too big but the proposal review process does need retooling to more efficiently and accurately select the best proposals.

      – Your point regarding potentially insufficient data is well intended. Again, in my limited experience, perhaps this is less of a concern with all the modern data-entry options. For example, at the CMC South conference there was plenty of data entered.

      – You bring an interesting perspective to the issue of less teacher presenters. Definitely worth reflecting on.

      – I truly believe that the current leadership with Diane Briars and Matt Larson are exactly the kind of people we need. My limited experiences with Diane showed me her number one priorities involved what was best for teachers and NCTM. Anything about prioritizing her own personal views were non-existent. I believe that Matt will continue down this path. Again, I am ever the optimist but I think the timing is perfect for this to progress.

      Reply

  6. I read back on the “part 1,” including the comment that NCTM was simply too expensive for many the classroom teacher.
    That and your comment that the NCTM folks hadn’t heard of major movers in theMTBoS world says that they may well be two different worlds.

    Reply

    • Hi Sue. From my perspective, I see this as more of a formative assessment than a summative one. I “suffer” from the kind of optimistic naivety where I honestly believe that we can work together to address these issues.

      The first step though is clearly defining what those issues are so we can think of the best solutions. Just like Math Practice 1, our problem solving strategies may not work the first time, but we can adjust and try again until we figure out a solution.

      Reply

  7. Thanks so much for taking the time to do this. It is wonderfully clear, detailed, and incredibly helpful to everyone involved in speaking and building a program.

    Reply

  8. What a complete explanation of the process for regional meetings. I have been on regional and annual program committees since the mid-80’s (including chairing the most recent Cleveland regional). I must say that when you read this from the perspective of 30 years of history — including the original work was all done with paper copies of proposals and as recently as 15years ago, proposals had to be in more than a year before the conference actually took place. — NCTM has come a long way (and still needs to work on becoming more efficient! Luckly we have a great staff in conference services.

    A few thoughts…I have always believed that NCTM is the teacher’s organization. Getting teachers involved on programs, committees, authoring for journals and publications is a responsibility of the NCTM board. Release time from schools for any of these activities has become more and more challenging for teachers and NCTM is trying to address that as it looks at its governance structure. The expense of attending an annual or regional meeting can also be prohibitive to classroom teachers who do not get financial support from their districts. These are all important issues that NCTM must address if we continue to be the voice of and support the work of teachers.

    Evaluations of speakers is always an interesting issue. CMC has a culture of evaluations — both when they used paper evaluations and now with online evaluations. NCTM has a broader audience and while some people will complete evaluations (if they can find them in the app) it often turns out that most of the evaluations come from people who didn’t like the session. At the same time that is why it is so important for the members of the program committee to have a diverse background in geography, grade level experience, and expertise around current and critical topics.

    Thanks for getting this information out and getting the conversation started. I know the NCTM staff and board will look at this discussion and take any suggestions for improving the quality of our conferences very seriously.

    Reply

    • Thank you Linda. I really appreciate your perspective because it is very important to realize how far we’ve come rather than solely focus on what still needs to happen.

      I am wondering whether the culture of evaluations has changed significantly as I would think that nowadays more people would leave all sorts of reviews.

      We are all looking forward to continuing this conversation. Thanks.

      Reply

  9. Lots of good thoughts and reflections. I want teachers (especially newer teachers) to be more involved in NCTM. Conferences are a good way to start that process, so I’ve mentored potential speakers about the process. Chris makes a good point about the description. For me the objectives are the most important part. . The Focus on math is important, but my takeaway has always been some sessions can’t address pure content. So, a session about equity will score the same as an equal level proposal on content, for me – but the latter’s focus has to talk about math pedagogy or learning principles. That is an issue that is being discussed so that it is clear for all reviewers. Workshops – they don’t need manipulatives, but there should be something to differentiate a session from a workshop other than there being more time. To me, a session in which people discuss problems and solutions isn’t what I consider to be a workshop. There are new room configurations which have tables (round or rectangular). I’ve had these in two regionals this year. It made my talk much more inviting for discussion and collaboration. I still consider what I did a session, though.
    As someone who has selected program committee members, NCTM has in place guidelines that include all of your suggestions. New teachers, diverse backgrounds, geographic spread are all part of the process. I hope your committee met those goals. If you remember “my” committee from our Reston meeting, I hope you could see the team for that regional met that balance.

    Feedback is a problem NCTM is trying to deal with. The feedback I’ve gotten from sessions is minimal – a number and maybe a sentence – nothing that helps me improve. From a session of 100+, I’ve never gotten more than 3 or 4 comments from attendees. I think the main thing is making feedback opportunity accessible, having clear expectations of what is helpful for speakers and future committees, and, possibly, making time for attendees to do the evaluation.

    Teachers and proposals – a tough one. I’ve had the opportunity to work with a lot of classroom teachers. I frequently tell them everyone would love to learn about their great ideas, and that they should share them. First, the teachers seeme shocked that anyone would want to listen to them. Then the reality of submitting a proposal over a year in advance and wondering if funding will be available sets in. The timeline is an issue. I just presented with a newer teacher in Richmond. He did a great job and enjoyed the experience. It was over a month after the cut off for next year’s regionals. He is ready to go again – but has to wait until next summer for the opportunity to apply. Having a shorter time span between proposal and conference would be very helpful for teachers.

    Your suggestions and comments are great. My experiences at NCTM are that the organization is constantly trying to improve its services/value to members. I think you can rest assured that this blog and its posts will have a positive impact.

    Reply

    • Thanks Fred for all the feedback. It sounds like my thoughts mirror your own experiences. I am sure you had a diverse committee but I didn’t really get to talk with all of them to learn more about their experiences.

      I am very curious as to why NCTM conferences tend to have minimal feedback. At the recent CMC South conference I had about 200-something people in attendance and got around 90 digital evaluations (http://robertkaplinsky.com/wp-content/uploads/2014/11/CMC-South-Evaluations.pdf). I wonder if it is worth reflecting on the method used to collect the data. I certainly believe it is possible and CMC South is currently using it.

      Thanks again!

      Reply

  11. Hi Robert,
    I’m so glad you are naively optimistic. Me too. I often fortify myself with the believe that the naive optimists help move change along because they don’t dwell under the misconception that what they are describing as possible can’t be done. The naive optimist believes it just hasn’t been done yet. II’ll keep fighting the good fight if you will. I am always inspired and moved that your choice of weapons in this fight are such thoughtful and gentle ones.

    Big hug from Georgia.

    Reply

    • My assumption has always been that something happens with the presenters that makes them unable to present at the conference. Perhaps it is a family emergency or transportation issues.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Post comment