August 2026

News

Weekly Wonder – Starting on July 18, this project has been delivering a free weekly email of “wonders” (puzzles and games) to over 1300 teachers who signed up on the EFM website. These emails are to be copied and sent to the homes of a teacher’s families. We want to make it easy for teachers to share this side of math with their families. Another use is in support of a teacher’s Puzzle of the Week program in the classroom. There are three grade level bands for the material: K-2, 1-4, and 2-5.

Website – The new version of the EFM website continues to improve and grab worldwide attention. We are now averaging over 1000 visitors a day and our visitors are coming from a greater variety of countries (due to our now having 100+ languages supported).

Studies Show … What Exactly?

The education world is filled with programs or methods that are “research backed.” There are many educational institutions that will not accept materials without studies backing them up (as if their staffs are incapable of evaluating them). The golden phrase “studies show” has become a necessary and automatic ticket for acceptance.

For my part, I have not been shy about quoting studies that show how important early math education is. As my colleague Clarence Ames likes to say, studies show that math preparation going into kindergarten is the best predictor of anything anyone cares about. More about this later.

However, I would urge you to be careful about what studies show and don’t show, and how you decide to apply them. Do the results make sense? Are there hidden limitations? Are there confounding features left out or not considered? Were the researchers driven by looking for some kinds of results and happy to ignore others?

There is a famous line from Mark Twain’s autobiography: “There are three kinds of lies: lies, damned lies, and statistics.” It’s tempting to say something comparable about some studies I’ve seen.

Example 1 – Early Math Education Studies

It has been recognized for at least 20 years now that a child’s math preparation going into kindergarten is the best predictor of their success throughout all their school years in all disciplines. Those initial studies have been elaborated on by independent research teams with reinforcing results. As further studies showed that very early math work was excellent at developing executive function, those early results made more and more sense. Developing a child’s executive function improves their ability to plan, focus attention, manage emotions, juggle tasks, and achieve goals – it also improves associated, yet surprising, characteristics such as kindness and empathy.

The world of those studies makes sense and feels very solid to me. However, what some caregivers have chosen to do with this information is not so solid. Caring parents, hearing about the importance of early math education, often decide to be “serious” about their child’s math training – they use flashcards, workbooks, and electronic gamifications of math drilling. While producing a child who will enter kindergarten with strong basic math skills, they are also producing a child who will hate math and see no beauty or joy in it. I cannot blame the studies for this kind of result, but I can ask that people think more deeply about what is really going on and how best to achieve it.

Example 2 – My Accidental Study of What was Important

When I was teaching in the classroom, I introduced problem-based learning (PBL) to my 7-12 grade school. After using it, and without any studies to back it up, I was convinced that this was a big improvement in my teaching. However, families wanted reassurance that this new unfamiliar method was not harming their children’s education trajectory. They wanted to look at measures, such as the SAT and AP scores, that were simple to research and understand. After many years our math department yielded to this pressure and did an internal study of our students that showed that their SAT scores were perhaps a little better than they had been, but were certainly not worse.

By accident, I ended up doing a small study of my own around the same time. Each year in my geometry classes I did a half-day lesson on the value of persistence and then followed it up with some reading for homework. I started that lesson by putting models of the five Platonic solids in front of the class and asked them why there were only these five. To the students, it felt like a problem they could do, but it was actually just a little too early in the year for them to have all the tools they needed. The point of the exercise was to see how long they would think about it before stopping and looking to me for the answer.

For the two years before I started using PBL, the length of time before students gave up ranged from 30 to 90 seconds. After PBL, my classes were still going strong working on this problem after five minutes! They had learned persistence and problem solving techniques that gave them the confidence and interest to keep going. They had learned to work with each other, discuss ideas, and go up to the whiteboards and work in groups. Hard problems were to be attacked and enjoyed, not seen as impossible and discarded if they took more than two minutes.

Admittedly this was a very small study. However, it was consistent year after year and it took my breath away. Here was evidence that my students were learning things far more valuable than what would show up on an SAT exam. If you only looked at SAT scores, you would not have thought much of this change in teaching methods.

You need to be very careful to look at the whole picture when you look at results of studies. Most of the time it is very hard for studies to measure some of the more qualitative characteristics.

Example 3 – Countries whose children test better

We have all seen comparisons of how countries rank in how well they do math education. PISA is an international test that is often used for this, and it is a pretty good test. The difficulty with these comparisons is that there is no way for a relatively short test to look at all the things we might care about for our children. Even if they could, how would you build a ranking on this? People want norms that are simple to understand and apply.

As you saw in example 2, there are things like persistence, grit, problem solving skills, logical communication of ideas, creativity, and even love of doing math, that are very hard to measure and compare. What good is it to produce students who are great at advancing to calculus only to be unimaginative and lacking insight when it comes to doing math?

When looking at these country comparisons, be sure to look beyond the initial scores and get a better feeling for what is really going on before being tempted to use a country’s methods.

Example 4 – Short term success?

I am familiar with an early math program for families that did a study of families using their materials. The program liked to brag that the children involved in the study made significant advances during a relatively short time frame compared to families that did not use their math program.

That sounds really good. But look deeper. One question that comes to mind is how much the families benefited from simply doing anything that caused them to focus more on math than they would have otherwise. The study didn’t compare this program with using other math programs. Perhaps other math interventions might have been much better?

An important point left out of the study was how long families stayed with the program. I talked with one researcher who told me that almost all programs of this type, including this one, have a dramatic participation fall off between 4 and 9 months. So, how useful is a program that families do not stay with? Perhaps there is a way to keep engagement going – we don’t know.

Because there is a shortage of research funding, often the only way to get a study of a program done is for a program to pay for it. Such a study is driven by wanting to show the efficacy of a certain method, so it typically has little motivation to describe all the possible shortcomings of the study once some positive results are obtained. If a program has enough financial backing to be able to fund a study that shows its methods are an improvement, how much confidence should that give you? Many studies are careful to avoid these pitfalls, but not all.

Example 5 – A better method?

Someone designed math flash cards with a fact family on each card. They did a study using these cards in some parts of Mexico. The study was very positive and showed much improvement in math fact fluency. Every school should go out and buy these cards. Or should they?

In the study, teachers were given training in this method. The use of these cards was compared to the existing minimal teaching of these math facts. It is no surprise that math fact fluency improved with the cards and the additional teacher training.

The use of the cards was not compared with other teaching methods. EFM is a strong advocate of using math puzzles and games to improve fluency in ways that we feel are engaging, enjoyable, and give deeper understanding (we don’t have a study to back that up) than using flash cards and worksheets. Using these new cards is better than doing very little, but perhaps not nearly as good as doing some other methods. We don’t know the answer to this because the study didn’t compare methods. Yet, the study might suggest to you that the cards are the way to do things (and perhaps they are).

Example 6 – EFM in Kenya

Succumbing to pressure to have studies done, EFM has partnered with another organization to do two small and inexpensive studies in Kenya. The studies lasted only six weeks, involved fewer than 100 families, and had inconsistent involvement from the families.

I can give you caregiver quotes from both studies that show wonderful things about how using the EFM mobile app changed their attitudes toward doing math in their home (were they saying what they thought the runners of the study wanted?). As much as I enjoyed the quotes and found them affirming, the actual studies were just too small and not done well enough. Still, I could hide all that easily enough and just say: “Our studies in Kenya show that using the EFM app improved math attitudes for families” – you would be unlikely to discover what I was hiding if I did.

Example 7 – Confounding factors

There are many influences that make comparing education methods tricky and make it difficult to do well-intentioned and careful studies. Students are not homogeneous. Students can vary from neighborhood to neighborhood, or even year to year in the same place. Teachers can vary tremendously – a good method in the hands of a weak teacher can lead to disaster, and weak methods forced on a strong teacher can still lead to good results. These confounding factors are not always obvious, but they are often crucial for understanding the results.

Wrap Up

I am not condemning all studies. There are many excellent studies that are well done, insightful, and add clarity and insights to complex situations. However, I am urging you to be considered and thoughtful when reading a new study. Ask yourself what factors it may be leaving out (intentionally or unintentionally) and who funded or controlled the running of the study. Also, look at the reasonableness of the conclusions and see if they are in line with your current understandings.

Finally, as someone running a small nonprofit that does not have the funding to run large, well-designed studies, I urge you to be open to using materials that are not supported with the magical phrase “research backed” – those materials and approaches might be just as good or better than their better funded siblings!