Ricochet is the best place on the internet to discuss the issues of the day, either through commenting on posts or writing your own for our active and dynamic community in a fully moderated environment. In addition, the Ricochet Audio Network offers over 50 original podcasts with new episodes released every day.
The Real Epidemic: Shoddy Research
It seems to me that we have an epidemic of poor quality research in this country and around the world. This is causing us to make bad decisions and bad public policy with an unfounded sense of confidence and righteousness.
I am a history teacher at an international school, and we recently had a mind-numbingly dull “professional development” day where we paid some grifter education consultant thousands of dollars to educate us about the newest research-approved methods of assessment. (My suggestion that we cancel the “professional learning” day and use the money for a schoolwide party was unfortunately rejected by the administration.)
The professional development day began with this consultant zooming in and lecturing us about the transformative power of new assessment strategies, all of which could be found in his overly priced books and a subscription to his consulting group webpage. Of course, everything he told us to do in his lecture was “research-based.” There was simply no room for disagreement, he claimed, because educational research had already proven that these new strategies worked to increase student achievement.
Now, educational research is notoriously unreliable. It is virtually impossible and often unethical to design controlled experiments on students as test subjects. Any teacher with just a smidgen of intelligence understands how to manipulate test cohorts and data collection to achieve any result he or she wishes. I could easily invent some new bogus strategy, conduct a research study on my students to show that it works, and then work the professional development circuit as a consultant for a few years and retire in comfort. But despite working in education, I still have some self-respect, so that is not an option for me.
The problem, it seems to me, is that we get caught in the bad habit of trusting individual research studies without considering the larger aggregate data trends. If it’s true, as our consultant claimed, that these strategies were revolutionary and increasingly adopted by schools across the western world, then why does the aggregate data show a decline in student achievement in reading over the past thirty years? This data set shows that the reading scores for high school seniors have dropped consistently and considerably since 1992. The scaled score for all students on the reading portion of the National Assessment of Educational Progress (NAEP) was 292 in 1992. By 2019, the average reading score dropped to 285. If these new research-proven strategies were as effective as advertised, shouldn’t we see an aggregate increase in reading performance as these practices become more widely adopted?
This fetish with the sloppy individual research study has been most damaging with covid policy. All of the tinpot tyrants can point to some contrived research study about the benefits of masking children. For example, this study “proves” that masks work incredibly well to stop disease transmission, but this study failed to mention that the masked schools which had fewer cases also met fewer days than those unmasked schools with more cases. Thus, it is impossible to reliably conclude that masks are the relevant variable here and not time in session.
Again, like in education research, the aggregate covid data shows that many of these individual research studies are just rank scientism masking political preference as truth. If masks are as effective as some researchers claim, shouldn’t we see differences in case counts and trends based on mask policy? Dr. Redfield, formerly of the CDC, ridiculously claimed that masks were our best defense against covid. Yet, aggregate data sets suggest that masks make little difference. I recommend everyone play “Pin the Mask Mandate on the Country” at Tom Woods’ Covid Chart Quiz website to show how little masks actually accomplish. Additionally, this data set compiled from Emily Oster at Brown University shows mask mandates have no effect whatsoever on case counts in schools.
I am increasingly tired of charlatans claiming the moral high ground on truth because they have “research” to back up their claims. There is a ton of garbage research emerging from the academy. The press is generally too stupid and/or uninterested to look at methodologies of these studies, so they often report invalid conclusions as fact when it confirms their political priors. Politicians and bureaucrats then use these laundered findings to justify bad public policy and demonize critics as ill-educated rubes who don’t make informed decisions.
We need more honesty and humility in academia, especially when findings at the micro level do not conform to macro trends. In the words of Justin Trudeau, this low-grade and misleading research is “unacceptable” and “it has to stop.”
Published in Education
Amen. Amen, amen.
It’s not just education. The PTB are relentlessly using worthless “research” to justify their authoritarianism. As you begin to show, every one of the studies that the CDC quotes to justify masking has fatal flaws which, in less politically-charged times, would have led to their being scrapped before publication.
Maybe it’s true, and maybe that’s why.
Not all revolutions are good.
Preach!
They’re not even making decisions based on relevant expertise. The experts know one section of one subfield of medicine or sociology. They don’t have expertise in the dozens of other areas of medicine and sociology that they would need to know, for example, that masks, mask mandates, vaccine mandates, or lockdowns are a net good.
The bigger issue is that they’re making moral decisions based on supposed medical expertise.
The deaf can’t read lips.
Small children are slower to learn to talk.
Education is disrupted.
Economies shrink.
The poor suffer and suffer–around the world. Millions risk starvation.
The unvaccinated are set apart as second-class citizens, considered unworthy of medical care and common decency.
Freedoms of speech, association, and religion are abandoned.
Lives are lost as side-effects of despair, poverty, and interruptions to regular medical care.
And any cost is deemed acceptable if public-health officials merely speculate that it might save some lives from Covid.
Fair point. If equity is the goal, as it increasingly is in education, then bringing down the top is the easiest way to do it. But I tend to think that their motives are genuine. I think they believe their own baloney and just can’t bring themselves to look at the reality that their ideas are a failure.
What bothers me is that when these flaws are exposed, the “experts” stand by the findings of their work, even when it is at odds with observed reality. I am pretty sure that the lefty governors are now relaxing restrictions only because they are craven politicians. Deep in their heart they still probably believe that masks and lockdowns work because they have research to prove it!
I dug through the CDC website in Feb 2020 and read their papers on masks. They had good research back then. The best paper was from 2015 and explained that masks were not effected for the general public against respiratory viruses, because it takes training to wear a mask correctly. The conclusion was that masks were an ineffective method to control spread of a respiratory disease. That paper got memory-holed when Covid came for political reasons.
The CDC spent many years studying drugs to see what would work against SARS-1. Several promising drugs were identified as being effective in the lab (in vitro), but all that research got memory-holed when Covid came for political reasons.
This was not shoddy research that put stuff down a memory hole. This was not shoddy research that redefined what a “case” was. This was not shoddy research that mis-used PCR tests in a way that the inventor said was inappropriate. These were political choices to create a scamdemic and the public health and medical industry mostly went along with it.
Say it, Brother!!!
There seems to be some kind of bizarre shift in the tone of your essay here, from dead serious to total sarcasm. Perhaps you could clarify. Justin Trudeau referenced in a concluding paragraph is a real curve ball.
Both equity and wearing masks are topics of discussion here but their nature and consequently any related research is very different.
What makes you assume that “assessment strategies” have anything at all to do with student performance results? We discover daily what students are actually being taught and it has little to do with reading.
I am not assuming that assessment strategies have anything to do with reading scores. I am criticizing the peddlers of “education research” who claim to have magic bullets to enhance student achievement. Some schools implement these “research-based” strategies, but there has been no overall increase in reading scores. If these strategies worked as advertised, achievement should increase. It hasn’t.
I don’t think this is a bizarre shift. Justin Trudeau is one of these phoneys who claims the science is settled, which is why he considers opposing views “unacceptable.” He is part of this class of arrogant busybodies who claims a monopoly on science. Like education consultants, he deserves to be mocked mercilessly.
We need
moresome honesty and humility in academia, especially when findings at the micro level do not conform to macro trends.Yes.
I have nothing–NOTHING– good to say about academia. They have lost credibility in so many areas that I think they’re irredeemable.
I think we have enough “research”, now. It’s time to give the poor, suffering tax payer some respite.
There is a serious problem with the replicability of soft science findings. A good experiment should be replicable, but many are not.
It could take a generation to fix the mess we are in.
Education research other than intersectional/Culture/Race studies has about the lowest standard in academia and it has for ages. Try getting a drug approved with their level of research methodology and you would be going to jail for mass manslaughter.
Um all the studies showed they did squat. Even the Demark study in 2020, was not statistically significant. They were completely ignoring all the public mask studies which they had published in their own periodicals or only using Hospitals studies.
It was only lab simulations that showed they worked (like a double mask). Only a politician or a moron (but I repeat myself) would think a closed environment lab simulation should ever be used for public policy. A lab simulation can help tell you what experiments you should set up first to save money but that is about it.
This is absolutely correct. Yet, the educationistas have true belief and never admit that their research-based findings are based on such suspect methodologies.
I agree with your post, especially the education consulting racket.
I worked at a very good, very effective, Catholic all-boys high school for 18 years. It seems as though every five years we were presented with the latest fad that would improve our teaching effectiveness. Usually, the school administrators would pay lip service to it until it died its natural death in a year or two. They were pretty good at protecting the faculty from such nonsense and letting us do our job.
One area where they did venture into the consulting morass concerned professional development. The previous method was very much sink or swim. A new teacher would be assigned a mentor, who usually did little more than tell you the copy machine codes. FNGs were pretty much on their own to figure it out.
Quite rightly, the administration thought the school could do a better job in this area, so they put an old hand, Mr. V, in charge of developing a program. Mr. V was a VERY highly regarded teacher by all who worked with him (including me). He taught my two boys, who both learned a tremendous amount and loved his classes. Well, Mr. V partnered with a company whose instruction was supposedly all research-based (they even put the word “research” in the company’s title). The training would take place over the course of a year (one week prior to school starting, and two days of training in the latter part of the year). All new teachers were required to take the course as a condition of employment, and veteran teachers were allowed to take it. I took it.
The company sent Ms. M to teach the course. The course had some good tidbits in it, mainly classroom management type stuff; but overall, its central point seemed to be that teachers should adopt many different “strategies” (all research based) to cajole students to learn. The focus seemed to be more on entertaining the students rather than teaching. That approach did not go over well with the veteran faculty in the course.
Worse, Ms. M, who had a credential list two feet long, stated early in the course that we should never use lecture as an primary instruction method, or at least use it sparingly. She then proceeded to teach strictly by lecture for the entire week. The faculty were, to a man, not impressed. But Ms. M had only one person to please, and that was Mr. V, who wrote the checks. She did and was thus was invited back for several years until money for the project dried up.
Thus, the old adage should be revised to read:
Those who can, do.
Those who can’t, teach.
And those who can’t teach, consult.
The cost and range of “education consultants” has grown exponentially.
When a school hires a consultant it is an admission that the leadership can’t do their jobs.
The way consultants run their workshops is embarrassing. I think most teachers would be fired if they ran their classes the same way most consultants run their trainings.
I refuse to call them school leaders exactly for that reason. They are administrators. But it is also a racket, and not just incompetence. School “leaders” support the consultant racket because they often aspire to be enter this lucrative world themselves. Gotta keep the gravy train rolling.
They dont want to admit that Parents and IQ are the two most important variables by a large factor over teachers. If I was a law makers. I would force all publicly funded education studies to normilzes aka require those variables be considered. I have a feeling there would be a massive comming to Jesus. That the whole admin and EDU phds industry would hate the conclusions because it would prove there work is basicly meaningless.