The Attack on Grad School Entrance Standards

 

The social justice activists have been making more waves in the sciences recently, such as my field of astronomy. The latest is a push to get rid of the use of standardized test scores (the Physics Graduate Record Exam, or physics GRE) in admission to physics graduate school. (Read the statement by our professional association’s president.) The claim is that the GRE is poorly correlated with success as a research astrophysicist and is more correlated with sex, race, and ethnicity.

Some astronomers have tried to quantify this by doing a study of “successful” astronomers; i.e., those who have received the prestigious postdoctoral research fellowships after grad school. They were able to use 149 responses (55%) out of the 271 questionnaires sent out. The rest didn’t respond or didn’t provide their GRE scores, and the study did not try to account for the biases this introduced.

The authors state that test scores don’t predict your success by this (limited) measure, and that the successful astronomers’ test scores “spans the full range of percentiles.” (If you are in the 30th percentile, you scored higher than 30% of the test-takers that year.) But if you look at their distribution of fellows’ scores (Fig. 2, right side from the study above), there’s actually a strong trend towards high scores among the fellows. Yes, there are a few who have low scores (that lets them talk about the “full range” of scores being represented), but it’s certainly not unrelated. The higher the score, the more fellowships awarded. I put the numbers into a spreadsheet and found that the Pearson correlation between the number of fellowships awarded and the score was 0.90, a significant correlation for this sample size. I’ve put my chart of the score distribution below.

Physics GRE distribution of all fellows

So how are they claiming this doesn’t correlate? They compare these results to an “expected” distribution of fellows’ scores if the score were correlated and if all grad schools rejected applicants who scored below the 50th percentile (Fig. 2, left side). This last part does not reflect reality. Some grad schools informally put in a minimum score cutoff, others may scrutinize students’ applications more carefully at the low-score end, and others might not use it at all. Different schools have different average scores among their grad students, and there’s no indication that any school puts a cut-off at the 50th percentile. Different schools have wildly different admissions practices, so the comparison case is already known not to exist. What good is this comparison, then?

Now take a look at the breakdown of the fellows’ scores by sex:

Screen Shot 2015-12-17 at 9.02.03 AM

I made these charts by separating the numbers in the paper’s Fig. 2. What I find interesting is the differences here. The men’s score distribution shows a very clear trend: a higher score is more likely to get you a prestigious fellowship. (Correlation=0.94) But women’s results show a peak in the middle percentiles; i.e., most of the women who got prestigious fellowships had roughly average scores. (We might also wonder what’s going on with the “other” and no-response sex choices, but a non-response has selection biases.)

Now, it’s impossible to know what the average women fellows’ scores being average means without knowing the distribution of all women’s physics scores. Though, I couldn’t find the latter distribution on the web, I wonder if there’s any preference going on in the selection of prestigious fellowships. One possibility is that the reviewers making the choices are — consciously or unconsciously — trying to bring more women into the field by awarding them fellowships even when they have lower scores than they expect of men. Or, it could simply be that good research skills are normally distributed among women of different test scores, while these skills are strongly correlated with test scores for men. It would be an interesting question to answer, but this study doesn’t do it.

So, if schools were to stop using test scores in admissions, what would they use? Undergraduate grades are an obvious choice, but it’s hard to control for the different levels of rigor among different colleges.  Letters of recommendation are another, but those are very subjective, both in how they’re written and how they’re interpreted. And I’m guessing that very few negative letters of recommendation are written. One suggestion is that applicants write a “personal statement,” but that might be the worst of all ideas, if we think of the awful, inflated, navel-gazing drivel already being written for undergraduate admissions. Our professional society will even consider recommending schools to explicitly take race and sex into account and, of course, I think that’s a big step backwards for society.

My own opinion is that the test scores be one part of the application and be used in conjunction with undergraduate grades, as a way to correct for the different rigor of different schools. Letters of recommendation should supplement this, and there are other pieces of evidence that could make a difference: undergraduate research projects, papers already published, and so on. But that’s pretty much how things work already.

Here’s my experience: I think I scored in the 36th percentile, which was much lower than I expected.  But my advisor said not to worry, because the percentiles are skewed by the number of foreign students coming in (they have already got a master’s degree before they start grad school in the United States, and schools tend to take that into account). The results from two of my applications might be instructive.  School A is one of the top graduate programs for astronomy. School B is highly ranked in physics overall, but not as prestigious for this program. My scores were almost exactly the average for school A but well below the average for school B.  Yet school A rejected me very quickly, while school B gave me the best offer of any, and I accepted and succeeded there.

Published in Education
Like this post? Want to comment? Join Ricochet’s community of conservatives and be part of the conversation. Join Ricochet for Free.

There are 20 comments.

Become a member to join the conversation. Or sign in if you're already a member.
  1. Mike H Inactive
    Mike H
    @MikeH

    My goal was to get into the 40’s. I got a 41. :)

    • #1
  2. Hoyacon Member
    Hoyacon
    @Hoyacon

    I hope your thoughts/work on this gets wider dissemination.  I’m only a backyard telescope kind of guy, but isn’t Ms. Urry rather well known as a gender activist?  Her presidency is a platform.

    • #2
  3. PHCheese Inactive
    PHCheese
    @PHCheese

    My goal was to get the heck out of college. I had no idea about grad school. My college was new and taking the test was mandatory. The were trying to set a base. I had a triple major, economists, history and political science.I mostly had A,s in these courses. I am dislexic and sucked at math and language. I also worked at least 50 hours a week. My over all QPA was only 2.8. I and my department heads were shocked when my results were the .99 percentile. Georgetown school of Foreign Service offered me a slot. Instead I volunteered for the draft and the Army. I don’t regret that decision but I do think what my life would be like if I had accepted. This was in 1967.

    • #3
  4. Richard Fulmer Inactive
    Richard Fulmer
    @RichardFulmer

    The fact is that standardized scholastic tests are culturally biased; they’re biased toward information-age cultures.  In our world, people who are able to deal with data – words, mathematics, and symbols – tend to do better in academia and in jobs that deal with information.

    These tests would clearly be out of place in stone-age or agrarian societies in which different kinds of knowledge and skills are important.

    While we could reorient the tests to check for knowledge of, say, “street smarts,” the possession of such knowledge would be of only marginal value for astronomers, physicists, and other hard-science professionals.

    The reason that the right and the left are talking past each other on the topic of standardized testing is that they don’t agree on the purpose of the tests.  Those of us on the right see them as a means of determining probable success or failure in school and on the job.  Those on the left see them as a devious way of discriminating on the basis of race and sex.  Therefore, the left wants to either eliminate them or modify them so that they don’t get in the way of hiring on the basis of their “politically correct” standards.

    • #4
  5. Tim H. Inactive
    Tim H.
    @TimH

    Hoyacon:I hope your thoughts/work on this gets wider dissemination. I’m only a backyard telescope kind of guy, but isn’t Ms. Urry rather well known as a gender activist? Her presidency is a platform.

    Yes, I think you could say that.  It’s certainly been a big issue for her for years.  I won’t speak badly of her—we’re friendly acquaintances, and I’ve known her since I was a grad student—but I disagree with the approach our society has taken on some of these issues.

    I see a number of other candidates for our professional society putting race/sex/etc. advocacy issues into their platforms, and I think that’s the wrong way for us to go.  They argue that unconscious biases against women (for example) are pervasive in the field.  But some of these will openly express contempt for (and favor job discrimination against)  conservatives, religious believers, and the like, arguing that that‘s not a problem.  I think that if biases against women in our field have been pushed underground to the point where they are mostly expressed “unconsciously,” then it’s not as bad as some claim.

    • #5
  6. Tim H. Inactive
    Tim H.
    @TimH

    Richard Fulmer:While we could reorient the tests to check for knowledge of, say, “street smarts,” the possession of such knowledge would be of only marginal value for astronomers, physicists, and other hard-science professionals.

    I think they sort of get at this when some of the activists argue we should look specifically at research skills and personality traits that produce job success:  optimism, “sticktoitiveness,” creative thinking, emotional self-control(!), and so on.  I would agree with looking for those skills.  But they’re awfully hard to measure in an application!  I agree with the principle there, but I don’t see much way to check for them until you’ve got experience with the person.

    …on the topic of standardized testing is that they don’t agree on the purpose of the tests. Those of us on the right see them as a means of determining probable success or failure in school and on the job. Those on the left see them as a devious way of discriminating on the basis of race and sex.

    I think that’s about right.  I don’t think standardized tests are an ideal way to measure all of the things you need for grad school or professional research, but it gives a much more objective measure of certain skills than the letters of recommendation do, and it controls for grade inflation.  Their recommended solutions are too subjective and would make admission even more capricious than it is already.

    • #6
  7. The Reticulator Member
    The Reticulator
    @TheReticulator

    I wonder if success is a good measure of success.  For example, if the successful ones bully their way to the top by playing one or more of the available victim cards, is that really what we want more of?

    • #7
  8. Z in MT Member
    Z in MT
    @ZinMT

    I agree wholeheartedly with this post. GRE’s particularly the subject exams are good ways to flatten the variances in readiness of different undergraduate programs. They are not a great predictor of research capability, but are a good predictor of course work and ability to pass Comprehensive Exams.

    The general GRE is a weird joke with a math section that is easier than the SAT, juxtaposed with a vocabulary/English section that would challenge many English majors. It is even worse since they got rid of the logic section in favor of an essay section.

    • #8
  9. Lidens Cheng Member
    Lidens Cheng
    @LidensCheng

    I’m about half way through grad school in physics. It’s a little tough to analyze this. I would say the majority of the students I’ve interacted with in the physics and astronomy departments here at University of Arizona will say very different things. The astronomy department here is top notch so for sure they don’t accept students or very rarely accept student who don’t do well on the physics GRE. The physics department here is more open and definitely doesn’t view it as a deciding factor. A lot of students in both departments think the exam isn’t a great judge of physics knowledge and I would agree. It relies quite a bit on memorizing things because it’s a multiple choice exam.  Physics students are used to taking long exams where we can derive things ourselves and then solve the problems.

    At the same time, I don’t think getting rid of the exam is a good idea either. I think a standard exam is important just like recommendation letters, grades, and research experience. I’m more of a proponent of modifying the exam to actually reasonably test physics concepts and the importance of the ability to solve problems. Of course, this is difficult because multiple choice. If you didn’t catch my drift, I hate multiple choice exams. Thank god that part of my life is over.

    • #9
  10. Lidens Cheng Member
    Lidens Cheng
    @LidensCheng

    Also I don’t really think physics GRE is necessarily a good indicator of the ability to pass grad classes or comprehensive exams. Again because class exams and comps are the normal kind of exams physics students take. You get a handful of long problems to solve. Our comprehensive exam here is pretty brutal. 5 problems in 3 hours for 3 days. But yeah this is the kind of exam that physics and astro people take and really I think these are the exams that actually matter.

    • #10
  11. Lidens Cheng Member
    Lidens Cheng
    @LidensCheng

    Third post: more rambling because this topic is relevant to my current state in life

    Everything I have to say is anecdotal though. All my friends here went through the same and all my friends at other physics/astro departments through out the country also. Some of us did really well on the physics GRE and some of us really didn’t, but we’ve all passed everything we needed to in grad school by now.

    Sooooo I don’t know what my point is. I guess I don’t know what the solution should be. No to omitting physics GRE but also no to keeping it in the exact same format it’s currently in.

    • #11
  12. Tim H. Inactive
    Tim H.
    @TimH

    Z in MT reminds me of something else I was thinking of. The claims are that this isn’t a good predictor of research success. (I’m willing to believe that, except their own data show it to be pretty good at the top end.) But it certainly seems related to classroom preparation, which is itself a requirement for getting your Ph.D. and becoming a scientist. The graduate departments take a big risk on the students they admit. Tuition in physics is usually free, and you get a stipend to live on, in return for working 20 hours a week as a TA. They DON’T want these people to fail out of the classes and not complete their Ph.D.’s…But a bunch of them will.

    So would the activists suggest getting rid of the classes you take in grad school? The American Ph.D. system has an advantage over the British and Canadian system, in that we wind up getting a broader classroom education before doing our dissertation research. It takes several more years, but I felt more prepared to do a wide variety of work, while friends who went through the British schools got a doctorate at 25 but felt that they had little training to do anything outside their research project.

    • #12
  13. Z in MT Member
    Z in MT
    @ZinMT

    Tim I know better than most the risk with Physics grad students, try supporting a Physics student on an RA while they are still taking courses. My area is laser physics and I have supported many students through a Master’s but yet to have gotten one to stay on for a PhD because they learn that they can get a job with just a Master’s in industry for more than they will make in the very unlikely event that they ever get a academic position after their PhD.

    • #13
  14. Tim H. Inactive
    Tim H.
    @TimH

    Lidens Cheng:I would say the majority of the students I’ve interacted with in the physics and astronomy departments here at University of Arizona will say very different things. The astronomy department here is top notch so for sure they don’t accept students or very rarely accept student who don’t do well on the physics GRE.

    By the way, it was Arizona where I was right at the average Physics GRE score but didn’t get in.  ;)  And this actually impresses me, because it means they weren’t simply going by my GRE.  For that matter, Pitt is where I did go to grad school, and I was a good bit below their average GRE.  So clearly, these two schools take other things into account, and I suspect every school does.

    It relies quite a bit on memorizing things because it’s a multiple choice exam. Physics students are used to taking long exams where we can derive things ourselves and then solve the problems.

    True that it’s multiple-choice, but I thought I remembered that you needed to derive the answer the normal way in order to select the correct one.  Well, that and some conceptual questions that didn’t require derivation, but those are important, too.

    An exam that had normal long-derivation answers might be a bit better, but imagine grading it!

    • #14
  15. Tim H. Inactive
    Tim H.
    @TimH

    Do any of y’all want to take a stab at explaining why the score distributions for “successful” men and women are so drastically different?  I’m really amazed and want to figure out what’s causing it.

    • #15
  16. EJHill Podcaster
    EJHill
    @EJHill

    Tim H.: Tim H. Do any of y’all want to take a stab at explaining why the score distributions for “successful” men and women are so drastically different?

    Success is the awarding of fellowships? Then don’t tell me the scores, describe the process of application for the fellowship, not the application for admission into the grad school. Those seem to be different things that are unrelated.

    An admissions test would seem to be a broad knowledge thing while fellowships might be narrow to a specific question or problem?

    And then there’s the people factor. Are there interviews in the fellowship process?

    • #16
  17. The Reticulator Member
    The Reticulator
    @TheReticulator

    Tim H.:Do any of y’all want to take a stab at explaining why the score distributions for “successful” men and women are so drastically different? I’m really amazed and want to figure out what’s causing it.

    I presume each point on the graphs represents the mean of a bunch of data points.  If so, I’d want to look at the number and distribution for each before getting too carried away with speculations.  But I’d need to come up with more motivation before I’d do that myself.  It’s easier to sit here and hope that somebody else will go to the work of presenting the information that way.

    • #17
  18. Douglas Inactive
    Douglas
    @Douglas

    Tim H.:

    Hoyacon:I hope your thoughts/work on this gets wider dissemination. I’m only a backyard telescope kind of guy, but isn’t Ms. Urry rather well known as a gender activist? Her presidency is a platform.

    Yes, I think you could say that. It’s certainly been a big issue for her for years. I won’t speak badly of her—we’re friendly acquaintances, and I’ve known her since I was a grad student—but I disagree with the approach our society has taken on some of these issues.

    May as well get used to it. Conventional wisdom said that math, science, and engineering would be immune from this nonsense because they were results-based. Along came “America’s premier astronomer”, Neil deGrasse Tyson, a guy who apparently hasn’t done any actual astronomy since his doctoral thesis, and yet pretty much rules the hearts of Millenials and IS the face of Astronomy now. Nothing is safe from the Everyone Gets a Trophy culture. Ability must be whipped and taught its place, underneath grievance.

    • #18
  19. Don Tillman Member
    Don Tillman
    @DonTillman

    One of my lefty Facebook friends posted a link to this:

    An open letter to SCOTUS from professional physicists
    drafted by the Equity & Inclusion in Physics & Astronomy group

    http://eblur.github.io/scotus/

    Same folks?

    • #19
  20. Tim H. Inactive
    Tim H.
    @TimH

    Don Tillman:One of my lefty Facebook friends posted a link to this:

    […]

    Same folks?

    Not explicitly, but I think there’s some overlap in the people involved.  Our professional discussion group unfortunately had someone post a link to this petition, along with some insults directed at Scalia.  As far as I can tell, that petition comes from this group, which looks like it’s directly connected with this other one somehow.

    I was originally going to tie this post in with the petition on Scalia, but I trimmed it.  It’s disappointing, to say the least, to see this emotional and poorly-thought-out attack on him, in part because they clearly didn’t understand what he was doing.  The advocates made it out as if Scalia were attacking “our colleagues,” when Scalia’s discussion was getting at those who didn’t go into our field in the first place.  These are by definition not our colleagues…but they might have been.  It’s about how to change the circumstances so that people won’t be discouraged from going into tougher majors because they attended a school that’s more rigorous than they’re ready for.

    The petition-waving activists have it so backwards.

    • #20
Become a member to join the conversation. Or sign in if you're already a member.