A guest post by Kevin Denny
The recent publication of the QS world rankings generated a lot of interest as well as criticism from various people, including me. A common response to such criticisms is to say “Like it or not, they matter to people so we need to pay attention to them”. But do they matter?
This paper looks at how the publication of US college rankings influences the demand for places but only when colleges are not listed alphabetically.
Salience in Quality Disclosure: Evidence from the U.S. News College Rankings
M. Luca & J. Smith
How do rankings affect demand? This paper investigates the impact of college rankings, and the visibility of those rankings, on students’ application decisions. Using natural experiments from U.S. News and World Report College Rankings, we present two main findings. First, we identify a causal impact of rankings on application decisions. When explicit rankings of colleges are published in U.S. News, a one-rank improvement leads to a 1-percentage-point increase in the number of applications to that college. Second, we show that the response to the information represented in rankings depends on the way in which that information is presented. Rankings have no effect on application decisions when colleges are listed alphabetically, even when readers are provided data on college quality and the methodology used to calculate rankings. This finding provides evidence that the salience of information is a central determinant of a firm’s demand function, even for purchases as large as college attendance.
[NOTE: The “rankings” tag leads to previous posts on this topic.]
UPDATE: Glenn Ellison has a cool paper that’s related:
A large literature following Hirsch (2005) has proposed citation-based indexes that could be used to rank academics. This paper examines how well several such indexes match labor market outcomes using data on the citation records of young tenured economists at 25 U.S. departments. Variants of Hirsch’s index that emphasize smaller numbers of highly-cited papers perform better than Hirsch’s original index and have substantial power to explain which economists are tenured at which departments. Adjustment factors for differences across fields and years of experience are presented.
14 replies on “Do college rankings matter for student choice?”
Kevin has long argued that many rankings contain little useful information. My response has always been that they are used nonetheless. My evidence was anecdotal. No longer.
It doesn’t follow from the fact that the long-established US News ranking (which, incidentally, is gamed by a great many institutions) affects student choice in the US that any of our johnny-come-lately hodge-podge of international rankings mean anything to anyone. People in the US also care about the ratings of cars that the magazine Consumer Reports publishes. Doesn’t mean they’ll care about any old car ratings that any outfit might deign to publish. So the meaningfulness or importance of QS or Shanghai Jiao-Tong or the ranking my granny came out with yesterday have not been established.
Of course there are also some very good reasons for colleges and universities to decline to participate and some of those that do refuse have hardly been penalised for it.
That one ranking matters does not indeed imply that others do too. There is also a big difference between a within-country ranking and an international one.
That said, it would be a reasonable conjecture that these various other rankings, including that of “Ernie Ball”‘s granny, matter too, to someone. All I can say is that they don’t tell me anything of interest so I don’t get excited. What does get me exercised is the largely uncritical approach of academics to these rankings. It is almost like they are afraid to complain. Academics should be expert at sniffing out faulty reasoning and dodgy data. Yet confronted with rather dubious numbers, most keep schtum. Danny Blanchflower did a nice job on the QS rankings, fair dues to him.
Economists in particular are remiss because we think of ourselves as pretty numerate, are used to dealing with messy data and have developed index number theory to a sophisticated level.
See update with Ellison’s paper.
I’ve never met an academic, economist or otherwise, who does not appreciate the many shortcomings of rankings. I guess many are constructivists: You don’t complain until you have a better alternative.
If Richard Tol lived in the 15th century, he’d be defending the ptolemaic model on the grounds that “it’s used.”
If Richard Tol’d been born in the 15th century, he’d been toiling in the Frisian clay.
An flat earther???
I’d imagine location would play a role i.e. easier, cheaper to travel. Peer pressure, friends attending, parents etc. Course availability would also be key, why attend higher ranked university if it doesn’t have the course you want when a lower ranked college has it.
Rankings may matter more for post graduate studies but not by much. Some might be inclined to a certain supervisor or identified speciality etc. Cost again would be a factor.
The Earth doesn’t come much flatter as it does in Frisia.
Richard: my experience of academics is certainly different from yours. Of course people mumble all sorts of things over their glass of port but, publicly at least, most are pretty uncritical. But I am also referring to the universities themselves who prefer the two-step: celebrate when you go up in the rankings and say nothing otherwise.
I don’t think having an alternative is a prerequisite for pointing out weaknesses in something. Anyway, my involvement in this debate was because I said that putting a weight, albeit small, on “foreign faculty” was absurd. So that obviously means I think the weight should be zero.
“T” is listed, alphabetically, before “U” … hmmm do I detect a touch of the ‘status anxiety twitches’?
I also hear that Frisian is listed in the VU Amsterdam index as a foreign language – but noone in Amsterdam has yet managed to decipher it .. hence, that chair has been vacant for a few hundred years …. and the Frisian Index remains open.
@All West Frisians
I would say that countries ranked high internationally are usually ranked high domestically. Foreign students are influenced by high international rankings to a greater extent than the natives.
Within a country people know a lot about their universities and the universities that are oversubscribed to the point where they turn away five out of six applicants are usually medium sized with a good reputation for teaching, a pleasant campus, lively night life, involved alumni and numerous other factors that do not enter into ranking criteria.
If you want the highly intelligent, wealthy Chinese, Applied Science and Science students it helps to be ranked high internally and externally.
One wonders whether the decisions by some universities to pursue “opt-in” rankings like the 4-star/5-star ones provided to (I decline to use “awarded to”) UCC and UL of late are less about student appeal and more about a desire by university bureaucrats to have something in the trophy cabinet…