In Defense of College Rankings

This paper was presented at the Association for the Study of Higher Education Annual Meeting in Louisville, Kentucky, on November 10, 2007.

People in higher education are famously disagreeable, in the sense that they'll disagree about almost anything. From small differences to weighty questions of politics, history, and truth, there is always room in the academy for vigorous debate.

Yet there's one topic that engenders near-universal agreement: college rankings. Everybody hates them. This is true to the point where rankings animus now provides useful conversational common ground, like the weather, or the need to stop global warming. Even the most vicious ideological battles and internecine feuds can be temporarily put aside to join arms and denounce U.S. News & World Report.

The level of ill will is hard to overstate—speaking to the New York Times, the president of Bard College once called the rankings "A catastrophic fraud. Corrupt, intellectually bankrupt and revolting."¹ This from the leader of an institution U.S. News ranks as "first tier."

There are legitimate critiques of college rankings. But on the whole, rankings are not nearly as bad as people believe. Rankings critiques are overstated for two reasons. First, because they tend to conflate critiques of specific rankings—nearly always those published by U.S. News—with critiques of rankings per se. Second, because real concerns about the downside of rankings are never properly weighed against their many benefits.

The Education Conservancy, a nonprofit group and ringleader of a recent revolt by some liberal arts colleges against U.S. News, offers a typical example of the first phenomenon. A letter promoted by the Conservancy and signed by the presidents of a number of colleges, including Skidmore, Trinity, Dickinson, and St. John's, included the following:

We are writing to seek your commitment (and the commitment of your institution) to a new approach to rankings of colleges and universities compiled by U.S. News and World Report. We believe these rankings are misleading and do not serve well the interests of prospective students in finding a college or university that is well suited to their education beyond high school. Among other reasons, we believe this because such rankings

• imply a false precision and authority that is not warranted by the data they use;
• obscure important differences in educational mission in aligning institutions on a single scale;
• say nothing or very little about whether students are actually learning at particular colleges or universities;
• encourage wasteful spending and gamesmanship in institutions' pursuing improved rankings;
• overlook the importance of a student in making education happen and overweight the importance of a university's prestige in that process;
• and degrade for students the educational value of the college search process.²

Only the third bullet, along with parts of the fourth and the fifth, are truly critiques of the specific rankings published by U.S. News. Those rankings are undoubtedly problematic, having little to do with the quality of education institutions provide. Instead, they are almost exclusively driven by three things: wealth, fame, and exclusivity.

Unsurprisingly, institutions have responded by trying to raise money, build prestige, and reject as many applicants as possible. For many institutions—particularly public universities—these actions betray their historic responsibility to provide a reasonably priced, high-quality education to a broad array of students. More subtly—but no less importantly—the U.S. News rankings reinforce a pre-existing status hierarchy that over-values the scholarly mission of higher education relative to the task of teaching students, particularly undergraduates. By giving institutional leaders more incentives to aggregate money and status, U.S. News makes it harder to create incentives for leaders to focus on anything else.

I have proposed (and here is where I part ways with most of my fellow U.S. News critics) that the only way to beat U.S. News is to join them, by creating a new college rankings system based on different, superior measures of institutional quality.³ In just the last 10 years, a host of new data sources have become available to fill this need, from measures of post-graduation employment to "value-added" assessments of analytic reasoning and critical thinking to high-quality surveys of institutional teaching practices and student engagement. These measures could be combined to create a rich, multi-dimensional portrait of institutional quality that would be far superior to what is provided by U.S. News.

But the case for alternative rankings is often stymied by generalized anti-rankings arguments like those contained in the rest of the Education Conservancy's bullet points. In the main, these arguments are much weaker than specific critiques of U.S. News. As a result, an opportunity to harness the undeniable power of rankings to influence institutional decision-making is being wasted.

Rankings Are Unavoidable

We are often told (more often, perhaps, than is strictly necessary) that higher education represents mankind’s highest, most complex ideals. Surely, it is said, such lofty aspirations can’t be boiled down to mere numbers, can’t be replicated by a few clever journalists with a moderate aptitude for Microsoft Excel. If they could, that would call into question the loftiness itself.

Yet at the same time, there exists a broadly shared and finely nuanced sense of quality and status in higher education—of ranking, in other words. These strongly held yet contradictory notions sometimes manifest in fairly spectacular displays of cognitive dissonance. In 1996, for example, the president of Stanford University, Gerhard Casper, sent a personal letter to James Fallows, the recently hired editor of U.S. News. Casper said:

"As the president of a university that is among the top-ranked universities [according to U.S. News], I hope I have the standing to persuade you that much about these rankings— particularly their specious formulas and spurious precision—is utterly misleading…I am extremely skeptical that the quality of a university—any more than the quality of a magazine—can be measured statistically. However, even if it can, the producers of the U.S. News rankings remain far from discovering the method. Let me offer as prima facie evidence two great public universities: the University of Michigan-Ann Arbor and the University of California-Berkeley. These clearly are among the very best universities in America—one could make a strong argument for either in the top half-dozen. Yet, in the last three years, the U.S. News formula has assigned them ranks that lead many readers to infer that they are second rate: Michigan 21-24-24, and Berkeley 23-26-27."⁴

In a single sentence, Casper denounced as "specious," "spurious," and "utterly misleading" rankings that identified Stanford as one of the very best universities in America, an assessment that, it is safe to assume, Casper believed to be entirely correct. Although Casper was “extremely skeptical” that the quality of a university can be measured “statistically," he apparently found university quality to be readily measurable through non-statistical methods, since he was able to confidently place both Michigan and Berkeley at or near the top six out of many hundreds of universities nationwide. Non-spurious precision turns out to be quite possible, but only using methods that Casper—through what could only have been an unfortunate oversight, given the pressing need for an alternative to U.S. News—declined to identify. One also marvels at the myopia that would cause someone to regard a ranking of 21st out of some 250 universities as "second rate." What does that make the bottom institutions? Thirteenth rate?

While Casper was pretending to argue with the idea of rankings generally, he was really just arguing about the result of the U.S. News method, which didn’t fit with the personal Top 20 list he apparently carried around in his head. Higher education is awash in differentiated status, in carefully constructed and generally recognized rationales for making distinctions along a variety of lines. When it comes to the scholarly mission of higher education, for example, people have no trouble identifying the most outstanding researchers, departments, and institutions. Not with unanimity, of course, but with enough consensus to matter, to provide a "strong argument" for people like Casper.

The distinction between this process and rankings like those produced by U.S. News is a difference of control, not kind. External rankings take the power of self-definition away from the academy. This is particularly disconcerting when those ranking are aimed at consumers, who hold the financial fate of many institutions in their hands.

The Role of Consumer Choice in Rankings

In this way, rankings represent the harsh competitive discipline of the marketplace. Many people are deeply concerned about the negative influence of markets and consumerism on higher education and see rankings as the culprit. John W. Boyer, dean of The College at the University of Chicago, said of the U.S. News rankings, "It's unfortunate that you get this hyper-commercialization and hyper-consumerization of academic life whereby a college education becomes the equivalent of an SUV."⁵ He's right about colleges not being the same as SUVs—SUVs cost a lot less. But this is a case of blaming the cart for the speed and momentum of the horse. While one can debate the pros and cons of students who increasingly see themselves as savvy consumers, rankings didn't cause that trend. They merely responded to it, filling a need that would have been met by someone, if not U.S. News.

Even as colleges, departments, and professors willingly sort themselves along scholarly lines, institutions like to promote (for admissions purposes) what could be termed the "fit paradigm." Syndicated columnist Thomas Sowell recently summed up this well-worn idea, echoed by guidance counselors and admissions officers across the land, when he said, "What you really want is not the 'best' college but the college that fits you best."⁶ By this way of thinking, there are no "better" or "worse" institutions, only thousands of sui generis colleges and universities waiting to be matched to millions of equally unique students. No competitive pressure, just a pleasant process of creating the perfect union.

To be sure, different institutions have distinct missions and offer students varied environments in which to learn. The basic idea of "fit" isn't illegitimate, but it can only be taken so far. In the end, most colleges and universities provide the same degrees in roughly the same way. They're a lot more alike than they are different, and some are obviously much better than others.

The unwillingness of people in higher education to admit this publicly, in reference to specific institutions, in a way that prospective students can understand, stems from the conspiracy of politeness that governs much of the public higher education dialogue. Just as Ronald Reagan's 11th Commandment directed Republicans to never speak ill of one another, so too are university representatives required to preface all references to their peers and competitors with assurances that such-and-such is a "fine" institution, or some similar encomium. This amounts to institutional grade inflation, a world where every college and university is "first tier" in its own way.

Rankings fly in the face of that comfortable fiction. They're based on the same principle that law schools use when ranking their students—the best criteria are comparative, and it's worth knowing who is better than whom.

Debatable Assumptions and Imperfect Data

Some critics of rankings recognize the need for competition and concede that some colleges are better than others, but insist that rankings make those distinctions too finely, or too bluntly, depending on how you look at it. This is the critique of "spurious" or "false" precision made by President Casper, the Education Conservancy, and many others. This line of criticism has several components, some of which are legitimate and need to be carefully addressed.

All rankings that aspire to be comprehensive are based on debatable assumptions and imperfect data. U.S. News has decided that spending per student should represent 10 percent of an institution's score. This is an arbitrary decision, not based on the result of any empirically demonstrated relationship between spending and student learning, or spending and anything else. Fifteen percent would yield a different rankings list, while being no more—or less—legitimate. The accuracy of "peer survey" component of the U.S. News list—arbitrarily set at 25 percent of the rankings—is limited by the imperfect knowledge and judgment of the deans and presidents who determine the survey results.

The issue of debatable assumptions is more easily addressed. To begin, they can be made less debatable. If higher education researchers could empirically estimate the relationship between financial inputs and university outcomes—which is presumably positive, given the amount of energy colleges put into increasing those inputs—then U.S. News could use those findings to weight their rankings non-arbitrarily. The same is true for other measures like class size, professor's salaries, etc. No such reliable estimates exist, which is the fault of the academy, not U.S. News.

Still, more research could mitigate but not eliminate debatable assumptions. All estimates of quality are based on underlying values rooted in beliefs as much as data. Rankings have the great virtue of making those assumptions explicit. One can disagree with the U.S. News criteria for quality, but at least everyone knows what they are—that's why one can disagree with them. The same is not true for President Casper's secret rankings, which are simply asserted behind the veil of expertise. There will never be unanimity about what higher education quality means, so the only thing to do is be crystal-clear about debatable assumptions and their consequences. Rankings facilitate such transparency.

The problem of imperfect data is trickier. One can argue whether financial resources are an appropriate measure for ranking colleges, and if so, how it should be weighted. But as an indicator, they at least have the virtue of being measurable to a high degree of accuracy—among the many virtues of dollars are that they're relatively easy to count. By contrast, a college's average SAT score—while also easy to calculate with precision—is a highly imperfect proxy for the thing U.S. News is actually trying to measure: the academic "quality" of a college's incoming freshman class. Similarly, the U.S. News reputational survey, like all surveys, is subject to measurement error—it's an imperfect estimate of the actual academic reputation of universities. Alumni giving rates don't provide a complete picture of alumni good will. And so on.

All of which means the U.S. News rankings are sometimes wrong, even under the terms of their highly debatable assumptions. This is true for any ranking based on factors beyond the narrowly quantifiable. The college ranked #10 might really be #12; a university in the third tier might actually be in the second, etc. etc.

Many rankings critics believe these imperfections make the presentation of rankings in ordinal (i.e. rank) order inappropriate. In other words, if you're not 100 percent sure #10 is actually #10 and not #12, you shouldn't say it's #10. The preferred alternative is usually something along the lines of what U.S. News already does with lower-rated institutions: arranging institution into groups or "tiers" and not making distinctions within them. This is an imperfect solution, since tiers still require absolute dividing lines, and measurement error could still put an institution on the wrong side of the line. More importantly, it's the wrong solution, one the runs against the principles that colleges and universities themselves use when presenting data to the world.

Nearly all measurements of complicated things contain the possibility of error. When university researchers present the results of surveys, experiments, and analyses, they don't hide the numbers out of concern that readers will infer a spurious precision that doesn't exist. Rather, they present the results along with a clear explanation of the assumptions and methods that created them, along with an estimate of how much measurement error is likely to exist. While U.S. News is clear about its assumptions and methods, it provides no information about the size of probable measurement error. It should. But the fact that it doesn't isn't an argument against rankings per se; it's only a critique of the way U.S. News has chosen to present them. When U.S. News or anyone else says that Big State University is #10, what they're really saying is that Big State University is more likely to be #10 than #11, or #9, or anything else.

Reasonable consumers of rankings understand this, just like they understand that the real-world difference between institutions on a ranking list doesn't necessarily correspond to the ordinal difference. The actual distance between #10 and #12, for example, might be bigger or smaller than the actual distance between #12 and #14. This isn't a particularly difficult concept to grasp, and it's well-understood by anyone who follows college or professional sports. If Ohio State's football team finishes the season ranked #1 in the country due its 14-0 record and 17-point average margin of victory, including a crushing 56-3 defeat of Michigan in front of 107,501 heartbroken fans in Ann Arbor, while the #2 and #3 ranked teams both finished at 12-2, people understand that the difference between #1 and #2 is bigger than the difference between #2 and #3. It should be noted that U.S. News does, in fact, display each ranked institution's overall score, which shows the real difference between institutions converted to a 100-point scale, in the first column of its ranking list. This allows readers to see that #2 Harvard (with a score of 99) is closer to #3 Yale (score=98) than Yale is to #4 Stanford (score=95).

Rankings are simply the most open, most comprehensive way of displaying information about university quality, the method that vests the most power in—and gives the most credence to—individual consumers. When rankings critics suggest that evaluation data should be suppressed by organizing institutions into tiers or otherwise obscuring the underlying results, they’re displaying an undeniable disregard for the ability of the general public to make reasonable inferences from data. They are essentially saying, "You can't handle the truth." This is especially troubling given that many consumers of rankings (like parents of college-age children) are, themselves, college graduates.

This disregard, coupled with the notion that universities are especially unquantifiable, runs through most critiques of college rankings, as well as broader criticisms of the idea of higher education accountability (rankings are simply one of accountability's many forms). In reality, universities are not as mysterious as their denizens would like to believe, and the outside public is not as ignorant as they fear.

Keeping Responsibility in Its Proper Place

Even as higher education distrusts the general public, it—oddly enough—seems to distrust itself just as much. Rankings critics often point to questionable behavior among institutions trying to improve their score—the "gamesmanship" described by the Education Conservancy. This is a legitimate concern. But there are two types of questionable behavior, and the distinction between them is important.

The first type could be termed "bad responses to bad incentives." A public university with an access mission that tries to boost its average SAT score by redirecting need-based financial aid to so-called "merit aid" in an attempt to recruit "better" students is one example. Redirecting resources from undergraduate teaching to attracting "star" researchers in order to burnish the institution's academic reputation is another. But these are really just symptoms of an underlying problem we have already discussed—the specific shortcomings of the U.S. News rankings. The best way to eliminate bad incentives is to create good ones.

That won't entirely solve the problem, though, because we are still left with the second type of questionable behavior, which could be termed "bad responses to good incentives." For example, U.S. News incorporates graduation rates into its rankings, a legitimate measure of student success, and thus a good incentive. But an institution seeking to improve its ranking by boosting graduation rates could respond in at least four possible ways, not all of them good:

• Lie about the numbers by reporting falsely inflated data to the U.S. Department of Education, which gathers graduation rate statistics. This would be relatively easy, since the Department doesn't perform comprehensive audits of graduation rate reports. There's precedent for this—numerous universities have been exposed lying about graduation rates, SAT scores, admission rates, and other statistics submitted to U.S. News.⁷
• Lower standards and thus make it easier for academically marginal students to complete their degree.
• Focus resources and priorities on recruiting "better" students who come to college from more academically rich, financially secure backgrounds, and are thus more likely to graduate.
• Focus resources and priorities on improving student counseling, expanding pre-college "summer bridge programs," providing prospective students with information about academic standards while they're still in high school, improving remedial courses, structuring the first year of college to better support at-risk freshman, and working to improve the quality of teaching and learning, especially for lower-division undergraduates.

In other words, a university could respond to the graduation rate incentive built into the U.S. News rankings by choosing to lie, compromise its academic integrity, betray its commitment to underserved students, or actually try to help its students graduate.

How much responsibility does U.S. News bear if a university fails to make the last choice? None whatsoever. Responsibility for institutional integrity lies with institutions, which is, one would hope, much to their preference. If you surrender responsibility for your choices, you eventually surrender your ability to make them.

Examples of bad responses to good incentives in the business world are instructive. When Enron's perfidy was discovered, its stock price took a nosedive, its executives were sent to prison, and its auditors were disgraced. There was plenty of blame to go around. But, crucially, nobody blamed the underlying system of reporting and, in effect, ranking businesses based on the financial results that Enron, along with fellow cheaters like Worldcom, obscured. Indeed, the response from Congress was to strengthen that system by passing the Sarbanes-Oxley Act, which required more in the way of stringent public reporting. Yet when universities lie to U.S. News, rankings critics blame U.S. News, not the lying university. It's strange that higher education doesn’t grant itself the honor of moral agency.

Bringing New Measures Into the Rankings Game: Pros and Cons

Incentives to increase one positive outcome by sacrificing another can be further mitigated by structuring rankings so that different incentives balance one other out. If a college is ranked on both graduation rates and academic quality, for example, then sacrificing one for the other becomes a zero-sum game. Comprehensive, robust, and multi-dimensional rankings reduce the potential for distortionary incentives and sub-optimal behavior. The more pillars holding up the system, the less the chance that one pillar will crumble and bring down the entire enterprise.

This puts a premium on finding new measures of things like teaching quality, student engagement, and student learning outcomes. In recent years, enterprises like the National Survey of Student Engagement (NSSE) and the Collegiate Learning Assessment (CLA) have arisen to fill this need, providing valid, meaningful, comparable measures of institutional quality. The fact that hundreds of institutions have voluntarily chosen to participate in NSSE and the CLA contradicts President Casper's notion that university quality is beyond the realm of quantitative measurement.⁸ The leaders of these groundbreaking instruments, however, are generally are opposed to their use in rankings. Their concerns deserve careful consideration.

George Kuh, founder and director of NSSE, recently objected to using NSSE data to characterize the whole of an institution, as all institution-level rankings do. Kuh said:

Simple displays based on institutional averages can be misleading if used by prospective students to "pick the right school." This is because—as Pascarella and Terenzini (1991, 2005) concluded—individual student performance typically varies much more within institutions than average performance does between institutions. This is true at every level of education. To illustrate: Results from [NSSE]…reveal that for all but one of 14 NSSE scales for both first-year and senior students, less than 10 percent of the total variance in student engagement is between institutions. The remaining variance—in several instances more than 95 percent—exists at the student level within a college or university.⁹

This is a variant on a common anti-rankings argument, one that Kuh describes more explicitly later in the same article:

"…rankings de facto reduce rich, complicated, and often distinctive patterns of student experiences and outcomes to a single number, masking much of what is informative about variations in student and institutional performance."

It's true that rankings boil down lots of complex data to a single number, but that doesn't necessarily mean that the complexity has been lost. Rather, rankings summarize complexity, a perfectly legitimate and unavoidable practice. No one can ever fully comprehend the totality of an institution as rich and complicated as a university, which necessitates choosing certain key elements and measurement methods to represent the unknowable whole, and then finding ways to represent those elements in a way that facilitates comparison and comprehension by non-experts. As long as the underlying data elements and weights that create rankings are kept transparent, then complexity isn't reduced or masked, but merely transformed.

The act of reducing institutions to a single number also reflects the reality of the consumer marketplace: Students can only choose a single university to attend.¹⁰ It may very well be the case that far more variance on NSSE and other measures occurs within institutions than between them. So what? Students are better off understanding the inter-institutional variance that exists than not understanding it. If institutions are different, and that difference can be accurately measured, then the variance can be used in rankings. If anything, the fact that university quality varies substantially between departments and for different students groups is an argument for more rankings—by department, for different student groups. It's hardly an argument against rankings per se.

That said, there are limits to how far the multiple rankings concept can be extended. It's a seductive notion, since it seems to represent a sensible middle ground between concerns like Kuh's and the need for robust consumer information. During the deliberations of U.S. Secretary of Education Margaret Spellings' Commission on the Future of Higher Education, some commissioners expressed desire for a system of "customized" rankings, where individual students could choose which factors to consider and how much weight they deserve.

But the real-world potential for such a system is limited at best. Most consumers don't want to decide which of NSSE's scores of questions are most important, or how much weight to give "value-added" scores on the CLA. Such decisions require more than just the reasonable ability to interpret information I ascribe to consumers above. They require deep expertise and judgment. This is one of the main services sold by U.S. News—not just providing access to information (most of which can be found elsewhere) but making meaning from information.

The other virtue of U.S. News is universality, which also cuts against the logic of multiple rankings. Colleges provide students with two things of great value. The first is the package of knowledge, skills, and ideas that students acquire and the person they come to be. The second is a portable, universally recognized, non-expiring credential that proves those things to the outside world—a diploma. The value of that diploma is augmented by the consensus public perception of the quality of education it signifies. In other words, it's not enough for a student to know he or she got a great education—everyone else needs to know it too.

For that reason, there can't be a vast number of conflicting rankings upon which nobody can agree. It doesn't follow that U.S. News should enjoy its current near-monopoly on the rankings business forever, but anything more than a handful of serious competitors would confuse the market to the point of dysfunction. Consumers want, and need, rankings that are valued by people other than themselves.

A similar, more institutionally focused argument is voiced by the Community College Survey of Student Engagement (CCSSE), which administers a survey very similar to NSSE and whose official policy on rankings is as follows:

"CCSSE opposes using its data to rank colleges. Each community college's performance should be considered in terms of its mission, institutional focus, and student characteristics. Because of differences in these areas—and variations in college resources—comparing survey results between individual institutions serves little constructive purpose and likely will be misleading."

Again—while institutional missions and circumstances vary, they don't vary infinitely. NSSE and CCSSE are carefully constructed to assess institutional practices that are—or should be—common to all institutions, like student-faculty interaction or "active and collaborative learning." That's why the CCSSE administers the same survey to every community college, and NSSE to every four-year college, regardless of "mission, institutional focus, and student characteristics." Both organizations display the results in terms of aggregate "benchmark" scores that are compiled in much the same way that rankings are compiled, and explicitly call for institutions to compare themselves to national averages or the average among broad groups of peers.

Students, meanwhile, should be indifferent to whether the quality of education a college provides is a function of "variations in college resources" or anything else. They just need the best education they can find and afford. Confining comparisons to small groups of like institutions—based on resources or other factors—obscures one of the most important choices students make: which small group to choose among. It also makes it hard to compare institutions that are in different groups altogether. Limiting comparisons to peer groups makes sense from the institutional perspective, but not the student perspective. (The dominance of the former over the latter is the underlying source of many higher education policy problems.)

It has also been suggested that using measures like NSSE for rankings would harm the survey process. This is a more serious concern. Participation in NSSE, the CLA, and other measures yet to be invented is voluntary, and predicated on an expectation of institutional control. Results are released only when institutions allow. (CCSSE, to its credit, makes all results public). The threat of a poor rankings result could cause some institutions to decline participation. As Peter Ewell, Vice President of the National Center for Higher Education Management Systems once said of NSSE, "People won't pay for the gun that shoots them in the head."¹¹ This problem is not unsolvable, but will require leadership beyond the institutional level. In one recent example, California State University System Chancellor Charlie Reed mandated that all system institutions begin using the CLA. The University of Texas system also has system-wide CLA participation, and publishes the results in its annual accountability report.

Survey designers also worry that attaching high stakes to their data will alter the data itself. There's a kind of implied Heisenberg Uncertainty Principle lying behind these concerns, as if the integrity of the measures depends on nobody ever seeing them. Still, it's a legitimate concern—there is some evidence that students responding to the Business Week survey used as part of that magazine's business school ranking are more apt to respond positively, in an effort to increase the value of their degree. It's probably safe to assume that business school students represent the extreme end of the distribution of student in terms of the likelihood of doing this, both because they're more likely by inclination to exhibit profit-maximizing behavior and because the potential profit to maximize is larger. A freshman at a large regional public university would probably be more forthcoming. Such negative incentives can also be reduced by making individual surveys one part of a large, diverse set of rankings measures, so the impact of gaming one piece of the system is reduced.

The Upside of Rankings

The potential downside of using surveys like NSSE and the CLA to rank institutions must also be weighed against the upside. Put simply, rankings matter, in a way that the status quo uses of these data do not. The current uses are based on a voluntary self-improvement paradigm: Institutions will examine their performance, compare themselves to their peers and national norms, and take action to improve where performance is low and/or where improvement is most needed.

This model has a fatal flaw: It assumes that institutional underperformance in student engagement, teaching, and undergraduate learning is always or often a result of mistakes that can be corrected. It is not. Rather, underperformance on behalf of students is often a function of choices that have been made. For example, NSSE indicates that freshmen attending the most research-intensive (and thus, usually the most prestigious) four-year universities are less likely to have frequent contact with their professors than students at less research-intensive universities, or liberal arts colleges. This is no surprise—prioritizing research over teaching is what "research-intensive" means, just like a focus on teaching, smaller class sizes, more frequent contact with professors, etc., is a hallmark of liberal arts colleges.

More broadly, the entire status hierarchy within higher education is based on value structures that prioritize research and scholarship over teaching. There is no reason to think higher education institutions will choose to change this, since they chose to make it that way in the first place. Only external pressure will cause them to change. And that's what rankings provide, by empowering consumers and exposing the consequences of institutional decision-making to public scrutiny. Fear of this loss of control, of the potential for upending the cherished value structure, lies at the heart of higher education's antipathy toward rankings.

Rankings are democratic and fair. Anyone can play the game, by the same set of rules. This can be seen even within the status-centric confines of the U.S. News system, where institutions like Duke and Washington University in St. Louis have fought their way into the upper echelons. Rankings also contribute to a more vibrant consumer market, by alerting consumers to a broader range of potential choices, and giving them the ability to evaluate a range of institutions. Many students, particularly those of limited financial means, don't have the luxury of being able to talk to alumni first-hand or travel far and wide to a range of prospective colleges.

As policymakers in Washington, D.C. and state capitols ratchet up pressure on higher education to be more accountable to the public, rankings arguably represent—to paraphrase Winston Churchill—the worst form of higher education accountability, except for all the others. Unlike regulatory schemes like the federal No Child Left Behind Act, rankings involve no government-set performance goals or mandatory interventions, no interference with the prerogatives of higher education leaders and educators to run their institutions as they see fit.

Most importantly, rankings are here to stay. President Casper's letter of 10 years ago was followed by a brief spate of anti-rankings activity that was excitedly labeled a "rankings revolt" by the media, using words remarkably similar to coverage of the Education Conservancy's actions today.¹² It quickly died out, as the most recent uprising surely will, for the simple reason that rankings can't be stopped. It's a free country; U.S. News can publish what it likes and consumers can buy what they want. Most of the information used by U.S. News is publicly available and all of it is replaceable—the magazine has already indicated that if university presidents and provosts boycott the reputational survey in sufficient numbers, the magazine will simply survey someone else. This would in no way compromise public acceptance of the rankings, since the end result will be exactly the same. Ask anyone what the "best" colleges are and they'll say what they've always said: "Harvard, Princeton, Yale, etc…"

The only way to "solve" this rankings problem is to replace the current flawed rankings system with a better one. Not to move "Beyond Rankings"—to use a popular term and the title of a recent USA Today article highlighting NSSE—since that's impossible.¹³ Rather, to make rankings reflect the best ideals of higher education and best interests of students and society at large. That will require both an enhanced commitment from policymakers to require more transparency and disclosure from higher education institutions, and more care in distinguishing between issues relating to rankings generally and those relating to U.S. News. For example, a statement on the NSSE Web site regarding the USA Today effort said:

"…college rankings are based primarily on measures of resources and reputation which research studies indicate are not related to desired student learning and personal development outcomes."

This is true for the specific rankings published by U.S. News, but not "college rankings" in general, and particularly not rankings based on measures like NSSE. The USA Today article quotes George Kuh as follows:

NSSE, in contrast [to the U.S. News rankings], is "about trying to get people to talk about things that matter," says NSSE director George Kuh, an Indiana University education professor.

Perhaps even more, he suggests, NSSE challenges the rankings-driven notion that only a sliver of colleges are worthy of consideration. "There are many innovative programs being offered today, often at a college or university right around the corner," he says.

Kuh is correct in saying that the U.S. News rankings drive the notion that only a few colleges can be top-drawer, since those rankings are mostly driven by things that are scarce by definition—status, elite students, and piles of money. This is not, however, true of rankings driven by measures like NSSE. Any college or university can promote student-faculty interaction, active and collaborative learning, and a supportive campus environment. If they can't, they shouldn't be enrolling students, because good practice isn't a scarce good.

Rankings based on the right measures would create healthier competition, more informed consumers, and greater opportunities for institutions of all kinds to be recognized for serving their students. They would reduce the false scarcity that Kuh rightly laments. They would be imperfect, and not without problems. But given the pressing need to give students a better higher education than they now receive, and the inevitability of status-based rankings until something better comes along, those costs would surely be exceeded by the manifest benefits new rankings would bring.

Endnotes

1. Alex Kuczynksi, "The Media Business, 'Best' List for Colleges by U.S. News is Under Fire," The New York Times, August 20, 2001.

2. http://www.educationconservancy.org/presidents_letter.html. Letter dated May 10, 2007, retrieved on November 5, 2007.

3. Kevin Carey, College Rankings Reformed: The Case for a New Order in Higher Education (Washington, DC: Education Sector, 2006).

4. Letter from Gerhard Gasper to James Fallows, September 23, 1996. Although the letter was sent as private correspondence, it has since entered the public domain with the permission of both parties. http://www.stanford.edu/dept/pres-provost/president/speeches/961206gcfallow.html.

5. Chris Smith, "News You Can Abuse," The University of Chicago Magazine, October 2001.

6. Thomas Sowell, "College Shell Game Ignores Best Fit," Creators Syndicate, October 29, 2007.

7. Steve Stecklow, "Cheat Sheets: Colleges Inflate SATs and Graduation Rates in Popular Guidebooks," The Wall Street Journal, April 5, 1995.

8. As does research demonstrating the relationship between survey results and widely accepted academic outcomes, see for example George D. Kuh, Ty Cruce, Rick Shoup, Jillian Kinzie, and Robert M. Gonyea, Unmasking the Effects of Student Engagement on College Grades and Persistence, Center for Postsecondary Research, Indiana University–Bloomington, presented at the Annual Meeting of the American Education Research Association, 2007.

9. George D. Kuh, "Promises and Pitfalls of Institutional Transparency," Change, September/October 2007.

10. This may change, if college credits become more portable and higher education providers who aren't bound to fixed geographic points become more prevalent. But it is still the case at the moment, particularly in the traditional higher education market to which rankings discussions are most applicable.

11.Amy Graham and Nicholas Thompson, "Broken Ranks," The Washington Monthly, September 2001.

12. See for example Peter Applebome, "Fighting the Rankings of a College Guide," The New York Times, January 5, 1997, which begins: "An unlikely revolt of students against grades—not their own, but the ones their colleges and universities get from a national magazine—is becoming a cause celebre on a growing number of campuses."

13. Mary Beth Marklein, "Beyond Rankings: A new way to look for a college," USA Today, November 5, 2007.