Ranking Canada’s law schools
They’re all hard to get in to. Which one will you get the most out of?
This is the second year that Maclean’s has ranked Canada’s law schools, and this year’s methodology follows the same approach as last year—with a few improvements. The goal remains the same: to objectively assess each school against recognized measures of faculty quality and graduate employment quality. Are a law school’s professors significant contributors to the intellectual life of their discipline? Do a law school’s graduates land the most sought-after jobs in government, the private sector and—new this year—academia?
The methodology behind the Maclean’s law school ranking was created in co-operation with professor Brian Leiter, the John P. Wilson Professor of Law and director of the Center for Law, Philosophy, and Human Values at the University of Chicago. Leiter is also America’s most prominent critic of that other, well-known law school ranking: the U.S. News and World Report ranking of American law schools. Leiter has long criticized U.S. News’s methodology as misguided and open to gaming. One of his blogs, at www.leiterrankings.com, features his own alternative rankings of law school quality, focusing on outcomes rather than inputs, and using data that cannot be manipulated by institutions. That’s why, in 2007, we asked Leiter to work with us to build a Canadian law school ranking based on his criticisms of and alternatives to the U.S. News approach. “My central motivation for undertaking this task,” says Leiter, “was to show that it’s actually possible to evaluate law schools in ways that are meaningful.”
All of the data used in the Maclean’s law rankings are publicly available. All focus on law school outputs. Fifty per cent of the overall ranking is determined by faculty quality, and 50 per cent by graduate quality. The four measures of graduate quality look at the success each law school has had producing graduates able to land the most competitive public and private sector jobs. The four indicators are:
Elite Firm Hiring: We calculated how many of each school’s graduates are serving as associates at law firms on Lexpert’s list of the largest firms in nine Canadian regions, or at one of the five leading New York firms, according to the employment website Vault. This was done by examining the online biographies of thousands of lawyers at dozens of law firms. To scale this measure to the size of each law school, the tally was divided by the size of each school’s first-year class, averaged over the past two years. This measure is worth 20 per cent.
National Reach: This indicator, based on the Elite Firm Hiring measure, is worth 10 per cent. It measures the proportion of each law school’s grads at leading firms who are working at firms other than the three that hired the most grads from this school. It’s a measure of the extent to which leading firms outside a school’s region hire its graduates.
Supreme Court Clerkships: A measure of how many of a school’s graduates have served as clerks at the Supreme Court of Canada—there are 27 clerks each year; it is one of the most competitive positions open to graduates. We looked at the last six years’ worth of clerks. As with the other measures of graduate quality, the tally was divided by each school’s average first-year enrolment.
Faculty Hiring: This new indicator is worth 10 per cent. It looks at how many of a school’s graduates are professors at Canadian law schools, with extra weight given to grads hired by faculties other than their alma mater.
page 1 page 2
One of the problems with this ranking is that it implicity assumes the students entering law school are homogeneous. They’re not. The top students enter the top schools, and we should expect them to garner better jobs even if the law schools are all identical.
What we really need is a ranking that somehow computes “value added” by law schools – or anything else one cares to evaluate. I have yet to see this accomplished and cannot think of how to do so without violating privacy laws.
Andrew, how does the ranking implicitly assume that students entering law school are homogeneous? I don’t follow.
Andrew raises a good point…. A very good point, in fact, which is why I’m going to devote such a long response to it.
But first of all, a qualifier: Andrew’s point doesn’t apply to the “Faculty Quality” portion of the law school ranking, worth 50%. Faculty Quality is an output measure that can’t be subjected to the “input subtraction” that he is suggesting. So put Faculty Quality aside. Let’s just talk about the remaining 50% of the ranking, namely “Graduate Quality.”
The Graduate Quality measures on the law ranking show that graduates of U of T law are more successful at landing the most competitive legal jobs than graduates of, say, U Moncton law. That is something worth knowing. But Andrew is pointing out that the law ranking doesn’t answer another question. It’s a question we didn’t set out to answer, but certainly a question worth pondering. Question: Do U of T grads enjoy greater labour market success than U Moncton grads due to the fact that U of T students may have already been ahead of U Moncton students before they even entered law school (with higher undergraduate averages, for example)? Or is the gap between the job market outcomes of U of T and Moncton grads partly or wholly attributable to some unknown X factor that we’ll call U of T’s valued added? The X factor might be explained by better teaching at U of T, or better career counseling, or job placement services, or a better alumni employer network or… or, well, who knows? If we could compare the entering quality of students at each Canadian law school, and then compared those inputs with the rankings outputs, which law schools’ graduates would we find went on enjoy better than expected job market outcomes, and which experienced worse than expected job market outcomes? All interesting questions. Just not the questions that we were asking.
Our ranking results tell and reveal a great deal -– much of which has been only discussed anecdotally until now, in the absence of this kind of hard data. But our ranking should not be taken as proof that U of T does a better job of increasing the intellectual capital of students than, say, Moncton. The fact that the profs at U of T are on average more prominent in their fields, as measured by the citations indicator, and the fact that grads of U of T are more likely to land the most competitive private and public sector jobs suggests that something positive is happening at U of T. But the ranking doesn’t explore the factors that went in to creating those positive outcomes, or the relative difference in student quality inputs that went into the outcomes. The ranking just measures the outcomes – and those outcomes are important.
How important? Well, if you were the parent of a 12 year old, and I told you that one high school in your neighbourhood sent 100% of its graduates to university and another high school sent 50% of its graduates to prison, I’m guessing that you would probably find this data to be of considerable interest in selecting which school your son or daughter should attend—even in the absence of a statistical analysis of the quality of each high school’s incoming students, and a calculation of each institution’s valued added as measured by the degree to which spending five years at high school X did or did not add to its students’ likelihood of achieving post-high school educational success.
But let’s go back to critiquing the ranking. The job market outputs that we measured in the ranking are to some degree — we obviously don’t know to what degree — dependent on the inputs, i.e. incoming student quality. Better incoming students to some degree equals better graduates. It might even be possible to measure this relationship between inputs and outputs.
Given a bit of time and money (can anyone spot me a few grad students, an economics professor and a SSHRC grant?) and willingness on the part of universities to open up several years or decades worth of admissions files, and an ability to track those graduates into the labour market, it should be possible to calculate an output-input difference at each law school, by doing a study similar to the one done a few years ago by Princeton economist Alan Krueger. It would be fantastic if someone were to do such a study in Canada. Krueger compared the career outcomes of students who studied at Harvard college with those who were accepted at Harvard, but instead went to less prestigious U.S. state schools. What he found is that inputs in that case were important, even determinative: two decades after graduation, the high performing high school students who chose not to go to Harvard enjoyed careers and incomes that were not substantially different than those of the high performing high schoolers who chose Harvard. It’s one of the reasons I’ve argued, in this magazine, that where you go to university for your undergrad degree matters and is important and will make a difference in your life in a host of ways, but going to university at all – and what kind of a student and person you are, and what you want to get out of university and are willing to do at university and after university to make the most of your life-– will have far more of an impact in terms of career success.
Until a few years ago, the annual Maclean’s University Rankings issue included a rough value-added measure of undergraduate programs. Our statisticians compared the quality of entering students at each university — as measured by the high school grade averages of entering undergrads and percentage of students with high school grades over 75% — with two output measures: the percentage who graduated on-time from university, and those who won national student awards. It was a rough attempt to establish a relationship between the quality of a university’s student inputs and the quality of its graduate outputs. The university that finished first on value added for several years in a row, Lakehead, even made “Canada’s best valued added university” a campus slogan and centerpiece of a national marketing campaign.
Lakehead was able to claim that title because, though it was among the least competitive universities in terms of the students it took in, when compared to other Canadian universities, the percentage of Lakehead students graduating on-time and winning national awards was higher than statistically expected.
Denouement: a few years ago, a number of Canadian universities unfortunately decided to stop making public some of the information that we used to calculate this measure. We can’t calculate it anymore. The data isn’t available.
But back to law schools. So what’s the bottom line? What does the Maclean’s law school ranking tell us? It tells us how Canadian law schools compare in terms of faculty quality, according to one metric, and how the most competitive employers in the private sector, academia and government believe the law schools and their graduates compare, as revealed by whom these employers have chosen to hire. All of which is worth knowing, whether one is a student, a potential student, a faculty member or an interested member of the public. Does the ranking tell you something important about each law school? Yes. Does it tell you everything? No.
[...] This year’s ranking – the second that Leiter has done – has just been published at Macleans (here). In many ways Leiter’s rankings are a useful contribution. As he has suggested of rankings [...]
[...] Here’s their Canadian law schools rankings. [...]
The measure “Elite firm hiring” is fairly misleading and seems to rest on a number of faulty assumptions. And it is not at all clear how it measures “graduate job quality” or “graduates able to land the most competitive public and private sector jobs”.
First of all, it rests on the assumption that “large” means “elite”. What evidence is there that the largest firms actually have the most competitive or sought after positions? In my experience, the most sought after firm positions were with boutiques. Public sector positions with federal and provincial justice were also highly sought after: neither is addressed in the survey. Not to mention that since the largest firms tend to be clustered in the larger centres, this probably biases the survey toward schools in larger centres: no surprise then that U of T, McGill and UBC come out on top.
Second of all, what evidence is there that large firms provide the best “job quality” In this regard, I would think that a job satisfaction survey would be a better measure.
The fact that this flawed measure is then extended into the “National Reach” category such that it ultimately determines 30% of a school’s raises serious questions about the utility of this ranking.
THe most successful lawyers aren’t always the ones who have entered expensive schools. Its all about their own preparations and development. Its all about the attitude and pressure that makes the best lawyers (which does appear to be some of the more expensive universities) if you fail you’ve just wasted money.
Many of the McGill Faculty publish in French. Does the Faculty Citations criterion take this into account?
[...] Every year, the Internationally Trained Lawyer Program will aim to help 100 lawyers who need accreditation to practice law in the province. Michael Chan, Ontario’s minister of citizenship and immigration, launched the program a ceremony last Tuesday. More: Ranking Canada’s law schools [...]
[...] Ranking Canada’s law schools : Macleans OnCampus [...]
It’s all about their ethics, and it appears, that they are desperately needed:
http://ahabit.com/letter.htm
New graduates will hopefully not tolerate the miscarriages of justice that destroy the lives of victims and innocent people.