| Summary: The Colley Matrix Basketball Rankings are compared to both the NCAA Tournament Selection Committee seedings and to the results in the NCAA Tournament itself. Surprisingly, in 2002-03, 2003-04 and 2004-05, the Colley seedings agree to within 21%, 22% and 22% of the committee's seeings (I say 5, they say 4; I say 10, they say 12). Perhaps more surprisingly, the committee's seedings agree with the actual tournament results only 3.3%, 8.6% and 0.55% better than seedings based purely on the Colley Matrix's pre-tournament rankings. Moreover, the 02-03, 03-04 and 04-05 tournaments featured 21, 16 and 19 upsets, in which the lower-seeded team beat a higher-seeded team, while 20, 16 and 15 of those games would have been upsets according to the Colley Matrix rankings... slightly better than the committee! |
The most obvious ranking against which any college basketball ranking should be compared is the NCAA Tournament Selection Committee seeding. The second most obvious thing is the actual results in the NCAA Tournament. The links above provide tables which do just that. However, some explanation is necessary to understand what the tables are telling you.
The teams are sorted according to their seeds in the NCAA Tournament. On each row, you'll also find the seeding Colley Matrix would have issued for that team and the round in which the team actually finished in the tournament. We list only teams seeded between 1 and 12 to avoid getting into the lesser conference champions with automatic bids.
The fifth column, "Finish Seed" is the seed the team should have had to get the actual result achieved in the tournament by the team. For teams that finished in the Final Four, the correct seed was obviously 1. For teams that finished in the Elite 8, the correct seed is simply 2, because, in an ideal world, all 2 seeds would presumably make the Regional Final, but lose to the 1 seed. Sweet 16 finishers, ideally, should have been 3 or 4 seeds (we don't know which), so the Finish Seed is reported as the geometric mean of 3 and 4. Similarly, Second Round finishers, ideally, should have been ranked anywhere from 5 to 8, and so the Finish Seed is reported as the geometric mean of 5, 6, 7 and 8. The Finish Seed for First Round finishers is the geometric mean of numbers from 9 to 16.
The last three columns are the results, which are given as the ratios of the Colley Seed to the Committee Seed, the Colley Seed to the Finish Seed, and the Committee Seed to the Finish Seed. All ratios are expressed as greater than one.
At the bottom of these three columns are the main results, denoted as "Absolute Mean Ratios," which are the geometric means of ratio columns over the 48 teams in the table. In practice, this statistic is the exponent of the mean of the absolute values of the logs of the ratios of the two seedings compared. The absolute value means that there is no cancellation between high and low errors; all errors are treated as "high," so all errors add up coherently. In symbols, the statistic is simply this,
This statistic is telling you the expected ratio of a seeding from one sytem vs. that of another. If the statistc were 1.1, for instance, we would expect the seedings from the two compared methods to agree to within 10% of each other (one says 10; the other says 9 or 11).
When we now examine the bottom of the tables, we discover that, incredibly, the Colley Matrix Seedings agree with the Committee's seedings within about 20%. That means that our seeings would only differ by one place at the #5 level; in other words, if the committee said #5, I might say either #4, or #6. If the committee said #10, I might say #8. This is a rather shocking agreement when one considers that my system is based purely on wins and losses over the whole season, and gives no explicit consideration to home vs. away, conference, conference title, "hotness," injuries, strong OOC wins, or any of the other factors that the committee considers strongly.
Finally, looking at the last two columns, we have the agreement between the actual results in the tournament vs. both the Colley Matrix seeds and the committee's seeds. In 2003, the committee was within 61% of the correct seeding; the Colley seed was within 66%. So the committee was doing 3.3% better. In 2004, the committee and the Colley seeding faired more poorly, with the committee off by 71% and the Colley seeding off by 86% (the committee was doing 8.6% better). In 2005, the Colley Matrix and committee had remarkably similar results at 74% and 73% error (only 0.55% difference!).
We should expect the committee, which has the benefit of many computer systems, and all the other factors mentioned above to do somewhat better than any one computer system. What's surprising is that the committee only doing about 10% better than the Colley Matrix in the last two years of the tournament. Furthermore, the fact that the Colley Matrix agrees with the committee better than either agrees with the acutal results tells us that most of the "mistakes" made by the Colley Matrix were also made by the committee.