clock menu more-arrow no yes mobile

Filed under:

NFL Draft Grades: No Longer in the Eye of the Beholder

Each year around this time, the NFL punditocracy mobilizes itself in pursuit of one goal: making absolute fools of themselves.  OK, so maybe that's not exactly their intention, but assigning grades seemingly seconds after NFL teams completed their drafts is an activity that just invites ridicule. And ridicule them we do! More often than not, the pundits end up like that guy in your fantasy league who spends his first 30 post-draft minutes standing in front of the big board, gracing everyone within earshot with his "expert" opinions of their drafts; which, of course, end up having no relationship whatsoever with your league's final standings.

Based on the myriad of "Re-Grading the [insert year here] Draft" articles that crop up 3 to 6 years after the fact, it's pretty apparent that even the pundits are in on their own joke. They know as well as we do that evaluating a draft before any of the picks actually play a down in the NFL is a mug's game. But, in my view, it's not just the rapid-onset factor that's driving the silliness of draft grades. Naturally, I think another problem is that their just so darn subjective. Looking back at the 2001 draft, I'm sure people writing about the Falcons think Michael Vick, and therefore their entire draft, was a disappointment. On the other hand, writers in Philadelphia these days think the Falcons made a great pick. The same can probably be said about what people in New Orleans think about the Chargers' 2001 draft as opposed to their counterparts in San Diego. To some degree, draft evaluations truly do seem to be in the eye of the beholder.

As I'm wont to do, I thought up a solution to the subjectivity problem, did a little hardcore statistics, and developed a way to objectively evaluate every pick by every team in every draft since the AFL-NFL merger. It's basically an extension of a series of posts I did last April, as well as an application that Doug Drinen of Pro Football Reference (PFR) suggested when he developed his Approximate Value (AV) stat. Basically, it all came down to answering 2 questions:

  1. How well did a draft pick perform?
  2. How well was he expected to perform given the selection in which he was picked?

If we know objective answers to those 2 questions, then we can look at all of a team's draft picks, and come up with a number that represents the extent to which that team drafted players who outperformed expectations. If, in the aggregate, a team's picks vastly outperformed expectations, we can then objectively say that the team had an excellent draft. On the flip side, if the picks vastly underperformed expectations, we can then objectively say the opposite. I suppose you could say that, after this series of posts, no longer will draft grades be in the eye of the beholder.

Speaking of the series, here's the schedule:

Monday - An introduction to my draft evaluation system

Tuesday - Overall team grades for the 2006 NFL Draft

Wednesday - Team grades for their picks in Rounds 1 and 2 of the 2006 NFL Draft

Thursday - Team grades for their picks in Rounds 3-7 of the 2006 NFL Draft

Friday - A discussion focused on the 49ers' 2006 draft

After the jump, the methods behind my draft evaluation madness...


Fundamentally, the key to all of this was coming up with a reliable indicator of the kind of career we should expect a given draft pick to have. That's because the subjectivity of expectations is what drives the draft grading problem. For instance, consider J.J. Stokes, whom Fooch anointed as the 49ers' 2nd-biggest draft bust of the past 20 years. As Niner fans, and as Fooch admitted, we look at Stokes with such disdain because we expected so much from him, and paid such a high price to get him. He was supposed to be the heir apparent to Jerry Rice (lofty expectations indeed!). He was the #10 pick in the draft, which the 49ers had to proverbially mortgage the future to acquire. Either he was going to be the best thing since sliced bread, or we were going to use the bread knife against him. As you'll see a bit later, though, it turns out that Stokes had a career that was about 95% as valuable as the career you'd expect from a #10 pick. To put that 95% number in perspective, 14 players taken by the 49ers in the first 2 rounds gave the team an even lower return on their investment than Stokes did. In other words, our subjective expectations lead us to view him as the 2nd-biggest bust, when objective expectations suggest he's actually the 15th-biggest bust.

So, if the key is reining in our expectations, how did I go about doing that? Good question. For the Niners Nation (NN) veterans, you might recall I did a post last April that showed how the performance of draft picks follows a predictable pattern represented by the logarithmic decay curve. In English, this basically means that, as you go deeper and deeper into the draft, the average performance of each successive pick - as measured by PFR's Career AV stat - gets worse and worse. Except that, rather than getting worse along a straight line, it gets worse and worse along an L-shaped curve, which is called a logarithmic decay curve because the average Career AV for a given pick is directly related to the natural logarithm of the pick number (I promise you a pretty graph of this in a minute).

For this post, I went ahead and redid the analysis that results in the equation for that L-shaped logarithmic decay curve; with two improvements. First, rather than using all draft drafts 1994-2005, I used a random sample of drafts from 1970-2004. Second, instead of using all of the picks in each draft, I used only the first 220. The fundamental reason for both of these improvements is that not all drafts are created equally. The most obvious difference, of course, is that the NFL draft hasn't been a static 255-pick affair since 1970. For instance, the 1976 NFL Draft had 17 rounds totaling 487 picks, whereas the 1994 version only had 222. Clearly, then, it's kind of difficult to come up with an average performance for, say, the 450th pick, when only 1 draft since the merger even had a 450th pick.

The other, more statistical way that post-merger drafts differ from each other is the extent to which the logarithmic decay curve represents the relationship between pick number and performance. In the graph below, you can see that, over time, the draft has become more and more logarithmic: 


This is an interesting finding in and of itself, but I won't dwell on it. Suffice it to say, though, another (less obtuse) way to put, "the draft is becoming more logarithmic" is that it's becoming more predictable over time (notice the white, upward-sloping trendline). As you can see, over the 35 years depicted in the graph, predictability has doubled. The implication of this for my analysis is that taking a random sample of drafts eliminates the potential for results that might be skewed by the trend. For instance, when looking at the graph, it sure appears that the results of my original analysis were probably skewed by the fact that the NFL draft had a sustained, high level of predictability from 1994-2005.


So, with that little methodological detour out of the way, let's move on to the fun stuff. Below is the first fruit of my labor, a graph showing how the average performance of a draft pick decreases as the draft proceeds, along with the equation that represents the relationship between performance and pick number (click to enlarge):


That R-squared value of .880 means that only 12% of the performance differences between picks has nothing to do with the order in which they were picked. Those of you who read my stuff regularly know that an R-squared that high is pretty much unheard of in NFL stat analysis. Those of you who don't read my stuff, first, welcome. Second, just understand that you can basically chalk up that "unexplained" 12% to things like voodoo, steroids, and illegally videotaping your opponents.

Now, at this point, let me just mention that PFR's Chase Stuart did an analysis just like this one - minus my methodological improvements - and found essentially the same thing. However, his analysis was in the context of the draft trade value chart (more on that some other day). To my knowledge, no one at PFR has ventured into the territory where I'm about to take you, which is to use the above equation as a means for quantifying (i.e., removing the subjectivity of) the performance expectation of each draft pick, and thereby allowing for an objective evaluation of the pick.

To explain how it works, let's return to one, J.J. Stokes. Remember how I told you that his career was about 95% as valuable as the career you'd expect from a #10 pick? Here's how I came up with that percentage. First, Stokes had a Career AV of 41 to show for his 9 seasons, which equals a Career AV per Year (AV/Yr) of 4.56. Next, I plugged the number 10 into the equation in the graph, which resulted in an expected AV/Yr of 4.85. The difference between 4.56 and 4.85 is 0.29, which means that Stokes underperformed expectations by 0.29 AV/Yr. Alternatively, you can represent this underachievement as a percentage: Stokes' actual performance of 4.56 AV/Yr was 93.93% of the 4.85 AV/Yr we should have expected from him. Pretty cool, huh?

Well, that's just the beginning. Now that we can objectively evaluate each individual draft pick, we can then, say, combine all of the draft picks for a given team in a given draft, and see how good of a return that team got on their draft investments. For example, take the 49ers' 1986 draft, which some consider to be the best draft post-merger. In that draft, they had 13 picks. If we add up the actual AV/Yrs for those 13 players, we get a total of 40.08 (i.e., AV/Yr for Larry Roberts = 3.00, plus AV/Yr for Tom Rathman = 4.44, and so on). Next, if we add up the expected AV/Yrs for the 13 slots where those players were taken (i.e., expected AV/Yr for pick 39 = 3.09, plus expected AV/Yr for pick 56 = 2.63, and so on), we get a total of 21.78, which means that the Niners got a 184.02% return on their investment in the 1986 draft (40.08 divided by 21.78). I haven't had the time to go back and evaluate every draft by every team to be able to tell you where that ranks post-merger, but, for the sake of comparison, the Steelers got a 216.78% return on their 1974 draft, another draft class mentioned as one of the best ever (40.69 actual AV/Yr divided by 18.77 expected AV/Yr).

What I've essentially created here are 2 measures of how well NFL teams get value above and beyond what's expected simply from the draft slots at which they picked. First, the difference between total performance and expected performance across all of a team's picks can be seen as a measure of drafting value added. The team was presented with various picks associated with various expectancies. By how much did the players they actually selected beat or fall short of those expectations? Second, the ratio of total performance to expected performance across all of a team's picks is a measure of drafting efficiency. Rather than absolute value above or below expectation, what kind of percentage return on their investment did they get from their actual selections?

Really, with these 2 arrows in one's quiver, the potential applications here are endless. For example, is it better to stockpile picks so that you're simply more likely to add more value or is it better to get a huge return from a small cache of picks? What do these stats have to say about my long-held view that the best teams have a knack for avoiding busts in the early rounds, and finding diamonds in the rough during later rounds? Unfortunately, I have yet to develop a device that gives me infinite time and space to explore these questions.

For now, though, Fooch has been kind enough to give me the rest of the week to test drive this system on the 2006 NFL Draft, from which we're now 5 years removed. Which teams added the most value with their picks? Which teams were the most efficient? Which teams' early picks woefully underperformed expectations? Which teams' late picks vastly overachieved? Where did the 49ers rank in all of these things? You're going to have to stay tuned to find out, but at the very least we can finally put to rest the notion that the only draft grades you're ever going to see are the ones spit out by pundits on a deadline. To put it differently, from this day forward, draft grades are no longer in the eye of the beholder.