To me, evaluating draft grades has always felt like some sort of voodoo; sports writers and other simply assign a grade to a team’s draft haul based on how they feel about the players drafted…it’s not very scientific or rigorous. Seeing all these grades got me thinking: can we make draft grading rigorous, quantitative, driven by data? I think the answer is ‘yes’.
I started with Pro Football Reference’s Career Approximate Value, or CarAV. CarAV was created by PFR founder Doug Drinen, and the method is an attempt to "put a single number on the seasonal value of a player at any position from any year (since 1950)". This article isn’t meant to be a discussion on the merits of CarAV, so you can explore it yourself if you wish – most analysts accept it as a valid statistic that can be applied to all positions, including offensive linemen.
To begin the analysis, I looked at every player drafted in the 2010 NFL draft by draft position and CarAV. I then created a logarithmic curve fit for the relationship – its well accepted that this relationship is logarithmic. Here is the 2010 fit -- the outlier at the top of the chart is NaVorro Bowman:
Next, I calculated expected CarAV, or expCarAV, based on a player’s draft position. From there, I can calculate net CarAV, or netCarAV. I can then sum up the netCarAVs for each team that year to determine how well that team did drafting. Alternate approaches can include simply summing up CarAV for that team in that draft, but if you do that, teams that drafted high automatically come out on top unless you blew that. But with this approach, if you take a player at #7 overall, he better play like a #7 overall pick, or you’ll get negative netCarAV.
The next step is to convert team netCarAV into some kind of grade. I decided early on that I’d grade on a curve; this makes sense because an NFL draft is zero-sum. So, first thing I did is determine a percentile score for each team based on a normal distribution. I then created a grading curve where the 50th percentile team received a 75, a C. Results are below for 2010:
As you can see, the 49ers come out on top, followed closely by New England and Denver. If everything above 96.66% is an A+, then the 49ers, Patriots, and Broncos all got an A+. The Jaguars, Bucs, Jets, Vikings, Bills, and Rams all get Fs. Notice how poor of a score the Rams got; if we went with the other method, they would have scored well because they had the #1 overall pick.
A final step I took is to assess draft performance over the 2010-2013 time period. Since CarAV goes up with number of seasons, CarAV is already weighted for a time-lag: the 2010 draft matters much more for the 4-year assessment period than 2013.
These are raw scores; for the 4-year period the 49ers get a B+/A-:
Here are draft grades for each team by year – feel free to make them letter grades if you want.
Note: teams get credit for drafting a player even if they trade him away. So, even though Tayler Mays is a bust in the big picture, he still has a CarAV of 7, significantly below expected CarAV of 14.17, but still something…even though it was with another team. The philosophy behind this is that even if a player didn’t have room on a team, the GM still evaluated the talent level effectively…we also received credit for Marcus Cooper last year.
A final consideration is undrafted free agents. Seahawks WR Doug Baldwin has performed like a first round draft pick (18 CarAV) despite being undrafted. This makes the 2011 class for them even better…but you won’t see it here.