With the NFL regular season over, it's high time to see how my preseason projections ultimately played out. I've previously put together an initial post, a quarter report, and a half-year report.
Rather than asking you to click through a lot of stuff (especially since there are some confusing issues with the first two posts), allow me to try to explain things here in a way that hopefully makes sense.
Before the season, I had the idea to test a basic principle of large samples on the one position in the NFL that really accrues them over a single year: quarterbacks. The principle: That with large enough samples over a short enough period, performance becomes reliably projectable. So, I took something I've picked up in baseball, that three years is enough time to accrue a sample large enough that luck mostly washes out, but that doesn't go back so far as to make the numbers irrelevant to the future, and made it one of the guiding pieces of my puzzle.
So with that, I had the basic structure for my QB projections: I would use three-year averages to make a projection for every likely potential starting quarterback in the NFL (I would later just ignore the quarterbacks who did not accrue an adequate sample during the season). The projections themselves would be based entirely on rates (specifically: completion percentage, TD percentage, INT percentage, sack percentage, yards per completion, and yards per sack), so that the counting stats could be prorated easily to any particular number of attempts, and would use ANY/A as the ultimate measure of performance (because, despite some specific issues I still have with it, I feel it's the most complete single stat we have right now to measure QB performance with).
For most quarterbacks, I could reliably just go more or less with the averages themselves. But I also wanted to try to account for growth in young players and decline in old players, though I resolved to be conservative with both. With all that in mind, I decided that my projections would rely on the following principles:
- Under most circumstances, quarterbacks must be regressed near their three-year average rates (Completion%, TD%, INT%, Y/C, Sack%).
- For quarterbacks younger than 28, I can project conservative positive or negative growth only if a) the trend is already apparent in the three-year sample, b) the available sample represents less than three years' worth of data and a typical development curve can be reasonably presumed, or c) the available sample represents less than three years' worth of data and a coaching or personnel change suggests it.
- For quarterbacks older than 32, I can project conservative decline only if the trend is already apparent in the three-year sample.
- For rookies, my best guess would have to do (I'll have to improve on the "wild-guess" system in the future, possibly by tapping the Lewin model).
In the original post, I projected out attempts numbers, but that was more or less a formality: Once the season actually got underway and quarterbacks actually started to accrue pass attempts, I could simply prorate the projection to match the actual number of attempts for each quarterback. Creating a projection for EVERY potential starter was also a bit of a formality, mostly done for the sake of thoroughness, as any reliability within the projections would depend on the players actually getting a large number of attempts (basically, a big deviation from a projection in 40 attempts doesn't tell you anything about how good or bad the projection was, but a big deviation from a projection in 400 attempts can tell you a lot). I decided that a quarterback who started 3 out of every 4 games for his team would have enough pass attempts to make comparing his performance to his projection worthwhile.
For ease of reference, I've changed the original spreadsheet to do two things: 1) It only reflects the projections I made for the 26 quarterbacks who ultimately started 12 or more games during the year, and 2) I've prorated the original projections to 500 attempts for everyone--this is simply to emphasize the importance of the rates and the unimportance of the totals for those original projections.
One important thing to keep in mind as we go: Because I'm tossing out quarterbacks who started fewer than 12 games, there's a minor survivor bias built in. I'm not sure if there's a good way to account for this other than to simply cop to it. My original projection for Michael Vick, for instance, was not accurate, but because he did not qualify, he's not included. Was the projection inaccurate because the sample never got large enough for Vick to adequately regress (as players with equally extreme starts, like Tony Romo, Christian Ponder, and Josh Freeman, ultimately did by seasons end), or was the projection simply poor? I'm not sure. I'm not sure there's a good way to be sure. It's also true that other projections were fairly good, but still got knocked out for playing time (it was projected that Matt Cassel and Blaine Gabbert, for example, would be very bad, and they were... they just weren't very bad enough for long enough to make the comparison reliable). Just keep these wrinkle in mind as you read.
Also keep in mind that 26 guys still managed to qualify, and that's a lot. It's not like I've knocked out half the league just to make myself look good.
Preamble complete, I want to give you the original spreadsheet, adjusted as I've outlined above, so you can actually see the rates I assigned all of these players in the preseason:
You should be able to click that to see the full version. Remember: I'm not actually saying that everyone would have 500 attempts. That's ridiculous. I just needed a nice round number so the equations could do their work. The important numbers here are the rates, and the really important numbers are the ANY/A ones, and I suppose the passer rating ones if that's your bag.
For the next step, I'll prorate those original projections against the actual attempts taken by every one of those quarterbacks, so we can see how close they actually came out. I won't include all of the right-most columns in the rest of these images, because you wouldn't see any meaningful changes there anyway and they would just serve to make the images really wide.
Previously, I ranked my projections in half-yard chunks. If the projected ANY/A was within half a yard of the actual ANY/A, I called it a good projection. If it was between half a yard and a full yard, I called it pretty good. And so on. This was very rough, and not really based on anything. This time, I've used the ANY/A range for the quarterbacks who started 12 or more games to come up with something better. Basically, the difference between Peyton Manning and Mark Sanchez was 3.5 ANY/A. If I assumed that 3.5 ANY/A was 100% of the useful range in 2012, then every .35 ANY/A would represent unit of 10%. By this, if my projection was 100%-90% accurate (0.00-0.35 ANY/A), it would get an A. 89%-80% (0.36-0.70), a B. 79%-70% (0.71-1.05), a C. 69-60% (1.06-1.40). a D. And anything worse, an F.
However, because I calculated the players ANY/A to the tenths place, I had to do a little rounding. Therefore, the ranges are as follows:
Deviation between 0.0 and 0.4 ANY/A = A
Deviation between 0.5 and 0.7 ANY/A = B
Deviation between 0.8 and 1.1 ANY/A = C
Deviation Between 1.2 and 1.4 ANY/A = D
Deviation of 1.5 ANY/A or greater = F
Let's get started on the results then. Here are the projections I got A's on. For every player, the top line is the projection and the bottom line is the actual season.
So out of 26 final projections, 10 get A's. That's... decent.
For a lot of these guys, this doesn't tell us a hell of a lot we didn't already know. Drew Brees, Joe Flacco, and Ben Roethlisberger are three of the most consistent quarterbacks in the league, so it would have been pretty embarrassing not to nail those ones. Assuming conservative leaps for young guys like Dalton and Ponder paid off big here (it pays off less well for a couple other guys later--but that's what happens when you don't have three-years' worth of data to project from).
The most interesting quarterback on this list to me is Sam Bradford. I projected maybe more improvement from career numbers for him than anybody else, because I presumed that both the extra year of experience and the new coaching staff would benefit him (nobody else had the benefit of both, as I recall). He represents perhaps my greatest success on the list. The question for him going forward is going to be: How much of his improvement came from the coaching change and how much came from actual development?
The rest of these guys are, for all intents and purposes, who we thought they were. Which is a good result for the projection system.
Let's do the B grades:
This is where things start to get interesting for me. The relatively accurate projection for Andrew Luck was, well, practically sheer luck. I mean, I based it on nothing. The others are more interesting.
I'm particularly happy with the projections here for Matt Ryan, Matt Stafford, Jay Cutler, and Aaron Rodgers.
Matt Ryan was supposed to have a breakout season based on preseason prognostication, but when all was said and done he was still pretty close to what he'd done the three previous seasons. The two biggest differences for him from his past were that he completed a much higher rate of his passes and that he threw almost 50 more passes than he ever had previously.
Matt Stafford was coming off of a record-setting season. A top-10 season. But the three-year averages demanded some negative regression. In fact, I projected that he would outperform his three-year average, because the positive trend was apparent in the sample, but that still came out to something well below his 2011. In fact, he was pretty close to his three-year averages, and much, much closer to my projection than to his atmospheric 2011. If anything, a more conservative projection would have served me better.
Jay Cutler is known for his consistent inconsistency, and should have been difficult to peg. There was a positive trend in his three-year sample, and I apparently was taken in by it and applied it to the projection. Because he was not younger than 28, this should have been against the rules, but I am stupid. His actual three-year ANY/A was 5.4. His actual 2012 ANY/A was 5.4. I didn't buy into my own system, and lost accuracy because of it. Lesson learned.
That said, the fact that it was still a B-range projection probably means I'm only kind of stupid and not completely stupid.
Aaron Rodgers's projection is one I'm especially proud of, because it's a perfect example of trusting regression. Early in the season, he was seriously under-performing his projection. Once that happened, regression did not tell us that he would hit the projection by the end of the year, but that he was very likely to approach it. That is exactly what happened. I'm very pleased with the fact that, despite his horrid start, he ended up as a B-level projection. Big numbers work. Hooray!
And, overall, I had 16 qualified projections in the A-B range. And I'm pretty happy with that.
Let's look at the C-grades:
LOL Mark Sanchez.
I'm actually pleased that my Mark Sanchez projection wasn't worse than this, just because he was THAT BAD. In a sane world, he would have been a victim of the survivor bias I talked about before. But this isn't a sane world. This is Rex Ryan's world. That said, it was still a poor projection.
This group is interesting overall, though. Tony Romo made a very valiant effort in the second half to make his projection respectable, and is another very good example of regression being a thing that is real. By the half-season report, I was commenting that maybe Tony Romo was simply toast. Maybe the camel's back had broken. Maybe you just can't project that kind of self-destruction. But, nope. In the second half, he was exactly the Tony Romo he's always been. The first half, of course, means that the overall projection ended up being poor, but I am again pleased that some basic rules of regression ultimately won the day.
Earlier I mentioned how conservative projections for Andy Dalton and Christian Ponder paid off big time, and now I have to mention how an equally conservative projection for another second-year player kind of bit it. Cam Newton had a very fine season in the end, rather significantly improving on his production from 2011--even with a lousy first half. One of the key predictors of pro success for college quarterbacks in the Lewin system is improvement from the junior to senior years. I wonder if less conservative development should be projected for qualified players who significantly outperform their Lewin expectations as rookies...
In any case, Cam Newton is a very exciting player. He's done nothing but get better every season he's been a starting quarterback at any high level, and he's already a very productive player. How high is that ceiling, exactly, at this point?
Peyton Manning is incredible. I mean it. The man is not credible. After a year of not playing football, over the age of 32, with a three-year sample that suggests possible decline... he outperformed what was probably an optimistic projection to begin with by almost a full ANY/A. He had his best statistical season since turning 30, arguably the best statistical season in the NFL, on a new team, without some his favorite receivers. I have no explanation for Peyton Manning. He defies logic. He's incredible.
And Josh Freeman. Josh Freeman. Josh Freeman. He's confusing. His inconsistency year-to-year is remarkable. After 2010, he looked for all intents and purposes like an elite prospect. Then his TD% nosedived and his INT% skyrocketed. His Y/C dropped, but his completion rate was at least improving. Until 2012, when his completion rate nosedived, his Y/C skyrocketed, and his TD and INT rates more or less stabilized. He had an extremely strange season statistically. Some things are elite. Some things are... what's the opposite of elite? And it's never the same thing from year to year. I have no idea what to make of him. He's weird. And my projection for him was bad. Was my projection for him bad because he's weird, or was it bad because I need to improve something? I don't know. But he's weird.
What's his deal, anyway?
Let's look at the one D grade, then:
Oh, look. A rookie. Nothing to see here. As I said above, I had no real system for projecting rookies. I was making wild guesses. These projections were destined to suck. Except for Luck. Suck except for Luck? Odd...
Oh goodie, the F grades:
Oh, look. Three rookies, including what are arguably the two best rookie seasons in NFL history. I'm okay with that, I guess. There's obviously a major problem with my ability to guess randomly, and I should probably develop some method by which I don't need to guess randomly. I think I can probably develop something reasonable using the Lewin system, but the biggest flaw was simply not recognizing the survivor bias sooner. The lowest ANY/A among quarterbacks who started at least 12 games in 2012 was 4.4. For Weeden, Tannehill, and Wilson, I was either guessing they would be way below that, or extremely close. I maybe should have presumed that were they to play enough games to qualify, they would need to be only reasonably below average, and not extremely below average. The more I know.
Still, I really don't mind that 4 out of my 5 really terrible projections were for rookies. It seems to me like that is how the system should work. You do well when you have lots of data available, and you do poorly when you have little, or none.
The exception was Philip Rivers, who really is terrible now. I mean, man. He's terrible. He's so bad I don't actually believe that he can keep being this bad. But then he keeps being this bad. Will he actually keep being this bad next year? I don't actually believe that he can. But...
Okay, I'm really tired of typing stuff.
