San Francisco 49ers Statistical Analysis Mailbag: Rookie Head Coaches

The first set of mailbag questions was definitely a varied one, with everything from DontForget's "When are the 49ers gonna win their 6th championship?" to John Hagee's - um, I mean Session9's - "How many of the 49ers players would survive the rapture under the differing Christian theologies?" Clearly, you guys hold my statistical abilities in high esteem, and I appreciate that.

THE QUESTION

Thankfully, for the sake of this reputation, mcwagner asked a question that I can actually answer, although it wasn't actually in the form of a question (yes, I am NN's Alex Trebek):

(What are) the success rates of rookie head coaches in a three year window and the current trends?

For the sake of brevity, and the fact that I'm working on a much (much!) larger, more detailed analysis predicting head coaching success in the NFL, I'm going to limit my answer in this post to the "current trends" part of mcwagner's question. Also, as it turns out, the average tenure of an NFL head coach is closer to 4 years than 3, so I'm therefore going to look at success in a 4-year window.

After the jump, some methodological and statistical mumbo jumbo, and then the answer...

THE METHOD

So, as is the case with any halfway-decent statistical analysis, the first thing we need to do is define terms. Specifically, for the purposes of answering this question, what do we mean by "rookie head coach" and "success rate?" Let's take these one at a time.

A rookie head coach in the NFL can be any of the following:

• 1. An NFL head coach who's in his first year with his current team
• 2. An NFL head coach who's never been an NFL head coach before
• 3. An NFL head coach who's never been a head coach at the NFL or NCAA level before

I'm going to go with definition #2 here for a couple of reasons. First, #1 is just the general case of #2 and #3, so it's redundant. Also, I'm pretty sure mcwagner wasn't thinking "a team's new head coach" when formulating the question given that we're obviously alluding to Jim Harbaugh here. Second, it turns out that, of the 22 "rookie NFL head coaches" I'm going to evaluate, only 3 had previously been a head coach at the college level. With a sample size of 3, there's no way to get a read on whether collegiate head coaching experience matters at all for a first-time NFL head coach.

If you're a long-time reader of my posts, you know I'm going to be using Football Outsiders' DVOA stats as a measure of success. For the newbies, I'll also a couple of standard measures of success (e.g., winning percentage). Those are the specifics. However, let's zoom out a bit. In a more general sense, the "success" of a head coach can be defined as either of these:

• 1. His team performed at a high level over the course of his tenure
• 2. His team performed at a higher level over the course of his tenure than the team he inherited

I'm going with definition #2 here for at least 2 reasons. First, aside from the occasional personality clashes (e.g., Al Davis vs. Jon Gruden, Lane Kiffin, Ghandi, Mother Teresa), it's almost by definition that coaching changes occur when the front office wants better performance than what they're getting at the moment. Sure, they want a guy who wins. But, they measure the "success" of their new head coach by whether or not he wins more than the last guy did. A perfect example of this is the Buccaneers replacing Tony Dungy with Jon Gruden after the 2001 season. Sure, Dungy won a lot of games. But Gruden's success as TB's head coach was going to be measured by whether or not he won a Super Bowl, not just whether or not he won a lot of games too.

Second, again, I think this better gets at that the underlying assumptions of mcwagner's question. As Niner fans, we want to know whether or not the hiring of Harbaugh is likely to make the team meaningfully better than what it was under Mike Singletary (and Mike Nolan for that matter). Sure, if the Niners went out and won 8 games each of the next 4 seasons, Harbaugh would have had more success - in an absolute sense - than half of the league. To us, though, that wouldn't be successful at all. Rather, we're looking for 10-to-16-win seasons, multiple playoff appearances, and another Lombardi trophy or 4.

Finally, it turns out that rookie head coaches inherit worse teams than NFL head coaching retreads. Specifically, among the 38 head coaches hired between 2002 and 2007, retreads were 5 times more likely to inherit a playoff team, took over teams that were, on average, 1 win better and ranked 5 spots higher in Total DVOA. Therefore, if I were to simply compare the performance of the retreads during their tenures to that of the rookies during their tenures, I'd erroneously conclude that the retreads did better because I'd be ignoring the fact that the retreads had a head start performance-wise. For you golfers out there, think of this as akin to the purpose of handicaps.

THE ANSWER

So, just to recap this is the analysis in a nutshell:

• 22 first-time NFL head coaches hired from 2002-2007 vs. 16 retreads hired from 2002-2007
• I'm looking at their performance relative to the team they inherited.
• Their performance is over the course of (at maximum) the first 4 years of their tenure.
• Relative performance is measured by winning percentage differential, Total DVOA differential, and playoff performance differential
• One little extra added thing I should mention up front is that Dom Capers took over an expansion team in 2002, and thus he did not technically inherit a team. The way I dealt with this was to substitute stats for "the worst NFL team in 2001" in place of stats for "the Texans in 2001." That 2002 HOU squad was pretty atrocious, so I think the equally atrocious 2001 Carolina Panthers is a reasonable proxy.

OK, so below is a table showing the basic data set I collected for the analysis. Remember, in this table, the performance stats are relative to the year before the head coach was hired. For instance, Nolan's stats are the difference between what happened in 2004 and what happened during his tenure from 2005 to 2008.

For winning percentage (WPct) and Estimated Winning Percentage (EstWPct, a stat that translates DVOA into a winning percentage), I've simply subtracted the team's winning percentage for the year before the coach's tenure from the team's average winning percentage during the coach's tenure. For Total DVOA Ranking (TRk), I've subtracted the team's ranking for the year before the coach's tenure from the team's best ranking during the coach's tenure. I did this because using the team's best ranking is an indicator of the coach's performance ceiling given the talent on the team.

Finally, for the various playoff improvement stats, I've used 1 to mean that the team improved during the coach's tenure (e.g., from a non-playoff team before he took over to making the playoffs at least once during his tenure), -1 to mean that the team got worse (e.g., from a playoff team before he took over to not making the playoffs at least once during his tenure), and a 0 to mean that the team treaded water (e.g., remaining a non-playoff team).

With that out of the way, here's the table (I've sorted both the rookies and the retreads in descending order of WPct):

Right off the bat, you can tell that there were a lot more instances of dramatic improvement when a team hired a coach who had never been an NFL head coach before than when it hired a retread. Furthermore, this is the case whether we look at winning percentages, statistical rankings, or playoff-related performance.

One thing the statistically savvy reader might point out here, though, is that this result can be chalked up to statistical ceiling and floor effects. For those who haven't read my prior explanations of the ceiling and floor effects, and aren't aware of them otherwise, the ceiling effect is basically the idea that it's harder to improve on something when you're already starting off at an above-average level. On the other side of things is the floor effect, which is the idea that it's harder to get worse on something when you're already starting off at a below-average level. Think of it this way. According to the ceiling effect, the Patriots can be 4 wins worse next season, but it's mathematically impossible for them to be 4 wins better, so it's more likely they're going win fewer games just because of math. In contrast, according to the floor effect, the Niners can be 8 wins better next season, but it's a mathematical impossibility for them to be 8 wins worse, so it's more likely they're going to win more games just because of math.

In rebuttal to this ceiling/floor effect argument, however, I'd remind everyone of what I mentioned earlier. Namely, for the head coaches I evaluated, rookies inherited teams that were, on average, 1 win worse than the teams retreads inherited. To give you the actual stats, rookies took over that were about 5-11, whereas retreads took over teams that were about 6-10. Given the actual numbers, then, it's not like we had a situation here where rookies are taking over doormats while retreads are taking over playoff teams. In other words, the retreads are starting out only slightly closer to the floor than the rookies, not dancing on the ceiling. Therefore, I'm not sure the whole ceiling/floor effect applies here.

To give a more concise picture of the improvements, first here's a summary table:

From the table, you can easily see that, although both rookies and retreads improve their teams (i.e., all numbers are positive) rookie head coaches improve their teams by about 10% in terms of winning percentage and about 5 slots more in terms of Total DVOA ranking. Combining this result with the fact that rookie head coaches generally command a lower salary, the conclusion here is that, in the current NFL, teams are more likely to get a bigger bang for their buck by hiring a head coach who's never held that position in the league.

Finally, here's a chart showing the differences between rookies and retreads in terms of playoff improvement (click to enlarge):

The ho hum result shown in the table is that rookies are more likely to turn non-playoff teams into playoff teams. That's to be expected given that they're more likely to inherit non-playoff teams to begin with. What I find interesting however, is what happens once rookies and retreads get into the playoffs. Specifically, rookies are nearly twice as likely as retreads to make a conference championship game participant out of a team that didn't go to the conference championship game the year before they got there. Similarly, rookies are about twice as likely to take them to the Super Bowl. So, the conclusion here is that, even if you ignore the fact that rookies are inherently more likely to improve a team in terms of getting them to the playoffs, they're still better at advancing deeper in the playoffs.*

Now, my question to you all for discussion in the comments section is, "Why have rookie head coaches been better recently?"

The first mailbag seems to have been quite a success, and I enjoyed doing the analysis for you guys immensely. Let's keep the questions coming, and I'll be back next week with a new answer. Remember, for now you can submit your questions in the comments section of this post, on NN's Facebook profile, and on NN's Twitter account. Fire away!

*Obviously, small-sample-size-related caveats apply here.