The NFL Draft is upon us and thus ends this year's cycle of talent evaluation for 32 teams and thousands of college prospects. Decisions will be made this week about who to pick, what players to pass on, and bets will be made on what players will bring the most impact for their respective teams. How those decisions are made vary from team to team, just as from company to company in the corporate world.
I make my living selling psychometrics. That is, all variety of tests and assessments that bring a level of measurement and analytic data that can be applied to making hiring decisions. And make no mistake about it, drafting talent - whether in the NFL or in any hiring situation, frankly - is truly the life's blood of any organization. Hire well and you can plan on years of high productivity and growth. Hire poorly and your organization will suffer just as badly.
The NFL's psychometric tool of choice is the Wonderlic test. Much has been written about it and MANY debates have ensued. For the purposes of my post, I'm not going to get too far into a debate about whether the Wonderlic itself is good or bad. I will, however, discuss methodology and its applicability to the NFL. The biggest question most people bring about Wonderlic (or any similar instrument) is: "Do they really work?"
Let's start by considering 2 main concepts for background purposes: 1) Validity of an assessment, 2) Predictive nature of an assessment.
"Valid" as defined by Industrial/Organizational Psychologists (the PhD's that design and deploy these things) is comprised of 3 items:
- The assessment measures what it is supposed to measure - in short, does the assessment do what it says it is designed to do?
- It is relevant to the job - A great example of this would be if you were giving a typing test to a high steel construction worker who will never see a keyboard. Sure.. they could pass or fail but it would never matter. More on this later.
- Established correlation between assessment scores and job performance - Do people who score higher on the assessment consistently perform better than those who score lower?
Predictive nature of an assessment is, basically, how directly a higher score on the assessment tied to a desired outcome on the job. Here are a few examples that will help to give you an idea of the correlation and predictive nature we're discussing and the scale (+1.0 = "perfect" prediction).
The correlation between:
- Taking ibuprofen and reduced pain = +0.14
- Drinking alcohol and aggressive behavior = +0.23
- Past behavior as indicator of future behavior =+0.39
- Weight and height = +0.44
- Nearness to the Equator and daily temperature = +0.60
As you can see, a number of these items are very strong predictors of the follow-on item, but none of them are perfect.
Hiring tools, similarly, are imperfect. The following items are tied to predicting successful job performance:
- Educational qualifications = +0.10
- Employment interviews = +0.15
- Ability tests/Job Samples = +0.45
- Assessment (testing) centers = +0.65
The point being that using well-designed and VALID assessments is more predictive than the correlation between higher daily temperatures and being closer to the Equator - a pretty powerful case. I hope there are only a few doubters with me at this point. The bottom line is that truly scientific assessments WORK. Now... back to the Wonderlic.
Wonderlic is a cognitive ability test - it tests how smart players are, the raw mental horsepower they possess, period. It doesn't measure a running back's ability to see holes in an opposing line, nor does it measure a quarterback's ability to read coverages. It does, however, give a good glimpse into the power of the player's processor, so to speak.
So, given all this psychology mumbo-jumbo, is the Wonderlic truly "valid." My answer would be that it depends on the position. Revisiting what "valid" means: Is it relevant to the JOB. "NFL Player" is NOT a job. Running Back, Tight End, Quarterback, Punter... those are JOBS as we would define them. In my opinion, the NFL is clearly using the wrong assessment because it really only applies directly to predicting the success of a few positions, most notably quarterback. That said, the NFL combine has many physical ability tests that are certainly indicators of NFL success but, again, not all of the tests are relevant for all NFL positions.
I don't possess the data to draw correlations between, say, 40 yard dash time and the ultimate success of a Cornerback in the NFL, but it's clear that faster is generally better. Same can be said for bench press reps and D-linemen.
So, if not the Wonderlic, then what? I would suggest working out what HR people call a "competency framework." Competencies are basically the skills, abilities, and attitudes that drive the desired results. Here is an example of my company's top-level competency framework:
As you can see, each competency is tied to specific behaviors. Translating this to the NFL, we could use the following example:
Desired outcome: More TD receptions from our WR corps
Competencies/behaviors of WRs that drive this outcome (not comprehensive, just suggestions):
- Physical characteristics: height, weight, arm length, speed, vertical leap (self explanatory)
- Cognitive ability - how smart is the guy?
- Resilience - how easy is it for this player to overcome failure, shake it off, and go again immediately?
- Character - do this player's values align with our organizational values?
The argument can be made that the NFL Combine does a great job of measuring the physical capabilities of the players, but what do they do to objectively measure "resilience?" Testing for resilience would have been a very valuable step to take when evaluating whether to choose Ryan Leaf or Peyton Manning! (I'm just saying)
In any data ecosystem, there are bound to be outliers and those that go against all scientific logic (ever seen a platypus??) A lot has been made of Frank Gore and his infamous score of a 6 on the Wonderlic. But, given the background I've laid for you here, it makes sense right? Running backs (especially the classic "north-south bell cow" backs) don't need to be the smartest guys on the field, but it sure helps if your quarterback is a smart guy. Oh... and never mind that Ryan Fitzpatrick scored a 48 out of 50 on the Wonderlic...and he completed the test in 9 minutes.
The bottom line, in my opinion, is that the use of the Wonderlic, while not the ideal instrument, adds at least a modicum of objectivity to an otherwise subjective evaluation system (outside the raw physical measurables, of course). What are your thoughts?