Kentucky Football: Paul Nelson Of Rank By Offers Responds to A Sea of Blue On Recruiting Evaluation

Below is an unedited response from the owner/operator of Rank By Offers, Paul Nelson, to our article of July 10th. You can re-read it to refresh your memory here.

I want to thank Paul for his response because it provides food for thought. He also told me in another e-mail that he's heard from a Kentucky football fan in Norway. The Big Blue Nation is spread out across the globe.

====Begin Response ===

I read your excellent article and comments. You have some very intelligent readers. They honed in on the shortcomings of the offer-based approach immediately. I’m always glad to hear constructive feedback, such as that you and your readers have provided. It will help direct the changes I’ll be making.

I tend to separate the problems into two categories … data quality issues and problems with the offer-based approach itself. I can’t do much about the data quality issues, but I do believe it will improve. Problems with the approach itself, I can address. Unfortunately, these are some pretty tricky problems, as you’ll see.

The data quality issues result from the haphazard way in which offers are currently made public. Recruits, their families, or their coaches tell someone from one of the recruiting services that the recruit has received an offer from a university. The university is prevented from confirming, denying, or making any reference to the recruit at all. With the exception of being published on the recruiting service websites, nothing is really done with the offers. There is no incentive, aside from pride, for recruits to diligently report offers. It’s easy to see by reviewing the data that more than a few recruits don’t report all of their offers. As wamarsh pointed out, many claim recruits report offers they haven’t received. I haven’t heard of any specific cases, so I’m reserving judgment. The method of reporting and publishing offers is about as bad as it can get right now. It can only get better. I do believe the data quality will improve the more that offer-based ranking is accepted. As I said before, several recruits have corrected their offers already after being exposed to RankByOffers.

If I were to design an ideal system for reporting offers, recruiting services, the recruits, their High School coaches, and their families would have nothing to do with it. The source of the offers, Division 1 programs, would be required to add their offers to a central repository sanctioned by the NCAA. That’s it. Nothing else would have to change. The college programs would still be prohibited from commenting publicly about recruits until National Signing Day. This would be a pretty simple system to implement. In addition to giving us a reliable source of data for offer-based ranking, it would solve a number of other problems related to the current offer reporting process. Not only would it prevent under-reporting and over-reporting of offers, it would also prevent some of the cases we’ve heard about recently in which recruits didn’t know they had offers from a particular school … in some cases for years. I’m going to propose that such a system be implemented to the NCAA. I think college Athletic Directors and coaches will support it. If you or your readers think of additional advantages to this kind of offer reporting system, please let me know. I’ll include them in my proposal.

Even if this ideal offer reporting system were to be implemented, the problems with the offer-based approach itself would remain. As hoboat33 and Dana Stinson pointed out, the problem of early commits is one that needs to be solved for this to become a ranking system that can stand on its own. The problem of early commits is a complicated one. There are some mitigating factors that make the problem somewhat less serious than it first appears. Recruits who are identified as being elite early in the process receive most of their offers extremely early these days. There are several players at the top of the offer-based rankings who were early commits. So it isn’t all early commits who are severely penalized. Another factor minimizing the impact of early commitment is the "intimidation factor". Early commitments to all schools are not equal. An early commit to Alabama is not likely to receive many more offers. Most other programs will be too intimidated to try to flip a commit from such a powerful program. They will consider it a waste of time and effort. An early commit to Georgia State, on the other hand, is likely to continue receiving offers. The intimidation factor just isn’t as strong. The team weightings used in RankByOffers are probably a fairly good representation of the intimidation factor.

Whatever adjustment is made for early commits will need to take all of this into consideration. It seems to me that the adjustment would need to consider the number of offers the early commit has received and also the intimidation factor of the school to which they are committed. An early commit with forty offers probably doesn’t need to be adjusted as much as an early commit with one offer. Similarly, an early commit with one offer from Georgia State (not to pick on the Panthers) probably doesn’t need to be adjusted as much as an early commit with one offer from Alabama. I’m not in favor of rules based upon "fuzzy" logic. Unfortunately, that kind of thinking is going to be required here. I’ll be trying to base the adjustment on fact and logic as much as possible, but I’m afraid those ideals will need to be sacrificed to some degree in this case. I’m open to suggestions.

Although I’m not convinced yet that the rankings should be adjusted for stated positions, it would be much easier to do. An average could be established for each stated position. The basic adjustment would be to align all of the medians … a fairly simple process. Here are the average offer points by position for the 2014 class:

Based upon this information, we would subtract 42.78 offer points for the average Defensive Back, and add 100.29 offer points for the average Quarterback. We couldn’t simply apply the positional adjustment to each player. If we did, a Quarterback with a single offer from a low level program would receive the same adjustment bonus as a Quarterback with 25 offers from top programs. The total adjustment for Quarterbacks would need to be distributed to each recruit in proportion to the offer points they already have. This is not difficult to do either. In short, it would be possible to adjust for positions. I’m wrestling with whether it should be done or not. There is a reason that coaches offer DBs more than QBs. I’m wondering whether it’s wise to undo the positional value they've assigned. I’d be interested to hear your reader’s opinions. Maybe when I get some time, I’ll model this adjustment and see if what comes out the other end makes sense.

Another problem with the approach mentioned by wamarsh is "uncommittable offers". Many would prefer that uncommittable offers not to be counted at all. I don’t think eliminating them completely would accurately represent reality. The school thought highly enough of the recruit to give them an offer, whether it is uncommittable or not. I would agree that they might not deserve to be given full credit, depending upon the reason the offer is uncommittable. Some elite programs give out offers that are uncommittable from the start and never become committable. Fortunately, only a few programs are in a position to play this game. There are many reasons that offers become uncommittable. I’m not going to go through them all. Suffice it to say that if I could, I would give full credit for any offer that was committable at any point, whether or not it is now committable. If the offer was never committable, I’d give the recruit partial credit. The net change to the current method would be to deduct half of the points for offers that were never commitable. Unfortunately, this information is not available. If someone knows how I could get that information, please let me know.

Lewke mentioned the weightings used to value individual offers. When I first published the ranking, a little less than two months ago, I was using a weighting from 1 to 125. An offer from Alabama was worth 125 because there are 125 FBS teams, an offer from the number 2 team was worth 124. This continued until the last ranked team, whose offers were worth 1 point. I was expecting huge changes in the player and team rankings when I switched over to the three year Elo weighting. To my surprise, the changes in the rankings were minor. I don’t agree with the observation that the Elo weighting doesn’t perfectly represent reality. But, I don’t think it’s too far off. I will revisit the weightings, once the more serious issues are taken care of.

Lewke mentioned the weightings used to value individual offers. When I first published the ranking, a little less than two months ago, I was using a weighting from 1 to 125. An offer from Alabama was worth 125 because there are 125 FBS teams, an offer from the number 2 team was worth 124. This continued until the last ranked team, whose offers were worth 1 point. I was expecting huge changes in the player and team rankings when I switched over to the three year Elo weighting. To my surprise, the changes in the rankings were minor. I agree with the observation that the Elo weighting doesn’t perfectly represent reality. But, I don’t think it’s too far off. I will revisit the weightings, once the more serious issues are taken care of.

One interesting proposal that both Lewke and JLeverenz mentioned is using only the top 5 or 6 offers for each recruit. I hadn’t thought of this approach, but I’m intrigued. We can get an indication of what would change by looking at the Top 25 offers received by the recruits on the 2014 offer-based ranking. Some recruits with lots of offers, but few offers from top programs would drop, like Deion Hallmon and D’Andre Payne. Others who have fewer reported offers, but more top 25 offers would rise, like Leonard Fournette and Jabril Peppers. This approach would also partially alleviate the early commitment problem since the total number of offers wouldn’t matter as much. I’ll model this one and see what happens.

It’s probably going to be a few more months before RankByOffers rankings can stand on their own. Until then, they might be of some use in adjusting ratings from other services. One thing that RankByOffers is already very good at is identifying players who have been left behind by the rankings services. Even just looking at the top 30 recruits, based on offers, we can see several players who have tons of offers, but have gone unnoticed by the services.

Thanks and Stay tuned!

Paul