Sunday, June 08, 2008

Accuracy analysis of 2007 fantasy football experts

Back in February, I posted an article to the NY Times pro football blog, The Fifth Down, about the accuracy of fantasy football experts when it comes to preseason overall rankings. Since then, I have tweaked my methodology for looking at the expert rankings and added a site (Rotoworld) to the analysis. And it's a good thing I added Rotoworld, too, since they led the way on almost every angle of this analysis. But before I get too deep into the results, I just want to put a big warning that these results have flaws; it is impossible (as far as I can tell!) to do a perfect analysis of this sort of thing since every site has a different methodology and likely different scoring systems that they're working off of. And perhaps more importantly, this is merely one year of analysis that I'm looking at; anyone who works with stats can tell you that one year of results doesn't hold very much weight...which is why I plan on continuing this (and hopefully adjusting my ranking process until I'm more comfortable with it) for years to come. Additionally, please keep in mind that I assume you have a brain, that you aren't going to just look at this list and never ever look at SI again; that would be ill-informed of you. This is just meant to be a supplementary tool as you start looking at various experts out there.

Also, there are some sites (like that I only had positional rankings for, not overall top 100 rankings. This makes a difference only because I am attempting to compare apples to apples as much as possible. Perhaps this coming year I'll switch my analysis around by looking at these positional rankings the sites churn out rather than the overall Top 100 types of rankings (and pulling out the positional piece from the overall). See below for more on the positional issues that I've tried to address.

So I welcome your comments and if anyone has a better way of looking at this (keeping in mind my limitations of using just Excel and of course having a full-time job), by all means, please let me know.

So without further ado, here are the results:
1. Expert rankings from an overall perspective (see this article for an explanation on how I did these calculations):
1. Rotoworld (6005)
Ask The Commish (6,031)
3. (6,142)
FF Cheatsheets (6,157)
5. (6,167)
FFToolbox (6,211)
CBS Sports (6,221)
Yahoo! Sports (6,253)
Fantasy Football Xtreme (6,301)
Fox Sports (6,345)
Pigskin Addiction (6,370)
SI (6,419)
ESPN (6,447)
ProFantasySports (6,557)

A few months ago many of you felt that looking at sites from an overall ranking and not a positional ranking was flawed (see the comments section of that article linked above for details). And I agree that positional analysis needs to be taken into account, but I am also very interested in how well a site can predict overall rankings as well. That still has value to me and in my book, it adds to what makes an expert an expert. But as requested, I took a stab at looking at the rankings within positions from the info in Top 100 rankings, so that I wasn't just looking at these from an overall rankings perspective. So this next list is an attempt to address some of the issues related to how overall rankings are created and taking positions into account a little more - but again, it's likely these results are flawed to some extent since I think that's a really tall order. But hey, read on anyway...

2. Expert rankings from a positional perspective (the number in parentheses is the total number of spots off a site's preseason ranking was from the final season outcome):
QB rankings
1. FootballGuys (192)
2. FFToolbox (196) and (196)
4. Ask The Commish (197)
5. Rotoworld (200)
6. FF Cheatsheets (203)
7. Pigskin Addiction (211)
8. CBS Sportsline (213)
9. FoxSports/Rotter (215)
10. Yahoo! (221)
11. SI (225)
12. ESPN (229)
13. ProFantasySports (246)
14. FFXtreme (261)

RB rankings
1. Rotoworld (984)
2. FoxSports/Rotter (1000)
3. FF Cheatsheets (1007)
4. FFToolbox (1010)
5. Yahoo! (1013)
6. Pigskin Addiction (1020)
7. Football Guys (1021)
8. ESPN (1023) and (1023)
10. ProFantasySports (1032)
11. SI (1037)
12 Ask The Commish (1039)
13. FFXtreme (1045)
14. CBS Sportsline (1060)

WR rankings
1. Rotoworld (1075)
2. FFXtreme (1078)
3. FF Cheatsheets (1082)
4. CBS Sportsline (1122)
5. Football Guys (1124)
6. FFToolbox (1127)
7. Pigskin Addiction (1150)
8. ESPN (1160) and Yahoo! (1160)
10. FoxSports/Rotter (1170)
11. ProFantasySports (1182)
12. Ask the Commish (1201)
13. SI (1237)
14. (1245)

TE rankings
1. Football Guys (94)
2. (95)
3. Rotoworld (96) and Yahoo! (96)
5. Ask the Commish (98)
6. FFXtreme (99) and FoxSports/Rotter (99)
8. Pigskin Addiction (100)
9. CBS Sportsline (104) and FF Cheatsheets (104) and FFToolbox (104)
12. ProFantasySports (105)
13. ESPN (107) and SI (107)

Now if you take those numbers and just do a quick averaging of the positional rankings (so for Football Guys, I'd average together their rankings of and 1 (QB), 7 (RB), 5 (WR), and 1 (TE) for a positional average of 3.5), you get the following:
1. Rotoworld (avg 2.5)
2. Football Guys (avg 3.5)
3. FF Cheatsheets (avg 5.25) tied with FF Toolbox (avg 5.25)
5. (avg 6.5) tied with Yahoo! (avg 6.5)
7. FoxSports/Rotter (avg 6.75)
8. Pigskin Addiction (avg 7)
9. Ask the Commish (avg 8.25)
10. CBS Sportsline (avg 8.75) tied with FFXtreme (avg 8.75)
12. ESPN (avg 10.25)
13. ProFantasySports (avg 11.5)
14. SI (avg 12)

One final note, after a lot of reflection I did not include fan favorite Football Outsiders in this analysis since they have such a unique way of looking at rankings (which I love, by the way) - additionally, their rankings can be adjusted based on the scoring system you use, which is awesome, but it therefore means they don't really have one overall top 100 ranking; they have many, many overall rankings you can choose from. So it's unclear how they did in comparison; I can certainly pull out some of the players that FO did a good or bad job predicting though, so if that's of interest, let me know.

If you want all the gory details, see this Google spreadsheet. If you have trouble with that spreadsheet though and just want me to e-mail you the excel version, I can do that, too. Just contact me.



Anonymous said...

Great analysis. The one downside to an overall list is position scarcity. There are probably a lot of RB, WR, and TEs that are chosen even when the drafter knows there is a QB on the board that will put up more points. Once you get beyond the elite QBs, most will still put up more points than an elite WR depending on the scoring system, but I'll take the elite WR knowing the drop-off for that position is coming.

Anonymous said...

Where is on this list? I've been a long time subscriber of theirs and find their rankings and projections to be quite accurate. Just my 2 cents, but it's a shame they've been left off the list

Fantasy Football Librarian said...

Thanks for the comments - ffexchange, you're right, position scarcity is an issue that needs to be accounted for and I'm hoping that most FFers do consider it when drafting.

And anonymous, I will get in touch with for their 2007 rankings - I'm open to adding any other sites as long as I can get my hands on their rankings. Problem is, a lot of them aren't freely available and now many of the 2007 rankings are hard to track down. But thanks for the suggestion - I'll see what I can do.

Anonymous said...

Nice job, FFL - thanks.

Anyone know what's up with That was a good resource to use throughout the season - hope it returns.

dbt said...

Pro Football Prospectus '07 has the KUBIAK projections for their top 200 players listed in order -- I realize that actually puts FO at a disadvantage since press-time is way before draft time, but I'd still be interested to see how it stacks up.

Anonymous said...

Just wondering if there's a way of factoring in injuries to this equation. It seems a bit unreasonable to punish websites who, say, had Marvin Harrison ranked over Terrell Owens when the inaccuracy of that ranking was not based directly on performance (esp. considering Harrison's lack of previous injury history)

Anonymous said...

I know that I'd be interested in a error squared type measurement on actual performances. ie if the site predicted someone to score 120 points and they actually scored 150, the error contribution from that prediction would be 900.

I know that not all sites actually provide estimated points per player, but the rankings list don't accurately reflect which errors in rankings actually mattered. Being off a lot in the later rounds is not as big of an impact as being off a lot in the early rounds due to the relative magnitude of the contribution later round players contribute.

I think it'd be another interesting way to measure the error for sites that divulge enough information to make it possible.

Anonymous said...

Did you look at Sporting News rankings? I haven't tried to do anything as in depth as you, but they seem to be fairly on target with my league settings.

Highlander said...

Too much emphasis on rankings..
with a BBA in statistical analysis and an MBA in Finance, i will leave you with 3 words Random Walk theory.
The random walk hypothesis is a financial theory stating that stock market prices evolve according to a random walk and thus the prices of the stock market cannot be predicted.
The random walk hypothesis was also applied to NBA basketball. Psychologists made a detailed study of every shot the Philadelphia 76ers made over one and one-half seasons of basketball. The psychologists found no positive correlation between the previous shots and the outcomes of the shots afterwards. Economists and believers in the random walk hypothesis apply this to the stock market. The actual lack of correlation of past and present can be easily seen. If a stock goes up one day, no stock market participant can accurately predict that it will rise again the next. Just as a basketball player with the “hot hand” can miss his or her next shot, the stock that seems to be on the rise can fall at any time, making it completely random.
Same for football, just my humble opinion

Anonymous said...

Football Outsiders would probably prefer you didn't score their 2007 season. KUBIAK had hiccups all over the map. I'm curious to see how they spin this in the 2008 annual.

Anonymous said...

This is invaluable info. Kudos.

Anonymous said...

FFMastermind and FFToday should be included in the analysis

Fantasy Football Librarian said...

Thanks for all of the feedback, everyone. I will look into adding Sporting News, FFToday and FFMastermind for this coming year.

Ethan - I'd rather not touch the injuries, to be honest. While Harrison's injury was unexpected, there are plenty of injury-prone players and I don't want to be in the position of judging which players' injuries could have been anticipated. For those who are injury-prone, I think you get what you pay for if that player gets injured and you ranked them highly. And for those players that aren't injury-prone (like Harrison), most sites are punished equally for that kind of situation - more or less.

Dan R - I had considered squaring rather than abs valuing the results for the exact reasons you pointed out - and will try that out to see what kind of a difference that makes. Thanks!

Highlander - you make some good points but I don't think there's any harm in looking at rankings. To base your entire draft on a site's rankings without interjecting your own gut or external knowledge would be less than ideal; but the rankings are there as a supplement, not a drafting bible. Though I do think most FFers realize that; am I being naive to make that assumption?

And anonymous, you're right, this was an off year for Football Outsiders and their KUBIAK rankings; which is exactly why these results really need to be taken fairly lightly - it's just one year of results.

Thanks for the input, keep it coming.

Sean Zlatnik said...

It would be nice if a stab was made at including FootballOutsiders- You can use the default values in the spread sheet. I used Kubiak for my initial picks. Some of what they said was right on (so glad I had nothing to do with players like Larry Johnson!) and some was way off. I'm probably not going to lean quite so heavily on Kubiak this next year, but I will still include it in my work.

Also, I don't see rotowire- that would really be useful, because I paid for them, and they gave me great stuff during the year. I thought it was worth the investment- I think I did a lot better because of their info.

Anonymous said...

I am interested in seeing how Football Outsiders does next year. The fact that it's customizable simply means that you can compare Apples to Apples easier, as you can compare them based on SI's criteria, Rotowire's criteria, etc.

Anonymous said...

Why wasn't the huddle included in your rankings and do you plan on including them next year?


Anonymous said...

It would also be interesting to see whether the "expert" rankings were any better than the rankings one come have come up with solely from Average Draft Position (ADP). That would be a real test of "The Wisdowm of Crowds" theory.

Fantasy Football Librarian said...

I like this last suggestion to see how the ADP/wisdom of crowds would hold up. Of course there are so many different ADP metrics to choose from, but I'll be sure to include ADP this year. Thanks for the suggestion!

Anonymous said...

Another twist on the wisdom of crowds approach would be to compare the accuracy of the pooled experts' rankings to any individual expert's rankings.