Skip to main content

How Accurate Were They in 2006?

A great question was brought up by Aaron in a recent comment about my Average Ranking Positions post. Aaron asked if I had any of the Top 100 lists from this same group of experts from 2006 and if so, how accurate the rankings were - so I've created a new spreadsheet, a 2006 Predicted Rankings vs Actual Fantasy Results spreadsheet. To find all of these 2006 rankings, I'm using the Internet Archive Wayback Machine, a website that has archived much of the Internet. Type in the website address you're interested and voila, a dynamic November 2005 version of that page is at your fingertips (or whatever other date you're looking for). Pretty cool, huh?

So I found ten of the same rankings I used this year for 2006 - of the ones I couldn't find, FF Toolbox had only the top 60 for 2006 so I had to throw it out; NFL.com didn't provide an 2006 overall ranking, just positional; and Fantasy Football Xtreme's 2006 version was not free. Otherwise, a ranking from each of the other sites was acquired and I did the same averaging thing for these rankings as I did for the 2007 rankings. Then I compiled the fantasy point totals for each of these players based on the standard scoring of one of my leagues (players who have retired or are not on active rosters were not included, like Tiki or Corey Dillon). The scoring system makes all the difference here, so the scoring rules I derived my fantasy points from are:

Passing Yards (25 yards per point)
Passing Touchdowns (6)
Interceptions (-2)
Rushing Yards (10 yards per point)
Rushing Touchdowns (6)
Reception Yards (10 yards per point)
Reception Touchdowns (6)
2-Point Conversions (2)
Fumbles Lost (-2)
Offensive Fumble Return TD (6)

Once I had all of the fantasy point totals for the season, I ranked all of the players based on fantasy points scored in the season. I compared this ranking of fantasy points scored to the 2006 pre-season rankings the experts provided. Then I took the absolute value of the difference between the two for each player and summed it up for each ranking. So the ranking that is the most accurate should have the smallest number. See the fuzzy chart below - I know it's hard to read the ranking names along the x axis, but the ranking with the smallest number, or best accuracy, was SI. So in order of greatest accuracy, the list goes:
SI
CBS Sportsline
ESPN
FootballGuys.com
Fox Sports
Pigskin Addiction
Ask The Commish
ProFantasySports.com
Yahoo Sports
FF Cheat Sheets

If you look at the spreadsheet (or above hard-to-read chart) you can see that ultimately they are all really close together - the span is not nearly as great as I thought it might be. Nor do any of them really seem to be all that accurate. This is also just one year's worth of data, so perhaps when time permits I will look into finding 2005 and 2004 data to solidify these findings.

I found it interesting to see which players had the most errant and accurate predictions from the expert rankings. Most errant included Jerry Porter, Chris Perry, Chris Brown, and Koren Robinson. The most accurate rankings were (not surprisingly) for LT, Larry Johnson, Reggie Wayne, and Steven Jackson.

Hope that was enlightening for some of you and I promise to provide posts with more timely fantasy information soon.



Comments

Anonymous said…
I always wondered about that! Awesome post. I wonder where the Bruno Boys rank?
Corey/FFGoat, I'll try to find the Bruno Boys 2006 ranking - if you already have it, send the URL my way, I'll add it to the spreadsheet and let you know the outcome. In fact if anyone else has a 2006 ranking they're curious about, just send me the ranking or URL and I will put it in the spreadsheet.
Unknown said…
This is great; thanks! I'm thinking of crunching the numbers a little differently. It would be interesting to see if experts vary across positions, like does SI do a better job at ranking qbs or rbs? Which expert is the best at picking wrs? I also was planning on throwing in a term that so misrankings the best players matters most. My intuition is that it's a lot worse to pick the second best running back as the first, rather than to rank the 12th best running back as the 11th (the difference in fantasy points/game between LT and Larry Johnson last year was 20%, between Rudi Johnson and Ahman Green: 2.3%). Ultimately, it would be nice to have a weighting function that punished in direct proportion to the amount of fantasy points the misranking cost you, but I probably won't get around to that. Whatever I do get around to, I'll post here.
Anonymous said…
Any chance you or a reader has access to last year's Football Outsiders' rankings? They're likely the most scientific of anything out there.

Does anyone have links to any other attempts to judge football projections? (Baseball projections is a big interest of mine.)
Unknown said…
I ran the numbers to give separate rankings for qb and rb. I also applied a linear weighting as mentioned above (it is more important to rank the #1 player correctly than the #10 player). Technically speaking, for each expert I took the top n players for each position, and compared them against the expert, such that
s= 1/r_max * Sigma [(r_max + 1 - r_a)/r_max * (1 - |r_a - r_p|/(r_max - 1)]
where r_max is the highest examined ranking (top 10 qbs, top 20 rbs), r_a is the actual rank based on yahoo point total for that player, and r_p is the expert's predicted rank for that player. If the expert didn't include a player ranked player with his top n players for that position, they received a score of zero for that prediction. This score is an ad hoc measurement of agreement score, since I did want to include the weighting.

For running backs, the best experts were (scores are normalized so 1.0 is perfect ranking):
1. ask the commish(.4107), 2. fantasy football cheat sheet(.4043), 3. football guys(.3786), 4. yahoo(.3655), 5. pro fantasy sports(.3720), 6.si(.3595), 7. fox(.3583), 8.cbs(.3506), 9. pigskin addiction (.3400), 10. espn(.3263)

For qbs:
1. si(.5980), 2. ask the commish(.5798), 3. yahoo(.5535), 4. espn(.5535), 5. football guys(.5434), 6. fox(.4989), 7. cbs(.4949),8. pro fantasy sports(.4829), 9. pigskin addiction (.4284), 10. ff cheat sheet(.4102)

Given one sample year, we really don't have any statistical power to say anything, but this is a start. And on a less technical note, let me just point out that none of the experts properly ranked LT #1, which was huge given the 97 point differential between him and Larry Johnson.
Aaron, this is awesome. Do you mind if I put into a post where it might have more visibility?

Quick question - WRs? Any reason you chose not to do them other than lack of time?

I had also noticed that no one ranked LT as #1 last year - stunning, isn't it? So whose to say that Frank Gore or Steven Jackson won't outscore LT this year. In fact, I've read a few things recently about LT's decline - or perhaps his return to normalcy is a better way of phrasing it. Interesting stuff.
Unknown said…
Hey Librarian,
I realized last night I actually calculated the weighting factor slightly different then I described above. Here are the scores for the method described in the post:

WR:yahoo(.6036), football guys(.5929), FF cheat sheet(0.5792), espn(0.5750), fox(0.5679), Ask the commmish(0.5465), cbs(0.5447), si(0.5387), pfs(0.5346), pigskin(0.5310)

QB:
si(.4889), commish(.4788), football guys(.4707), pfs(.4586), yahoo, (.4485), espn(.4404), cbs(.4222), fox(.3939), ffcs(.3899), pigskin(.3636)

RB:
commish(.5072), football guys(0.4947), pfs(0.4875), pigskin(.4453), yahoo(0.4417), ffcs(.4393), si(.4232), cbs(0.4066), fox(.3798), espn(.3667)

If you can arrange these posts into something coherent, you are more than welcome to do so. Good luck with your draft.
Is not it amazing - one last year I realized that was # 1 and LT? So Frank Gore or Steven Jackson this year, not outscore LT. Or maybe a better way to ruin a normal return groan choose the thug - in fact, I read something recently about the decline of the LT. Interesting stuff.

Popular posts from this blog

Labor Day FF links

Happy long weekend! Here's hoping most of you are free to enjoy the 3-day weekend and the relaxation and fun that comes with it. I spent my Sunday participating in 3 fantasy football drafts scattered throughout the day and night, but I think I've finally wrapped up my drafts for the year. So like many of you I'm now turning my attention to the Week 1 games! But before we get to some player rankings and prep for Week 1, I want to make sure that all you experts out there are aware of the upcoming deadline for the annual accuracy contest that I run with the Fantasy Sports Trade Association. I compare preseason positional rankings from experts to the final outcome of the fantasy season to see which site had the most accurate preseason rankings, and this year's deadline is fast approaching: September 9th by kickoff. Check out the info on the FSTA site for more details and be sure to e-mail me your submissions. Now, onto Week 1! First up, let's get to some of this

2010 Week 2 consensus start/sit

Greetings and happy weekend. I've got this week's consensus start/sit for you which is a tally of all the start/sit votes that players received in the various start/sit articles that I linked to earlier in the week. So while these aren't my personal recommendations, I do think knowing what the consensus is could be of help when making lineup decisions...so enjoy! Note: the players are listed in no particular order. QB Start: Michael Vick, Kyle Orton, Matt Ryan, Jason Campbell Sit: David Garrard, Alex Smith, Jay Cutler, Vince Young, Carson Palmer RB Start: LeSean McCoy, Darren McFadden, Ahmad Bradshaw, Brandon Jackson Sit: Tim Hightower, C.J. Spiller, Beanie Wells, Ronnie Brown WR Start: Santana Moss, Legedu Naanee, Malcolm Floyd, Mike Williams (TB), Dez Bryant Sit: Randy Moss, Michael Crabtree TE Start: Chris Cooley, Todd Heap, John Carlson, Jermichael Finley Sit: Jermaine Gresham, Owen Daniels, Tony Scheffler, Jeremy Shockey, Bo Scaife Kicker Start: La

Start/Sit Week 4 and a smattering of rankings

It's been awhile, my fantasy football friends. I have lofty intentions of getting several posts up here each week, but life is busy and I will do my best. So in the intervening 6 years since I've posted regularly on here, our family added another child (now there are 3!), moved back to Boulder for a sabbatical year and then returned to Knoxville, and we have basically been chugging along like many of the rest of you. My two boys (who turn 12 and 9 soon) are in a fantasy football league for the first time this year with their buddies, which ostensibly is the just the dads (and mom!) playing one another of course. But my oldest is taking it super seriously and what kind of Fantasy Football Librarian would I be if I didn't step up my game for him? So I'm back - and am interested in the fantasy landscape - lots of the same awesome sites out there, but many new ones as well. I'll try to incorporate a mix. Please be sure to let me know if my 2006 ways aren't working i