Wednesday, August 08, 2007

How Accurate Were They in 2006?

A great question was brought up by Aaron in a recent comment about my Average Ranking Positions post. Aaron asked if I had any of the Top 100 lists from this same group of experts from 2006 and if so, how accurate the rankings were - so I've created a new spreadsheet, a 2006 Predicted Rankings vs Actual Fantasy Results spreadsheet. To find all of these 2006 rankings, I'm using the Internet Archive Wayback Machine, a website that has archived much of the Internet. Type in the website address you're interested and voila, a dynamic November 2005 version of that page is at your fingertips (or whatever other date you're looking for). Pretty cool, huh?

So I found ten of the same rankings I used this year for 2006 - of the ones I couldn't find, FF Toolbox had only the top 60 for 2006 so I had to throw it out; didn't provide an 2006 overall ranking, just positional; and Fantasy Football Xtreme's 2006 version was not free. Otherwise, a ranking from each of the other sites was acquired and I did the same averaging thing for these rankings as I did for the 2007 rankings. Then I compiled the fantasy point totals for each of these players based on the standard scoring of one of my leagues (players who have retired or are not on active rosters were not included, like Tiki or Corey Dillon). The scoring system makes all the difference here, so the scoring rules I derived my fantasy points from are:

Passing Yards (25 yards per point)
Passing Touchdowns (6)
Interceptions (-2)
Rushing Yards (10 yards per point)
Rushing Touchdowns (6)
Reception Yards (10 yards per point)
Reception Touchdowns (6)
2-Point Conversions (2)
Fumbles Lost (-2)
Offensive Fumble Return TD (6)

Once I had all of the fantasy point totals for the season, I ranked all of the players based on fantasy points scored in the season. I compared this ranking of fantasy points scored to the 2006 pre-season rankings the experts provided. Then I took the absolute value of the difference between the two for each player and summed it up for each ranking. So the ranking that is the most accurate should have the smallest number. See the fuzzy chart below - I know it's hard to read the ranking names along the x axis, but the ranking with the smallest number, or best accuracy, was SI. So in order of greatest accuracy, the list goes:
CBS Sportsline
Fox Sports
Pigskin Addiction
Ask The Commish
Yahoo Sports
FF Cheat Sheets

If you look at the spreadsheet (or above hard-to-read chart) you can see that ultimately they are all really close together - the span is not nearly as great as I thought it might be. Nor do any of them really seem to be all that accurate. This is also just one year's worth of data, so perhaps when time permits I will look into finding 2005 and 2004 data to solidify these findings.

I found it interesting to see which players had the most errant and accurate predictions from the expert rankings. Most errant included Jerry Porter, Chris Perry, Chris Brown, and Koren Robinson. The most accurate rankings were (not surprisingly) for LT, Larry Johnson, Reggie Wayne, and Steven Jackson.

Hope that was enlightening for some of you and I promise to provide posts with more timely fantasy information soon.


Anonymous said...

I always wondered about that! Awesome post. I wonder where the Bruno Boys rank?

Fantasy Football Librarian said...

Corey/FFGoat, I'll try to find the Bruno Boys 2006 ranking - if you already have it, send the URL my way, I'll add it to the spreadsheet and let you know the outcome. In fact if anyone else has a 2006 ranking they're curious about, just send me the ranking or URL and I will put it in the spreadsheet.

Unknown said...

This is great; thanks! I'm thinking of crunching the numbers a little differently. It would be interesting to see if experts vary across positions, like does SI do a better job at ranking qbs or rbs? Which expert is the best at picking wrs? I also was planning on throwing in a term that so misrankings the best players matters most. My intuition is that it's a lot worse to pick the second best running back as the first, rather than to rank the 12th best running back as the 11th (the difference in fantasy points/game between LT and Larry Johnson last year was 20%, between Rudi Johnson and Ahman Green: 2.3%). Ultimately, it would be nice to have a weighting function that punished in direct proportion to the amount of fantasy points the misranking cost you, but I probably won't get around to that. Whatever I do get around to, I'll post here.

Anonymous said...

Any chance you or a reader has access to last year's Football Outsiders' rankings? They're likely the most scientific of anything out there.

Does anyone have links to any other attempts to judge football projections? (Baseball projections is a big interest of mine.)

Unknown said...

I ran the numbers to give separate rankings for qb and rb. I also applied a linear weighting as mentioned above (it is more important to rank the #1 player correctly than the #10 player). Technically speaking, for each expert I took the top n players for each position, and compared them against the expert, such that
s= 1/r_max * Sigma [(r_max + 1 - r_a)/r_max * (1 - |r_a - r_p|/(r_max - 1)]
where r_max is the highest examined ranking (top 10 qbs, top 20 rbs), r_a is the actual rank based on yahoo point total for that player, and r_p is the expert's predicted rank for that player. If the expert didn't include a player ranked player with his top n players for that position, they received a score of zero for that prediction. This score is an ad hoc measurement of agreement score, since I did want to include the weighting.

For running backs, the best experts were (scores are normalized so 1.0 is perfect ranking):
1. ask the commish(.4107), 2. fantasy football cheat sheet(.4043), 3. football guys(.3786), 4. yahoo(.3655), 5. pro fantasy sports(.3720),, 7. fox(.3583),, 9. pigskin addiction (.3400), 10. espn(.3263)

For qbs:
1. si(.5980), 2. ask the commish(.5798), 3. yahoo(.5535), 4. espn(.5535), 5. football guys(.5434), 6. fox(.4989), 7. cbs(.4949),8. pro fantasy sports(.4829), 9. pigskin addiction (.4284), 10. ff cheat sheet(.4102)

Given one sample year, we really don't have any statistical power to say anything, but this is a start. And on a less technical note, let me just point out that none of the experts properly ranked LT #1, which was huge given the 97 point differential between him and Larry Johnson.

Fantasy Football Librarian said...

Aaron, this is awesome. Do you mind if I put into a post where it might have more visibility?

Quick question - WRs? Any reason you chose not to do them other than lack of time?

I had also noticed that no one ranked LT as #1 last year - stunning, isn't it? So whose to say that Frank Gore or Steven Jackson won't outscore LT this year. In fact, I've read a few things recently about LT's decline - or perhaps his return to normalcy is a better way of phrasing it. Interesting stuff.

Unknown said...

Hey Librarian,
I realized last night I actually calculated the weighting factor slightly different then I described above. Here are the scores for the method described in the post:

WR:yahoo(.6036), football guys(.5929), FF cheat sheet(0.5792), espn(0.5750), fox(0.5679), Ask the commmish(0.5465), cbs(0.5447), si(0.5387), pfs(0.5346), pigskin(0.5310)

si(.4889), commish(.4788), football guys(.4707), pfs(.4586), yahoo, (.4485), espn(.4404), cbs(.4222), fox(.3939), ffcs(.3899), pigskin(.3636)

commish(.5072), football guys(0.4947), pfs(0.4875), pigskin(.4453), yahoo(0.4417), ffcs(.4393), si(.4232), cbs(0.4066), fox(.3798), espn(.3667)

If you can arrange these posts into something coherent, you are more than welcome to do so. Good luck with your draft.

live football scores said...

Is not it amazing - one last year I realized that was # 1 and LT? So Frank Gore or Steven Jackson this year, not outscore LT. Or maybe a better way to ruin a normal return groan choose the thug - in fact, I read something recently about the decline of the LT. Interesting stuff.

Unknown said...

Excellent blog you’ve got here.It’s difficult to find high-quality writing like yours nowadays. I really appreciate individuals like you! Take care!! Please check out my site.
Daily Fantasy Sports Strategy