The Socratic approach to PECOTA (and why we don’t hate Bryce Harper)
From SABR member Ben Lindbergh and Colin Wyers at Baseball Prospectus on February 21, 2013:
When the PECOTA spreadsheet appears, one of the first things people do is pick out the players projected to make the greatest gains or suffer the largest declines. Then the questions start: Why does PECOTA like/dislike so-and-so so much? Is there a problem with the projections? Or is the system just picking up on something I’m not seeing?
Behind the scenes, the BP staff goes through the same thought process. Before we publish the projections, we approach PECOTA’s output with a skeptical eye, on the lookout for anything that could be a bug. But even after we’re satisfied with the spreadsheet and release it to our subscribers, PECOTA retains the capacity to surprise.
That PECOTA sometimes spits out surprising projections is a feature, not a bug. In that sense, it satisfies Bill James’ “80/20 Rule”, which says that a good statistic should conform to our expectations most of the time but still surprise often enough to be interesting. After all, if PECOTA simply confirmed what we already knew about player performance, it wouldn’t be worth the work. But James also (probably) said something like “No statistical finding is immune to the laws of common sense.” Granted, “common” sense might be more loophole than law, but the lesson seems sound: we shouldn’t blindly accept what every statistic says. A surprising stat might be right, but regardless, we’ll learn more by asking why it says what it says than we would by accepting it without question.
Read the full article here: http://www.baseballprospectus.com/article.php?articleid=19673
Originally published: February 21, 2013. Last Updated: February 21, 2013.