Think about it. One absolutely cannot tell, by watching, the difference between a .300 hitter and a .275 hitter. The difference is one hit every two weeks. It might be that a reporter, seeing every game that the team plays, could sense that difference over the course of the year if no records were kept, but I doubt it. Certainly the average fan, seeing perhaps a tenth of the team's games, could never gauge two performances that accurately -- in fact if you see both 15 games a year, there is a 40% chance that the .275 hitter will have more hits than the .300 hitter in the games that you see. The difference between a good hitter and an average hitter is simply not visible -- it is a matter of record.
But the hitter is the center of attention. We notice what he does, bend over the scorecard with his name in mind. If he hits a smash down the third base line and the third baseman makes a diving stop and throws the runner out, then we notice and applaud the third baseman. But until the smash is hit, who is watching the third baseman? If he anticipates, if he adjusts for the hitter and moves over just two steps, then the same smash is a routine backhand stop -- and nobody applauds.
So if we can't tell who the good fielders are accurately from the record books, and we can't tell accurately from watching, how can we tell?
Bill James, 1977 Baseball Abstract, quoted in Michael Lewis, Moneyball