Today's pick in the exaggeration category wins the title strongest program to use exaggerations. The plural form is intentional, there is so much stuff on that website that is incorrect that I can't list it all; I will stick with just two claims. If you haven't guessed by now, the program is World Championship Checkers. The name is slightly ironic - WCC was the only top program that didn't show up at the 2002 computer world championship. Let's see what its authors claim on their website: The first thing you notice is the big banner on top that claims that WCC is the program that eats other checkers programs for breakfast. Of course, they have nothing to back up this wild claim. I played 10 games with WCC gold against Cake Sans Souci - an older version of Cake, also using only the 6-piece database for this match - and WCC lost 3 games and drew 7. That's not exactly what I would call eating for breakfast! Of course, with only 10 games, this is not a statistically significant result; a repetition of the match with different openings would produce a different result. However, it is likely that WCC is a bit weaker than the top programs. 4 games of that match can be replayed online at my games page. Both WCC gold and Cake are free downloads, you can repeat this kind of experiment any time you like.
Reading further on the WCC website, I read the next wild claim: Accelerated database lookup code makes WCC outperform all checkers programs on the market. The WCC databases are much smaller and are accessed three to four times faster than those of all other available checkers programs. Again, there is nothing to back this up of course. I have no idea how fast the database lookup of WCC really is, but vice versa the WCC authors have no idea how fast my database lookup (and that of other programs) is. How can they know they are 3-4 times faster? They can't, their statement is ridiculous. Their database, it is true, is 14.5% smaller than mine, and I don't want to pretend that is not good. It is good! But does it make a significant difference? The answer is no, it doesn't. This can be shown by running engine matches with different database cache sizes for one engine, and the results are not significantly different. Besides, there is something more to be said about endgame databases: A database not only uses RAM for the database itself, but also for some tables it needs. The amount it needs for its tables, the overhead, can be different for different databases. I don't know how much overhead WCC has, and I won't start claiming it has a larger overhead than my database. But any comparison between databases without including the overhead is rather pointless.
Subscribe to:
Post Comments (Atom)
3 comments:
Martin, you were too kind to WCC -- the program is considerably weaker than cake or kingsrow! The misinformation on their web site has always irked me, and is an insult to the authors of better programs.
BTW, I agree with you about the size of the endgame databases, it is a small 2nd order effect of little consequence. The db lookup speed is much more important than db size. You could compress your db with Lempel-Ziv, the algorithm used in WinZip, and make it considerably smaller, but the lookup speed would be terrible. Size alone is meaningless.
-- Ed
Ed, the only reason I didn't write something like "at least a bit weaker" or "clearly weaker than..." is an email of yours! I remember telling you about the 3-0 in ten games, and you wrote back and said something like that you were surprised by the result, WCC wasn't doing that bad against KingsRow. At least that's how I remember that conversation. From the games, I remember clearly that I wasn't impressed at all with WCC, it was outsearched and out-evaluated by Cake at the time. Do you have any hard results to back up your statement "considerably weaker"? Sorry for asking, but I don't want to make the same kind of mistakes here that I'm laughing about ;-)
Martin, your memory is apparently better than mine. I had to go back and find some old emails to see what I had written. I found two, one where I noted some strange behaviors in wcc's search, and commented on its missing some important features, and a second where I reported playing a few games with books turned off. Although wcc came out on the losing end in these games, the results were less lopsided than yours, and it was not enough games to really give it a good test. I'll retract my statement, because even though it still is my opinion, I dont have enough evidence to back it up. I dont want to be guilty of the same exaggerations that the wcc web site is just full of.
-- Ed
Post a Comment