I just finished debugging a Java checkers move generator - thanks to perft I found some bugs, and just watched the first game of my Java checkers engine played against itself (not a nice sight). Not surprisingly (at least to me), the Java code runs much slower than the C code - I found a factor 3.2 difference. Is that just me, or is it Java??
C (Cake's code without move ordering)
depth 1 positions 7 time 0.000000 1.$ kN/s
depth 2 positions 49 time 0.000000 1.$ kN/s
depth 3 positions 302 time 0.000000 1.$ kN/s
depth 4 positions 1469 time 0.015000 97.9 kN/s
depth 5 positions 7361 time 0.000000 1.$ kN/s
depth 6 positions 36768 time 0.000000 1.$ kN/s
depth 7 positions 179740 time 0.000000 1.$ kN/s
depth 8 positions 845931 time 0.063000 13427.5 kN/s
depth 9 positions 3963680 time 0.156000 25408.2 kN/s
depth 10 positions 18391564 time 0.547000 33622.6 kN/s
depth 11 positions 85242128 time 2.656000 32094.2 kN/s
depth 12 positions 388623673 time 12.266000 31683.0 kN/s
Java
perft 1:7 1ms 7000positions/sec
perft 2:49 1ms 49000positions/sec
perft 3:302 1ms 302000positions/sec
perft 4:1469 17ms 86000positions/sec
perft 5:7361 1ms 7361000positions/sec
perft 6:36768 16ms 2298000positions/sec
perft 7:179740 17ms 10572000positions/sec
perft 8:845931 79ms 10707000positions/sec
perft 9:3963680 392ms 10111000positions/sec
perft 10:18391564 1813ms 10144000positions/sec
perft 11:85242128 8486ms 10045000positions/sec
perft 12:388623673 39141ms 9928000positions/sec
A postscript a couple of days later: I managed to improve my Java code a bit, it is now only 2.3 times slower than the C code. I suppose that is about the best one can hope for with Java - or??
Tuesday, March 03, 2009
Sunday, February 15, 2009
perft for checkers
A couple of days ago I got a mail from Aart Bik, a programmer working at Google. He is currently developing a checkers application (among others) for the Google android platform (aka gPhone). He asked me about perft numbers for checkers. Perft is an invention of the chess programming community, a test to verify the correctness of the move generator, by printing how many legal moves exist in a search tree to depth N from the current position. Although I have this function in my chess engine, I never thought of adding it to Cake until now. However, Aart is right, and a perft function is always good to have for debugging purposes, so I added it to Cake. In the next CheckerBoard/Cake release, Cake will have this functionality. For starters, here are the numbers Cake reports from the initial position:
Aart is happy too, since his numbers match. If you are interested in his applications for the gPhone, then check out his blog.
depth 1 positions 7
depth 2 positions 49
depth 3 positions 302
depth 4 positions 1469
depth 5 positions 7361
depth 6 positions 36768
depth 7 positions 179740
depth 8 positions 845931
depth 9 positions 3963680
depth 10 positions 18391564
depth 11 positions 85242128
depth 12 positions 388623673
Aart is happy too, since his numbers match. If you are interested in his applications for the gPhone, then check out his blog.
Monday, January 26, 2009
Dweeb!
I just heard that Jonathan Schaeffer published a revised version of his book on Chinook, One Jump Ahead. I own it, and it was one of the things that motivated me to work on checkers. In the new version, some new material was added on the solution of checkers by the Chinook team, which probably also makes good reading. I also get mentioned in the book - although I'm rather on the receiving end:
"Darse Billings and many other people were upset at the junk that was being posted on the web. Darse sent out the following message:
"Darse Billings and many other people were upset at the junk that was being posted on the web. Darse sent out the following message:
Who the hell is this dweeb? I read the report of the Las Vegas tournament and was appalled with his obnoxious and slanderous comments against Jonathan. Someone should teach him some manners.
Personally, I see no reason for Jonathan to accept a challenge from NEMESIS - CHINOOK doesn't have to prove anything to anywone. The program did enough talking eight years ago - long before the pipsqueaks started squeaking. But if he does grant them a match, I hope it will only be after sufficient preparation to blow them out of the water. Punks."
Darse (and Schaeffer with his Junk) is referring to my report on the Las Vegas world championship (As a side note, Darse was not on the Chinook team, and not a checkers programmer - exactly what his qualifications would be to judge the claims in my report remains a mystery to me).
Well, at least I learned some new English curse-words :-)
Thursday, August 07, 2008
4 in a row source code
I just published the source code of my 4 in a row program on my four in a row page. While this doesn't appear to fit in a computer checkers blog, I think it does belong for two reasons: first, it is an example of how to write a high-performance game program. Second, this is the second part of my plan to publish some of my source code (CheckerBoard came in first). More to follow...
Tuesday, July 22, 2008
CheckerBoard source code release
My interest in computer checkers is waning - partly because the game is solved, partly because other interests are taking more of my time these days. I still sometimes get mail asking for new features in CheckerBoard, but I haven't done much lately. Since I don't think this will change anytime soon, I have made the source code of CheckerBoard (in version 1.651) public. This code can be used in any way you might see fit; I'm not going to bother with some kind of public license.
I am very much aware that the code is neither particularly good nor clean; CheckerBoard was - like perhaps most software - not really designed well, i.e. it started out as a simple program and then I kept on adding more and more things to it, and so the whole program structure isn't nice. Some of the source code is really old, written at a time when my programming skills were even less developed than they are now. And of course, I'm not a trained programmer who learned how to properly develop software. All of this will make the CheckerBoard source code rather hard to understand, but if you have a lot of patience and a decent working knowledge of standard C, then you can probably get to grips with it.
I am very much aware that the code is neither particularly good nor clean; CheckerBoard was - like perhaps most software - not really designed well, i.e. it started out as a simple program and then I kept on adding more and more things to it, and so the whole program structure isn't nice. Some of the source code is really old, written at a time when my programming skills were even less developed than they are now. And of course, I'm not a trained programmer who learned how to properly develop software. All of this will make the CheckerBoard source code rather hard to understand, but if you have a lot of patience and a decent working knowledge of standard C, then you can probably get to grips with it.
Friday, July 18, 2008
CheckerBoard 1.651
I just published CheckerBoard 1.651 on my webpage. This release fixes four things:
1) a serious synchronization bug between the multiple threads of CheckerBoard, which could show up in autoplay and engine match mode; on my two newer machines that I usually use for development, this bug didn't show, but on my oldest machine it was very obvious. Even if the bug didn't show clearly on the newer machines, it is possible that it affects engine match outcomes - an upgrade is therefore strongly recommended if you run engine matches.
2) a small bug with graphics updating when changing the piece set (the board would update, the pieces only on the next window resizing).
3) export to HTML now also works from setup positions for Italian checkers (it always worked for English checkers)
4) I forgot to include the file 11man_FEN.txt which is used for 11-man engine matches. It contains 2400 of the 2500 possible 11-man starting positions. 100 of the positions are omitted, because they are likely wins for one side. The newest release includes this file, so that 11-man engine matches with 4800 games can be run (if you have the patience...) for better statistics than with standard engine matches.
If I get no negative feedback on this release, I will also publish the source code of CheckerBoard, since I find it increasingly difficult to find the time to work on CB - making the source public is a better alternative to stopping the development altogether.
But now, enjoy your bug-free 11-man matches!
1) a serious synchronization bug between the multiple threads of CheckerBoard, which could show up in autoplay and engine match mode; on my two newer machines that I usually use for development, this bug didn't show, but on my oldest machine it was very obvious. Even if the bug didn't show clearly on the newer machines, it is possible that it affects engine match outcomes - an upgrade is therefore strongly recommended if you run engine matches.
2) a small bug with graphics updating when changing the piece set (the board would update, the pieces only on the next window resizing).
3) export to HTML now also works from setup positions for Italian checkers (it always worked for English checkers)
4) I forgot to include the file 11man_FEN.txt which is used for 11-man engine matches. It contains 2400 of the 2500 possible 11-man starting positions. 100 of the positions are omitted, because they are likely wins for one side. The newest release includes this file, so that 11-man engine matches with 4800 games can be run (if you have the patience...) for better statistics than with standard engine matches.
If I get no negative feedback on this release, I will also publish the source code of CheckerBoard, since I find it increasingly difficult to find the time to work on CB - making the source public is a better alternative to stopping the development altogether.
But now, enjoy your bug-free 11-man matches!
Sunday, July 13, 2008
CB 1.65 bug
It has been nearly 3 months since I released CheckerBoard 1.65, and I hate to admit that I got bug reports nearly instantly after its release. I finally took the time to look into them (partly because my girlfriend is in Spitzbergen). I cleaned up a lot of my source code in CB 1.65 with the goal of making it public with this release, and unfortunately, I managed to add a rather serious synchronization bug to it. I believe to have found it, some final testing is still required, but probably there will soon be an update available.
Friday, February 22, 2008
Engine match mania
Over the last week, I tested several versions of my latest checkers engine (which carries the rather bland name Cake 1.8) - and I did this on all of my three computers. I own a AMD64 desktop, a CoreDuo laptop and a CoreQuad Desktop. These machines are all about equally fast; the CoreQuad is a bit faster than the other two, it also runs a 64-bit windows. Here are the results of my engine matches against KingsRow 1.16c:
The results are a mess! I ordered the different versions of Cake 1.8 according to their overall result, and while you can see that v1 is best overall, it is worse than v2 on the AMD64 and worse than Cake Manchester on the CoreDuo. Which means that if I had only been testing on the CoreDuo, I would have been disgusted with the performance of my new program, because it's worse than the old one, and if I had been testing on the AMD64, I would have chosen v2 rather than v1. Which of course means that running engine matches on a single computer (as I did for the past 7 years or so) is clearly insufficient, and it also casts some doubt on the current methodology of using 3 computers - how do I know that using 3 computers is enough? Wouldn't 10 be better? All in all, this is quite a disgusting discovery, and I'm at a bit of a loss of how to proceed when optimizing my engine :-(
Engine CoreQuad CoreDuo AMD64 total
-------------------------------------------------
Cake 1.8 v1 +28-12 +27-18 +27-20 82-50
Cake 1.8 v2 +19-17 +25-17 +32-15 76-49
Cake 1.8 v3 +23-15 +24-19 +27-17 74-51
Cake 1.8 v4 +21-15 +23-17 +26-17 70-49
Cake Manchester +23-11 +24-13 +24-21 71-45
The results are a mess! I ordered the different versions of Cake 1.8 according to their overall result, and while you can see that v1 is best overall, it is worse than v2 on the AMD64 and worse than Cake Manchester on the CoreDuo. Which means that if I had only been testing on the CoreDuo, I would have been disgusted with the performance of my new program, because it's worse than the old one, and if I had been testing on the AMD64, I would have chosen v2 rather than v1. Which of course means that running engine matches on a single computer (as I did for the past 7 years or so) is clearly insufficient, and it also casts some doubt on the current methodology of using 3 computers - how do I know that using 3 computers is enough? Wouldn't 10 be better? All in all, this is quite a disgusting discovery, and I'm at a bit of a loss of how to proceed when optimizing my engine :-(
Thursday, February 07, 2008
Bug squashing
Yesterday and today I found bugs in Cake's evaluation. I was looking at some games that Cake had lost, and made a change in the evaluation which should have improved the situation. However, although the evaluation was different, the output of the engine remained the same - the exact same number of nodes searched with the same value for the position. That made me *very* suspicious, and I looked at the code a bit more closely, and discovered that during my code cleanup of last summer I had broken a really important piece of knowledge. Being suspicious, I continued looking for bugs and found one minor bug and one cosmetic bug. The match result improved drastically after stomping the first bug, from +19-16 @5s/move to +26-11. I don't have much hope that the minor bug will make a big difference but soon I will see!
Monday, February 04, 2008
Multi-Core speed
Nowadays, CPUs aren't getting much faster in terms of clock speed, but the manufacturers are putting multiple cores in one CPU. While this gives a lot more bang for the buck, it is a challenge for programmers to make use of the newly available resources. Some problems are easy to parallelize, while others are much more difficult. Unfortunately, tree searching such as in two-player-strategy-games belongs to the second class: it's not straightforward at all to implement a parallel game tree search. Of course there are solutions for this, the best-known is called YBWC for young brother wait concept. This is quite easy to explain with words, but really programming it is something different :-(
I haven't managed to find any sensible description of the algorithm with some real code, except for the source code of Crafty, a strong free chess engine. While this is documented quite well on the source code level, there are very many source code files and many of them have the required changes for the parallel search, and it's all very confusing to me. I wonder if there is any sensible YBWC tutorial anywhere on the net?
Anyway, my own plans were different to start with: All I wanted to do was to make my book generator work in parallel - it is quite easy to have multiple threads running on a multi-core-CPU when each thread is just doing a standard search on its own position. The big difficulty only arises when all threads are supposed to help search the same position, because they have to start communicating with each other.
The only problem in having multiple threads running simultaneously is the following: the endgame database access code has to be protected from being accessed by multiple threads, since confusion could occur when two threads try to load data from disk at the same time. In such situations programmers use a "lock" to prevent other threads from accessing critical code when one thread starts using it. This thread has to free up the critical code again once it is done, so that the other threads can also use it. Obviously, this whole process might slow down the program a lot, because one thread can be blocking the others - and the only way of knowing whether this is a problem is to try! I made Cake thread-safe last summer, and finally tested the speed on my new quadcore yesterday. Here is the result:
1 thread: 2050 kN/s
2 threads: 2000 kN/s
4 threads: 1815 kN/s
Running 2 threads simultaneously thus results in a speed loss of about 2.5%, while running 4 threads gives a loss of 11%. Of course, that is not a big price to pay, since in total you are getting 4x89% = 356% out of your CPU compared to having a single thread running. I also have to admit that I just put the whole database lookup call in a critical section; perhaps it is possible to make that more efficient by only putting the parts of the database lookup that affect the database cache in a critical section. Nevertheless, I'm afraid I have no excuses any more and should be working on a multi-threaded book generator now...
I haven't managed to find any sensible description of the algorithm with some real code, except for the source code of Crafty, a strong free chess engine. While this is documented quite well on the source code level, there are very many source code files and many of them have the required changes for the parallel search, and it's all very confusing to me. I wonder if there is any sensible YBWC tutorial anywhere on the net?
Anyway, my own plans were different to start with: All I wanted to do was to make my book generator work in parallel - it is quite easy to have multiple threads running on a multi-core-CPU when each thread is just doing a standard search on its own position. The big difficulty only arises when all threads are supposed to help search the same position, because they have to start communicating with each other.
The only problem in having multiple threads running simultaneously is the following: the endgame database access code has to be protected from being accessed by multiple threads, since confusion could occur when two threads try to load data from disk at the same time. In such situations programmers use a "lock" to prevent other threads from accessing critical code when one thread starts using it. This thread has to free up the critical code again once it is done, so that the other threads can also use it. Obviously, this whole process might slow down the program a lot, because one thread can be blocking the others - and the only way of knowing whether this is a problem is to try! I made Cake thread-safe last summer, and finally tested the speed on my new quadcore yesterday. Here is the result:
1 thread: 2050 kN/s
2 threads: 2000 kN/s
4 threads: 1815 kN/s
Running 2 threads simultaneously thus results in a speed loss of about 2.5%, while running 4 threads gives a loss of 11%. Of course, that is not a big price to pay, since in total you are getting 4x89% = 356% out of your CPU compared to having a single thread running. I also have to admit that I just put the whole database lookup call in a critical section; perhaps it is possible to make that more efficient by only putting the parts of the database lookup that affect the database cache in a critical section. Nevertheless, I'm afraid I have no excuses any more and should be working on a multi-threaded book generator now...
Subscribe to:
Posts (Atom)
